Too many open files at 1000 req/sec

[Maxim Dounin]

Hello!

On Sun, Aug 10, 2025 at 07:28:20AM -0700, Zsolt Ero wrote:

>  Hello Maxim and thank you for your detailed answer.
> 
> First, about `multi_accept`: I can confirm that it indeed distributes
> requests super unevenly.
> Luckily I have 2 servers, handling 50-50% of the requests, so I could
> experiment by turning it off on one and restarting the nginx service on
> both.
> 
> multi_accept: on
> 
> for pid in $(pgrep -f "nginx: worker"); do echo "PID $pid: $(lsof -p $pid |
> wc -l) open files"; done
> PID 1761825: 66989 open files
> PID 1761827: 8766 open files
> PID 1761828: 962 open files
> PID 1761830: 184 open files
> PID 1761832: 46 open files
> PID 1761833: 81 open files
> PID 1761834: 47 open files
> PID 1761835: 40 open files
> PID 1761836: 45 open files
> PID 1761837: 44 open files
> PID 1761838: 40 open files
> PID 1761839: 40 open files
> 
> multi_accept: off
> PID 1600658: 11137 open files
> PID 1600659: 10988 open files
> PID 1600660: 10974 open files
> PID 1600661: 11116 open files
> PID 1600662: 10937 open files
> PID 1600663: 10891 open files
> PID 1600664: 10934 open files
> PID 1600665: 10944 open files
> 
> This is from an everyday, low-load situation, not CDN "Purge Cache" or
> similar flood.
> 
> Based on this, multi_accept: on clearly makes no sense, I wonder why it's
> written in so many guides. Back then, I went through blog
> posts/optimisation guides/StackOverflow before I ended up on the config I
> used. multi_accept: on was in many of them.

The "multi_accept on;" can be useful when there are lots of 
connections to handle, and its effect can be easily seen in 
benchmarks (and that's probably why "guides" tend to recommend 
it).  Uniform distribution of requests between worker processes, 
however, isn't something it provides.

On the other hand, in situations where "multi_accept" is 
beneficial, distribution likely will be much better than you've 
observed, since all worker processes will be busy handling 
connections.  Also, uniform distribution can be achieved by other 
means, such as with "listen .. reuseport".

[...]

> 3. What I didn't say was that the files are from a read-only mounted
> disk-image (btrfs loop,ro 0 0). I guess modern Linux kernel level caching
> should be quite optimised for this scenario, shouldn't it? I believe
> open_file_cache in my situation introduces a huge complexity surface with
> possibly no upside? I'll be definitely turning it off altogether.

Yes, exactly.  Most likely it saves you some CPU seconds due to 
smaller number of syscalls made, but provides little to no real 
benefits.

> 4. Now for the limits/connections, I feel it's bit of a deeper water.
> Currently (at normal load) I have this on the multi accept off server:
> 
> ss -Htnp state established '( sport = :80 or sport = :443 )' \
>   | awk 'match($0,/pid=([0-9]+)/,m){c[m[1]]++} END{for (p in c) printf
> "nginx worker pid %s: %d ESTAB\n", p, c[p]}' \
>   | sort -k6,6nr
> nginx worker pid 1600658: 203 ESTAB
> nginx worker pid 1600659: 213 ESTAB
> nginx worker pid 1600660: 211 ESTAB
> nginx worker pid 1600661: 201 ESTAB
> nginx worker pid 1600662: 220 ESTAB
> nginx worker pid 1600663: 214 ESTAB
> nginx worker pid 1600664: 213 ESTAB
> nginx worker pid 1600665: 212 ESTAB
> 
> and this on the multi accept on one:
> 
> nginx worker pid 1761825: 1388 ESTAB
> nginx worker pid 1761827: 114 ESTAB
> nginx worker pid 1761828: 6 ESTAB

So it looks like even the default worker_connections (512) will 
work for you under normal conditions.  But, obviously enough, 
there should be some reserve to handle load spikes.

> Isn't the default http2 128 streams a bit of a too high value?

Yep, it comes from HTTP/2 specification, which says
(https://datatracker.ietf.org/doc/html/rfc9113#section-6.5.2-2.6.1):

: It is recommended that this value be no smaller than 100, so as to 
: not unnecessarily limit parallelism.

Yet I think that it might be a good idea to actually limit 
parallelism to a much smaller value by default.

> What do you think about
> 
> worker_connections 8192;
> 
> http2_max_concurrent_streams 32;
> 
> => 8192 * (32+1) = 270,336 < 300k

Looks good to me.

> Also what do you think about adding http2_idle_timeout 30s;

There is no http2_idle_timeout since nginx 1.19.7, 
keepalive_timeout is used instead for both HTTP/1.x and HTTP/2, 
and the default is 75s.

Also, idle connections in HTTP/2, as well as keepalive 
connections in HTTP/1.x, are subject to LRU-based reuse if there 
aren't enough worker_connections, effectively auto-tuning 
keepalive_timeout.  As such, manual tuning to reduce connection 
usage isn't really needed.

-- 
Maxim Dounin
http://mdounin.ru/