From zsolt.ero at gmail.com Sat Aug 9 16:57:40 2025 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Sat, 9 Aug 2025 09:57:40 -0700 Subject: Too many open files at 1000 req/sec Message-ID: Hi, I'm seeking advice on the most robust way to configure Nginx for a specific scenario that led to a caching issue. I run a free vector tile map service (https://openfreemap.org/). The server's primary job is to serve a massive number of small (~70 kB), pre-gzipped PBF files. To optimize for ocean areas, tiles that don't exist on disk should be served as a 200 OK with an empty body. These are then rendered as empty space on the map. Recently, the server experienced an extremely high load: 100k req/sec on Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, Nginx started serving some *existing* tiles as empty bodies. Because these responses included cache-friendly headers (expires 10y), the CDN cached the incorrect empty responses, effectively making parts of the map disappear until a manual cache purge was performed. My goal is to prevent this from happening again. A temporary server overload should result in a server error (e.g., 5xx), not incorrect content that gets permanently cached. The Nginx error logs clearly showed the root cause of the system error: 2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files), client: 172.69.122.170, server: ... It appears my try_files directive interpreted this "Too many open files" error as a "file not found" condition and fell back to serving the empty tile. System and Nginx Diagnostic Information Here is the relevant information about the system and Nginx process state (captured at normal load, after I solved the high traffic incident, still showing high FD usage on one worker). - *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server (not VPS) - *nginx version*: nginx/1.27.4 - *Systemd ulimit for nofile:* # cat /etc/security/limits.d/limits1m.conf - soft nofile 1048576 - hard nofile 1048576 - *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done Max open files 300000 300000 files Max open files 300000 300000 files ... (all 8 workers show the same limit) - *Open File Descriptor Count per Worker:* # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done nginx PID 1090: 57 open files nginx PID 1091: 117 open files nginx PID 1092: 931 open files nginx PID 1093: 65027 open files nginx PID 1094: 7449 open files ... (Note the one worker with a very high count, ~98% of which are regular files). - sysctl fs.file-max: fs.file-max = 9223372036854775807 - systemctl show nginx | grep LimitNOFILE: LimitNOFILE=524288 LimitNOFILESoft=1024 Relevant Nginx Configuration Here are the key parts of my configuration that led to the issue. worker_processes auto; worker_rlimit_nofile 300000; ? events { worker_connections 40000; multi_accept on; } ? http { open_file_cache max=1000000 inactive=60m; open_file_cache_valid 60m; open_file_cache_min_uses 1; open_file_cache_errors on; # ... *server block tile serving logic:* location ^~ /monaco/20250806_231001_pt/ { alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; try_files $uri @empty_tile; add_header Content-Encoding gzip; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; ? add_header x-ofm-debug 'specific PBF monaco 20250806_231001_pt'; } ? location @empty_tile { return 200 ''; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; ? add_header x-ofm-debug 'empty tile'; } Full generated config is uploaded here: https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf Questions 1. I think multi_accept + open_file_cache > worker_rlimit_nofile is causing the whole trouble by not distributing the requests across workers, and then reaching the limit. Can you confirm if this is the correct take? 2. How should I handle the "missing file should be empty response, server error should be 5xx" scenario? I've asked 5 LLMs and each gave different answers, which I'm including below. I'd like to ask your expert opinion, and not trust LLMs in this regard. *o3* error_page 404 = @empty_tile; *Gemini* if (!-f $request_filename) { return 200 ''; } *Opus* location ^~ /{area}/{version}/ { # Check if file exists without opening it if (!-f $request_filename) { return 404; } ? # File exists, try to serve it try_files $uri =503; # Return 503 if can't open (system error) ? add_header Content-Encoding gzip; expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; add_header x-ofm-debug 'specific PBF {area} {version}'; } ? # Handle 404s (file doesn't exist) - serve empty tile error_page 404 = @empty_tile_safe; ? # Handle 503s (system errors) - don't cache! error_page 503 = @system_error; ? location @empty_tile_safe { return 200 ''; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; add_header x-ofm-debug 'empty tile (intentional)'; } ? location @system_error { return 503 'Service temporarily unavailable'; ? # SHORT cache for errors - don't poison the CDN cache! expires 5s; ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control "no-cache, must-revalidate"; add_header Retry-After "5" always; add_header x-ofm-debug 'system error - temporary'; } 3. *open_file_cache Tuning:* My current open_file_cache settings are clearly too aggressive and caused the problem. For a workload of millions of tiny, static files, what would be considered a good configuration for max, inactive, and min_uses? 4. *open_file_cache_errors:* Should this be on or off? My intent for having it on was to cache the "not found" status for ocean tiles to reduce disk checks. I want to cache file-not-found scenarios, but not server errors. What is the correct usage in this context? 5. *Limits:* What values would you recommend for values like worker_rlimit_nofile and worker_connections? Should I raise LimitNOFILESoft? Finally, since this is the freenginx list: does freenginx offer anything over stock nginx which would help me in this use case? Even just a monitoring page with FD values would help. Best regards, Zsolt -------------- next part -------------- An HTML attachment was scrubbed... URL: From showfom at gmail.com Sun Aug 10 06:32:10 2025 From: showfom at gmail.com (Xiufeng Guo) Date: Sun, 10 Aug 2025 15:32:10 +0900 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hi, What?s the average server load? Best Regards, Xiufeng Guo On Sun, Aug 10, 2025 at 02:04 Zsolt Ero wrote: > Hi, > > I'm seeking advice on the most robust way to configure Nginx for a > specific scenario that led to a caching issue. > > I run a free vector tile map service (https://openfreemap.org/). The > server's primary job is to serve a massive number of small (~70 kB), > pre-gzipped PBF files. > > To optimize for ocean areas, tiles that don't exist on disk should be > served as a 200 OK with an empty body. These are then rendered as empty > space on the map. > > Recently, the server experienced an extremely high load: 100k req/sec on > Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, > Nginx started serving some *existing* tiles as empty bodies. Because > these responses included cache-friendly headers (expires 10y), the CDN > cached the incorrect empty responses, effectively making parts of the map > disappear until a manual cache purge was performed. > > My goal is to prevent this from happening again. A temporary server > overload should result in a server error (e.g., 5xx), not incorrect > content that gets permanently cached. > > The Nginx error logs clearly showed the root cause of the system error: > > 2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files), client: 172.69.122.170, server: ... > > It appears my try_files directive interpreted this "Too many open files" > error as a "file not found" condition and fell back to serving the empty > tile. > System and Nginx Diagnostic Information > > Here is the relevant information about the system and Nginx process state > (captured at normal load, after I solved the high traffic incident, still > showing high FD usage on one worker). > > - > > *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server > (not VPS) > - > > *nginx version*: nginx/1.27.4 > - > > *Systemd ulimit for nofile:* > > # cat /etc/security/limits.d/limits1m.conf > - soft nofile 1048576 > - hard nofile 1048576 > > - > > *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* > > # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done > Max open files 300000 300000 files > Max open files 300000 300000 files > ... (all 8 workers show the same limit) > > - > > *Open File Descriptor Count per Worker:* > > # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done > nginx PID 1090: 57 open files > nginx PID 1091: 117 open files > nginx PID 1092: 931 open files > nginx PID 1093: 65027 open files > nginx PID 1094: 7449 open files > ... > > (Note the one worker with a very high count, ~98% of which are regular > files). > - > > sysctl fs.file-max: > > fs.file-max = 9223372036854775807 > > - > > systemctl show nginx | grep LimitNOFILE: > > LimitNOFILE=524288 > LimitNOFILESoft=1024 > > > Relevant Nginx Configuration > > Here are the key parts of my configuration that led to the issue. > > worker_processes auto; > worker_rlimit_nofile 300000; > ? > events { > worker_connections 40000; > multi_accept on; > } > ? > http { > open_file_cache max=1000000 inactive=60m; > open_file_cache_valid 60m; > open_file_cache_min_uses 1; > open_file_cache_errors on; > # ... > > *server block tile serving logic:* > > location ^~ /monaco/20250806_231001_pt/ { > alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; > try_files $uri @empty_tile; > add_header Content-Encoding gzip; > ? > expires 10y; > ? > types { > application/vnd.mapbox-vector-tile pbf; > } > ? > add_header 'Access-Control-Allow-Origin' '*' always; > > >> It appears my try_files directive interpreted this "Too many open files" >> error as a "file not found" condition and fell back to serving the empty >> tile. >> System and Nginx Diagnostic Information >> >> Here is the relevant information about the system and Nginx process state >> (captured at normal load, after I solved the high traffic incident, still >> showing high FD usage on one worker). >> >> - >> >> *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server >> (not VPS) >> - >> >> *nginx version*: nginx/1.27.4 >> - >> >> *Systemd ulimit for nofile:* >> >> # cat /etc/security/limits.d/limits1m.conf >> - soft nofile 1048576 >> - hard nofile 1048576 >> >> - >> >> *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* >> >> # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done >> Max open files 300000 300000 files >> Max open files 300000 300000 files >> ... (all 8 workers show the same limit) >> >> - >> >> *Open File Descriptor Count per Worker:* >> >> # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done >> nginx PID 1090: 57 open files >> nginx PID 1091: 117 open files >> nginx PID 1092: 931 open files >> nginx PID 1093: 65027 open files >> nginx PID 1094: 7449 open files >> ... >> >> (Note the one worker with a very high count, ~98% of which are >> regular files). >> - >> >> sysctl fs.file-max: >> >> fs.file-max = 9223372036854775807 >> >> - >> >> systemctl show nginx | grep LimitNOFILE: >> >> LimitNOFILE=524288 >> LimitNOFILESoft=1024 >> >> >> Relevant Nginx Configuration >> >> Here are the key parts of my configuration that led to the issue. >> >> worker_processes auto; >> worker_rlimit_nofile 300000; >> ? >> events { >> worker_connections 40000; >> multi_accept on; >> } >> ? >> http { >> open_file_cache max=1000000 inactive=60m; >> open_file_cache_valid 60m; >> open_file_cache_min_uses 1; >> open_file_cache_errors on; >> # ... >> >> *server block tile serving logic:* >> >> location ^~ /monaco/20250806_231001_pt/ { >> alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; >> try_files $uri @empty_tile; >> add_header Content-Encoding gzip; >> ? >> expires 10y; >> ? >> types { >> application/vnd.mapbox-vector-tile pbf; >> } >> ? >> add_header 'Access-Control-Allow-Origin' '*' always; >> error should be 5xx" scenario? I've asked 5 LLMs and each gave different > answers, which I'm including below. I'd like to ask your expert opinion, > and not trust LLMs in this regard. > > *o3* > > error_page 404 = @empty_tile; That's what I would recommend as well. You may also want to use "log_not_found off;" to avoid excessive logging. Also, it may make sense to actually rethink how empty tiles are stored. With "no file means empty title" approach you are still risking the same issue even with "error_page 404" if files will be lost somehow - such as due to disk issues, during incomplete synchronization, or whatever. [...] > 3. *open_file_cache Tuning:* My current open_file_cache settings are > clearly too aggressive and caused the problem. For a workload of millions > of tiny, static files, what would be considered a good configuration for max, > inactive, and min_uses? I don't think that open_file_cache would be beneficial for your use case. Rather, it may make sense to tune OS namei(9) cache (dentry cache on Linux; not sure there are any settings other than vm.vfs_cache_pressure) to the number of files. On the other hand, given the 1k r/s request rate, most systems should be good enough without any tuning. > 4. *open_file_cache_errors:* Should this be on or off? My intent for having > it on was to cache the "not found" status for ocean tiles to reduce disk > checks. I want to cache file-not-found scenarios, but not server errors. > What is the correct usage in this context? The "open_file_cache_errors" directive currently caches all file system errors, and doesn't make any distinction between what exactly gone wrong - either the file or directory cannot be found, or there is a permissions error, or something else. If you want to make sure that no unexpected errors will be cached, consider keeping it off. On the other hand, it may make sense to explicitly exclude EMFILE, ENFILE, and may be ENOMEM from caching. I'll take a look. Note though, that as suggested above, my recommendation would be to avoid using "open_file_cache" at all. > 5. *Limits:* What values would you recommend for values like > worker_rlimit_nofile and worker_connections? Should I raise LimitNOFILESoft? In general, "worker_connections" should be set depending on the expected load (and "worker_processes"). Total number of connections nginx will be able to handle is worker_processes * worker_connections. Given you use "worker_connections 40000;" and at least 5 worker processes, your server is already able to handle more than 200k connections, and it is likely more than enough. Looking into stub_status numbers (and/or system connections stats) might give you an idea if you needed more connections. Note that using many worker connection might require OS tuning (but it looks like you've already set fs.file-max to an arbitrary high value). And the RLIMIT_NOFILE limit should be set to a value needed for your worker processes. It doesn't matter how do you set it, either in system (such as with LimitNOFILESoft in systemd) or with worker_rlimit_nofile in nginx itself (assuming it's under the hard limit set in the system). The basic idea is that worker processes shouldn't hit the RLIMIT_NOFILE limit, but should hit worker_connections limit instead. This way workers will be able to reuse least recently used connections to free some resources, and will be able to actively avoid accepting new connections to let other worker processes do this. Given each connection uses at least one file for the socket, and can use many (for the file it returns, for upstream connections, for various temporary files, subrequests, streams in HTTP/2, and so on), it is usually a good idea to keep RLIMIT_NOFILE several times higher than worker_connections. Since you have HTTP/2 enabled with the default max_concurrent_streams (128), and no proxying or subrequests, a reasonable limit would be worker_connections * (128 + 1) or so, that's about 5 mln open files (or you could consider reducing max_concurrent_streams, or worker_connections, or both). And of course you'll have to add some for various files not related to connections, such as logs and open_file_cache if you'll decide to keep it. > Finally, since this is the freenginx list: does freenginx offer anything > over stock nginx which would help me in this use case? Even just a > monitoring page with FD values would help. I don't think there is a significant difference in this particular use case. While freenginx provides various fixes and improvements, including fixes in open_file_cache, they won't make a difference here - the root cause of the issue you've hit is fragile configuration combined with too low resource limits. -- Maxim Dounin http://mdounin.ru/ From zsolt.ero at gmail.com Sun Aug 10 14:28:20 2025 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Sun, 10 Aug 2025 07:28:20 -0700 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hello Maxim and thank you for your detailed answer. First, about `multi_accept`: I can confirm that it indeed distributes requests super unevenly. Luckily I have 2 servers, handling 50-50% of the requests, so I could experiment by turning it off on one and restarting the nginx service on both. multi_accept: on for pid in $(pgrep -f "nginx: worker"); do echo "PID $pid: $(lsof -p $pid | wc -l) open files"; done PID 1761825: 66989 open files PID 1761827: 8766 open files PID 1761828: 962 open files PID 1761830: 184 open files PID 1761832: 46 open files PID 1761833: 81 open files PID 1761834: 47 open files PID 1761835: 40 open files PID 1761836: 45 open files PID 1761837: 44 open files PID 1761838: 40 open files PID 1761839: 40 open files multi_accept: off PID 1600658: 11137 open files PID 1600659: 10988 open files PID 1600660: 10974 open files PID 1600661: 11116 open files PID 1600662: 10937 open files PID 1600663: 10891 open files PID 1600664: 10934 open files PID 1600665: 10944 open files This is from an everyday, low-load situation, not CDN "Purge Cache" or similar flood. Based on this, multi_accept: on clearly makes no sense, I wonder why it's written in so many guides. Back then, I went through blog posts/optimisation guides/StackOverflow before I ended up on the config I used. multi_accept: on was in many of them. 2. Thanks, so I'll be rewriting it as error_page 404 = @empty_tile; log_not_found off; 3. What I didn't say was that the files are from a read-only mounted disk-image (btrfs loop,ro 0 0). I guess modern Linux kernel level caching should be quite optimised for this scenario, shouldn't it? I believe open_file_cache in my situation introduces a huge complexity surface with possibly no upside? I'll be definitely turning it off altogether. 4. Now for the limits/connections, I feel it's bit of a deeper water. Currently (at normal load) I have this on the multi accept off server: ss -Htnp state established '( sport = :80 or sport = :443 )' \ | awk 'match($0,/pid=([0-9]+)/,m){c[m[1]]++} END{for (p in c) printf "nginx worker pid %s: %d ESTAB\n", p, c[p]}' \ | sort -k6,6nr nginx worker pid 1600658: 203 ESTAB nginx worker pid 1600659: 213 ESTAB nginx worker pid 1600660: 211 ESTAB nginx worker pid 1600661: 201 ESTAB nginx worker pid 1600662: 220 ESTAB nginx worker pid 1600663: 214 ESTAB nginx worker pid 1600664: 213 ESTAB nginx worker pid 1600665: 212 ESTAB and this on the multi accept on one: nginx worker pid 1761825: 1388 ESTAB nginx worker pid 1761827: 114 ESTAB nginx worker pid 1761828: 6 ESTAB Isn't the default http2 128 streams a bit of a too high value? What do you think about worker_connections 8192; http2_max_concurrent_streams 32; => 8192 * (32+1) = 270,336 < 300k Also what do you think about adding http2_idle_timeout 30s; Best regards, Zsolt On 10. Aug 2025 at 12:34:55, Maxim Dounin wrote: > Hello! > > On Sat, Aug 09, 2025 at 09:57:40AM -0700, Zsolt Ero wrote: > > I'm seeking advice on the most robust way to configure Nginx for a specific > > scenario that led to a caching issue. > > > I run a free vector tile map service (https://openfreemap.org/). The > > server's primary job is to serve a massive number of small (~70 kB), > > pre-gzipped PBF files. > > > To optimize for ocean areas, tiles that don't exist on disk should be > > served as a 200 OK with an empty body. These are then rendered as empty > > space on the map. > > > Recently, the server experienced an extremely high load: 100k req/sec on > > Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, > > Nginx started serving some *existing* tiles as empty bodies. Because these > > responses included cache-friendly headers (expires 10y), the CDN cached the > > incorrect empty responses, effectively making parts of the map disappear > > until a manual cache purge was performed. > > > My goal is to prevent this from happening again. A temporary server > > overload should result in a server error (e.g., 5xx), not incorrect content > > that gets permanently cached. > > > [...] > > Full generated config is uploaded here: > > https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf > > Questions > > > 1. I think multi_accept + open_file_cache > worker_rlimit_nofile is causing > > the whole trouble by not distributing the requests across workers, and then > > reaching the limit. Can you confirm if this is the correct take? > > > The root cause is definitely open_file_cache configured > with maximum number of cached files higher than allowed by the > number of open files resource limit. > > Using multi_accept makes this easier to hit by making request > distribution between worker processes worse than it could be. > > Overall, I would recommend to: > > - Remove multi_accept, it's not needed unless you have very high > connection rates (and with small connection rates it'll waste > resources). Even assuming 1k r/s translates to 1k connections per > second, using multi_accept is unlikely to be beneficial. > > - Remove open_file_cache. It is only beneficial if opening files > requires significant resources, and this is unlikely for local > files on Unix systems. On the other hand, it is very likely to > introduce various issues, either by itself due to bugs (e.g., I've > recently fixed several open_file_cache bugs related to caching > files with directio enabled), or by exposing and magnifying other > issues, such as non-atomic file updates or misconfigurations like > this one. > > 2. How should I handle the "missing file should be empty response, server > > error should be 5xx" scenario? I've asked 5 LLMs and each gave different > > answers, which I'm including below. I'd like to ask your expert opinion, > > and not trust LLMs in this regard. > > > *o3* > > > error_page 404 = @empty_tile; > > > That's what I would recommend as well. > > You may also want to use "log_not_found off;" to avoid excessive > logging. > > Also, it may make sense to actually rethink how empty tiles are > stored. With "no file means empty title" approach you are still > risking the same issue even with "error_page 404" if files will be > lost somehow - such as due to disk issues, during incomplete > synchronization, or whatever. > > [...] > > 3. *open_file_cache Tuning:* My current open_file_cache settings are > > clearly too aggressive and caused the problem. For a workload of millions > > of tiny, static files, what would be considered a good configuration for > max, > > inactive, and min_uses? > > > I don't think that open_file_cache would be beneficial for your > use case. Rather, it may make sense to tune OS namei(9) cache > (dentry cache on Linux; not sure there are any settings other than > vm.vfs_cache_pressure) to the number of files. On the other hand, > given the 1k r/s request rate, most systems should be good enough > without any tuning. > > 4. *open_file_cache_errors:* Should this be on or off? My intent for having > > it on was to cache the "not found" status for ocean tiles to reduce disk > > checks. I want to cache file-not-found scenarios, but not server errors. > > What is the correct usage in this context? > > > The "open_file_cache_errors" directive currently caches all file > system errors, and doesn't make any distinction between what > exactly gone wrong - either the file or directory cannot be found, > or there is a permissions error, or something else. If you want > to make sure that no unexpected errors will be cached, consider > keeping it off. > > On the other hand, it may make sense to explicitly exclude EMFILE, > ENFILE, and may be ENOMEM from caching. I'll take a look. > > Note though, that as suggested above, my recommendation would be > to avoid using "open_file_cache" at all. > > 5. *Limits:* What values would you recommend for values like > > worker_rlimit_nofile and worker_connections? Should I raise > LimitNOFILESoft? > > > In general, "worker_connections" should be set depending on the > expected load (and "worker_processes"). Total number of > connections nginx will be able to handle is worker_processes * > worker_connections. > > Given you use "worker_connections 40000;" and at least 5 worker > processes, your server is already able to handle more than 200k > connections, and it is likely more than enough. Looking into > stub_status numbers (and/or system connections stats) might give > you an idea if you needed more connections. Note that using > many worker connection might require OS tuning (but it looks like > you've already set fs.file-max to an arbitrary high value). > > And the RLIMIT_NOFILE limit should be set to a value needed for > your worker processes. It doesn't matter how do you set it, > either in system (such as with LimitNOFILESoft in systemd) or with > worker_rlimit_nofile in nginx itself (assuming it's under the hard > limit set in the system). > > The basic idea is that worker processes shouldn't hit the > RLIMIT_NOFILE limit, but should hit worker_connections limit > instead. This way workers will be able to reuse least recently > used connections to free some resources, and will be able to > actively avoid accepting new connections to let other worker > processes do this. > > Given each connection uses at least one file for the socket, and > can use many (for the file it returns, for upstream connections, > for various temporary files, subrequests, streams in HTTP/2, and > so on), it is usually a good idea to keep RLIMIT_NOFILE several > times higher than worker_connections. > > Since you have HTTP/2 enabled with the default > max_concurrent_streams (128), and no proxying or subrequests, a > reasonable limit would be worker_connections * (128 + 1) or so, > that's about 5 mln open files (or you could consider reducing > max_concurrent_streams, or worker_connections, or both). And of > course you'll have to add some for various files not related to > connections, such as logs and open_file_cache if you'll decide to > keep it. > > Finally, since this is the freenginx list: does freenginx offer anything > > over stock nginx which would help me in this use case? Even just a > > monitoring page with FD values would help. > > > I don't think there is a significant difference in this particular > use case. While freenginx provides various fixes and > improvements, including fixes in open_file_cache, they won't make > a difference here - the root cause of the issue you've hit is > fragile configuration combined with too low resource limits. > > -- > Maxim Dounin > http://mdounin.ru/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Sun Aug 10 16:25:27 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 10 Aug 2025 19:25:27 +0300 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hello! On Sun, Aug 10, 2025 at 07:28:20AM -0700, Zsolt Ero wrote: > Hello Maxim and thank you for your detailed answer. > > First, about `multi_accept`: I can confirm that it indeed distributes > requests super unevenly. > Luckily I have 2 servers, handling 50-50% of the requests, so I could > experiment by turning it off on one and restarting the nginx service on > both. > > multi_accept: on > > for pid in $(pgrep -f "nginx: worker"); do echo "PID $pid: $(lsof -p $pid | > wc -l) open files"; done > PID 1761825: 66989 open files > PID 1761827: 8766 open files > PID 1761828: 962 open files > PID 1761830: 184 open files > PID 1761832: 46 open files > PID 1761833: 81 open files > PID 1761834: 47 open files > PID 1761835: 40 open files > PID 1761836: 45 open files > PID 1761837: 44 open files > PID 1761838: 40 open files > PID 1761839: 40 open files > > multi_accept: off > PID 1600658: 11137 open files > PID 1600659: 10988 open files > PID 1600660: 10974 open files > PID 1600661: 11116 open files > PID 1600662: 10937 open files > PID 1600663: 10891 open files > PID 1600664: 10934 open files > PID 1600665: 10944 open files > > This is from an everyday, low-load situation, not CDN "Purge Cache" or > similar flood. > > Based on this, multi_accept: on clearly makes no sense, I wonder why it's > written in so many guides. Back then, I went through blog > posts/optimisation guides/StackOverflow before I ended up on the config I > used. multi_accept: on was in many of them. The "multi_accept on;" can be useful when there are lots of connections to handle, and its effect can be easily seen in benchmarks (and that's probably why "guides" tend to recommend it). Uniform distribution of requests between worker processes, however, isn't something it provides. On the other hand, in situations where "multi_accept" is beneficial, distribution likely will be much better than you've observed, since all worker processes will be busy handling connections. Also, uniform distribution can be achieved by other means, such as with "listen .. reuseport". [...] > 3. What I didn't say was that the files are from a read-only mounted > disk-image (btrfs loop,ro 0 0). I guess modern Linux kernel level caching > should be quite optimised for this scenario, shouldn't it? I believe > open_file_cache in my situation introduces a huge complexity surface with > possibly no upside? I'll be definitely turning it off altogether. Yes, exactly. Most likely it saves you some CPU seconds due to smaller number of syscalls made, but provides little to no real benefits. > 4. Now for the limits/connections, I feel it's bit of a deeper water. > Currently (at normal load) I have this on the multi accept off server: > > ss -Htnp state established '( sport = :80 or sport = :443 )' \ > | awk 'match($0,/pid=([0-9]+)/,m){c[m[1]]++} END{for (p in c) printf > "nginx worker pid %s: %d ESTAB\n", p, c[p]}' \ > | sort -k6,6nr > nginx worker pid 1600658: 203 ESTAB > nginx worker pid 1600659: 213 ESTAB > nginx worker pid 1600660: 211 ESTAB > nginx worker pid 1600661: 201 ESTAB > nginx worker pid 1600662: 220 ESTAB > nginx worker pid 1600663: 214 ESTAB > nginx worker pid 1600664: 213 ESTAB > nginx worker pid 1600665: 212 ESTAB > > and this on the multi accept on one: > > nginx worker pid 1761825: 1388 ESTAB > nginx worker pid 1761827: 114 ESTAB > nginx worker pid 1761828: 6 ESTAB So it looks like even the default worker_connections (512) will work for you under normal conditions. But, obviously enough, there should be some reserve to handle load spikes. > Isn't the default http2 128 streams a bit of a too high value? Yep, it comes from HTTP/2 specification, which says (https://datatracker.ietf.org/doc/html/rfc9113#section-6.5.2-2.6.1): : It is recommended that this value be no smaller than 100, so as to : not unnecessarily limit parallelism. Yet I think that it might be a good idea to actually limit parallelism to a much smaller value by default. > What do you think about > > worker_connections 8192; > > http2_max_concurrent_streams 32; > > => 8192 * (32+1) = 270,336 < 300k Looks good to me. > Also what do you think about adding http2_idle_timeout 30s; There is no http2_idle_timeout since nginx 1.19.7, keepalive_timeout is used instead for both HTTP/1.x and HTTP/2, and the default is 75s. Also, idle connections in HTTP/2, as well as keepalive connections in HTTP/1.x, are subject to LRU-based reuse if there aren't enough worker_connections, effectively auto-tuning keepalive_timeout. As such, manual tuning to reduce connection usage isn't really needed. -- Maxim Dounin http://mdounin.ru/