From zsolt.ero at gmail.com Sat Aug 9 16:57:40 2025 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Sat, 9 Aug 2025 09:57:40 -0700 Subject: Too many open files at 1000 req/sec Message-ID: Hi, I'm seeking advice on the most robust way to configure Nginx for a specific scenario that led to a caching issue. I run a free vector tile map service (https://openfreemap.org/). The server's primary job is to serve a massive number of small (~70 kB), pre-gzipped PBF files. To optimize for ocean areas, tiles that don't exist on disk should be served as a 200 OK with an empty body. These are then rendered as empty space on the map. Recently, the server experienced an extremely high load: 100k req/sec on Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, Nginx started serving some *existing* tiles as empty bodies. Because these responses included cache-friendly headers (expires 10y), the CDN cached the incorrect empty responses, effectively making parts of the map disappear until a manual cache purge was performed. My goal is to prevent this from happening again. A temporary server overload should result in a server error (e.g., 5xx), not incorrect content that gets permanently cached. The Nginx error logs clearly showed the root cause of the system error: 2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files), client: 172.69.122.170, server: ... It appears my try_files directive interpreted this "Too many open files" error as a "file not found" condition and fell back to serving the empty tile. System and Nginx Diagnostic Information Here is the relevant information about the system and Nginx process state (captured at normal load, after I solved the high traffic incident, still showing high FD usage on one worker). - *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server (not VPS) - *nginx version*: nginx/1.27.4 - *Systemd ulimit for nofile:* # cat /etc/security/limits.d/limits1m.conf - soft nofile 1048576 - hard nofile 1048576 - *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done Max open files 300000 300000 files Max open files 300000 300000 files ... (all 8 workers show the same limit) - *Open File Descriptor Count per Worker:* # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done nginx PID 1090: 57 open files nginx PID 1091: 117 open files nginx PID 1092: 931 open files nginx PID 1093: 65027 open files nginx PID 1094: 7449 open files ... (Note the one worker with a very high count, ~98% of which are regular files). - sysctl fs.file-max: fs.file-max = 9223372036854775807 - systemctl show nginx | grep LimitNOFILE: LimitNOFILE=524288 LimitNOFILESoft=1024 Relevant Nginx Configuration Here are the key parts of my configuration that led to the issue. worker_processes auto; worker_rlimit_nofile 300000; ? events { worker_connections 40000; multi_accept on; } ? http { open_file_cache max=1000000 inactive=60m; open_file_cache_valid 60m; open_file_cache_min_uses 1; open_file_cache_errors on; # ... *server block tile serving logic:* location ^~ /monaco/20250806_231001_pt/ { alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; try_files $uri @empty_tile; add_header Content-Encoding gzip; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; ? add_header x-ofm-debug 'specific PBF monaco 20250806_231001_pt'; } ? location @empty_tile { return 200 ''; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; ? add_header x-ofm-debug 'empty tile'; } Full generated config is uploaded here: https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf Questions 1. I think multi_accept + open_file_cache > worker_rlimit_nofile is causing the whole trouble by not distributing the requests across workers, and then reaching the limit. Can you confirm if this is the correct take? 2. How should I handle the "missing file should be empty response, server error should be 5xx" scenario? I've asked 5 LLMs and each gave different answers, which I'm including below. I'd like to ask your expert opinion, and not trust LLMs in this regard. *o3* error_page 404 = @empty_tile; *Gemini* if (!-f $request_filename) { return 200 ''; } *Opus* location ^~ /{area}/{version}/ { # Check if file exists without opening it if (!-f $request_filename) { return 404; } ? # File exists, try to serve it try_files $uri =503; # Return 503 if can't open (system error) ? add_header Content-Encoding gzip; expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; add_header x-ofm-debug 'specific PBF {area} {version}'; } ? # Handle 404s (file doesn't exist) - serve empty tile error_page 404 = @empty_tile_safe; ? # Handle 503s (system errors) - don't cache! error_page 503 = @system_error; ? location @empty_tile_safe { return 200 ''; ? expires 10y; ? types { application/vnd.mapbox-vector-tile pbf; } ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control public; add_header X-Robots-Tag "noindex, nofollow" always; add_header x-ofm-debug 'empty tile (intentional)'; } ? location @system_error { return 503 'Service temporarily unavailable'; ? # SHORT cache for errors - don't poison the CDN cache! expires 5s; ? add_header 'Access-Control-Allow-Origin' '*' always; add_header Cache-Control "no-cache, must-revalidate"; add_header Retry-After "5" always; add_header x-ofm-debug 'system error - temporary'; } 3. *open_file_cache Tuning:* My current open_file_cache settings are clearly too aggressive and caused the problem. For a workload of millions of tiny, static files, what would be considered a good configuration for max, inactive, and min_uses? 4. *open_file_cache_errors:* Should this be on or off? My intent for having it on was to cache the "not found" status for ocean tiles to reduce disk checks. I want to cache file-not-found scenarios, but not server errors. What is the correct usage in this context? 5. *Limits:* What values would you recommend for values like worker_rlimit_nofile and worker_connections? Should I raise LimitNOFILESoft? Finally, since this is the freenginx list: does freenginx offer anything over stock nginx which would help me in this use case? Even just a monitoring page with FD values would help. Best regards, Zsolt -------------- next part -------------- An HTML attachment was scrubbed... URL: From showfom at gmail.com Sun Aug 10 06:32:10 2025 From: showfom at gmail.com (Xiufeng Guo) Date: Sun, 10 Aug 2025 15:32:10 +0900 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hi, What?s the average server load? Best Regards, Xiufeng Guo On Sun, Aug 10, 2025 at 02:04 Zsolt Ero wrote: > Hi, > > I'm seeking advice on the most robust way to configure Nginx for a > specific scenario that led to a caching issue. > > I run a free vector tile map service (https://openfreemap.org/). The > server's primary job is to serve a massive number of small (~70 kB), > pre-gzipped PBF files. > > To optimize for ocean areas, tiles that don't exist on disk should be > served as a 200 OK with an empty body. These are then rendered as empty > space on the map. > > Recently, the server experienced an extremely high load: 100k req/sec on > Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, > Nginx started serving some *existing* tiles as empty bodies. Because > these responses included cache-friendly headers (expires 10y), the CDN > cached the incorrect empty responses, effectively making parts of the map > disappear until a manual cache purge was performed. > > My goal is to prevent this from happening again. A temporary server > overload should result in a server error (e.g., 5xx), not incorrect > content that gets permanently cached. > > The Nginx error logs clearly showed the root cause of the system error: > > 2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files), client: 172.69.122.170, server: ... > > It appears my try_files directive interpreted this "Too many open files" > error as a "file not found" condition and fell back to serving the empty > tile. > System and Nginx Diagnostic Information > > Here is the relevant information about the system and Nginx process state > (captured at normal load, after I solved the high traffic incident, still > showing high FD usage on one worker). > > - > > *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server > (not VPS) > - > > *nginx version*: nginx/1.27.4 > - > > *Systemd ulimit for nofile:* > > # cat /etc/security/limits.d/limits1m.conf > - soft nofile 1048576 > - hard nofile 1048576 > > - > > *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* > > # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done > Max open files 300000 300000 files > Max open files 300000 300000 files > ... (all 8 workers show the same limit) > > - > > *Open File Descriptor Count per Worker:* > > # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done > nginx PID 1090: 57 open files > nginx PID 1091: 117 open files > nginx PID 1092: 931 open files > nginx PID 1093: 65027 open files > nginx PID 1094: 7449 open files > ... > > (Note the one worker with a very high count, ~98% of which are regular > files). > - > > sysctl fs.file-max: > > fs.file-max = 9223372036854775807 > > - > > systemctl show nginx | grep LimitNOFILE: > > LimitNOFILE=524288 > LimitNOFILESoft=1024 > > > Relevant Nginx Configuration > > Here are the key parts of my configuration that led to the issue. > > worker_processes auto; > worker_rlimit_nofile 300000; > ? > events { > worker_connections 40000; > multi_accept on; > } > ? > http { > open_file_cache max=1000000 inactive=60m; > open_file_cache_valid 60m; > open_file_cache_min_uses 1; > open_file_cache_errors on; > # ... > > *server block tile serving logic:* > > location ^~ /monaco/20250806_231001_pt/ { > alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; > try_files $uri @empty_tile; > add_header Content-Encoding gzip; > ? > expires 10y; > ? > types { > application/vnd.mapbox-vector-tile pbf; > } > ? > add_header 'Access-Control-Allow-Origin' '*' always; > > >> It appears my try_files directive interpreted this "Too many open files" >> error as a "file not found" condition and fell back to serving the empty >> tile. >> System and Nginx Diagnostic Information >> >> Here is the relevant information about the system and Nginx process state >> (captured at normal load, after I solved the high traffic incident, still >> showing high FD usage on one worker). >> >> - >> >> *OS:* Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server >> (not VPS) >> - >> >> *nginx version*: nginx/1.27.4 >> - >> >> *Systemd ulimit for nofile:* >> >> # cat /etc/security/limits.d/limits1m.conf >> - soft nofile 1048576 >> - hard nofile 1048576 >> >> - >> >> *Nginx Worker Process Limits (worker_rlimit_nofile is set to 300000):* >> >> # for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done >> Max open files 300000 300000 files >> Max open files 300000 300000 files >> ... (all 8 workers show the same limit) >> >> - >> >> *Open File Descriptor Count per Worker:* >> >> # for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done >> nginx PID 1090: 57 open files >> nginx PID 1091: 117 open files >> nginx PID 1092: 931 open files >> nginx PID 1093: 65027 open files >> nginx PID 1094: 7449 open files >> ... >> >> (Note the one worker with a very high count, ~98% of which are >> regular files). >> - >> >> sysctl fs.file-max: >> >> fs.file-max = 9223372036854775807 >> >> - >> >> systemctl show nginx | grep LimitNOFILE: >> >> LimitNOFILE=524288 >> LimitNOFILESoft=1024 >> >> >> Relevant Nginx Configuration >> >> Here are the key parts of my configuration that led to the issue. >> >> worker_processes auto; >> worker_rlimit_nofile 300000; >> ? >> events { >> worker_connections 40000; >> multi_accept on; >> } >> ? >> http { >> open_file_cache max=1000000 inactive=60m; >> open_file_cache_valid 60m; >> open_file_cache_min_uses 1; >> open_file_cache_errors on; >> # ... >> >> *server block tile serving logic:* >> >> location ^~ /monaco/20250806_231001_pt/ { >> alias /mnt/ofm/monaco-20250806_231001_pt/tiles/; >> try_files $uri @empty_tile; >> add_header Content-Encoding gzip; >> ? >> expires 10y; >> ? >> types { >> application/vnd.mapbox-vector-tile pbf; >> } >> ? >> add_header 'Access-Control-Allow-Origin' '*' always; >> error should be 5xx" scenario? I've asked 5 LLMs and each gave different > answers, which I'm including below. I'd like to ask your expert opinion, > and not trust LLMs in this regard. > > *o3* > > error_page 404 = @empty_tile; That's what I would recommend as well. You may also want to use "log_not_found off;" to avoid excessive logging. Also, it may make sense to actually rethink how empty tiles are stored. With "no file means empty title" approach you are still risking the same issue even with "error_page 404" if files will be lost somehow - such as due to disk issues, during incomplete synchronization, or whatever. [...] > 3. *open_file_cache Tuning:* My current open_file_cache settings are > clearly too aggressive and caused the problem. For a workload of millions > of tiny, static files, what would be considered a good configuration for max, > inactive, and min_uses? I don't think that open_file_cache would be beneficial for your use case. Rather, it may make sense to tune OS namei(9) cache (dentry cache on Linux; not sure there are any settings other than vm.vfs_cache_pressure) to the number of files. On the other hand, given the 1k r/s request rate, most systems should be good enough without any tuning. > 4. *open_file_cache_errors:* Should this be on or off? My intent for having > it on was to cache the "not found" status for ocean tiles to reduce disk > checks. I want to cache file-not-found scenarios, but not server errors. > What is the correct usage in this context? The "open_file_cache_errors" directive currently caches all file system errors, and doesn't make any distinction between what exactly gone wrong - either the file or directory cannot be found, or there is a permissions error, or something else. If you want to make sure that no unexpected errors will be cached, consider keeping it off. On the other hand, it may make sense to explicitly exclude EMFILE, ENFILE, and may be ENOMEM from caching. I'll take a look. Note though, that as suggested above, my recommendation would be to avoid using "open_file_cache" at all. > 5. *Limits:* What values would you recommend for values like > worker_rlimit_nofile and worker_connections? Should I raise LimitNOFILESoft? In general, "worker_connections" should be set depending on the expected load (and "worker_processes"). Total number of connections nginx will be able to handle is worker_processes * worker_connections. Given you use "worker_connections 40000;" and at least 5 worker processes, your server is already able to handle more than 200k connections, and it is likely more than enough. Looking into stub_status numbers (and/or system connections stats) might give you an idea if you needed more connections. Note that using many worker connection might require OS tuning (but it looks like you've already set fs.file-max to an arbitrary high value). And the RLIMIT_NOFILE limit should be set to a value needed for your worker processes. It doesn't matter how do you set it, either in system (such as with LimitNOFILESoft in systemd) or with worker_rlimit_nofile in nginx itself (assuming it's under the hard limit set in the system). The basic idea is that worker processes shouldn't hit the RLIMIT_NOFILE limit, but should hit worker_connections limit instead. This way workers will be able to reuse least recently used connections to free some resources, and will be able to actively avoid accepting new connections to let other worker processes do this. Given each connection uses at least one file for the socket, and can use many (for the file it returns, for upstream connections, for various temporary files, subrequests, streams in HTTP/2, and so on), it is usually a good idea to keep RLIMIT_NOFILE several times higher than worker_connections. Since you have HTTP/2 enabled with the default max_concurrent_streams (128), and no proxying or subrequests, a reasonable limit would be worker_connections * (128 + 1) or so, that's about 5 mln open files (or you could consider reducing max_concurrent_streams, or worker_connections, or both). And of course you'll have to add some for various files not related to connections, such as logs and open_file_cache if you'll decide to keep it. > Finally, since this is the freenginx list: does freenginx offer anything > over stock nginx which would help me in this use case? Even just a > monitoring page with FD values would help. I don't think there is a significant difference in this particular use case. While freenginx provides various fixes and improvements, including fixes in open_file_cache, they won't make a difference here - the root cause of the issue you've hit is fragile configuration combined with too low resource limits. -- Maxim Dounin http://mdounin.ru/ From zsolt.ero at gmail.com Sun Aug 10 14:28:20 2025 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Sun, 10 Aug 2025 07:28:20 -0700 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hello Maxim and thank you for your detailed answer. First, about `multi_accept`: I can confirm that it indeed distributes requests super unevenly. Luckily I have 2 servers, handling 50-50% of the requests, so I could experiment by turning it off on one and restarting the nginx service on both. multi_accept: on for pid in $(pgrep -f "nginx: worker"); do echo "PID $pid: $(lsof -p $pid | wc -l) open files"; done PID 1761825: 66989 open files PID 1761827: 8766 open files PID 1761828: 962 open files PID 1761830: 184 open files PID 1761832: 46 open files PID 1761833: 81 open files PID 1761834: 47 open files PID 1761835: 40 open files PID 1761836: 45 open files PID 1761837: 44 open files PID 1761838: 40 open files PID 1761839: 40 open files multi_accept: off PID 1600658: 11137 open files PID 1600659: 10988 open files PID 1600660: 10974 open files PID 1600661: 11116 open files PID 1600662: 10937 open files PID 1600663: 10891 open files PID 1600664: 10934 open files PID 1600665: 10944 open files This is from an everyday, low-load situation, not CDN "Purge Cache" or similar flood. Based on this, multi_accept: on clearly makes no sense, I wonder why it's written in so many guides. Back then, I went through blog posts/optimisation guides/StackOverflow before I ended up on the config I used. multi_accept: on was in many of them. 2. Thanks, so I'll be rewriting it as error_page 404 = @empty_tile; log_not_found off; 3. What I didn't say was that the files are from a read-only mounted disk-image (btrfs loop,ro 0 0). I guess modern Linux kernel level caching should be quite optimised for this scenario, shouldn't it? I believe open_file_cache in my situation introduces a huge complexity surface with possibly no upside? I'll be definitely turning it off altogether. 4. Now for the limits/connections, I feel it's bit of a deeper water. Currently (at normal load) I have this on the multi accept off server: ss -Htnp state established '( sport = :80 or sport = :443 )' \ | awk 'match($0,/pid=([0-9]+)/,m){c[m[1]]++} END{for (p in c) printf "nginx worker pid %s: %d ESTAB\n", p, c[p]}' \ | sort -k6,6nr nginx worker pid 1600658: 203 ESTAB nginx worker pid 1600659: 213 ESTAB nginx worker pid 1600660: 211 ESTAB nginx worker pid 1600661: 201 ESTAB nginx worker pid 1600662: 220 ESTAB nginx worker pid 1600663: 214 ESTAB nginx worker pid 1600664: 213 ESTAB nginx worker pid 1600665: 212 ESTAB and this on the multi accept on one: nginx worker pid 1761825: 1388 ESTAB nginx worker pid 1761827: 114 ESTAB nginx worker pid 1761828: 6 ESTAB Isn't the default http2 128 streams a bit of a too high value? What do you think about worker_connections 8192; http2_max_concurrent_streams 32; => 8192 * (32+1) = 270,336 < 300k Also what do you think about adding http2_idle_timeout 30s; Best regards, Zsolt On 10. Aug 2025 at 12:34:55, Maxim Dounin wrote: > Hello! > > On Sat, Aug 09, 2025 at 09:57:40AM -0700, Zsolt Ero wrote: > > I'm seeking advice on the most robust way to configure Nginx for a specific > > scenario that led to a caching issue. > > > I run a free vector tile map service (https://openfreemap.org/). The > > server's primary job is to serve a massive number of small (~70 kB), > > pre-gzipped PBF files. > > > To optimize for ocean areas, tiles that don't exist on disk should be > > served as a 200 OK with an empty body. These are then rendered as empty > > space on the map. > > > Recently, the server experienced an extremely high load: 100k req/sec on > > Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, > > Nginx started serving some *existing* tiles as empty bodies. Because these > > responses included cache-friendly headers (expires 10y), the CDN cached the > > incorrect empty responses, effectively making parts of the map disappear > > until a manual cache purge was performed. > > > My goal is to prevent this from happening again. A temporary server > > overload should result in a server error (e.g., 5xx), not incorrect content > > that gets permanently cached. > > > [...] > > Full generated config is uploaded here: > > https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf > > Questions > > > 1. I think multi_accept + open_file_cache > worker_rlimit_nofile is causing > > the whole trouble by not distributing the requests across workers, and then > > reaching the limit. Can you confirm if this is the correct take? > > > The root cause is definitely open_file_cache configured > with maximum number of cached files higher than allowed by the > number of open files resource limit. > > Using multi_accept makes this easier to hit by making request > distribution between worker processes worse than it could be. > > Overall, I would recommend to: > > - Remove multi_accept, it's not needed unless you have very high > connection rates (and with small connection rates it'll waste > resources). Even assuming 1k r/s translates to 1k connections per > second, using multi_accept is unlikely to be beneficial. > > - Remove open_file_cache. It is only beneficial if opening files > requires significant resources, and this is unlikely for local > files on Unix systems. On the other hand, it is very likely to > introduce various issues, either by itself due to bugs (e.g., I've > recently fixed several open_file_cache bugs related to caching > files with directio enabled), or by exposing and magnifying other > issues, such as non-atomic file updates or misconfigurations like > this one. > > 2. How should I handle the "missing file should be empty response, server > > error should be 5xx" scenario? I've asked 5 LLMs and each gave different > > answers, which I'm including below. I'd like to ask your expert opinion, > > and not trust LLMs in this regard. > > > *o3* > > > error_page 404 = @empty_tile; > > > That's what I would recommend as well. > > You may also want to use "log_not_found off;" to avoid excessive > logging. > > Also, it may make sense to actually rethink how empty tiles are > stored. With "no file means empty title" approach you are still > risking the same issue even with "error_page 404" if files will be > lost somehow - such as due to disk issues, during incomplete > synchronization, or whatever. > > [...] > > 3. *open_file_cache Tuning:* My current open_file_cache settings are > > clearly too aggressive and caused the problem. For a workload of millions > > of tiny, static files, what would be considered a good configuration for > max, > > inactive, and min_uses? > > > I don't think that open_file_cache would be beneficial for your > use case. Rather, it may make sense to tune OS namei(9) cache > (dentry cache on Linux; not sure there are any settings other than > vm.vfs_cache_pressure) to the number of files. On the other hand, > given the 1k r/s request rate, most systems should be good enough > without any tuning. > > 4. *open_file_cache_errors:* Should this be on or off? My intent for having > > it on was to cache the "not found" status for ocean tiles to reduce disk > > checks. I want to cache file-not-found scenarios, but not server errors. > > What is the correct usage in this context? > > > The "open_file_cache_errors" directive currently caches all file > system errors, and doesn't make any distinction between what > exactly gone wrong - either the file or directory cannot be found, > or there is a permissions error, or something else. If you want > to make sure that no unexpected errors will be cached, consider > keeping it off. > > On the other hand, it may make sense to explicitly exclude EMFILE, > ENFILE, and may be ENOMEM from caching. I'll take a look. > > Note though, that as suggested above, my recommendation would be > to avoid using "open_file_cache" at all. > > 5. *Limits:* What values would you recommend for values like > > worker_rlimit_nofile and worker_connections? Should I raise > LimitNOFILESoft? > > > In general, "worker_connections" should be set depending on the > expected load (and "worker_processes"). Total number of > connections nginx will be able to handle is worker_processes * > worker_connections. > > Given you use "worker_connections 40000;" and at least 5 worker > processes, your server is already able to handle more than 200k > connections, and it is likely more than enough. Looking into > stub_status numbers (and/or system connections stats) might give > you an idea if you needed more connections. Note that using > many worker connection might require OS tuning (but it looks like > you've already set fs.file-max to an arbitrary high value). > > And the RLIMIT_NOFILE limit should be set to a value needed for > your worker processes. It doesn't matter how do you set it, > either in system (such as with LimitNOFILESoft in systemd) or with > worker_rlimit_nofile in nginx itself (assuming it's under the hard > limit set in the system). > > The basic idea is that worker processes shouldn't hit the > RLIMIT_NOFILE limit, but should hit worker_connections limit > instead. This way workers will be able to reuse least recently > used connections to free some resources, and will be able to > actively avoid accepting new connections to let other worker > processes do this. > > Given each connection uses at least one file for the socket, and > can use many (for the file it returns, for upstream connections, > for various temporary files, subrequests, streams in HTTP/2, and > so on), it is usually a good idea to keep RLIMIT_NOFILE several > times higher than worker_connections. > > Since you have HTTP/2 enabled with the default > max_concurrent_streams (128), and no proxying or subrequests, a > reasonable limit would be worker_connections * (128 + 1) or so, > that's about 5 mln open files (or you could consider reducing > max_concurrent_streams, or worker_connections, or both). And of > course you'll have to add some for various files not related to > connections, such as logs and open_file_cache if you'll decide to > keep it. > > Finally, since this is the freenginx list: does freenginx offer anything > > over stock nginx which would help me in this use case? Even just a > > monitoring page with FD values would help. > > > I don't think there is a significant difference in this particular > use case. While freenginx provides various fixes and > improvements, including fixes in open_file_cache, they won't make > a difference here - the root cause of the issue you've hit is > fragile configuration combined with too low resource limits. > > -- > Maxim Dounin > http://mdounin.ru/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Sun Aug 10 16:25:27 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 10 Aug 2025 19:25:27 +0300 Subject: Too many open files at 1000 req/sec In-Reply-To: References: Message-ID: Hello! On Sun, Aug 10, 2025 at 07:28:20AM -0700, Zsolt Ero wrote: > Hello Maxim and thank you for your detailed answer. > > First, about `multi_accept`: I can confirm that it indeed distributes > requests super unevenly. > Luckily I have 2 servers, handling 50-50% of the requests, so I could > experiment by turning it off on one and restarting the nginx service on > both. > > multi_accept: on > > for pid in $(pgrep -f "nginx: worker"); do echo "PID $pid: $(lsof -p $pid | > wc -l) open files"; done > PID 1761825: 66989 open files > PID 1761827: 8766 open files > PID 1761828: 962 open files > PID 1761830: 184 open files > PID 1761832: 46 open files > PID 1761833: 81 open files > PID 1761834: 47 open files > PID 1761835: 40 open files > PID 1761836: 45 open files > PID 1761837: 44 open files > PID 1761838: 40 open files > PID 1761839: 40 open files > > multi_accept: off > PID 1600658: 11137 open files > PID 1600659: 10988 open files > PID 1600660: 10974 open files > PID 1600661: 11116 open files > PID 1600662: 10937 open files > PID 1600663: 10891 open files > PID 1600664: 10934 open files > PID 1600665: 10944 open files > > This is from an everyday, low-load situation, not CDN "Purge Cache" or > similar flood. > > Based on this, multi_accept: on clearly makes no sense, I wonder why it's > written in so many guides. Back then, I went through blog > posts/optimisation guides/StackOverflow before I ended up on the config I > used. multi_accept: on was in many of them. The "multi_accept on;" can be useful when there are lots of connections to handle, and its effect can be easily seen in benchmarks (and that's probably why "guides" tend to recommend it). Uniform distribution of requests between worker processes, however, isn't something it provides. On the other hand, in situations where "multi_accept" is beneficial, distribution likely will be much better than you've observed, since all worker processes will be busy handling connections. Also, uniform distribution can be achieved by other means, such as with "listen .. reuseport". [...] > 3. What I didn't say was that the files are from a read-only mounted > disk-image (btrfs loop,ro 0 0). I guess modern Linux kernel level caching > should be quite optimised for this scenario, shouldn't it? I believe > open_file_cache in my situation introduces a huge complexity surface with > possibly no upside? I'll be definitely turning it off altogether. Yes, exactly. Most likely it saves you some CPU seconds due to smaller number of syscalls made, but provides little to no real benefits. > 4. Now for the limits/connections, I feel it's bit of a deeper water. > Currently (at normal load) I have this on the multi accept off server: > > ss -Htnp state established '( sport = :80 or sport = :443 )' \ > | awk 'match($0,/pid=([0-9]+)/,m){c[m[1]]++} END{for (p in c) printf > "nginx worker pid %s: %d ESTAB\n", p, c[p]}' \ > | sort -k6,6nr > nginx worker pid 1600658: 203 ESTAB > nginx worker pid 1600659: 213 ESTAB > nginx worker pid 1600660: 211 ESTAB > nginx worker pid 1600661: 201 ESTAB > nginx worker pid 1600662: 220 ESTAB > nginx worker pid 1600663: 214 ESTAB > nginx worker pid 1600664: 213 ESTAB > nginx worker pid 1600665: 212 ESTAB > > and this on the multi accept on one: > > nginx worker pid 1761825: 1388 ESTAB > nginx worker pid 1761827: 114 ESTAB > nginx worker pid 1761828: 6 ESTAB So it looks like even the default worker_connections (512) will work for you under normal conditions. But, obviously enough, there should be some reserve to handle load spikes. > Isn't the default http2 128 streams a bit of a too high value? Yep, it comes from HTTP/2 specification, which says (https://datatracker.ietf.org/doc/html/rfc9113#section-6.5.2-2.6.1): : It is recommended that this value be no smaller than 100, so as to : not unnecessarily limit parallelism. Yet I think that it might be a good idea to actually limit parallelism to a much smaller value by default. > What do you think about > > worker_connections 8192; > > http2_max_concurrent_streams 32; > > => 8192 * (32+1) = 270,336 < 300k Looks good to me. > Also what do you think about adding http2_idle_timeout 30s; There is no http2_idle_timeout since nginx 1.19.7, keepalive_timeout is used instead for both HTTP/1.x and HTTP/2, and the default is 75s. Also, idle connections in HTTP/2, as well as keepalive connections in HTTP/1.x, are subject to LRU-based reuse if there aren't enough worker_connections, effectively auto-tuning keepalive_timeout. As such, manual tuning to reduce connection usage isn't really needed. -- Maxim Dounin http://mdounin.ru/ From ltning-nginx at anduin.net Wed Aug 20 11:06:33 2025 From: ltning-nginx at anduin.net (ltning-nginx at anduin.net) Date: Wed, 20 Aug 2025 13:06:33 +0200 Subject: Status of shared memory zones Message-ID: Hi, Let me start with a (belated) thank you to all contributors for the continued work on freenginx. We are in the process of switching to it - late but hopefully good! In the commercial F5 nginx, there has for some time been ways to monitor the status/usage of shared memory zones and (presumably) some other run-time stats that remain opaque to users of the community version. I'm wondering if there are any such facilities in freenginx - implemented or planned? It would be nice if the VTS plugin could emit some statistics, for example. We would be quite interested in such a feature, but realise it will be a fair amount of work for a relatively narrow audience. Is there any way we can "encourage" such functionality to be added, e.g. through financial support or some other means? We might even be able to produce code to do it, but would prefer if it was done by someone who is already familiar with (free)nginx and its architecture - we have no such people on-hand. Thanks again, wbr /Eirik From mdounin at mdounin.ru Wed Aug 20 16:08:07 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 20 Aug 2025 19:08:07 +0300 Subject: Status of shared memory zones In-Reply-To: References: Message-ID: Hello! On Wed, Aug 20, 2025 at 01:06:33PM +0200, Eirik ?verby via nginx wrote: > Let me start with a (belated) thank you to all contributors for > the continued work on freenginx. We are in the process of > switching to it - late but hopefully good! > > In the commercial F5 nginx, there has for some time been ways to > monitor the status/usage of shared memory zones and (presumably) > some other run-time stats that remain opaque to users of the > community version. I'm wondering if there are any such > facilities in freenginx - implemented or planned? It would be > nice if the VTS plugin could emit some statistics, for example. > > We would be quite interested in such a feature, but realise it > will be a fair amount of work for a relatively narrow audience. > Is there any way we can "encourage" such functionality to be > added, e.g. through financial support or some other means? We > might even be able to produce code to do it, but would prefer if > it was done by someone who is already familiar with (free)nginx > and its architecture - we have no such people on-hand. Providing better status information might indeed be an interesting area for improvements. Right now, the only status information provided by freenginx is that shown by stub_status, which is quite limited and basically only shows connections in various states and total accepts/requests counters. Everything else needs to be monitored either via logs or with 3rd party modules, like VTS you've mentioned. It would be great if you could share which stats you use/monitor, and how. In particular, it would be interesting to know preferred data formats. It would also be interesting to know how you expect to use information about shared memory zones usage. As for improving the VTS module, I don't think it's specific to freenginx. Information about shared memory zone usage is in the ngx_slab_pool_t structure since nginx 1.11.7, and it's more or less a question of showing this information properly. Just in case, relevant commits are: changeset: 6828:99770a5ea14f user: Ruslan Ermilov date: Wed Dec 07 22:25:37 2016 +0300 summary: Slab: slots statistics. changeset: 6829:6e757036e588 user: Ruslan Ermilov date: Wed Dec 07 22:25:37 2016 +0300 summary: Slab: free pages statistics. https://freenginx.org/hg/nginx/rev/99770a5ea14f https://freenginx.org/hg/nginx/rev/6e757036e588 -- Maxim Dounin http://mdounin.ru/ From jsabater at gmail.com Thu Aug 21 07:38:16 2025 From: jsabater at gmail.com (Jaume Sabater) Date: Thu, 21 Aug 2025 09:38:16 +0200 Subject: Status of shared memory zones In-Reply-To: References: Message-ID: Related to the information provided by NGINX via stub_status, I would like to suggest implementing ways to get feedback on the upstream servers, so one could figure out when they are done with requests and updates on the upstream node can start again. For example, let's say I have an upstream "myapp" with two servers, " myapp1.domain.com" and "myapp2.domain.com". I add the "down" option to the first one and reload. Then I have to wait for the server to finish off processing requests to that first server before deploying the new version of the "myapp" application. Once completed, I need to remove the "down" option, add it to the other server, then reload. Not sure if this is the expected way to do things, of course, but right now I need to play with console tools to check for connections, which is not as reliable as having NGINX inform about, say, they number of active requests with " myapp1.domain.com". Thanks. On Wed, Aug 20, 2025 at 6:15?PM Maxim Dounin wrote: > Hello! > > On Wed, Aug 20, 2025 at 01:06:33PM +0200, Eirik ?verby via nginx wrote: > > > Let me start with a (belated) thank you to all contributors for > > the continued work on freenginx. We are in the process of > > switching to it - late but hopefully good! > > > > In the commercial F5 nginx, there has for some time been ways to > > monitor the status/usage of shared memory zones and (presumably) > > some other run-time stats that remain opaque to users of the > > community version. I'm wondering if there are any such > > facilities in freenginx - implemented or planned? It would be > > nice if the VTS plugin could emit some statistics, for example. > > > > We would be quite interested in such a feature, but realise it > > will be a fair amount of work for a relatively narrow audience. > > Is there any way we can "encourage" such functionality to be > > added, e.g. through financial support or some other means? We > > might even be able to produce code to do it, but would prefer if > > it was done by someone who is already familiar with (free)nginx > > and its architecture - we have no such people on-hand. > > Providing better status information might indeed be an interesting > area for improvements. > > Right now, the only status information provided by freenginx is > that shown by stub_status, which is quite limited and basically > only shows connections in various states and total > accepts/requests counters. Everything else needs to be monitored > either via logs or with 3rd party modules, like VTS you've > mentioned. > > It would be great if you could share which stats you use/monitor, > and how. In particular, it would be interesting to know preferred > data formats. > > It would also be interesting to know how you expect to use > information about shared memory zones usage. > > As for improving the VTS module, I don't think it's specific to > freenginx. Information about shared memory zone usage is in the > ngx_slab_pool_t structure since nginx 1.11.7, and it's more or > less a question of showing this information properly. Just in > case, relevant commits are: > > changeset: 6828:99770a5ea14f > user: Ruslan Ermilov > date: Wed Dec 07 22:25:37 2016 +0300 > summary: Slab: slots statistics. > > changeset: 6829:6e757036e588 > user: Ruslan Ermilov > date: Wed Dec 07 22:25:37 2016 +0300 > summary: Slab: free pages statistics. > > https://freenginx.org/hg/nginx/rev/99770a5ea14f > https://freenginx.org/hg/nginx/rev/6e757036e588 > > -- > Maxim Dounin > http://mdounin.ru/ > -- Jaume Sabater "Ubi sapientas ibi libertas" -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu Aug 21 14:05:24 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 21 Aug 2025 17:05:24 +0300 Subject: Status of shared memory zones In-Reply-To: References: Message-ID: Hello! On Thu, Aug 21, 2025 at 09:38:16AM +0200, Jaume Sabater wrote: > Related to the information provided by NGINX via stub_status, I would like > to suggest implementing ways to get feedback on the upstream servers, so > one could figure out when they are done with requests and updates on the > upstream node can start again. > > For example, let's say I have an upstream "myapp" with two servers, " > myapp1.domain.com" and "myapp2.domain.com". I add the "down" option to the > first one and reload. Then I have to wait for the server to finish off > processing requests to that first server before deploying the new version > of the "myapp" application. Once completed, I need to remove the "down" > option, add it to the other server, then reload. Not sure if this is the > expected way to do things, of course, but right now I need to play with > console tools to check for connections, which is not as reliable as having > NGINX inform about, say, they number of active requests with " > myapp1.domain.com". Thanks for the input. With reloads, I don't think implementing this would be trivial and can be done efficiently, as this information needs to be collected and updated in shared memory. Even when using upstream shared memory zones, since these are recreated on configuration reloads, so the new worker processes don't have access to data of old upstream servers. Just in case, the most simple approach for the particular use case would be to monitor old worker processes - and when they are all done, you can be sure the upstream server in question is no longer used. This should work even if the same server is used elsewhere, and you cannot rely on checking connections and/or status of the upstream server itself. -- Maxim Dounin http://mdounin.ru/ From jsabater at gmail.com Thu Aug 21 14:19:24 2025 From: jsabater at gmail.com (Jaume Sabater) Date: Thu, 21 Aug 2025 16:19:24 +0200 Subject: Status of shared memory zones In-Reply-To: References: Message-ID: Hi, Maxim, and thanks for the feedback. My approach is to monitor the downstream NGINX server, whereas I think that your approach points at monitoring the upstream Gunicorn, isn't it? If so, then yes, that's another way to do it. I have never checked whether Gunicorn (in my particular case) offers that information, but it would definitely be one way to do it. Anyhow, I still would love to be able to query NGINX for much more information about active requests and upstreams but, unfortunately, I cannot be of much help with the internals of NGINX. Thanks. On Thu, Aug 21, 2025 at 4:05?PM Maxim Dounin wrote: > Hello! > > On Thu, Aug 21, 2025 at 09:38:16AM +0200, Jaume Sabater wrote: > > > Related to the information provided by NGINX via stub_status, I would > like > > to suggest implementing ways to get feedback on the upstream servers, so > > one could figure out when they are done with requests and updates on the > > upstream node can start again. > > > > For example, let's say I have an upstream "myapp" with two servers, " > > myapp1.domain.com" and "myapp2.domain.com". I add the "down" option to > the > > first one and reload. Then I have to wait for the server to finish off > > processing requests to that first server before deploying the new version > > of the "myapp" application. Once completed, I need to remove the "down" > > option, add it to the other server, then reload. Not sure if this is the > > expected way to do things, of course, but right now I need to play with > > console tools to check for connections, which is not as reliable as > having > > NGINX inform about, say, they number of active requests with " > > myapp1.domain.com". > > Thanks for the input. > > With reloads, I don't think implementing this would be trivial and > can be done efficiently, as this information needs to be collected > and updated in shared memory. Even when using upstream shared > memory zones, since these are recreated on configuration reloads, > so the new worker processes don't have access to data of old > upstream servers. > > Just in case, the most simple approach for the particular use case > would be to monitor old worker processes - and when they are all > done, you can be sure the upstream server in question is no longer > used. This should work even if the same server is used elsewhere, > and you cannot rely on checking connections and/or status of the > upstream server itself. > > -- > Maxim Dounin > http://mdounin.ru/ > -- Jaume Sabater "Ubi sapientas ibi libertas" -------------- next part -------------- An HTML attachment was scrubbed... URL: From ltning-nginx at anduin.net Thu Aug 21 14:29:00 2025 From: ltning-nginx at anduin.net (=?UTF-8?Q?Eirik_=C3=98verby?=) Date: Thu, 21 Aug 2025 16:29:00 +0200 Subject: Status of shared memory zones In-Reply-To: References: Message-ID: <5d105010-9280-4ea4-8c31-029e9aa08a3d@anduin.net> On 8/20/25 18:08, Maxim Dounin wrote: > On Wed, Aug 20, 2025 at 01:06:33PM +0200, Eirik ?verby via nginx wrote: >> In the commercial F5 nginx, there has for some time been ways to >> monitor the status/usage of shared memory zones and (presumably) >> some other run-time stats that remain opaque to users of the >> community version. I'm wondering if there are any such >> facilities in freenginx - implemented or planned? It would be >> nice if the VTS plugin could emit some statistics, for example. >> >> We would be quite interested in such a feature, but realise it >> will be a fair amount of work for a relatively narrow audience. >> Is there any way we can "encourage" such functionality to be >> added, e.g. through financial support or some other means? We >> might even be able to produce code to do it, but would prefer if >> it was done by someone who is already familiar with (free)nginx >> and its architecture - we have no such people on-hand. > > Providing better status information might indeed be an interesting > area for improvements. > > Right now, the only status information provided by freenginx is > that shown by stub_status, which is quite limited and basically > only shows connections in various states and total > accepts/requests counters. Everything else needs to be monitored > either via logs or with 3rd party modules, like VTS you've > mentioned. > > It would be great if you could share which stats you use/monitor, > and how. In particular, it would be interesting to know preferred > data formats. We use the VTS extension to monitor a number of things, including connection counts, latencies, etc. We use Prometheus for this, so the data is collected by the extension and then scraped by Prometheus at regular intervals. The VTS extensions emits a lot of information, we are not even close to making use of all of it, but collecting it allows us to drill into the data e.g. after an incident or when trying to understand application behaviour. Simplified example config we use: -- vhost_traffic_status_zone; vhost_traffic_status_histogram_buckets 0.005 0.01 0.05 0.1 0.2 0.5 1 5 10; ... location ~ ^/foo/(.*)/bar { set $loc $1; # Get stats for individual locations and server ports vhost_traffic_status_filter_by_set_key $server_port $loc:$request_method; .... } location /status { vhost_traffic_status_display; } -- We are currently not using the nginx-internal status endpoint, as I believe VTS gives us everything it provides - but if the shared memory zone info would be available there, we'd of course start using it. :) > It would also be interesting to know how you expect to use > information about shared memory zones usage. I'm mostly concerned about the (apparent?) total lack of information available to guide sizing of shared memory zones. For rate limiting based purely on source IP it's "easy" to calculate a worst-case scenario, but in our case we do rate limiting on a number of different keys - like an md5 of the request string (hard lesson learned there..), combinations of IP and other request properties, etc. As for what might be interesting to know, off the top of my head: - information about utilization (%/bytes) vs max size - information about number of entries - if possible the age distribution of entries (to know if entries likely to be evicted next could be too young, therefore suggesting a larger zone is needed) - something to see/predict possible fragmentation problems (not sure exactly what that would be, but knowing what the max size of a new element is at any given time might be an indicator. If it's low, it might be a problem; if it fluctuates a lot, that might be a different problem..) > As for improving the VTS module, I don't think it's specific to > freenginx. Information about shared memory zone usage is in the > ngx_slab_pool_t structure since nginx 1.11.7, and it's more or > less a question of showing this information properly. Just in > case, relevant commits are: > > changeset: 6828:99770a5ea14f > user: Ruslan Ermilov > date: Wed Dec 07 22:25:37 2016 +0300 > summary: Slab: slots statistics. > > changeset: 6829:6e757036e588 > user: Ruslan Ermilov > date: Wed Dec 07 22:25:37 2016 +0300 > summary: Slab: free pages statistics. > > https://freenginx.org/hg/nginx/rev/99770a5ea14f > https://freenginx.org/hg/nginx/rev/6e757036e588 Are you saying I should speak to the VTS author/project? I can try that of course, but it would be good to know what data is available first, so I don't send anyone (else) on a wild goose chase :) Thanks, /Eirik From mdounin at mdounin.ru Thu Aug 21 16:33:55 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 21 Aug 2025 19:33:55 +0300 Subject: Status of shared memory zones In-Reply-To: <5d105010-9280-4ea4-8c31-029e9aa08a3d@anduin.net> References: <5d105010-9280-4ea4-8c31-029e9aa08a3d@anduin.net> Message-ID: Hello! On Thu, Aug 21, 2025 at 04:29:00PM +0200, Eirik ?verby via nginx wrote: > On 8/20/25 18:08, Maxim Dounin wrote: > > On Wed, Aug 20, 2025 at 01:06:33PM +0200, Eirik ?verby via nginx wrote: > > > In the commercial F5 nginx, there has for some time been ways to > > > monitor the status/usage of shared memory zones and (presumably) > > > some other run-time stats that remain opaque to users of the > > > community version. I'm wondering if there are any such > > > facilities in freenginx - implemented or planned? It would be > > > nice if the VTS plugin could emit some statistics, for example. > > > > > > We would be quite interested in such a feature, but realise it > > > will be a fair amount of work for a relatively narrow audience. > > > Is there any way we can "encourage" such functionality to be > > > added, e.g. through financial support or some other means? We > > > might even be able to produce code to do it, but would prefer if > > > it was done by someone who is already familiar with (free)nginx > > > and its architecture - we have no such people on-hand. > > > > Providing better status information might indeed be an interesting > > area for improvements. > > > > Right now, the only status information provided by freenginx is > > that shown by stub_status, which is quite limited and basically > > only shows connections in various states and total > > accepts/requests counters. Everything else needs to be monitored > > either via logs or with 3rd party modules, like VTS you've > > mentioned. > > > > It would be great if you could share which stats you use/monitor, > > and how. In particular, it would be interesting to know preferred > > data formats. > > We use the VTS extension to monitor a number of things, > including connection counts, latencies, etc. We use Prometheus > for this, so the data is collected by the extension and then > scraped by Prometheus at regular intervals. The VTS extensions > emits a lot of information, we are not even close to making use > of all of it, but collecting it allows us to drill into the data > e.g. after an incident or when trying to understand application > behaviour. > > Simplified example config we use: > > -- > vhost_traffic_status_zone; > vhost_traffic_status_histogram_buckets 0.005 0.01 0.05 0.1 0.2 > 0.5 1 5 10; > > ... > > location ~ ^/foo/(.*)/bar { > set $loc $1; > # Get stats for individual locations and server ports > vhost_traffic_status_filter_by_set_key $server_port > $loc:$request_method; > .... > } > > location /status { > vhost_traffic_status_display; > } > -- Thanks for the details, interesting. > We are currently not using the nginx-internal status endpoint, > as I believe VTS gives us everything it provides - but if the > shared memory zone info would be available there, we'd of course > start using it. :) As far as I can see, VTS incorporates exiting stub_status data, so this is quite logical. :) > > It would also be interesting to know how you expect to use > > information about shared memory zones usage. > > I'm mostly concerned about the (apparent?) total lack of > information available to guide sizing of shared memory zones. > For rate limiting based purely on source IP it's "easy" to > calculate a worst-case scenario, but in our case we do rate > limiting on a number of different keys - like an md5 of the > request string (hard lesson learned there..), combinations of IP > and other request properties, etc. Ok, understood. That's indeed might be interesting to look into shared memory stats when using limit_req with such a different keys. > As for what might be interesting to know, off the top of my > head: > - information about utilization (%/bytes) vs max size > - information about number of entries > - if possible the age distribution of entries (to know if > entries likely to be evicted next could be too young, therefore > suggesting a larger zone is needed) > - something to see/predict possible fragmentation problems (not > sure exactly what that would be, but knowing what the max size > of a new element is at any given time might be an indicator. If > it's low, it might be a problem; if it fluctuates a lot, that > might be a different problem..) The "age distribution" part seems to be specific to limit_req, but for everything else I think existing slab allocator stats should be enough. > > As for improving the VTS module, I don't think it's specific > > to > > freenginx. Information about shared memory zone usage is in > > the > > ngx_slab_pool_t structure since nginx 1.11.7, and it's more or > > less a question of showing this information properly. Just in > > case, relevant commits are: > > > > changeset: 6828:99770a5ea14f > > user: Ruslan Ermilov > > date: Wed Dec 07 22:25:37 2016 +0300 > > summary: Slab: slots statistics. > > > > changeset: 6829:6e757036e588 > > user: Ruslan Ermilov > > date: Wed Dec 07 22:25:37 2016 +0300 > > summary: Slab: free pages statistics. > > > > https://freenginx.org/hg/nginx/rev/99770a5ea14f > > https://freenginx.org/hg/nginx/rev/6e757036e588 > > Are you saying I should speak to the VTS author/project? I can > try that of course, but it would be good to know what data is > available first, so I don't send anyone (else) on a wild goose > chase :) Well, if the goal is to get it in VTS, as you've stated in the original message ("would be nice if the VTS plugin could emit some statistics"), then yes. Otherwise, these probably will be eventually included into stub_status along with improvements, but no ETA for now. Commit logs of the referenced changesets mention the information available: its the number of free pages remained in the slab pool (that is, completely free memory in shared memory zone allocator; total number of pages can be calculated from the slab pool size), as well as total and allocated entries in each slab slot (set of allocations of a particular size). It might be somewhat risky for a 3rd party module to use it though, at least per-slot statistics, since these relies heavily on the implementation details of the slab allocator, which might change at some point, and the existing per-slot stats structures provide no safety belts. OTOH, in the worst case it will just show garbage. Also, the number of free pages seems completely safe and should be enough to provide basic information about shared memory zone usage. No free pages doesn't necessary mean that the next allocation will fail, since there might be quite a few free entries in the already allocated pages, but a good indicator that the zone is almost full. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Tue Aug 26 15:23:43 2025 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 26 Aug 2025 18:23:43 +0300 Subject: freenginx-1.29.1 Message-ID: Changes with freenginx 1.29.1 26 Aug 2025 *) Change: now responses from proxied servers over HTTP/0.9 are rejected as invalid by default; the "proxy_allow_http09" directive allows processing of such responses. *) Change: stricter syntax checks are now applied to the "Host" request header line. *) Feature: now interim 1xx responses of proxied servers are ignored. *) Feature: certificates for IP addresses are now supported when verifying backend SSL certificates. *) Feature: the "proxy_allow_duplicate_chunked" directive. Thanks to Gennady Bekasov. *) Bugfix: when using a host name in the request line, the "_" and some other characters were not allowed, as well as zone identifiers in IPv6 addresses. *) Workaround: "shutdown() failed (22: Invalid argument)" alerts might appear in logs on NetBSD. *) Bugfix: in the mail proxy module. -- Maxim Dounin http://freenginx.org/ From paul at stormy.ca Fri Aug 29 00:13:26 2025 From: paul at stormy.ca (Paul) Date: Thu, 28 Aug 2025 20:13:26 -0400 Subject: Using 444 Message-ID: I'm looking for advice, please. Using Nginx v1.18.0 (Ubuntu) which is "old" but security updated by Canonical, rock solid and very fast, for several static html sites and as proxy to a couple of other sites using python or perl. Total ~250k requests/day Recently logs have started showing ~10k php requests in rapid bursts. On a proxy to a perl box, this is a serious slow down I've added the following, appears to work well location ~ \.php$ { if ($request_method = GET) { return 444; # Drop } } I'm considering editing to ^(GET|HEAD|POST)$) { Any thoughts, downsides, recommendations? Tnx and warmest regards to all, Paul From bctrainers at gmail.com Fri Aug 29 01:36:52 2025 From: bctrainers at gmail.com (Brett Cooper) Date: Fri, 29 Aug 2025 01:36:52 +0000 Subject: Using 444 In-Reply-To: References: Message-ID: Hi Paul, If that server block is only serving Perl and nothing else at all, you should be fine to just strip out anything PHP-extension request-like without using the if statement. Judging with what you've stated, and without having seen access/error logs, it might be best to simply use the following for the Perl server configuration block: location ~ \.php$ { return 444; } It may also be pertinent to utilize `access_log off; ` and `log_not_found off;` within that location clause if you are also having resource/log-storage issues. Additionally, if the bogus PHP requests are coming from clients not sending a user agent (which from my POV, has been a common theme with probing bots lately), you could also configure this within the overall server {} block: if ($http_user_agent = "") { return 444; } Regards, Brett ------ Original Message ------ >From "Paul" To nginx at freenginx.org Date 08/28/2025 07:13:26 P Subject Using 444 >I'm looking for advice, please. Using Nginx v1.18.0 (Ubuntu) which is "old" but security updated by Canonical, rock solid and very fast, for several static html sites and as proxy to a couple of other sites using python or perl. Total ~250k requests/day > >Recently logs have started showing ~10k php requests in rapid bursts. On a proxy to a perl box, this is a serious slow down > >I've added the following, appears to work well > >location ~ \.php$ { > if ($request_method = GET) { > return 444; # Drop > } >} > >I'm considering editing to ^(GET|HEAD|POST)$) { > >Any thoughts, downsides, recommendations? > >Tnx and warmest regards to all, >Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: