<html><body><div dir="ltr">Hi,</div><div dir="ltr"><br></div><div dir="ltr">The peak was CPU 9%, IOWait, 4.7%, User: 2%, System: 2.65%</div><div dir="ltr">Ram is 6% used, 90% cached.</div><div dir="ltr"><br></div><div dir="ltr">Zsolt</div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On 10. Aug 2025 at 08:32:10, Xiufeng Guo <<a href="mailto:showfom@gmail.com">showfom@gmail.com</a>> wrote:<br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" type="cite">
<div dir="auto">Hi,</div><div dir="auto"><br></div><div dir="auto">What’s the average server load?<br clear="all"><br clear="all"><div dir="auto"><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Best Regards,</div><div>Xiufeng Guo<br></div></div></div></div></div><div><br></div><div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sun, Aug 10, 2025 at 02:04 Zsolt Ero <<a href="mailto:zsolt.ero@gmail.com">zsolt.ero@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div><div><span style="font-family:ui-sans-serif;font-size:16px;white-space:pre-wrap">Hi,</span><br><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">I'm seeking advice on the most robust way to configure Nginx for a specific scenario that led to a caching issue.</span></p><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">I run a free vector tile map service (</span><span style="font-family:ui-sans-serif"><a href="https://openfreemap.org/" target="_blank" style="font-family:ui-sans-serif">https://openfreemap.org/</a></span><span style="font-family:ui-sans-serif">). The server's primary job is to serve a massive number of small (~70 kB), pre-gzipped PBF files.</span></p><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">To optimize for ocean areas, tiles that don't exist on disk should be served as a </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">200 OK</code></span><span style="font-family:ui-sans-serif"> with an empty body. These are then rendered as empty space on the map.</span></p><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Recently, the server experienced an extremely high load: 100k req/sec on Cloudflare, and 1k req/sec on my two Hetzner servers. During this peak, Nginx started serving some </span><span style="font-family:ui-sans-serif"><em style="font-family:ui-sans-serif">existing</em></span><span style="font-family:ui-sans-serif"> tiles as empty bodies. Because these responses included cache-friendly headers (</span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">expires 10y</code></span><span style="font-family:ui-sans-serif">), the CDN cached the incorrect empty responses, effectively making parts of the map disappear until a manual cache purge was performed.</span></p><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">My goal is to prevent this from happening again. A temporary server overload should result in a server error (e.g., </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">5xx</code></span><span style="font-family:ui-sans-serif">), not incorrect content that gets permanently cached.</span></p><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">The Nginx error logs clearly showed the root cause of the system error:</span></p><pre lang="" style="box-sizing:border-box;overflow:visible;font-size:0.9em;break-inside:avoid;background-image:inherit;background-position:inherit;background-size:inherit;background-repeat:inherit;background-origin:inherit;background-clip:inherit;border:1px solid rgb(231,234,237);border-radius:3px;padding:8px 4px 6px;margin-bottom:15px;margin-top:15px;width:inherit;background-color:rgb(248,248,248)"><span role="presentation">2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files), client: 172.69.122.170, server: ...</span></pre><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">It appears my </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">try_files</code></span><span style="font-family:ui-sans-serif"> directive interpreted this "Too many open files" error as a "file not found" condition and fell back to serving the empty tile.</span></p><h4 style="white-space:pre-wrap;box-sizing:border-box;break-after:avoid-page;break-inside:avoid;font-size:1.25em;margin-top:1rem;margin-bottom:1rem;line-height:1.4;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">System and Nginx Diagnostic Information</span></h4><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Here is the relevant information about the system and Nginx process state (captured at normal load, after I solved the high traffic incident, still showing high FD usage on one worker).</span></p><ul style="font-size:16px;box-sizing:border-box;margin:0.8em 0px;padding-left:30px;font-family:ui-sans-serif"><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif">OS:</strong></span><span style="font-family:ui-sans-serif"> Ubuntu 22.04 LTS, 64 GB RAM, local NVME SSD, physical server (not VPS)</span></p></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif">nginx version</strong></span><span style="font-family:ui-sans-serif">: nginx/1.27.4</span></p></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Systemd </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">ulimit</code></span><span style="font-family:ui-sans-serif"> for </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">nofile</code></span><span style="font-family:ui-sans-serif">:</span></strong></span></p><pre lang="" style="font-family:monospace"><span role="presentation" style="font-family:monospace"># cat /etc/security/limits.d/limits1m.conf</span><br><span role="presentation" style="font-family:monospace">- soft nofile 1048576</span><br><span role="presentation" style="font-family:monospace">- hard nofile 1048576</span></pre></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Nginx Worker Process Limits (</span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">worker_rlimit_nofile</code></span><span style="font-family:ui-sans-serif"> is set to </span><span style="font-family:ui-sans-serif"><code style="font-family:monospace">300000</code></span><span style="font-family:ui-sans-serif">):</span></strong></span></p><pre lang="bash" style="font-family:monospace"><span role="presentation" style="font-family:monospace"><span style="font-family:monospace"># for pid in $(pgrep -f "nginx: worker"); do sudo cat /proc/$pid/limits | grep "Max open files"; done</span></span><br><span role="presentation" style="font-family:monospace">Max open files <span style="font-family:monospace">300000</span> <span style="font-family:monospace">300000</span> files</span><br><span role="presentation" style="font-family:monospace">Max open files <span style="font-family:monospace">300000</span> <span style="font-family:monospace">300000</span> files</span><br><span role="presentation" style="font-family:monospace">... (all <span style="font-family:monospace">8</span> workers show the same limit)</span></pre></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif">Open File Descriptor Count per Worker:</strong></span></p><pre lang="bash" style="font-family:monospace"><span role="presentation" style="font-family:monospace"><span style="font-family:monospace"># for pid in $(pgrep -f "nginx: worker"); do count=$(sudo lsof -p $pid 2>/dev/null | wc -l); echo "nginx PID $pid: $count open files"; done</span></span><br><span role="presentation" style="font-family:monospace">nginx PID <span style="font-family:monospace">1090</span>: <span style="font-family:monospace">57</span> open files</span><br><span role="presentation" style="font-family:monospace">nginx PID <span style="font-family:monospace">1091</span>: <span style="font-family:monospace">117</span> open files</span><br><span role="presentation" style="font-family:monospace">nginx PID <span style="font-family:monospace">1092</span>: <span style="font-family:monospace">931</span> open files</span><br><span role="presentation" style="font-family:monospace">nginx PID <span style="font-family:monospace">1093</span>: <span style="font-family:monospace">65027</span> open files</span><br><span role="presentation" style="font-family:monospace">nginx PID <span style="font-family:monospace">1094</span>: <span style="font-family:monospace">7449</span> open files</span><br><span role="presentation" style="font-family:monospace">...</span></pre><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">(Note the one worker with a very high count, ~98% of which are regular files).</span></p></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><code style="font-family:monospace">sysctl fs.file-max</code></span><span style="font-family:ui-sans-serif">:</span></p><pre lang="" style="font-family:monospace"><span role="presentation" style="font-family:monospace">fs.file-max = 9223372036854775807</span></pre></li><li style="font-family:ui-sans-serif"><p style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><code style="font-family:monospace">systemctl show nginx | grep LimitNOFILE</code></span><span style="font-family:ui-sans-serif">:</span></p><pre lang="" style="font-family:monospace"><span role="presentation" style="font-family:monospace">LimitNOFILE=524288</span><br><span role="presentation" style="font-family:monospace">LimitNOFILESoft=1024</span></pre></li></ul><h3 style="white-space:pre-wrap;box-sizing:border-box;break-after:avoid-page;break-inside:avoid;font-size:1.5em;margin-top:1rem;margin-bottom:1rem;line-height:1.43;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Relevant Nginx Configuration</span></h3><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif">Here are the key parts of my configuration that led to the issue.</span></p><pre lang="nginx" style="box-sizing:border-box;overflow:visible;font-size:0.9em;break-inside:avoid;background-image:inherit;background-position:inherit;background-size:inherit;background-repeat:inherit;background-origin:inherit;background-clip:inherit;border:1px solid rgb(231,234,237);border-radius:3px;padding:8px 4px 6px;margin-bottom:15px;margin-top:15px;width:inherit;background-color:rgb(248,248,248)"><span role="presentation"><span>worker_processes</span> auto;</span><br><span role="presentation"><span>worker_rlimit_nofile</span> 300000;</span><br><span role="presentation"><span></span></span><br><span role="presentation"><span>events</span> {</span><br><span role="presentation"> <span>worker_connections</span> 40000;</span><br><span role="presentation"> <span>multi_accept</span> on;</span><br><span role="presentation">}</span><br><span role="presentation"><span></span></span><br><span role="presentation"><span>http</span> {</span><br><span role="presentation"> <span>open_file_cache</span> max=<span>1000000 inactive</span>=60m;</span><br><span role="presentation"> <span>open_file_cache_valid</span> 60m;</span><br><span role="presentation"> <span>open_file_cache_min_uses</span> 1;</span><br><span role="presentation"> <span>open_file_cache_errors</span> on;</span><br><span role="presentation"> <span># ...</span></span></pre><p style="font-size:16px;white-space:pre-wrap;box-sizing:border-box;line-height:inherit;margin:0.8em 0px;font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><strong style="font-family:ui-sans-serif"><span style="font-family:ui-sans-serif"><code style="font-family:monospace">server</code></span><span style="font-family:ui-sans-serif"> block tile serving logic:</span></strong></span></p><pre lang="nginx" style="box-sizing:border-box;overflow:visible;font-size:0.9em;break-inside:avoid;background-image:inherit;background-position:inherit;background-size:inherit;background-repeat:inherit;background-origin:inherit;background-clip:inherit;border:1px solid rgb(231,234,237);border-radius:3px;padding:8px 4px 6px;margin-bottom:15px;margin-top:15px;width:inherit;background-color:rgb(248,248,248)"><span role="presentation"><span>location</span> <span>^~</span> /monaco/20250806_231001_pt/ {</span><br><span role="presentation"> <span>alias</span> /mnt/ofm/<span>monaco-20250806_231001_pt</span>/tiles/;</span><br><span role="presentation"> <span>try_files</span> <span>$uri </span><span>@empty_tile</span>;</span><br><span role="presentation"> <span>add_header</span> <span>Content-Encoding</span> <span>gzip</span>;</span><br><span role="presentation"><span></span></span><br><span role="presentation"> <span>expires</span> 10y;</span><br><span role="presentation"><span></span></span><br><span role="presentation"> <span>types</span> {</span><br><span role="presentation"> application/vnd.<span>mapbox-vector-tile</span> pbf;</span><br><span role="presentation"> }</span><br><span role="presentation"><span></span></span><br><span role="presentation"> <span>add_header</span> <span>'Access-Control-Allow-Origin'</span> <span>'*'</span> always;</span><br><span role="presentation" style="box-sizing: border-box; --tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; --tw-rotate: 0; --tw-skew-x: 0; --tw-skew-y: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-pan-x: ; --tw-pan-y: ; --tw-pinch-zoom: ; --tw-scroll-snap-strictness: proximity; --tw-ordinal: ; --tw-slashed-zero: ; --tw-numeric-figure: ; --tw-numeric-spacing: ; --tw-numeric-fraction: ; --tw-ring-inset: ; --tw-ring-offset-width: 0px; --tw-ring-offset-color: #fff; --tw-ring-color: rgb(59 130 246 / .5); --tw-ring-offset-shadow: 0 0 #0000; --tw-ring-shadow: 0 0 #0000; --tw-shadow: 0 0 #0000; --tw-shadow-colored: 0 0 #0000; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; </pre></div></div></blockquote></div></div>
</blockquote>
</div></body></html>