Status of shared memory zones

[Maxim Dounin]

Hello!

On Thu, Aug 21, 2025 at 04:29:00PM +0200, Eirik Øverby via nginx wrote:

> On 8/20/25 18:08, Maxim Dounin wrote:
> > On Wed, Aug 20, 2025 at 01:06:33PM +0200, Eirik Øverby via nginx wrote:
> > > In the commercial F5 nginx, there has for some time been ways to
> > > monitor the status/usage of shared memory zones and (presumably)
> > > some other run-time stats that remain opaque to users of the
> > > community version. I'm wondering if there are any such
> > > facilities in freenginx - implemented or planned? It would be
> > > nice if the VTS plugin could emit some statistics, for example.
> > > 
> > > We would be quite interested in such a feature, but realise it
> > > will be a fair amount of work for a relatively narrow audience.
> > > Is there any way we can "encourage" such functionality to be
> > > added, e.g. through financial support or some other means? We
> > > might even be able to produce code to do it, but would prefer if
> > > it was done by someone who is already familiar with (free)nginx
> > > and its architecture - we have no such people on-hand.
> > 
> > Providing better status information might indeed be an interesting
> > area for improvements.
> > 
> > Right now, the only status information provided by freenginx is
> > that shown by stub_status, which is quite limited and basically
> > only shows connections in various states and total
> > accepts/requests counters.  Everything else needs to be monitored
> > either via logs or with 3rd party modules, like VTS you've
> > mentioned.
> > 
> > It would be great if you could share which stats you use/monitor,
> > and how.  In particular, it would be interesting to know preferred
> > data formats.
> 
> We use the VTS extension to monitor a number of things, 
> including connection counts, latencies, etc. We use Prometheus 
> for this, so the data is collected by the extension and then 
> scraped by Prometheus at regular intervals. The VTS extensions 
> emits a lot of information, we are not even close to making use 
> of all of it, but collecting it allows us to drill into the data 
> e.g. after an incident or when trying to understand application 
> behaviour.
> 
> Simplified example config we use:
> 
> --
> vhost_traffic_status_zone;
> vhost_traffic_status_histogram_buckets 0.005 0.01 0.05 0.1 0.2 
> 0.5 1 5 10;
> 
> ...
> 
> location ~ ^/foo/(.*)/bar {
>     set $loc $1;
>     # Get stats for individual locations and server ports
>     vhost_traffic_status_filter_by_set_key $server_port 
>     $loc:$request_method;
>     ....
> }
> 
> location /status {
>     vhost_traffic_status_display;
> }
> --

Thanks for the details, interesting.

> We are currently not using the nginx-internal status endpoint, 
> as I believe VTS gives us everything it provides - but if the 
> shared memory zone info would be available there, we'd of course 
> start using it. :)

As far as I can see, VTS incorporates exiting stub_status data, 
so this is quite logical. :)

> > It would also be interesting to know how you expect to use
> > information about shared memory zones usage.
> 
> I'm mostly concerned about the (apparent?) total lack of 
> information available to guide sizing of shared memory zones. 
> For rate limiting based purely on source IP it's "easy" to 
> calculate a worst-case scenario, but in our case we do rate 
> limiting on a number of different keys - like an md5 of the 
> request string (hard lesson learned there..), combinations of IP 
> and other request properties, etc.

Ok, understood.

That's indeed might be interesting to look into shared memory 
stats when using limit_req with such a different keys.

> As for what might be interesting to know, off the top of my 
> head:
>  - information about utilization (%/bytes) vs max size
>  - information about number of entries
>  - if possible the age distribution of entries (to know if 
>  entries likely to be evicted next could be too young, therefore 
>  suggesting a larger zone is needed)
>  - something to see/predict possible fragmentation problems (not 
>  sure exactly what that would be, but knowing what the max size 
>  of a new element is at any given time might be an indicator. If 
>  it's low, it might be a problem; if it fluctuates a lot, that 
>  might be a different problem..)

The "age distribution" part seems to be specific to limit_req, but 
for everything else I think existing slab allocator stats should 
be enough.

> > As for improving the VTS module, I don't think it's specific 
> > to
> > freenginx.  Information about shared memory zone usage is in 
> > the
> > ngx_slab_pool_t structure since nginx 1.11.7, and it's more or
> > less a question of showing this information properly.  Just in
> > case, relevant commits are:
> > 
> > changeset:   6828:99770a5ea14f
> > user:        Ruslan Ermilov <ru at nginx.com>
> > date:        Wed Dec 07 22:25:37 2016 +0300
> > summary:     Slab: slots statistics.
> > 
> > changeset:   6829:6e757036e588
> > user:        Ruslan Ermilov <ru at nginx.com>
> > date:        Wed Dec 07 22:25:37 2016 +0300
> > summary:     Slab: free pages statistics.
> > 
> > https://freenginx.org/hg/nginx/rev/99770a5ea14f
> > https://freenginx.org/hg/nginx/rev/6e757036e588
> 
> Are you saying I should speak to the VTS author/project? I can 
> try that of course, but it would be good to know what data is 
> available first, so I don't send anyone (else) on a wild goose 
> chase :)

Well, if the goal is to get it in VTS, as you've stated in the 
original message ("would be nice if the VTS plugin could emit some 
statistics"), then yes.  Otherwise, these probably will be 
eventually included into stub_status along with improvements, but 
no ETA for now.

Commit logs of the referenced changesets mention the information 
available: its the number of free pages remained in the slab pool 
(that is, completely free memory in shared memory zone allocator; 
total number of pages can be calculated from the slab pool size), 
as well as total and allocated entries in each slab slot (set of 
allocations of a particular size).

It might be somewhat risky for a 3rd party module to use it 
though, at least per-slot statistics, since these relies heavily 
on the implementation details of the slab allocator, which might 
change at some point, and the existing per-slot stats structures 
provide no safety belts.  OTOH, in the worst case it will just 
show garbage.  Also, the number of free pages seems completely 
safe and should be enough to provide basic information about 
shared memory zone usage.  No free pages doesn't necessary mean 
that the next allocation will fail, since there might be quite a 
few free entries in the already allocated pages, but a good 
indicator that the zone is almost full.

-- 
Maxim Dounin
http://mdounin.ru/