From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] QUIC: fixed close timer processing with early data.
Message-ID: <hg.da400acf3756.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/da400acf3756
branches:  
changeset: 9280:da400acf3756
user:      Vladimir Khomutov <vl at wbsrv.ru>
date:      Wed Apr 10 09:38:10 2024 +0300
description:
QUIC: fixed close timer processing with early data.

The ngx_quic_run() function uses qc->close timer to limit the handshake
duration.  Normally it is removed by ngx_quic_do_init_streams() which is
called once when we are done with initial SSL processing.

The problem happens when the client sends early data and streams are
initialized in the ngx_quic_run() -> ngx_quic_handle_datagram() call.
The order of set/remove timer calls is now reversed; the close timer is
set up and the timer fires when assigned, starting the unexpected connection
close process.

The fix is to skip setting the timer if streams were initialized during
handling of the initial datagram.  The idle timer for quic is set anyway,
and stream-related timeouts are managed by application layer.

diffstat:

 src/event/quic/ngx_event_quic.c |  5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diffs (15 lines):

diff --git a/src/event/quic/ngx_event_quic.c b/src/event/quic/ngx_event_quic.c
--- a/src/event/quic/ngx_event_quic.c
+++ b/src/event/quic/ngx_event_quic.c
@@ -211,7 +211,10 @@ ngx_quic_run(ngx_connection_t *c, ngx_qu
     qc = ngx_quic_get_connection(c);
 
     ngx_add_timer(c->read, qc->tp.max_idle_timeout);
-    ngx_add_timer(&qc->close, qc->conf->handshake_timeout);
+
+    if (!qc->streams.initialized) {
+        ngx_add_timer(&qc->close, qc->conf->handshake_timeout);
+    }
 
     ngx_quic_connstate_dbg(c);
 


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] QUIC: client transport parameter data length checking.
Message-ID: <hg.081d4beeb591.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/081d4beeb591
branches:  
changeset: 9281:081d4beeb591
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Tue May 28 17:17:19 2024 +0400
description:
QUIC: client transport parameter data length checking.

diffstat:

 src/event/quic/ngx_event_quic_transport.c |  8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diffs (18 lines):

diff --git a/src/event/quic/ngx_event_quic_transport.c b/src/event/quic/ngx_event_quic_transport.c
--- a/src/event/quic/ngx_event_quic_transport.c
+++ b/src/event/quic/ngx_event_quic_transport.c
@@ -1750,6 +1750,14 @@ ngx_quic_parse_transport_params(u_char *
             return NGX_ERROR;
         }
 
+        if ((size_t) (end - p) < len) {
+            ngx_log_error(NGX_LOG_INFO, log, 0,
+                          "quic failed to parse"
+                          " transport param id:0x%xL, data length %uL too long",
+                          id, len);
+            return NGX_ERROR;
+        }
+
         rc = ngx_quic_parse_transport_param(p, p + len, id, tp);
 
         if (rc == NGX_ERROR) {


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] HTTP/3: fixed dynamic table overflow.
Message-ID: <hg.acb8548c00e9.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/acb8548c00e9
branches:  
changeset: 9282:acb8548c00e9
user:      Roman Arutyunyan <arut at nginx.com>
date:      Tue May 28 17:18:50 2024 +0400
description:
HTTP/3: fixed dynamic table overflow.

While inserting a new entry into the dynamic table, first the entry is added,
and then older entries are evicted until table size is within capacity.  After
the first step, the number of entries may temporarily exceed the maximum
calculated from capacity by one entry, which previously caused table overflow.

The easiest way to trigger the issue is to keep adding entries with empty names
and values until first eviction.

The issue was introduced by 987bee4363d1.

diffstat:

 src/http/v3/ngx_http_v3_table.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/src/http/v3/ngx_http_v3_table.c b/src/http/v3/ngx_http_v3_table.c
--- a/src/http/v3/ngx_http_v3_table.c
+++ b/src/http/v3/ngx_http_v3_table.c
@@ -308,7 +308,7 @@ ngx_http_v3_set_capacity(ngx_connection_
     prev_max = dt->capacity / 32;
 
     if (max > prev_max) {
-        elts = ngx_alloc(max * sizeof(void *), c->log);
+        elts = ngx_alloc((max + 1) * sizeof(void *), c->log);
         if (elts == NULL) {
             return NGX_ERROR;
         }


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] QUIC: ignore CRYPTO frames after handshake completion.
Message-ID: <hg.bbdcab20d67e.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/bbdcab20d67e
branches:  
changeset: 9283:bbdcab20d67e
user:      Roman Arutyunyan <arut at nginx.com>
date:      Tue May 28 17:19:08 2024 +0400
description:
QUIC: ignore CRYPTO frames after handshake completion.

Sending handshake-level CRYPTO frames after the client's Finished message could
lead to memory disclosure and a potential segfault, if those frames are sent in
one packet with the Finished frame.

diffstat:

 src/event/quic/ngx_event_quic_ssl.c |  5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diffs (15 lines):

diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c
--- a/src/event/quic/ngx_event_quic_ssl.c
+++ b/src/event/quic/ngx_event_quic_ssl.c
@@ -326,6 +326,11 @@ ngx_quic_handle_crypto_frame(ngx_connect
     ngx_quic_crypto_frame_t  *f;
 
     qc = ngx_quic_get_connection(c);
+
+    if (!ngx_quic_keys_available(qc->keys, pkt->level, 0)) {
+        return NGX_OK;
+    }
+
     ctx = ngx_quic_get_send_ctx(qc, pkt->level);
     f = &frame->u.crypto;
 


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] QUIC: ngx_quic_buffer_t use-after-free protection.
Message-ID: <hg.5c6649b4308f.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/5c6649b4308f
branches:  
changeset: 9284:5c6649b4308f
user:      Roman Arutyunyan <arut at nginx.com>
date:      Tue May 28 17:19:21 2024 +0400
description:
QUIC: ngx_quic_buffer_t use-after-free protection.

Previously the last chain field of ngx_quic_buffer_t could still reference freed
chains and buffers after calling ngx_quic_free_buffer().  While normally an
ngx_quic_buffer_t object should not be used after freeing, resetting last_chain
field would prevent a potential use-after-free.

diffstat:

 src/event/quic/ngx_event_quic_frames.c |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (11 lines):

diff --git a/src/event/quic/ngx_event_quic_frames.c b/src/event/quic/ngx_event_quic_frames.c
--- a/src/event/quic/ngx_event_quic_frames.c
+++ b/src/event/quic/ngx_event_quic_frames.c
@@ -648,6 +648,7 @@ ngx_quic_free_buffer(ngx_connection_t *c
     ngx_quic_free_chain(c, qb->chain);
 
     qb->chain = NULL;
+    qb->last_chain = NULL;
 }
 
 


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] HTTP/3: fixed handling of zero-length literal field line.
Message-ID: <hg.4c7a9355bcae.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/4c7a9355bcae
branches:  
changeset: 9285:4c7a9355bcae
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Tue May 28 17:20:45 2024 +0400
description:
HTTP/3: fixed handling of zero-length literal field line.

Previously, st->value was passed with NULL data pointer to header handlers.

diffstat:

 src/http/v3/ngx_http_v3_parse.c |  3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diffs (27 lines):

diff --git a/src/http/v3/ngx_http_v3_parse.c b/src/http/v3/ngx_http_v3_parse.c
--- a/src/http/v3/ngx_http_v3_parse.c
+++ b/src/http/v3/ngx_http_v3_parse.c
@@ -810,6 +810,7 @@ ngx_http_v3_parse_field_lri(ngx_connecti
 
             st->literal.length = st->pint.value;
             if (st->literal.length == 0) {
+                st->value.data = (u_char *) "";
                 goto done;
             }
 
@@ -932,6 +933,7 @@ ngx_http_v3_parse_field_l(ngx_connection
 
             st->literal.length = st->pint.value;
             if (st->literal.length == 0) {
+                st->value.data = (u_char *) "";
                 goto done;
             }
 
@@ -1072,6 +1074,7 @@ ngx_http_v3_parse_field_lpbi(ngx_connect
 
             st->literal.length = st->pint.value;
             if (st->literal.length == 0) {
+                st->value.data = (u_char *) "";
                 goto done;
             }
 


From mdounin at mdounin.ru  Mon Jun  3 01:46:30 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 04:46:30 +0300
Subject: [nginx] HTTP/3: protection from recursion during connection reuse.
Message-ID: <hg.d9fe808c1841.1717379190.8630137477818021854@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/d9fe808c1841
branches:  
changeset: 9286:d9fe808c1841
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Sun Jun 02 23:51:55 2024 +0300
description:
HTTP/3: protection from recursion during connection reuse.

When draining a connection associated with an HTTP/3 stream, calling
ngx_http_v3_send_cancel_stream() might result in an attempt to obtain
a connection for the decoder stream.  This in turn will trigger draining
of the very same connection.  Depending on the client settings, this
might either lead to stack overflow or will end up in decoder stream
creation error and destroying the connection at some point, potentially
resulting in use-after-free on stack.

Fix is to make sure that connection reuse is disabled in
ngx_http_v3_reset_stream(), so the recursion in question won't happen
regardless of what called functions do.

diffstat:

 src/http/v3/ngx_http_v3_request.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff --git a/src/http/v3/ngx_http_v3_request.c b/src/http/v3/ngx_http_v3_request.c
--- a/src/http/v3/ngx_http_v3_request.c
+++ b/src/http/v3/ngx_http_v3_request.c
@@ -401,6 +401,8 @@ ngx_http_v3_reset_stream(ngx_connection_
     ngx_http_v3_session_t   *h3c;
     ngx_http_v3_srv_conf_t  *h3scf;
 
+    ngx_reusable_connection(c, 0);
+
     h3scf = ngx_http_v3_get_module_srv_conf(c, ngx_http_v3_module);
 
     h3c = ngx_http_v3_get_session(c);


From mdounin at mdounin.ru  Mon Jun  3 02:24:16 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 05:24:16 +0300
Subject: [PATCH] Tests: optimized processing of large QUIC packets with padding
Message-ID: <c7315caf211007b32788.1717381456@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1717377448 -10800
#      Mon Jun 03 04:17:28 2024 +0300
# Node ID c7315caf211007b327886fe7cd544be018c0a398
# Parent  fb25cbe9d4ec378e7a98ac83d20a82a7e5a835da
Tests: optimized processing of large QUIC packets with padding.

Path MTU discovery packets might contain a lot of padding, and creating
a copy of the whole buffer for each PADDING frame, which is just one
byte with type 0, consumes lots of resources.  This was seen to result
in flapping of at least h3_keepalive.t and h3_ssl_early_data.t tests.

Fix is to copy at most 8 bytes for parse_int() calls when parsing
frame types.

diff --git a/lib/Test/Nginx/HTTP3.pm b/lib/Test/Nginx/HTTP3.pm
--- a/lib/Test/Nginx/HTTP3.pm
+++ b/lib/Test/Nginx/HTTP3.pm
@@ -1361,7 +1361,7 @@ sub parse_frames {
 	my $offset = 0;
 
 	while ($offset < length($buf)) {
-		my ($tlen, $type) = parse_int(substr($buf, $offset));
+		my ($tlen, $type) = parse_int(substr($buf, $offset, 8));
 		$offset += $tlen;
 		next if $type == 0;
 		my $frame = { type => $type };



From robm at fastmailteam.com  Mon Jun  3 11:12:12 2024
From: robm at fastmailteam.com (Robert Mueller)
Date: Mon, 03 Jun 2024 21:12:12 +1000
Subject: [nginx] Add support for XOAUTH2 and OAUTHBEARER authentication
In-Reply-To: <Zkv11aTD96ViUulu@mdounin.ru>
References: <c5d80668-2fcc-4cb3-b47f-2234a9ef8611@app.fastmail.com>
 <Zkv11aTD96ViUulu@mdounin.ru>
Message-ID: <217bff3f-53b4-4f53-97a3-8cd13504e051@app.fastmail.com>

Hi

> What do you think about this approach?

Thanks for looking at this patch. My nginx coding experience is very limited, so I appreciate that you spent the time to look at it carefully. I haven't looked closely at the updated patch, but the changes you listed all sound like improvements and/or fixes which is great. I'm happy to see this integrated upstream, so that others can take advantage of it in the future.

Cheers

-- 
Rob Mueller
robm at fastmailteam.com


From mdounin at mdounin.ru  Mon Jun  3 15:15:19 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:15:19 +0300
Subject: [nginx-tests] Tests: optimized processing of large QUIC packets ...
Message-ID: <hg.c7315caf2110.1717427719.-8454516869073360228@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/c7315caf2110
branches:  
changeset: 1983:c7315caf2110
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 04:17:28 2024 +0300
description:
Tests: optimized processing of large QUIC packets with padding.

Path MTU discovery packets might contain a lot of padding, and creating
a copy of the whole buffer for each PADDING frame, which is just one
byte with type 0, consumes lots of resources.  This was seen to result
in flapping of at least h3_keepalive.t and h3_ssl_early_data.t tests.

Fix is to copy at most 8 bytes for parse_int() calls when parsing
frame types.

diffstat:

 lib/Test/Nginx/HTTP3.pm |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/lib/Test/Nginx/HTTP3.pm b/lib/Test/Nginx/HTTP3.pm
--- a/lib/Test/Nginx/HTTP3.pm
+++ b/lib/Test/Nginx/HTTP3.pm
@@ -1361,7 +1361,7 @@ sub parse_frames {
 	my $offset = 0;
 
 	while ($offset < length($buf)) {
-		my ($tlen, $type) = parse_int(substr($buf, $offset));
+		my ($tlen, $type) = parse_int(substr($buf, $offset, 8));
 		$offset += $tlen;
 		next if $type == 0;
 		my $frame = { type => $type };


From mdounin at mdounin.ru  Mon Jun  3 15:16:21 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:21 +0300
Subject: [nginx] Mail: fixed EXTERNAL to be accepted only if enabled.
Message-ID: <hg.32d4582c484d.1717427781.-1356550532874038143@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/32d4582c484d
branches:  
changeset: 9287:32d4582c484d
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:03:05 2024 +0300
description:
Mail: fixed EXTERNAL to be accepted only if enabled.

As originally implemented in 6774:bcb107bb89cd, it wasn't possible to
disable the EXTERNAL authentication method: it was always accepted
(but not advertised unless enabled).  It is, however, believed that
it is better to reject attempts to use the disabled method, hence in
6869:b2915d99ee8d an attempt was made to address this.  This attempt
was insufficient though: it was still possible to use the method as long
as initial SASL response was used.

With this patch both challenge-response and initial response forms are
disabled.  Additionally, initial response handling for the PLAIN
authentication is removed from ngx_mail_auth_parse(), for consistency
and to don't provoke such bugs.

diffstat:

 src/mail/ngx_mail_imap_handler.c |   8 ++++++++
 src/mail/ngx_mail_parse.c        |  12 ++++--------
 src/mail/ngx_mail_pop3_handler.c |   8 ++++++++
 src/mail/ngx_mail_smtp_handler.c |   8 ++++++++
 4 files changed, 28 insertions(+), 8 deletions(-)

diffs (110 lines):

diff --git a/src/mail/ngx_mail_imap_handler.c b/src/mail/ngx_mail_imap_handler.c
--- a/src/mail/ngx_mail_imap_handler.c
+++ b/src/mail/ngx_mail_imap_handler.c
@@ -388,6 +388,10 @@ ngx_mail_imap_authenticate(ngx_mail_sess
 
     case NGX_MAIL_AUTH_PLAIN:
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_plain(s, c, 1);
+        }
+
         ngx_str_set(&s->out, imap_plain_next);
         s->mail_state = ngx_imap_auth_plain;
 
@@ -420,6 +424,10 @@ ngx_mail_imap_authenticate(ngx_mail_sess
             return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_external(s, c, 1);
+        }
+
         ngx_str_set(&s->out, imap_username);
         s->mail_state = ngx_imap_auth_external;
 
diff --git a/src/mail/ngx_mail_parse.c b/src/mail/ngx_mail_parse.c
--- a/src/mail/ngx_mail_parse.c
+++ b/src/mail/ngx_mail_parse.c
@@ -934,13 +934,11 @@ ngx_mail_auth_parse(ngx_mail_session_t *
 
         if (ngx_strncasecmp(arg[0].data, (u_char *) "PLAIN", 5) == 0) {
 
-            if (s->args.nelts == 1) {
+            if (s->args.nelts == 1 || s->args.nelts == 2) {
                 return NGX_MAIL_AUTH_PLAIN;
             }
 
-            if (s->args.nelts == 2) {
-                return ngx_mail_auth_plain(s, c, 1);
-            }
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
 
         return NGX_MAIL_PARSE_INVALID_COMMAND;
@@ -959,13 +957,11 @@ ngx_mail_auth_parse(ngx_mail_session_t *
 
         if (ngx_strncasecmp(arg[0].data, (u_char *) "EXTERNAL", 8) == 0) {
 
-            if (s->args.nelts == 1) {
+            if (s->args.nelts == 1 || s->args.nelts == 2) {
                 return NGX_MAIL_AUTH_EXTERNAL;
             }
 
-            if (s->args.nelts == 2) {
-                return ngx_mail_auth_external(s, c, 1);
-            }
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
 
         return NGX_MAIL_PARSE_INVALID_COMMAND;
diff --git a/src/mail/ngx_mail_pop3_handler.c b/src/mail/ngx_mail_pop3_handler.c
--- a/src/mail/ngx_mail_pop3_handler.c
+++ b/src/mail/ngx_mail_pop3_handler.c
@@ -517,6 +517,10 @@ ngx_mail_pop3_auth(ngx_mail_session_t *s
 
     case NGX_MAIL_AUTH_PLAIN:
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_plain(s, c, 1);
+        }
+
         ngx_str_set(&s->out, pop3_next);
         s->mail_state = ngx_pop3_auth_plain;
 
@@ -541,6 +545,10 @@ ngx_mail_pop3_auth(ngx_mail_session_t *s
             return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_external(s, c, 1);
+        }
+
         ngx_str_set(&s->out, pop3_username);
         s->mail_state = ngx_pop3_auth_external;
 
diff --git a/src/mail/ngx_mail_smtp_handler.c b/src/mail/ngx_mail_smtp_handler.c
--- a/src/mail/ngx_mail_smtp_handler.c
+++ b/src/mail/ngx_mail_smtp_handler.c
@@ -701,6 +701,10 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
 
     case NGX_MAIL_AUTH_PLAIN:
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_plain(s, c, 1);
+        }
+
         ngx_str_set(&s->out, smtp_next);
         s->mail_state = ngx_smtp_auth_plain;
 
@@ -733,6 +737,10 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
             return NGX_MAIL_PARSE_INVALID_COMMAND;
         }
 
+        if (s->args.nelts == 2) {
+            return ngx_mail_auth_external(s, c, 1);
+        }
+
         ngx_str_set(&s->out, smtp_username);
         s->mail_state = ngx_smtp_auth_external;
 


From mdounin at mdounin.ru  Mon Jun  3 15:16:21 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:21 +0300
Subject: [nginx] Mail: fixed EXTERNAL auth to clear s->passwd.
Message-ID: <hg.f83cb031a4a4.1717427781.-1356550532874038143@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/f83cb031a4a4
branches:  
changeset: 9288:f83cb031a4a4
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:03:07 2024 +0300
description:
Mail: fixed EXTERNAL auth to clear s->passwd.

The s->passwd field might be set after previous (failed) authentication
in the same session, and since EXTERNAL authentication did not touch it,
it was sent to the auth server.

diffstat:

 src/mail/ngx_mail_handler.c |  2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diffs (12 lines):

diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -744,6 +744,8 @@ ngx_mail_auth_external(ngx_mail_session_
     s->login.len = external.len;
     s->login.data = external.data;
 
+    ngx_str_null(&s->passwd);
+
     ngx_log_debug1(NGX_LOG_DEBUG_MAIL, c->log, 0,
                    "mail auth external: \"%V\"", &s->login);
 


From mdounin at mdounin.ru  Mon Jun  3 15:16:22 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:22 +0300
Subject: [nginx] Mail: added some parsing debug logging.
Message-ID: <hg.20017bff0de8.1717427782.-1356550532874038143@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/20017bff0de8
branches:  
changeset: 9289:20017bff0de8
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:03:09 2024 +0300
description:
Mail: added some parsing debug logging.

diffstat:

 src/mail/ngx_mail_parse.c |  9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diffs (33 lines):

diff --git a/src/mail/ngx_mail_parse.c b/src/mail/ngx_mail_parse.c
--- a/src/mail/ngx_mail_parse.c
+++ b/src/mail/ngx_mail_parse.c
@@ -30,6 +30,9 @@ ngx_mail_pop3_parse_command(ngx_mail_ses
 
     state = s->state;
 
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, s->connection->log, 0,
+                   "pop3 parse: %d", state);
+
     for (p = s->buffer->pos; p < s->buffer->last; p++) {
         ch = *p;
 
@@ -248,6 +251,9 @@ ngx_mail_imap_parse_command(ngx_mail_ses
 
     state = s->state;
 
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, s->connection->log, 0,
+                   "imap parse: %d", state);
+
     for (p = s->buffer->pos; p < s->buffer->last; p++) {
         ch = *p;
 
@@ -692,6 +698,9 @@ ngx_mail_smtp_parse_command(ngx_mail_ses
 
     state = s->state;
 
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, s->connection->log, 0,
+                   "smtp parse: %d", state);
+
     for (p = s->buffer->pos; p < s->buffer->last; p++) {
         ch = *p;
 


From mdounin at mdounin.ru  Mon Jun  3 15:16:22 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:22 +0300
Subject: [nginx] Mail: added support for XOAUTH2 and OAUTHBEARER authenti...
Message-ID: <hg.4538c1ffb0f8.1717427782.-1356550532874038143@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/4538c1ffb0f8
branches:  
changeset: 9290:4538c1ffb0f8
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:03:11 2024 +0300
description:
Mail: added support for XOAUTH2 and OAUTHBEARER authentication.

This patch adds support for the OAUTHBEARER SASL mechanism as defined
by RFC 7628, as well as pre-RFC XOAUTH2 SASL mechanism.  For both
mechanisms, the "Auth-User" header is set to the client identity
obtained from the initial SASL response sent by the client, and the
"Auth-Pass" header is set to the Bearer token itself.

The auth server may return the "Auth-Error-SASL" header, which is
passed to the client as an additional SASL challenge.  It is expected
to contain mechanism-specific error details, base64-encoded.  After
the client responds (with an empty SASL response for XAUTH2, or with
"AQ==" dummy response for OAUTHBEARER), the error message from the
"Auth-Status" header is sent.

Based on a patch by Rob Mueller.

diffstat:

 src/mail/ngx_mail.h                  |   35 +++-
 src/mail/ngx_mail_auth_http_module.c |  140 +++++++++++++----
 src/mail/ngx_mail_handler.c          |  282 ++++++++++++++++++++++++++++++++++-
 src/mail/ngx_mail_imap_handler.c     |   40 ++++
 src/mail/ngx_mail_imap_module.c      |    8 +-
 src/mail/ngx_mail_parse.c            |   28 +++
 src/mail/ngx_mail_pop3_handler.c     |   40 ++++
 src/mail/ngx_mail_pop3_module.c      |   12 +-
 src/mail/ngx_mail_smtp_handler.c     |   40 ++++
 src/mail/ngx_mail_smtp_module.c      |    8 +-
 10 files changed, 578 insertions(+), 55 deletions(-)

diffs (945 lines):

diff --git a/src/mail/ngx_mail.h b/src/mail/ngx_mail.h
--- a/src/mail/ngx_mail.h
+++ b/src/mail/ngx_mail.h
@@ -141,7 +141,9 @@ typedef enum {
     ngx_pop3_auth_login_password,
     ngx_pop3_auth_plain,
     ngx_pop3_auth_cram_md5,
-    ngx_pop3_auth_external
+    ngx_pop3_auth_external,
+    ngx_pop3_auth_xoauth2,
+    ngx_pop3_auth_oauthbearer
 } ngx_pop3_state_e;
 
 
@@ -152,6 +154,8 @@ typedef enum {
     ngx_imap_auth_plain,
     ngx_imap_auth_cram_md5,
     ngx_imap_auth_external,
+    ngx_imap_auth_xoauth2,
+    ngx_imap_auth_oauthbearer,
     ngx_imap_login,
     ngx_imap_user,
     ngx_imap_passwd
@@ -165,6 +169,8 @@ typedef enum {
     ngx_smtp_auth_plain,
     ngx_smtp_auth_cram_md5,
     ngx_smtp_auth_external,
+    ngx_smtp_auth_xoauth2,
+    ngx_smtp_auth_oauthbearer,
     ngx_smtp_helo,
     ngx_smtp_helo_xclient,
     ngx_smtp_helo_auth,
@@ -212,8 +218,9 @@ typedef struct {
     unsigned                no_sync_literal:1;
     unsigned                starttls:1;
     unsigned                esmtp:1;
-    unsigned                auth_method:3;
+    unsigned                auth_method:4;
     unsigned                auth_wait:1;
+    unsigned                auth_quit:1;
 
     ngx_str_t               login;
     ngx_str_t               passwd;
@@ -229,6 +236,8 @@ typedef struct {
     ngx_str_t               smtp_from;
     ngx_str_t               smtp_to;
 
+    ngx_str_t               auth_err;
+
     ngx_str_t               cmd;
 
     ngx_uint_t              command;
@@ -303,15 +312,19 @@ typedef struct {
 #define NGX_MAIL_AUTH_APOP              3
 #define NGX_MAIL_AUTH_CRAM_MD5          4
 #define NGX_MAIL_AUTH_EXTERNAL          5
-#define NGX_MAIL_AUTH_NONE              6
+#define NGX_MAIL_AUTH_XOAUTH2           6
+#define NGX_MAIL_AUTH_OAUTHBEARER       7
+#define NGX_MAIL_AUTH_NONE              8
 
 
-#define NGX_MAIL_AUTH_PLAIN_ENABLED     0x0002
-#define NGX_MAIL_AUTH_LOGIN_ENABLED     0x0004
-#define NGX_MAIL_AUTH_APOP_ENABLED      0x0008
-#define NGX_MAIL_AUTH_CRAM_MD5_ENABLED  0x0010
-#define NGX_MAIL_AUTH_EXTERNAL_ENABLED  0x0020
-#define NGX_MAIL_AUTH_NONE_ENABLED      0x0040
+#define NGX_MAIL_AUTH_PLAIN_ENABLED        0x0002
+#define NGX_MAIL_AUTH_LOGIN_ENABLED        0x0004
+#define NGX_MAIL_AUTH_APOP_ENABLED         0x0008
+#define NGX_MAIL_AUTH_CRAM_MD5_ENABLED     0x0010
+#define NGX_MAIL_AUTH_EXTERNAL_ENABLED     0x0020
+#define NGX_MAIL_AUTH_XOAUTH2_ENABLED      0x0040
+#define NGX_MAIL_AUTH_OAUTHBEARER_ENABLED  0x0080
+#define NGX_MAIL_AUTH_NONE_ENABLED         0x0100
 
 
 #define NGX_MAIL_PARSE_INVALID_COMMAND  20
@@ -399,6 +412,10 @@ ngx_int_t ngx_mail_auth_cram_md5_salt(ng
 ngx_int_t ngx_mail_auth_cram_md5(ngx_mail_session_t *s, ngx_connection_t *c);
 ngx_int_t ngx_mail_auth_external(ngx_mail_session_t *s, ngx_connection_t *c,
     ngx_uint_t n);
+ngx_int_t ngx_mail_auth_xoauth2(ngx_mail_session_t *s, ngx_connection_t *c,
+    ngx_uint_t n);
+ngx_int_t ngx_mail_auth_oauthbearer(ngx_mail_session_t *s, ngx_connection_t *c,
+    ngx_uint_t n);
 ngx_int_t ngx_mail_auth_parse(ngx_mail_session_t *s, ngx_connection_t *c);
 
 void ngx_mail_send(ngx_event_t *wev);
diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c
--- a/src/mail/ngx_mail_auth_http_module.c
+++ b/src/mail/ngx_mail_auth_http_module.c
@@ -53,6 +53,7 @@ struct ngx_mail_auth_http_ctx_s {
     ngx_str_t                       err;
     ngx_str_t                       errmsg;
     ngx_str_t                       errcode;
+    ngx_str_t                       errsasl;
 
     time_t                          sleep;
 
@@ -67,6 +68,7 @@ static void ngx_mail_auth_http_ignore_st
 static void ngx_mail_auth_http_process_headers(ngx_mail_session_t *s,
     ngx_mail_auth_http_ctx_t *ctx);
 static void ngx_mail_auth_sleep_handler(ngx_event_t *rev);
+static void ngx_mail_auth_send_error(ngx_mail_session_t *s);
 static ngx_int_t ngx_mail_auth_http_parse_header_line(ngx_mail_session_t *s,
     ngx_mail_auth_http_ctx_t *ctx);
 static void ngx_mail_auth_http_block_read(ngx_event_t *rev);
@@ -152,6 +154,8 @@ static ngx_str_t   ngx_mail_auth_http_me
     ngx_string("apop"),
     ngx_string("cram-md5"),
     ngx_string("external"),
+    ngx_string("xoauth2"),
+    ngx_string("oauthbearer"),
     ngx_string("none")
 };
 
@@ -677,6 +681,51 @@ ngx_mail_auth_http_process_headers(ngx_m
                 continue;
             }
 
+            if (len == sizeof("Auth-Error-SASL") - 1
+                && ngx_strncasecmp(ctx->header_name_start,
+                                   (u_char *) "Auth-Error-SASL",
+                                   sizeof("Auth-Error-SASL") - 1)
+                   == 0)
+            {
+                if (s->auth_method != NGX_MAIL_AUTH_XOAUTH2
+                    && s->auth_method != NGX_MAIL_AUTH_OAUTHBEARER)
+                {
+                    continue;
+                }
+
+                len = ctx->header_end - ctx->header_start;
+
+                if (s->protocol == NGX_MAIL_SMTP_PROTOCOL) {
+                    size = len + sizeof("334 " CRLF) - 1;
+
+                } else {
+                    size = len + sizeof("+ " CRLF) - 1;
+                }
+
+                p = ngx_pnalloc(s->connection->pool, size);
+                if (p == NULL) {
+                    ngx_close_connection(ctx->peer.connection);
+                    ngx_destroy_pool(ctx->pool);
+                    ngx_mail_session_internal_server_error(s);
+                    return;
+                }
+
+                ctx->errsasl.len = size;
+                ctx->errsasl.data = p;
+
+                if (s->protocol == NGX_MAIL_SMTP_PROTOCOL) {
+                    *p++ = '3'; *p++ = '3'; *p++ = '4'; *p++ = ' ';
+
+                } else {
+                    *p++ = '+'; *p++ = ' ';
+                }
+
+                p = ngx_cpymem(p, ctx->header_start, len);
+                *p++ = CR; *p = LF;
+
+                continue;
+            }
+
             /* ignore other headers */
 
             continue;
@@ -717,14 +766,15 @@ ngx_mail_auth_http_process_headers(ngx_m
                     *p++ = CR; *p = LF;
                 }
 
-                s->out = ctx->err;
+                s->out = ctx->errsasl;
+                s->auth_err = ctx->err;
                 timer = ctx->sleep;
 
                 ngx_destroy_pool(ctx->pool);
 
                 if (timer == 0) {
-                    s->quit = 1;
-                    ngx_mail_send(s->connection->write);
+                    s->auth_quit = 1;
+                    ngx_mail_auth_send_error(s);
                     return;
                 }
 
@@ -858,9 +908,8 @@ ngx_mail_auth_http_process_headers(ngx_m
 static void
 ngx_mail_auth_sleep_handler(ngx_event_t *rev)
 {
-    ngx_connection_t          *c;
-    ngx_mail_session_t        *s;
-    ngx_mail_core_srv_conf_t  *cscf;
+    ngx_connection_t    *c;
+    ngx_mail_session_t  *s;
 
     ngx_log_debug0(NGX_LOG_DEBUG_MAIL, rev->log, 0, "mail auth sleep handler");
 
@@ -877,33 +926,7 @@ ngx_mail_auth_sleep_handler(ngx_event_t 
             return;
         }
 
-        cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
-
-        rev->handler = cscf->protocol->auth_state;
-
-        s->mail_state = 0;
-        s->auth_method = NGX_MAIL_AUTH_PLAIN;
-        s->tag.len = 0;
-
-        c->log->action = "in auth state";
-
-        ngx_mail_send(c->write);
-
-        if (c->destroyed) {
-            return;
-        }
-
-        ngx_add_timer(rev, cscf->timeout);
-
-        if (rev->ready) {
-            rev->handler(rev);
-            return;
-        }
-
-        if (ngx_handle_read_event(rev, 0) != NGX_OK) {
-            ngx_mail_close_connection(c);
-        }
-
+        ngx_mail_auth_send_error(s);
         return;
     }
 
@@ -915,6 +938,57 @@ ngx_mail_auth_sleep_handler(ngx_event_t 
 }
 
 
+static void
+ngx_mail_auth_send_error(ngx_mail_session_t *s)
+{
+    ngx_event_t               *rev;
+    ngx_connection_t          *c;
+    ngx_mail_core_srv_conf_t  *cscf;
+
+    c = s->connection;
+    rev = c->read;
+
+    cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
+
+    rev->handler = cscf->protocol->auth_state;
+
+    s->auth_method = NGX_MAIL_AUTH_PLAIN;
+
+    c->log->action = "in auth state";
+
+    if (s->out.len == 0) {
+        s->out = s->auth_err;
+        s->quit = s->auth_quit;
+        ngx_str_null(&s->auth_err);
+
+        s->state = 0;
+        s->mail_state = 0;
+        s->tag.len = 0;
+
+    } else {
+        s->auth_err.len -= s->tag.len;
+        s->auth_err.data += s->tag.len;
+    }
+
+    ngx_mail_send(c->write);
+
+    if (c->destroyed) {
+        return;
+    }
+
+    ngx_add_timer(rev, cscf->timeout);
+
+    if (rev->ready) {
+        rev->handler(rev);
+        return;
+    }
+
+    if (ngx_handle_read_event(rev, 0) != NGX_OK) {
+        ngx_mail_close_connection(c);
+    }
+}
+
+
 static ngx_int_t
 ngx_mail_auth_http_parse_header_line(ngx_mail_session_t *s,
     ngx_mail_auth_http_ctx_t *ctx)
diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -755,6 +755,274 @@ ngx_mail_auth_external(ngx_mail_session_
 }
 
 
+ngx_int_t
+ngx_mail_auth_xoauth2(ngx_mail_session_t *s, ngx_connection_t *c, ngx_uint_t n)
+{
+    u_char     *p, *last;
+    ngx_str_t  *arg, oauth;
+
+    arg = s->args.elts;
+
+    if (s->auth_err.len) {
+        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                       "mail auth xoauth2 cancel");
+
+        if (s->args.nelts == 1 && arg[0].len == 0) {
+            s->out = s->auth_err;
+            s->quit = s->auth_quit;
+            s->state = 0;
+            s->mail_state = 0;
+            ngx_str_null(&s->auth_err);
+            return NGX_OK;
+        }
+
+        s->quit = s->auth_quit;
+        ngx_str_null(&s->auth_err);
+
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                   "mail auth xoauth2: \"%V\"", &arg[n]);
+
+    oauth.data = ngx_pnalloc(c->pool, ngx_base64_decoded_length(arg[n].len));
+    if (oauth.data == NULL) {
+        return NGX_ERROR;
+    }
+
+    if (ngx_decode_base64(&oauth, &arg[n]) != NGX_OK) {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid base64 encoding in "
+                      "AUTH XOAUTH2 command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    /*
+     * https://developers.google.com/gmail/imap/xoauth2-protocol
+     * "user=" {User} "^Aauth=Bearer " {token} "^A^A"
+     */
+
+    p = oauth.data;
+    last = p + oauth.len;
+
+    while (p < last) {
+        if (*p++ == '\1') {
+            s->login.len = p - oauth.data - 1;
+            s->login.data = oauth.data;
+            s->passwd.len = last - p;
+            s->passwd.data = p;
+            break;
+        }
+    }
+
+    if (s->login.len < sizeof("user=") - 1
+        || ngx_strncasecmp(s->login.data, (u_char *) "user=",
+                           sizeof("user=") - 1)
+           != 0)
+    {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid login in AUTH XOAUTH2 command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    s->login.len -= sizeof("user=") - 1;
+    s->login.data += sizeof("user=") - 1;
+
+    if (s->passwd.len < sizeof("auth=Bearer ") - 1
+        || ngx_strncasecmp(s->passwd.data, (u_char *) "auth=Bearer ",
+                           sizeof("auth=Bearer ") - 1)
+           != 0)
+    {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid token in AUTH XOAUTH2 command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    s->passwd.len -= sizeof("auth=Bearer ") - 1;
+    s->passwd.data += sizeof("auth=Bearer ") - 1;
+
+    if (s->passwd.len < 2
+        || s->passwd.data[s->passwd.len - 2] != '\1'
+        || s->passwd.data[s->passwd.len - 1] != '\1')
+    {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid token in AUTH XOAUTH2 command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    s->passwd.len -= 2;
+
+    ngx_log_debug2(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                   "mail auth xoauth2: \"%V\" \"%V\"", &s->login, &s->passwd);
+
+    s->auth_method = NGX_MAIL_AUTH_XOAUTH2;
+
+    return NGX_DONE;
+}
+
+
+ngx_int_t
+ngx_mail_auth_oauthbearer(ngx_mail_session_t *s, ngx_connection_t *c,
+    ngx_uint_t n)
+{
+    u_char     *p, *d, *last, *prev;
+    ngx_str_t  *arg, oauth;
+
+    arg = s->args.elts;
+
+    if (s->auth_err.len) {
+        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                       "mail auth oauthbearer cancel");
+
+        if (s->args.nelts == 1
+            && ngx_strncmp(arg[0].data, (u_char *) "AQ==", 4) == 0)
+        {
+            s->out = s->auth_err;
+            s->quit = s->auth_quit;
+            s->state = 0;
+            s->mail_state = 0;
+            ngx_str_null(&s->auth_err);
+            return NGX_OK;
+        }
+
+        s->quit = s->auth_quit;
+        ngx_str_null(&s->auth_err);
+
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                   "mail auth oauthbearer: \"%V\"", &arg[n]);
+
+    oauth.data = ngx_pnalloc(c->pool, ngx_base64_decoded_length(arg[n].len));
+    if (oauth.data == NULL) {
+        return NGX_ERROR;
+    }
+
+    if (ngx_decode_base64(&oauth, &arg[n]) != NGX_OK) {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid base64 encoding in "
+                      "AUTH OAUTHBEARER command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    /*
+     * RFC 7628
+     * "n,a=user at example.com,^A...^Aauth=Bearer <token>^A^A"
+     */
+
+    p = oauth.data;
+    last = p + oauth.len;
+
+    s->login.len = 0;
+    prev = NULL;
+
+    while (p < last) {
+        if (*p == ',') {
+            if (prev
+                && (size_t) (p - prev) > sizeof("a=") - 1
+                && ngx_strncasecmp(prev, (u_char *) "a=", sizeof("a=") - 1)
+                   == 0)
+            {
+                s->login.len = p - prev - (sizeof("a=") - 1);
+                s->login.data = prev + sizeof("a=") - 1;
+                break;
+            }
+
+            p++;
+            prev = p;
+            continue;
+        }
+
+        if (*p == '\1') {
+            break;
+        }
+
+        p++;
+    }
+
+    if (s->login.len == 0) {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid login in AUTH OAUTHBEARER command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    s->passwd.len = 0;
+    prev = NULL;
+
+    while (p < last) {
+        if (*p == '\1') {
+            if (prev
+                && (size_t) (p - prev) > sizeof("auth=Bearer ") - 1
+                && ngx_strncasecmp(prev, (u_char *) "auth=Bearer ",
+                                   sizeof("auth=Bearer ") - 1)
+                   == 0)
+            {
+                s->passwd.len = p - prev - (sizeof("auth=Bearer ") - 1);
+                s->passwd.data = prev + sizeof("auth=Bearer ") - 1;
+                break;
+            }
+
+            p++;
+            prev = p;
+            continue;
+        }
+
+        p++;
+    }
+
+    if (s->passwd.len == 0) {
+        ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                      "client sent invalid token in AUTH OAUTHBEARER command");
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
+    /* decode =2C =3D in login */
+
+    p = s->login.data;
+    d = s->login.data;
+    last = s->login.data + s->login.len;
+
+    while (p < last) {
+        if (*p == '=') {
+
+            /*
+             * login is always followed by other data,
+             * so p[1] and p[2] can be checked directly
+             */
+
+            if (p[1] == '2' && (p[2] == 'C' || p[2] == 'c')) {
+                *d++ = ',';
+
+            } else if (p[1] == '3' && (p[2] == 'D' || p[2] == 'd')) {
+                *d++ = '=';
+
+            } else {
+                ngx_log_error(NGX_LOG_INFO, c->log, 0,
+                              "client sent invalid login in "
+                              "AUTH OAUTHBEARER command");
+                return NGX_MAIL_PARSE_INVALID_COMMAND;
+            }
+
+            p += 3;
+            continue;
+        }
+
+        *d++ = *p++;
+    }
+
+    s->login.len = d - s->login.data;
+
+    ngx_log_debug2(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                   "mail auth oauthbearer: \"%V\" \"%V\"",
+                   &s->login, &s->passwd);
+
+    s->auth_method = NGX_MAIL_AUTH_OAUTHBEARER;
+
+    return NGX_DONE;
+}
+
+
 void
 ngx_mail_send(ngx_event_t *wev)
 {
@@ -919,13 +1187,17 @@ ngx_mail_auth(ngx_mail_session_t *s, ngx
 {
     s->args.nelts = 0;
 
-    if (s->buffer->pos == s->buffer->last) {
-        s->buffer->pos = s->buffer->start;
-        s->buffer->last = s->buffer->start;
+    if (s->state) {
+        /* preserve tag */
+        s->arg_start = s->buffer->pos;
+
+    } else {
+        if (s->buffer->pos == s->buffer->last) {
+            s->buffer->pos = s->buffer->start;
+            s->buffer->last = s->buffer->start;
+        }
     }
 
-    s->state = 0;
-
     if (c->read->timer_set) {
         ngx_del_timer(c->read);
     }
diff --git a/src/mail/ngx_mail_imap_handler.c b/src/mail/ngx_mail_imap_handler.c
--- a/src/mail/ngx_mail_imap_handler.c
+++ b/src/mail/ngx_mail_imap_handler.c
@@ -220,6 +220,14 @@ ngx_mail_imap_auth_state(ngx_event_t *re
         case ngx_imap_auth_external:
             rc = ngx_mail_auth_external(s, c, 0);
             break;
+
+        case ngx_imap_auth_xoauth2:
+            rc = ngx_mail_auth_xoauth2(s, c, 0);
+            break;
+
+        case ngx_imap_auth_oauthbearer:
+            rc = ngx_mail_auth_oauthbearer(s, c, 0);
+            break;
         }
 
     } else if (rc == NGX_IMAP_NEXT) {
@@ -432,6 +440,38 @@ ngx_mail_imap_authenticate(ngx_mail_sess
         s->mail_state = ngx_imap_auth_external;
 
         return NGX_OK;
+
+    case NGX_MAIL_AUTH_XOAUTH2:
+
+        if (!(iscf->auth_methods & NGX_MAIL_AUTH_XOAUTH2_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_imap_auth_xoauth2;
+            return ngx_mail_auth_xoauth2(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, imap_plain_next);
+        s->mail_state = ngx_imap_auth_xoauth2;
+
+        return NGX_OK;
+
+    case NGX_MAIL_AUTH_OAUTHBEARER:
+
+        if (!(iscf->auth_methods & NGX_MAIL_AUTH_OAUTHBEARER_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_imap_auth_oauthbearer;
+            return ngx_mail_auth_oauthbearer(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, imap_plain_next);
+        s->mail_state = ngx_imap_auth_oauthbearer;
+
+        return NGX_OK;
     }
 
     return rc;
diff --git a/src/mail/ngx_mail_imap_module.c b/src/mail/ngx_mail_imap_module.c
--- a/src/mail/ngx_mail_imap_module.c
+++ b/src/mail/ngx_mail_imap_module.c
@@ -30,6 +30,8 @@ static ngx_conf_bitmask_t  ngx_mail_imap
     { ngx_string("login"), NGX_MAIL_AUTH_LOGIN_ENABLED },
     { ngx_string("cram-md5"), NGX_MAIL_AUTH_CRAM_MD5_ENABLED },
     { ngx_string("external"), NGX_MAIL_AUTH_EXTERNAL_ENABLED },
+    { ngx_string("xoauth2"), NGX_MAIL_AUTH_XOAUTH2_ENABLED },
+    { ngx_string("oauthbearer"), NGX_MAIL_AUTH_OAUTHBEARER_ENABLED },
     { ngx_null_string, 0 }
 };
 
@@ -40,6 +42,8 @@ static ngx_str_t  ngx_mail_imap_auth_met
     ngx_null_string,  /* APOP */
     ngx_string("AUTH=CRAM-MD5"),
     ngx_string("AUTH=EXTERNAL"),
+    ngx_string("AUTH=XOAUTH2"),
+    ngx_string("AUTH=OAUTHBEARER"),
     ngx_null_string   /* NONE */
 };
 
@@ -182,7 +186,7 @@ ngx_mail_imap_merge_srv_conf(ngx_conf_t 
     }
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (m & conf->auth_methods) {
@@ -208,7 +212,7 @@ ngx_mail_imap_merge_srv_conf(ngx_conf_t 
     auth = p;
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (m & conf->auth_methods) {
diff --git a/src/mail/ngx_mail_parse.c b/src/mail/ngx_mail_parse.c
--- a/src/mail/ngx_mail_parse.c
+++ b/src/mail/ngx_mail_parse.c
@@ -953,6 +953,20 @@ ngx_mail_auth_parse(ngx_mail_session_t *
         return NGX_MAIL_PARSE_INVALID_COMMAND;
     }
 
+    if (arg[0].len == 7) {
+
+        if (ngx_strncasecmp(arg[0].data, (u_char *) "XOAUTH2", 7) == 0) {
+
+            if (s->args.nelts == 1 || s->args.nelts == 2) {
+                return NGX_MAIL_AUTH_XOAUTH2;
+            }
+
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
     if (arg[0].len == 8) {
 
         if (ngx_strncasecmp(arg[0].data, (u_char *) "CRAM-MD5", 8) == 0) {
@@ -976,5 +990,19 @@ ngx_mail_auth_parse(ngx_mail_session_t *
         return NGX_MAIL_PARSE_INVALID_COMMAND;
     }
 
+    if (arg[0].len == 11) {
+
+        if (ngx_strncasecmp(arg[0].data, (u_char *) "OAUTHBEARER", 11) == 0) {
+
+            if (s->args.nelts == 1 || s->args.nelts == 2) {
+                return NGX_MAIL_AUTH_OAUTHBEARER;
+            }
+
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        return NGX_MAIL_PARSE_INVALID_COMMAND;
+    }
+
     return NGX_MAIL_PARSE_INVALID_COMMAND;
 }
diff --git a/src/mail/ngx_mail_pop3_handler.c b/src/mail/ngx_mail_pop3_handler.c
--- a/src/mail/ngx_mail_pop3_handler.c
+++ b/src/mail/ngx_mail_pop3_handler.c
@@ -260,6 +260,14 @@ ngx_mail_pop3_auth_state(ngx_event_t *re
         case ngx_pop3_auth_external:
             rc = ngx_mail_auth_external(s, c, 0);
             break;
+
+        case ngx_pop3_auth_xoauth2:
+            rc = ngx_mail_auth_xoauth2(s, c, 0);
+            break;
+
+        case ngx_pop3_auth_oauthbearer:
+            rc = ngx_mail_auth_oauthbearer(s, c, 0);
+            break;
         }
     }
 
@@ -553,6 +561,38 @@ ngx_mail_pop3_auth(ngx_mail_session_t *s
         s->mail_state = ngx_pop3_auth_external;
 
         return NGX_OK;
+
+    case NGX_MAIL_AUTH_XOAUTH2:
+
+        if (!(pscf->auth_methods & NGX_MAIL_AUTH_XOAUTH2_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_pop3_auth_xoauth2;
+            return ngx_mail_auth_xoauth2(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, pop3_next);
+        s->mail_state = ngx_pop3_auth_xoauth2;
+
+        return NGX_OK;
+
+    case NGX_MAIL_AUTH_OAUTHBEARER:
+
+        if (!(pscf->auth_methods & NGX_MAIL_AUTH_OAUTHBEARER_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_pop3_auth_oauthbearer;
+            return ngx_mail_auth_oauthbearer(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, pop3_next);
+        s->mail_state = ngx_pop3_auth_oauthbearer;
+
+        return NGX_OK;
     }
 
     return rc;
diff --git a/src/mail/ngx_mail_pop3_module.c b/src/mail/ngx_mail_pop3_module.c
--- a/src/mail/ngx_mail_pop3_module.c
+++ b/src/mail/ngx_mail_pop3_module.c
@@ -30,6 +30,8 @@ static ngx_conf_bitmask_t  ngx_mail_pop3
     { ngx_string("apop"), NGX_MAIL_AUTH_APOP_ENABLED },
     { ngx_string("cram-md5"), NGX_MAIL_AUTH_CRAM_MD5_ENABLED },
     { ngx_string("external"), NGX_MAIL_AUTH_EXTERNAL_ENABLED },
+    { ngx_string("xoauth2"), NGX_MAIL_AUTH_XOAUTH2_ENABLED },
+    { ngx_string("oauthbearer"), NGX_MAIL_AUTH_OAUTHBEARER_ENABLED },
     { ngx_null_string, 0 }
 };
 
@@ -40,6 +42,8 @@ static ngx_str_t  ngx_mail_pop3_auth_met
     ngx_null_string,  /* APOP */
     ngx_string("CRAM-MD5"),
     ngx_string("EXTERNAL"),
+    ngx_string("XOAUTH2"),
+    ngx_string("OAUTHBEARER"),
     ngx_null_string   /* NONE */
 };
 
@@ -183,7 +187,7 @@ ngx_mail_pop3_merge_srv_conf(ngx_conf_t 
     size += sizeof("SASL") - 1 + sizeof(CRLF) - 1;
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (ngx_mail_pop3_auth_methods_names[i].len == 0) {
@@ -214,7 +218,7 @@ ngx_mail_pop3_merge_srv_conf(ngx_conf_t 
     p = ngx_cpymem(p, "SASL", sizeof("SASL") - 1);
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (ngx_mail_pop3_auth_methods_names[i].len == 0) {
@@ -254,7 +258,7 @@ ngx_mail_pop3_merge_srv_conf(ngx_conf_t 
            + sizeof("." CRLF) - 1;
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (ngx_mail_pop3_auth_methods_names[i].len == 0) {
@@ -279,7 +283,7 @@ ngx_mail_pop3_merge_srv_conf(ngx_conf_t 
                    sizeof("+OK methods supported:" CRLF) - 1);
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (ngx_mail_pop3_auth_methods_names[i].len == 0) {
diff --git a/src/mail/ngx_mail_smtp_handler.c b/src/mail/ngx_mail_smtp_handler.c
--- a/src/mail/ngx_mail_smtp_handler.c
+++ b/src/mail/ngx_mail_smtp_handler.c
@@ -548,6 +548,14 @@ ngx_mail_smtp_auth_state(ngx_event_t *re
         case ngx_smtp_auth_external:
             rc = ngx_mail_auth_external(s, c, 0);
             break;
+
+        case ngx_smtp_auth_xoauth2:
+            rc = ngx_mail_auth_xoauth2(s, c, 0);
+            break;
+
+        case ngx_smtp_auth_oauthbearer:
+            rc = ngx_mail_auth_oauthbearer(s, c, 0);
+            break;
         }
     }
 
@@ -745,6 +753,38 @@ ngx_mail_smtp_auth(ngx_mail_session_t *s
         s->mail_state = ngx_smtp_auth_external;
 
         return NGX_OK;
+
+    case NGX_MAIL_AUTH_XOAUTH2:
+
+        if (!(sscf->auth_methods & NGX_MAIL_AUTH_XOAUTH2_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_smtp_auth_xoauth2;
+            return ngx_mail_auth_xoauth2(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, smtp_next);
+        s->mail_state = ngx_smtp_auth_xoauth2;
+
+        return NGX_OK;
+
+    case NGX_MAIL_AUTH_OAUTHBEARER:
+
+        if (!(sscf->auth_methods & NGX_MAIL_AUTH_OAUTHBEARER_ENABLED)) {
+            return NGX_MAIL_PARSE_INVALID_COMMAND;
+        }
+
+        if (s->args.nelts == 2) {
+            s->mail_state = ngx_smtp_auth_oauthbearer;
+            return ngx_mail_auth_oauthbearer(s, c, 1);
+        }
+
+        ngx_str_set(&s->out, smtp_next);
+        s->mail_state = ngx_smtp_auth_oauthbearer;
+
+        return NGX_OK;
     }
 
     return rc;
diff --git a/src/mail/ngx_mail_smtp_module.c b/src/mail/ngx_mail_smtp_module.c
--- a/src/mail/ngx_mail_smtp_module.c
+++ b/src/mail/ngx_mail_smtp_module.c
@@ -22,6 +22,8 @@ static ngx_conf_bitmask_t  ngx_mail_smtp
     { ngx_string("login"), NGX_MAIL_AUTH_LOGIN_ENABLED },
     { ngx_string("cram-md5"), NGX_MAIL_AUTH_CRAM_MD5_ENABLED },
     { ngx_string("external"), NGX_MAIL_AUTH_EXTERNAL_ENABLED },
+    { ngx_string("xoauth2"), NGX_MAIL_AUTH_XOAUTH2_ENABLED },
+    { ngx_string("oauthbearer"), NGX_MAIL_AUTH_OAUTHBEARER_ENABLED },
     { ngx_string("none"), NGX_MAIL_AUTH_NONE_ENABLED },
     { ngx_null_string, 0 }
 };
@@ -33,6 +35,8 @@ static ngx_str_t  ngx_mail_smtp_auth_met
     ngx_null_string,  /* APOP */
     ngx_string("CRAM-MD5"),
     ngx_string("EXTERNAL"),
+    ngx_string("XOAUTH2"),
+    ngx_string("OAUTHBEARER"),
     ngx_null_string   /* NONE */
 };
 
@@ -210,7 +214,7 @@ ngx_mail_smtp_merge_srv_conf(ngx_conf_t 
     auth_enabled = 0;
 
     for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-         m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+         m < NGX_MAIL_AUTH_NONE_ENABLED;
          m <<= 1, i++)
     {
         if (m & conf->auth_methods) {
@@ -253,7 +257,7 @@ ngx_mail_smtp_merge_srv_conf(ngx_conf_t 
         *p++ = 'A'; *p++ = 'U'; *p++ = 'T'; *p++ = 'H';
 
         for (m = NGX_MAIL_AUTH_PLAIN_ENABLED, i = 0;
-             m <= NGX_MAIL_AUTH_EXTERNAL_ENABLED;
+             m < NGX_MAIL_AUTH_NONE_ENABLED;
              m <<= 1, i++)
         {
             if (m & conf->auth_methods) {


From mdounin at mdounin.ru  Mon Jun  3 15:16:43 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:43 +0300
Subject: [nginx-tests] Tests: added test that EXTERNAL mail auth clears o...
Message-ID: <hg.81519d01f238.1717427803.-5811498871723959039@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/81519d01f238
branches:  
changeset: 1984:81519d01f238
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:15:22 2024 +0300
description:
Tests: added test that EXTERNAL mail auth clears old password.

diffstat:

 mail_imap.t |  19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)

diffs (36 lines):

diff --git a/mail_imap.t b/mail_imap.t
--- a/mail_imap.t
+++ b/mail_imap.t
@@ -93,7 +93,7 @@ http {
 EOF
 
 $t->run_daemon(\&Test::Nginx::IMAP::imap_test_daemon);
-$t->run()->plan(29);
+$t->run()->plan(30);
 
 $t->waitforsocket('127.0.0.1:' . port(8144));
 
@@ -184,6 +184,23 @@ my $s = Test::Nginx::IMAP->new();
 $s->send('1 AUTHENTICATE EXTERNAL ' . encode_base64('test at example.com', ''));
 $s->ok('auth external with username');
 
+# auth external after failed plain
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.1');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+
+$s->send('1 AUTHENTICATE PLAIN '
+	. encode_base64("\0test\@example.com\0bad", ''));
+$s->read();
+
+$s->send('1 AUTHENTICATE EXTERNAL ' . encode_base64('test at example.com', ''));
+$s->ok('auth external after plain');
+
+}
+
 # quoted strings
 
 $s = Test::Nginx::IMAP->new();


From mdounin at mdounin.ru  Mon Jun  3 15:16:43 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Mon, 03 Jun 2024 18:16:43 +0300
Subject: [nginx-tests] Tests: added tests for OAUTHBEARER and XOAUTH2 aut...
Message-ID: <hg.b5e2609d34a3.1717427803.-5811498871723959039@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/b5e2609d34a3
branches:  
changeset: 1985:b5e2609d34a3
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Jun 03 18:15:28 2024 +0300
description:
Tests: added tests for OAUTHBEARER and XOAUTH2 auth methods.

Based on a patch by Rob Mueller.

diffstat:

 mail_oauth.t |  338 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 338 insertions(+), 0 deletions(-)

diffs (343 lines):

diff --git a/mail_oauth.t b/mail_oauth.t
new file mode 100644
--- /dev/null
+++ b/mail_oauth.t
@@ -0,0 +1,338 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+
+# Tests for mail module, XOAUTH2 and OAUTHBEARER authentication.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+use MIME::Base64;
+use Socket qw/ CRLF /;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::IMAP;
+use Test::Nginx::POP3;
+use Test::Nginx::SMTP;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+local $SIG{PIPE} = 'IGNORE';
+
+my $t = Test::Nginx->new()->has(qw/mail imap pop3 smtp http map rewrite/)
+	->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+mail {
+    proxy_pass_error_message  on;
+    proxy_timeout  15s;
+    timeout  2s;
+    auth_http  http://127.0.0.1:8080/mail/auth;
+
+    server {
+        listen     127.0.0.1:8143;
+        protocol   imap;
+        imap_auth  plain oauthbearer xoauth2;
+    }
+    server {
+        listen     127.0.0.1:8110;
+        protocol   pop3;
+        pop3_auth  plain oauthbearer xoauth2;
+    }
+    server {
+        listen     127.0.0.1:8025;
+        protocol   smtp;
+        smtp_auth  plain oauthbearer xoauth2;
+    }
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    map $http_auth_protocol $proxy_port {
+	imap %%PORT_8144%%;
+	pop3 %%PORT_8111%%;
+	smtp %%PORT_8026%%;
+    }
+
+    map $http_auth_user:$http_auth_pass $reply {
+	test at example.com:secretok OK;
+	test=, at example.com:secretok OK;
+	default auth-failed;
+    }
+
+    map $http_auth_pass $passw {
+	secretok secret;
+    }
+
+    map $http_auth_pass $sasl {
+	saslfail "eyJzY2hlbWVzIjoiQmVhcmVyIiwic3RhdHVzIjoiNDAwIn0=";
+    }
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location = /mail/auth {
+            add_header Auth-Status $reply;
+            add_header Auth-Server 127.0.0.1;
+            add_header Auth-Port $proxy_port;
+            add_header Auth-Pass $passw;
+            add_header Auth-Wait 1;
+            add_header Auth-Error-SASL $sasl;
+            return 204;
+        }
+    }
+}
+
+EOF
+
+$t->run_daemon(\&Test::Nginx::IMAP::imap_test_daemon);
+$t->run_daemon(\&Test::Nginx::POP3::pop3_test_daemon);
+$t->run_daemon(\&Test::Nginx::SMTP::smtp_test_daemon);
+$t->try_run('no oauth support')->plan(48);
+
+$t->waitforsocket('127.0.0.1:' . port(8144));
+$t->waitforsocket('127.0.0.1:' . port(8111));
+$t->waitforsocket('127.0.0.1:' . port(8026));
+
+###############################################################################
+
+# AUTHBEARER SASL mechanism
+# https://datatracker.ietf.org/doc/html/rfc7628
+
+# XOAUTH2 SASL mechanism
+# https://developers.google.com/gmail/imap/xoauth2-protocol
+
+my $s;
+my $token = encode_base64(
+	"n,a=test\@example.com,\001auth=Bearer secretok\001\001", '');
+my $token_escaped = encode_base64(
+	"n,a=test=3D=2C\@example.com,\001auth=Bearer secretok\001\001", '');
+my $token_saslfail = encode_base64(
+	"n,a=test\@example.com,\001auth=Bearer saslfail\001\001", '');
+my $token_bad = encode_base64(
+	"n,a=test\@example.com,\001auth=Bearer bad\001\001", '');
+
+my $token_xoauth2 = encode_base64(
+	"user=test\@example.com\001auth=Bearer secretok\001\001", '');
+my $token_xoauth2_saslfail = encode_base64(
+	"user=test\@example.com\001auth=Bearer saslfail\001\001", '');
+my $token_xoauth2_bad = encode_base64(
+	"user=test\@example.com\001auth=Bearer bad\001\001", '');
+
+# IMAP
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token);
+$s->ok('imap oauthbearer success');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_escaped);
+$s->ok('imap oauthbearer escaped login');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE OAUTHBEARER');
+$s->check(qr/\+ /, 'imap oauthbearer challenge');
+$s->send($token);
+$s->ok('imap oauthbearer success after challenge');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_bad);
+$s->check(qr/^1 NO auth-failed/, 'imap oauthbearer non-sasl error');
+
+sleep(3);
+
+my @ready = $s->can_read(0);
+is(scalar @ready, 1, "imap ready for reading");
+ok($s->eof(), "imap session closed");
+
+# fail, sasl failure method
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+my $start = time;
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap oauthbearer sasl failure');
+my $wait_time = time - $start;
+ok($wait_time >= 1, 'imap oauthbearer error delayed');
+$s->send('AQ==');
+$s->check(qr/^1 NO auth-failed/,
+	'imap oauthbearer auth failure after dummy response');
+
+# fail, sasl failure method, invalid client response
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap oauthbearer sasl failure');
+$s->send('foo');
+$s->check(qr/^1 BAD /, 'imap oauthbearer invalid command after invalid line');
+
+# fail, sasl failure method, multiple attempts, then success
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap oauthbearer sasl failure');
+$s->send('AQ==');
+$s->check(qr/^1 NO auth-failed/,
+	'imap oauthbearer auth failure after dummy response');
+
+$s->send('1 AUTHENTICATE OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap oauthbearer sasl failure next');
+$s->send('foo');
+$s->check(qr/^1 BAD/, 'imap oauthbearer invalid command after invalid line');
+
+$s->send('1 AUTHENTICATE OAUTHBEARER');
+$s->check(qr/\+ /, 'imap oauthbearer challenge after fail');
+$s->send($token);
+$s->ok('imap oauthbearer success after fail');
+
+# IMAP XOAUTH2
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE XOAUTH2 ' . $token_xoauth2);
+$s->ok('imap xoauth2 success');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE XOAUTH2');
+$s->check(qr/^\+ /, 'imap xoauth2 challenge');
+$s->send($token_xoauth2);
+$s->ok('imap xoauth2 success after challenge');
+
+$s = Test::Nginx::IMAP->new();
+$s->read();
+$s->send('1 AUTHENTICATE XOAUTH2 ' . $token_xoauth2_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap xoauth2 with bad token');
+$s->send('');
+$s->check(qr/^1 NO auth-failed/, 'imap xoauth2 auth failure after empty line');
+
+$s->send('1 AUTHENTICATE XOAUTH2 ' . $token_xoauth2_saslfail);
+$s->check(qr/^\+ eyJz/, 'imap xoauth2 with bad token next');
+$s->send('foo');
+$s->check(qr/^1 BAD/, 'imap xoauth2 invalid command after invalid line');
+
+$s->send('1 AUTHENTICATE XOAUTH2 ' . $token_xoauth2);
+$s->ok('imap xoauth2 success after fail');
+
+# POP3
+
+$s = Test::Nginx::POP3->new();
+$s->read();
+$s->send('AUTH OAUTHBEARER ' . $token);
+$s->ok('pop3 oauthbearer success');
+
+$s = Test::Nginx::POP3->new();
+$s->read();
+$s->send('AUTH OAUTHBEARER');
+$s->check(qr/^\+ /, 'pop3 oauthbearer challenge');
+$s->send($token);
+$s->ok('pop3 oauthbearer success after challenge');
+
+$s = Test::Nginx::POP3->new();
+$s->read();
+$s->send('AUTH OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'pop3 oauthbearer sasl failure');
+$s->send('AQ==');
+$s->check(qr/^-ERR /, 'pop3 oauthbearer auth failure after dummy response');
+
+$s->send('AUTH OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^\+ eyJz/, 'pop3 oauthbearer sasl failure next');
+$s->send('');
+$s->check(qr/^-ERR /, 'pop3 oauthbearer invalid command after invalid line');
+
+$s->send('AUTH OAUTHBEARER ' . $token);
+$s->ok('pop3 oauthbearer success after fail');
+
+# POP3 XOAUTH2
+
+$s = Test::Nginx::POP3->new();
+$s->read();
+$s->send('AUTH XOAUTH2 ' . $token_xoauth2);
+$s->ok('pop3 xoauth2 success');
+
+$s = Test::Nginx::POP3->new();
+$s->read();
+$s->send('AUTH XOAUTH2');
+$s->check(qr/^\+ /, 'pop3 xoauth2 challenge');
+$s->send($token_xoauth2);
+$s->ok('pop3 xoauth2 success after challenge');
+
+# SMTP
+
+$s = Test::Nginx::SMTP->new();
+$s->read();
+$s->send('EHLO example.com');
+$s->read();
+$s->send('AUTH OAUTHBEARER ' . $token);
+$s->authok('smtp oauthbearer success');
+
+$s = Test::Nginx::SMTP->new();
+$s->read();
+$s->send('EHLO example.com');
+$s->read();
+$s->send('AUTH OAUTHBEARER');
+$s->check(qr/^334 /, 'smtp oauthbearer challenge');
+$s->send($token);
+$s->authok('smtp oauthbearer success after challenge');
+
+$s = Test::Nginx::SMTP->new();
+$s->read();
+$s->send('EHLO example.com');
+$s->read();
+$s->send('AUTH OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^334 eyJz/, 'smtp oauthbearer sasl failure');
+$s->send('AQ==');
+$s->check(qr/^535 /, 'smtp oauthbearer auth failure after dummy response');
+
+$s->send('AUTH OAUTHBEARER ' . $token_saslfail);
+$s->check(qr/^334 eyJz/, 'smtp oauthbearer sasl failure next');
+$s->send('foo');
+$s->check(qr/^500 /, 'smtp oauthbearer invalid command after invalid line');
+
+$s->send('AUTH OAUTHBEARER ' . $token);
+$s->authok('smtp oauthbearer success after fail');
+
+# SMTP XOAUTH2
+
+$s = Test::Nginx::SMTP->new();
+$s->read();
+$s->send('EHLO example.com');
+$s->read();
+$s->send('AUTH XOAUTH2 ' . $token_xoauth2);
+$s->authok('smtp xoauth2 success');
+
+$s = Test::Nginx::SMTP->new();
+$s->read();
+$s->send('EHLO example.com');
+$s->read();
+$s->send('AUTH XOAUTH2');
+$s->check(qr/^334 /, 'smtp xoauth2 challenge');
+$s->send($token_xoauth2);
+$s->authok('smtp xoauth2 success after challenge');
+
+###############################################################################


From mdounin at mdounin.ru  Mon Jun  3 15:17:32 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 3 Jun 2024 18:17:32 +0300
Subject: [nginx] Add support for XOAUTH2 and OAUTHBEARER authentication
In-Reply-To: <217bff3f-53b4-4f53-97a3-8cd13504e051@app.fastmail.com>
References: <c5d80668-2fcc-4cb3-b47f-2234a9ef8611@app.fastmail.com>
 <Zkv11aTD96ViUulu@mdounin.ru>
 <217bff3f-53b4-4f53-97a3-8cd13504e051@app.fastmail.com>
Message-ID: <Zl3ejO3l_VC13wGW@mdounin.ru>

Hello!

On Mon, Jun 03, 2024 at 09:12:12PM +1000, Robert Mueller wrote:

> > What do you think about this approach?
> 
> Thanks for looking at this patch. My nginx coding experience is 
> very limited, so I appreciate that you spent the time to look at 
> it carefully. I haven't looked closely at the updated patch, but 
> the changes you listed all sound like improvements and/or fixes 
> which is great. I'm happy to see this integrated upstream, so 
> that others can take advantage of it in the future.

Thanks for the feedback, and thanks for the patch, committed.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Mon Jun  3 22:26:56 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 4 Jun 2024 01:26:56 +0300
Subject: freengin-1.27.1 changes draft
Message-ID: <Zl5DMKqNWMmVao5R@mdounin.ru>

Hello!

Below are changes draft for freenginx 1.27.1.
Comments are welcome.


Changes with freenginx 1.27.1                                    04 Jun 2024

    *) Feature: the "max_headers" directive.
       Thanks to Maksim Yevmenkin.

    *) Feature: the $upstream_cache_key variable.
       Thanks to Kirill A. Korinsky.

    *) Feature: XOAUTH2 and OAUTHBEARER authentication mechanisms support in
       the mail proxy module.
       Thanks to Rob Mueller.

    *) Bugfix: graceful shutdown of old worker processes might be delayed
       when using HTTP/2.
       Thanks to Kasei Wang.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/3.

    *) Bugfix: in HTTP/3.

    *) Bugfix: in the mail proxy module.


????????? ? freenginx 1.27.1                                      04.06.2024

    *) ??????????: ????????? max_headers.
       ??????? ??????? ?????????.

    *) ??????????: ?????????? $upstream_cache_key.
       ??????? ??????? ??????????.

    *) ??????????: ????????? ??????? ?????????????? XOAUTH2 ? OAUTHBEARER ?
       ???????? ??????-???????.
       ??????? Rob Mueller.

    *) ???????????: ??????? ?????????? ?????? ??????? ????????? ?????
       ????????????? ??? ????????????? HTTP/2.
       ??????? Kasei Wang.

    *) ???????????: ??? ????????????? HTTP/3 ? ??????? ???????? ???
       ????????? segmentation fault.

    *) ???????????: ? HTTP/3.

    *) ???????????: ? ???????? ??????-???????.


-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Tue Jun  4 14:06:45 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 17:06:45 +0300
Subject: [nginx] freenginx-1.27.1-RELEASE
Message-ID: <hg.ee3eb2b9705f.1717510005.-2981570580444711699@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/ee3eb2b9705f
branches:  
changeset: 9291:ee3eb2b9705f
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 16:55:53 2024 +0300
description:
freenginx-1.27.1-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  82 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 82 insertions(+), 0 deletions(-)

diffs (92 lines):

diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml
+++ b/docs/xml/nginx/changes.xml
@@ -7,6 +7,88 @@
 <section title="freenginx">
 
 
+<changes ver="1.27.1" date="2024-06-04">
+
+<change type="feature">
+<para lang="ru">
+????????? max_headers.<br/>
+??????? ??????? ?????????.
+</para>
+<para lang="en">
+the "max_headers" directive.<br/>
+Thanks to Maksim Yevmenkin.
+</para>
+</change>
+
+<change type="feature">
+<para lang="ru">
+?????????? $upstream_cache_key.<br/>
+??????? ??????? ??????????.
+</para>
+<para lang="en">
+the $upstream_cache_key variable.<br/>
+Thanks to Kirill A. Korinsky.
+</para>
+</change>
+
+<change type="feature">
+<para lang="ru">
+????????? ??????? ?????????????? XOAUTH2 ? OAUTHBEARER
+? ???????? ??????-???????.<br/>
+??????? Rob Mueller.
+</para>
+<para lang="en">
+XOAUTH2 and OAUTHBEARER authentication mechanisms support
+in the mail proxy module.<br/>
+Thanks to Rob Mueller.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??????? ?????????? ?????? ??????? ????????? ????? ?????????????
+??? ????????????? HTTP/2.<br/>
+??????? Kasei Wang.
+</para>
+<para lang="en">
+graceful shutdown of old worker processes might be delayed
+when using HTTP/2.<br/>
+Thanks to Kasei Wang.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/3
+? ??????? ???????? ??? ????????? segmentation fault.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process
+when using HTTP/3.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+? HTTP/3.
+</para>
+<para lang="en">
+in HTTP/3.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+? ???????? ??????-???????.
+</para>
+<para lang="en">
+in the mail proxy module.
+</para>
+</change>
+
+</changes>
+
+
 <changes ver="1.27.0" date="2024-05-14">
 
 <change type="feature">

From mdounin at mdounin.ru  Tue Jun  4 14:06:46 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 17:06:46 +0300
Subject: [nginx] release-1.27.1 tag
Message-ID: <hg.7654ad1366ef.1717510006.-2981570580444711699@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/7654ad1366ef
branches:  
changeset: 9292:7654ad1366ef
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 16:55:54 2024 +0300
description:
release-1.27.1 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -479,3 +479,4 @@ 294a3d07234f8f65d7b0e0b0e2c5b05c12c5da0a
 ab948bfa042d7a7b20c3e730d7e9675cc172324f release-1.25.4
 2956b59565c91baa79d13d6411f2404614c0134e release-1.25.5
 8c4e2b7de093d357b2f462399d6d395b899ffe76 release-1.27.0
+ee3eb2b9705f0c913a1bf4b9fe74def31411e8bf release-1.27.1


From mdounin at mdounin.ru  Tue Jun  4 14:07:05 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 17:07:05 +0300
Subject: [nginx-site] freenginx-1.27.1
Message-ID: <hg.f7c8eeeaeafe.1717510025.6434924221447703544@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-site/rev/f7c8eeeaeafe
branches:  
changeset: 3087:f7c8eeeaeafe
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 17:04:23 2024 +0300
description:
freenginx-1.27.1

diffstat:

 text/en/CHANGES    |  24 ++++++++++++++++++++++++
 text/ru/CHANGES.ru |  24 ++++++++++++++++++++++++
 xml/index.xml      |   7 +++++++
 xml/versions.xml   |   1 +
 4 files changed, 56 insertions(+), 0 deletions(-)

diffs (92 lines):

diff --git a/text/en/CHANGES b/text/en/CHANGES
--- a/text/en/CHANGES
+++ b/text/en/CHANGES
@@ -1,4 +1,28 @@
 
+Changes with freenginx 1.27.1                                    04 Jun 2024
+
+    *) Feature: the "max_headers" directive.
+       Thanks to Maksim Yevmenkin.
+
+    *) Feature: the $upstream_cache_key variable.
+       Thanks to Kirill A. Korinsky.
+
+    *) Feature: XOAUTH2 and OAUTHBEARER authentication mechanisms support in
+       the mail proxy module.
+       Thanks to Rob Mueller.
+
+    *) Bugfix: graceful shutdown of old worker processes might be delayed
+       when using HTTP/2.
+       Thanks to Kasei Wang.
+
+    *) Bugfix: a segmentation fault might occur in a worker process when
+       using HTTP/3.
+
+    *) Bugfix: in HTTP/3.
+
+    *) Bugfix: in the mail proxy module.
+
+
 Changes with freenginx 1.27.0                                    14 May 2024
 
     *) Feature: updated descriptions of HTTP status codes.
diff --git a/text/ru/CHANGES.ru b/text/ru/CHANGES.ru
--- a/text/ru/CHANGES.ru
+++ b/text/ru/CHANGES.ru
@@ -1,4 +1,28 @@
 
+????????? ? freenginx 1.27.1                                      04.06.2024
+
+    *) ??????????: ????????? max_headers.
+       ??????? ??????? ?????????.
+
+    *) ??????????: ?????????? $upstream_cache_key.
+       ??????? ??????? ??????????.
+
+    *) ??????????: ????????? ??????? ?????????????? XOAUTH2 ? OAUTHBEARER ?
+       ???????? ??????-???????.
+       ??????? Rob Mueller.
+
+    *) ???????????: ??????? ?????????? ?????? ??????? ????????? ?????
+       ????????????? ??? ????????????? HTTP/2.
+       ??????? Kasei Wang.
+
+    *) ???????????: ??? ????????????? HTTP/3 ? ??????? ???????? ???
+       ????????? segmentation fault.
+
+    *) ???????????: ? HTTP/3.
+
+    *) ???????????: ? ???????? ??????-???????.
+
+
 ????????? ? freenginx 1.27.0                                      14.05.2024
 
     *) ??????????: ????????? ????????? ???????? ????? ???????.
diff --git a/xml/index.xml b/xml/index.xml
--- a/xml/index.xml
+++ b/xml/index.xml
@@ -8,6 +8,13 @@
 
 <news name="freenginx news" link="/" lang="en">
 
+<event date="2024-06-04">
+<para>
+<link doc="en/download.xml">freenginx-1.27.1</link>
+mainline version has been released.
+</para>
+</event>
+
 <event date="2024-05-14">
 <para>
 <link doc="en/download.xml">freenginx-1.27.0</link>
diff --git a/xml/versions.xml b/xml/versions.xml
--- a/xml/versions.xml
+++ b/xml/versions.xml
@@ -9,6 +9,7 @@
 
 <download tag="mainline" changes="">
 
+<item ver="1.27.1" name="freenginx" />
 <item ver="1.27.0" name="freenginx" />
 
 </download>

From mdounin at mdounin.ru  Tue Jun  4 15:35:47 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 18:35:47 +0300
Subject: [PATCH] Documented XOAUTH2 and OAUTHBEARER authentication methods
Message-ID: <7b7dbaa7d777cd8cacae.1717515347@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1717515321 -10800
#      Tue Jun 04 18:35:21 2024 +0300
# Node ID 7b7dbaa7d777cd8cacaebb177d76559f84736b94
# Parent  f7c8eeeaeafe54cabde349154d234272180f9044
Documented XOAUTH2 and OAUTHBEARER authentication methods.

diff --git a/xml/en/docs/mail/ngx_mail_auth_http_module.xml b/xml/en/docs/mail/ngx_mail_auth_http_module.xml
--- a/xml/en/docs/mail/ngx_mail_auth_http_module.xml
+++ b/xml/en/docs/mail/ngx_mail_auth_http_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_auth_http_module"
         link="/en/docs/mail/ngx_mail_auth_http_module.html"
         lang="en"
-        rev="11">
+        rev="12">
 
 <section id="directives" name="Directives">
 
@@ -203,6 +203,18 @@ Auth-SMTP-To: RCPT TO: &lt;postmaster at ma
 </para>
 
 <para>
+For the XOAUTH2 and OAUTHBEARER authentication methods (1.27.1),
+the <header>Auth-Error-SASL</header> header
+could be used to return an error response
+in the form of an additional base64-encoded SASL challenge
+(<link
+url="https://developers.google.com/gmail/imap/xoauth2-protocol#error_response"
+>XOAUTH2</link>,
+<link url="https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.2"
+>OAUTHBEARER</link>).
+</para>
+
+<para>
 For the SSL/TLS client connection (1.7.11),
 the <header>Auth-SSL</header> header is added, and
 <header>Auth-SSL-Verify</header> will contain
diff --git a/xml/en/docs/mail/ngx_mail_imap_module.xml b/xml/en/docs/mail/ngx_mail_imap_module.xml
--- a/xml/en/docs/mail/ngx_mail_imap_module.xml
+++ b/xml/en/docs/mail/ngx_mail_imap_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_imap_module"
         link="/en/docs/mail/ngx_mail_imap_module.html"
         lang="en"
-        rev="7">
+        rev="8">
 
 <section id="directives" name="Directives">
 
@@ -47,6 +47,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH=EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH=XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH=OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/en/docs/mail/ngx_mail_pop3_module.xml b/xml/en/docs/mail/ngx_mail_pop3_module.xml
--- a/xml/en/docs/mail/ngx_mail_pop3_module.xml
+++ b/xml/en/docs/mail/ngx_mail_pop3_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_pop3_module"
         link="/en/docs/mail/ngx_mail_pop3_module.html"
         lang="en"
-        rev="5">
+        rev="6">
 
 <section id="directives" name="Directives">
 
@@ -49,6 +49,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/en/docs/mail/ngx_mail_smtp_module.xml b/xml/en/docs/mail/ngx_mail_smtp_module.xml
--- a/xml/en/docs/mail/ngx_mail_smtp_module.xml
+++ b/xml/en/docs/mail/ngx_mail_smtp_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_smtp_module"
         link="/en/docs/mail/ngx_mail_smtp_module.html"
         lang="en"
-        rev="8">
+        rev="9">
 
 <section id="directives" name="Directives">
 
@@ -48,6 +48,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 <tag-name><literal>none</literal></tag-name>
 <tag-desc>
 Authentication is not required.
diff --git a/xml/ru/docs/mail/ngx_mail_auth_http_module.xml b/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
--- a/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_auth_http_module"
         link="/ru/docs/mail/ngx_mail_auth_http_module.html"
         lang="ru"
-        rev="11">
+        rev="12">
 
 <section id="directives" name="?????????">
 
@@ -201,6 +201,18 @@ Auth-SMTP-To: RCPT TO: &lt;postmaster at ma
 </para>
 
 <para>
+??? ??????? ?????????????? XOAUTH2 and OAUTHBEARER (1.27.1)
+? ????????? <header>Auth-Error-SASL</header>
+????? ??????? ?????????? ?? ??????
+? ????? ??????????????? SASL challenge ? base64
+(<link
+url="https://developers.google.com/gmail/imap/xoauth2-protocol#error_response"
+>XOAUTH2</link>,
+<link url="https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.2"
+>OAUTHBEARER</link>).
+</para>
+
+<para>
 ??? ??????????? ?????????? ?? ????????? SSL/TLS (1.7.11)
 ??????????? ????????? <header>Auth-SSL</header>, ? ???? ?????????
 <link doc="ngx_mail_ssl_module.xml" id="ssl_verify_client"/> ????????,
diff --git a/xml/ru/docs/mail/ngx_mail_imap_module.xml b/xml/ru/docs/mail/ngx_mail_imap_module.xml
--- a/xml/ru/docs/mail/ngx_mail_imap_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_imap_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_imap_module"
         link="/ru/docs/mail/ngx_mail_imap_module.html"
         lang="ru"
-        rev="7">
+        rev="8">
 
 <section id="directives" name="?????????">
 
@@ -47,6 +47,18 @@
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH=EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH=XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH=OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/ru/docs/mail/ngx_mail_pop3_module.xml b/xml/ru/docs/mail/ngx_mail_pop3_module.xml
--- a/xml/ru/docs/mail/ngx_mail_pop3_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_pop3_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_pop3_module"
         link="/ru/docs/mail/ngx_mail_pop3_module.html"
         lang="ru"
-        rev="5">
+        rev="6">
 
 <section id="directives" name="?????????">
 
@@ -49,6 +49,18 @@
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/ru/docs/mail/ngx_mail_smtp_module.xml b/xml/ru/docs/mail/ngx_mail_smtp_module.xml
--- a/xml/ru/docs/mail/ngx_mail_smtp_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_smtp_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_smtp_module"
         link="/ru/docs/mail/ngx_mail_smtp_module.html"
         lang="ru"
-        rev="8">
+        rev="9">
 
 <section id="directives" name="?????????">
 
@@ -48,6 +48,18 @@ SMTP-????????.
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 <tag-name><literal>none</literal></tag-name>
 <tag-desc>
 ?????????????? ?? ?????????.

From mdounin at mdounin.ru  Tue Jun  4 15:36:48 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 18:36:48 +0300
Subject: [PATCH] Tests: reworked HTTP/2 tests to use "http2 on"
Message-ID: <11463d3795703442f320.1717515408@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1717466882 -10800
#      Tue Jun 04 05:08:02 2024 +0300
# Node ID 11463d3795703442f320ef21a58733e74408cd7c
# Parent  b5e2609d34a385461bc0f2eaf6388f018acc1ce5
Tests: reworked HTTP/2 tests to use "http2 on".

diff --git a/grpc.t b/grpc.t
--- a/grpc.t
+++ b/grpc.t
@@ -44,9 +44,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 128k;
         large_client_header_buffers 4 32k;
 
@@ -90,11 +91,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_next_upstream.t b/grpc_next_upstream.t
--- a/grpc_next_upstream.t
+++ b/grpc_next_upstream.t
@@ -69,9 +69,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
+        listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 404;
         }
@@ -91,9 +93,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 200 "TEST-OK-IF-YOU-SEE-THIS\n";
         }
@@ -106,11 +110,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_pass.t b/grpc_pass.t
--- a/grpc_pass.t
+++ b/grpc_pass.t
@@ -63,10 +63,12 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
-        listen       127.0.0.1:8082 http2 ssl;
+        listen       127.0.0.1:8081;
+        listen       127.0.0.1:8082 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate_key localhost.key;
         ssl_certificate localhost.crt;
 
@@ -98,11 +100,7 @@ foreach my $name ('localhost') {
 
 $t->run_daemon(\&dns_daemon, port(8982), $t);
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run()->plan(5);
-open STDERR, ">&", \*OLDERR;
 
 $t->waitforfile($t->testdir . '/' . port(8982));
 
diff --git a/grpc_request_buffering.t b/grpc_request_buffering.t
--- a/grpc_request_buffering.t
+++ b/grpc_request_buffering.t
@@ -38,10 +38,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location /mirror { }
 
         location / {
@@ -64,11 +66,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_ssl.t b/grpc_ssl.t
--- a/grpc_ssl.t
+++ b/grpc_ssl.t
@@ -44,7 +44,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2 ssl;
+        listen       127.0.0.1:8081 ssl;
         server_name  localhost;
 
         ssl_certificate_key localhost.key;
@@ -53,6 +53,7 @@ http {
         ssl_verify_client optional;
         ssl_client_certificate client.crt;
 
+        http2 on;
         http2_body_preread_size 128k;
 
         location / {
@@ -62,9 +63,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 128k;
 
         location / {
@@ -129,11 +131,7 @@ sleep 1 if $^O eq 'MSWin32';
 
 $t->write_file('password', 'client');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2.t b/h2.t
--- a/h2.t
+++ b/h2.t
@@ -40,8 +40,10 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
+
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
@@ -88,26 +90,26 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
         return 200   first;
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost2;
         return 200   second;
     }
 
     server {
-        listen       127.0.0.1:8083 http2;
+        listen       127.0.0.1:8083;
         server_name  localhost;
 
         http2_max_concurrent_streams 1;
     }
 
     server {
-        listen       127.0.0.1:8086 http2;
+        listen       127.0.0.1:8086;
         server_name  localhost;
 
         send_timeout 1s;
@@ -115,7 +117,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8087 http2;
+        listen       127.0.0.1:8087;
         server_name  localhost;
 
         client_header_timeout 1s;
@@ -132,11 +134,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 # file size is slightly beyond initial window size: 2**16 + 80 bytes
 
diff --git a/h2_absolute_redirect.t b/h2_absolute_redirect.t
--- a/h2_absolute_redirect.t
+++ b/h2_absolute_redirect.t
@@ -36,10 +36,11 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
     absolute_redirect off;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  on;
 
         absolute_redirect on;
@@ -75,7 +76,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  off;
 
         location / { }
@@ -107,11 +108,7 @@ EOF
 mkdir($t->testdir() . '/dir');
 mkdir($t->testdir() . '/dir sp');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run()->plan(23);
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_auth_request.t b/h2_auth_request.t
--- a/h2_auth_request.t
+++ b/h2_auth_request.t
@@ -39,10 +39,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 200;
         }
@@ -66,11 +68,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_error_page.t b/h2_error_page.t
--- a/h2_error_page.t
+++ b/h2_error_page.t
@@ -37,9 +37,10 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         lingering_close off;
 
         error_page 400 = /close;
@@ -54,11 +55,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_fastcgi_request_buffering.t b/h2_fastcgi_request_buffering.t
--- a/h2_fastcgi_request_buffering.t
+++ b/h2_fastcgi_request_buffering.t
@@ -38,9 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             fastcgi_request_buffering off;
             fastcgi_pass 127.0.0.1:8081;
@@ -52,11 +54,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_headers.t b/h2_headers.t
--- a/h2_headers.t
+++ b/h2_headers.t
@@ -37,11 +37,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
-        listen       127.0.0.1:8082 http2 sndbuf=128;
+        listen       127.0.0.1:8082 sndbuf=128;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 2 64k;
 
         location / {
@@ -89,31 +90,35 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8084 http2;
+        listen       127.0.0.1:8084;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 4 512;
     }
 
     server {
-        listen       127.0.0.1:8085 http2;
+        listen       127.0.0.1:8085;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 1 512;
     }
 
     server {
-        listen       127.0.0.1:8086 http2;
+        listen       127.0.0.1:8086;
         server_name  localhost;
 
+        http2 on;
         underscores_in_headers on;
         add_header X-Sent-Foo $http_x_foo always;
     }
 
     server {
-        listen       127.0.0.1:8087 http2;
+        listen       127.0.0.1:8087;
         server_name  localhost;
 
+        http2 on;
         ignore_invalid_headers off;
         add_header X-Sent-Foo $http_x_foo always;
     }
@@ -123,11 +128,7 @@ EOF
 
 $t->run_daemon(\&http_daemon);
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 $t->waitforsocket('127.0.0.1:' . port(8083));
 
diff --git a/h2_keepalive.t b/h2_keepalive.t
--- a/h2_keepalive.t
+++ b/h2_keepalive.t
@@ -39,27 +39,30 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2 sndbuf=1m;
+        listen       127.0.0.1:8080 sndbuf=1m;
         server_name  localhost;
 
+        http2 on;
         keepalive_requests 2;
 
         location / { }
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
+        listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         keepalive_timeout 0;
 
         location / { }
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
         keepalive_time 1s;
 
         add_header X-Conn $connection_requests:$connection_time;
@@ -73,11 +76,7 @@ EOF
 $t->write_file('index.html', 'SEE-THAT' x 50000);
 $t->write_file('t.html', 'SEE-THAT');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_limit_conn.t b/h2_limit_conn.t
--- a/h2_limit_conn.t
+++ b/h2_limit_conn.t
@@ -39,9 +39,11 @@ http {
     limit_conn_zone  $binary_remote_addr  zone=conn:1m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /t.html {
             limit_conn conn 1;
         }
@@ -51,12 +53,7 @@ http {
 EOF
 
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_limit_req.t b/h2_limit_req.t
--- a/h2_limit_req.t
+++ b/h2_limit_req.t
@@ -41,10 +41,12 @@ http {
     limit_req_zone   $binary_remote_addr  zone=req:1m rate=1r/s;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / { }
         location /limit_req {
             limit_req  zone=req burst=2;
@@ -64,12 +66,7 @@ EOF
 
 $t->write_file('index.html', '');
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_priority.t b/h2_priority.t
--- a/h2_priority.t
+++ b/h2_priority.t
@@ -37,18 +37,15 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
+        http2 on;
     }
 }
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 # file size is slightly beyond initial window size: 2**16 + 80 bytes
 
diff --git a/h2_proxy_cache.t b/h2_proxy_cache.t
--- a/h2_proxy_cache.t
+++ b/h2_proxy_cache.t
@@ -39,10 +39,12 @@ http {
     proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location /cache {
             proxy_pass http://127.0.0.1:8081/;
             proxy_cache NAME;
@@ -68,12 +70,7 @@ EOF
 
 $t->write_file('t.html', 'SEE-THIS');
 $t->write_file('slow.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_max_temp_file_size.t b/h2_proxy_max_temp_file_size.t
--- a/h2_proxy_max_temp_file_size.t
+++ b/h2_proxy_max_temp_file_size.t
@@ -38,9 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         proxy_buffer_size 4k;
         proxy_buffers 8 4k;
 
@@ -66,12 +68,7 @@ http {
 EOF
 
 $t->write_file('1', 'X' x (1024 * 1024));
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_protocol.t b/h2_proxy_protocol.t
--- a/h2_proxy_protocol.t
+++ b/h2_proxy_protocol.t
@@ -39,9 +39,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 proxy_protocol http2;
+        listen       127.0.0.1:8080 proxy_protocol;
         server_name  localhost;
 
+        http2 on;
+
         location /pp {
             set_real_ip_from 127.0.0.1/32;
             real_ip_header proxy_protocol;
@@ -54,12 +56,7 @@ http {
 EOF
 
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering.t b/h2_proxy_request_buffering.t
--- a/h2_proxy_request_buffering.t
+++ b/h2_proxy_request_buffering.t
@@ -40,10 +40,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_request_buffering off;
             proxy_pass http://127.0.0.1:8081/;
@@ -65,11 +67,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering_redirect.t b/h2_proxy_request_buffering_redirect.t
--- a/h2_proxy_request_buffering_redirect.t
+++ b/h2_proxy_request_buffering_redirect.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         proxy_http_version 1.1;
 
         location / {
@@ -68,11 +69,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering_ssl.t b/h2_proxy_request_buffering_ssl.t
--- a/h2_proxy_request_buffering_ssl.t
+++ b/h2_proxy_request_buffering_ssl.t
@@ -41,9 +41,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_request_buffering off;
             proxy_pass https://127.0.0.1:8082;
@@ -98,11 +100,7 @@ foreach my $name ('localhost') {
 		or die "Can't create certificate for $name: $!\n";
 }
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_ssl.t b/h2_proxy_ssl.t
--- a/h2_proxy_ssl.t
+++ b/h2_proxy_ssl.t
@@ -39,10 +39,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate_key localhost.key;
         ssl_certificate localhost.crt;
 
@@ -74,12 +76,7 @@ foreach my $name ('localhost') {
 }
 
 $t->write_file('index.html', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body.t b/h2_request_body.t
--- a/h2_request_body.t
+++ b/h2_request_body.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         error_page 400 /proxy2/t.html;
 
         location / {
@@ -77,12 +78,7 @@ EOF
 $t->write_file('index.html', '');
 $t->write_file('t.html', 'SEE-THIS');
 $t->write_file('slow.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body_extra.t b/h2_request_body_extra.t
--- a/h2_request_body_extra.t
+++ b/h2_request_body_extra.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         client_header_buffer_size 1k;
         client_body_buffer_size 2k;
 
@@ -88,12 +89,7 @@ http {
 EOF
 
 $t->plan(50);
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body_preread.t b/h2_request_body_preread.t
--- a/h2_request_body_preread.t
+++ b/h2_request_body_preread.t
@@ -40,10 +40,11 @@ http {
     limit_req_zone   $binary_remote_addr  zone=req:1m rate=20r/m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 10;
 
         location /t { }
@@ -59,9 +60,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 0;
 
         location / {
@@ -76,9 +78,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8083 http2;
+        listen       127.0.0.1:8083;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             add_header X-Body $request_body;
             proxy_pass http://127.0.0.1:8081/t;
@@ -89,12 +93,7 @@ http {
 EOF
 
 $t->write_file('t', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_server_tokens.t b/h2_server_tokens.t
--- a/h2_server_tokens.t
+++ b/h2_server_tokens.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /200 {
             return 200;
         }
@@ -88,11 +90,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_ssl_verify_client.t b/h2_ssl_verify_client.t
--- a/h2_ssl_verify_client.t
+++ b/h2_ssl_verify_client.t
@@ -38,6 +38,7 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
     ssl_certificate_key localhost.key;
     ssl_certificate localhost.crt;
 
@@ -46,7 +47,7 @@ http {
     add_header X-Verify $ssl_client_verify;
 
     server {
-        listen       127.0.0.1:8080 ssl http2;
+        listen       127.0.0.1:8080 ssl;
         server_name  localhost;
 
         ssl_client_certificate client.crt;
@@ -55,7 +56,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 ssl http2;
+        listen       127.0.0.1:8080 ssl;
         server_name  example.com;
 
         location / { }
@@ -84,9 +85,7 @@ foreach my $name ('localhost', 'client')
 
 $t->write_file('t', 'SEE-THIS');
 
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 my $s = get_ssl_socket();
 plan(skip_all => 'no alpn') unless $s->alpn_selected();
diff --git a/h2_trailers.t b/h2_trailers.t
--- a/h2_trailers.t
+++ b/h2_trailers.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             add_trailer X-Var $host;
         }
@@ -60,12 +62,7 @@ EOF
 $t->write_file('index.html', 'SEE-THIS');
 $t->write_file('empty', '');
 $t->write_file('continuation', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_variables.t b/h2_variables.t
--- a/h2_variables.t
+++ b/h2_variables.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /h2 {
             return 200 $http2;
         }
@@ -60,11 +62,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h3_server_name.t b/h3_server_name.t
--- a/h3_server_name.t
+++ b/h3_server_name.t
@@ -44,10 +44,12 @@ http {
     ssl_certificate localhost.crt;
 
     server {
-        listen       127.0.0.1:8443 ssl http2;
+        listen       127.0.0.1:8443 ssl;
         listen       127.0.0.1:%%PORT_8980_UDP%% quic;
         server_name  ~^(?P<name>.+)\.example\.com$;
 
+        http2 on;
+
         location / {
             return 200 $name;
         }
@@ -74,11 +76,7 @@ foreach my $name ('localhost') {
 		or die "Can't create certificate for $name: $!\n";
 }
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/proxy_ssl_conf_command.t b/proxy_ssl_conf_command.t
--- a/proxy_ssl_conf_command.t
+++ b/proxy_ssl_conf_command.t
@@ -72,9 +72,11 @@ http {
 
     server {
         listen       127.0.0.1:8081 ssl;
-        listen       127.0.0.1:8082 ssl http2;
+        listen       127.0.0.1:8082 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate localhost.crt;
         ssl_certificate_key localhost.key;
         ssl_verify_client optional_no_ca;
@@ -106,12 +108,7 @@ foreach my $name ('localhost', 'override
 }
 
 $t->write_file('index.html', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/worker_shutdown_timeout_h2.t b/worker_shutdown_timeout_h2.t
--- a/worker_shutdown_timeout_h2.t
+++ b/worker_shutdown_timeout_h2.t
@@ -39,9 +39,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_pass http://127.0.0.1:8081;
             proxy_read_timeout 5s;
@@ -51,13 +53,7 @@ http {
 EOF
 
 $t->run_daemon(\&http_silent_daemon);
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
-
 $t->waitforsocket('127.0.0.1:' . port(8081));
 
 ###############################################################################



From mdounin at mdounin.ru  Tue Jun  4 15:38:24 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Tue, 04 Jun 2024 18:38:24 +0300
Subject: [PATCH] Tests: removed TODO and try_run() checks for legacy versions
Message-ID: <a095b971fbcc99a77206.1717515504@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1717515481 -10800
#      Tue Jun 04 18:38:01 2024 +0300
# Node ID a095b971fbcc99a77206173f6130d5ff2681c389
# Parent  11463d3795703442f320ef21a58733e74408cd7c
Tests: removed TODO and try_run() checks for legacy versions.

For h2_http2.t, try_run() is preserved to ensure that deprecation
warnings for "listen ... http2" are suppressed, yet plan() is reported
before try_run(), so failure to start will be properly reported.

diff --git a/auth_request.t b/auth_request.t
--- a/auth_request.t
+++ b/auth_request.t
@@ -203,14 +203,9 @@ like(http_post_big('/proxy-double'), qr/
 
 # Multiple WWW-Authenticate headers (ticket #485).
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/proxy-multi-auth'), qr/WWW-Authenticate: foo.*bar/s,
 	'multiple www-authenticate headers');
 
-}
-
 SKIP: {
 	eval { require FCGI; };
 	skip 'FCGI not installed', 2 if $@;
diff --git a/autoindex_win32.t b/autoindex_win32.t
--- a/autoindex_win32.t
+++ b/autoindex_win32.t
@@ -76,9 +76,6 @@ my $r = http_get('/');
 like($r, qr!href="test-file"!ms, 'file');
 like($r, qr!href="test-dir/"!ms, 'directory');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 like($r, qr!href="test-file-(%d0%bc%d0%b8){3}"!msi, 'utf file link');
 like($r, qr!test-file-(\xd0\xbc\xd0\xb8){3}</a>!ms, 'utf file name');
 
@@ -92,8 +89,6 @@ like($r, qr!Index of /test-dir-(\xd0\xbc
 like($r, qr!href="test-subfile-(%d0%bc%d0%b8){3}"!msi, 'utf subdir link');
 like($r, qr!test-subfile-(\xd0\xbc\xd0\xb8){3}</a>!msi, 'utf subdir name');
 
-}
-
 ###############################################################################
 
 sub win32_mkdir {
diff --git a/body_chunked.t b/body_chunked.t
--- a/body_chunked.t
+++ b/body_chunked.t
@@ -179,9 +179,6 @@ like(
 	qr/ 200 /, 'chunk extensions'
 );
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -194,8 +191,6 @@ like(
 	qr/ 413 /, 'too many chunk extensions'
 );
 
-}
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -208,9 +203,6 @@ like(
 	qr/ 200 /, 'trailers'
 );
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -223,8 +215,6 @@ like(
 	qr/ 413 /, 'too many trailers'
 );
 
-}
-
 # proxy_next_upstream
 
 like(http_get_body('/next', '0123456789'),
diff --git a/dav_utf8.t b/dav_utf8.t
--- a/dav_utf8.t
+++ b/dav_utf8.t
@@ -57,8 +57,6 @@ EOF
 
 ###############################################################################
 
-local $TODO = 'not yet' if $^O eq 'MSWin32' and !$t->has_version('1.23.4');
-
 my $d = $t->testdir();
 my $r;
 
diff --git a/fastcgi_header_params.t b/fastcgi_header_params.t
--- a/fastcgi_header_params.t
+++ b/fastcgi_header_params.t
@@ -59,9 +59,6 @@ EOF
 like(http_get_headers('/'), qr/SEE-THIS/,
 	'fastcgi request with many ignored headers');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r;
 
 $r = http(<<EOF);
@@ -88,8 +85,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'fastcgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {
diff --git a/h2.t b/h2.t
--- a/h2.t
+++ b/h2.t
@@ -1014,15 +1014,10 @@ is($frame->{headers}->{':status'}, 200, 
 
 # invalid connection preface
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.1');
-
 like(http('x' x 16), qr/400 Bad Request/, 'invalid preface');
 like(http('PRI * HTTP/2.0' . CRLF . CRLF . 'x' x 8), qr/400 Bad Request/,
 	'invalid preface 2');
 
-}
-
 # GOAWAY on SYN_STREAM with even StreamID
 
 $s = Test::Nginx::HTTP2->new();
diff --git a/h2_error_page.t b/h2_error_page.t
--- a/h2_error_page.t
+++ b/h2_error_page.t
@@ -85,19 +85,13 @@ my $s2 = Test::Nginx::HTTP2->new();
 $sid = $s2->new_stream({ method => 'foo' });
 $frames = $s2->read(all => [{ type => 'RST_STREAM' }]);
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 ($frame) = grep { $_->{type} eq "RST_STREAM" } @$frames;
 is($frame->{sid}, $sid, 'error 400 return 444 - invalid header');
 
-}
-
 # while keeping $s1 and $s2, stop nginx; this should result in
 # "open socket ... left in connection ..." alerts if any of these
 # sockets are still open
 
 $t->stop();
-$t->todo_alerts() unless $t->has_version('1.23.4');
 
 ###############################################################################
diff --git a/h2_headers.t b/h2_headers.t
--- a/h2_headers.t
+++ b/h2_headers.t
@@ -959,13 +959,8 @@ ok($frame, 'HPACK table boundary - heade
 
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 isnt($frame->{headers}->{'x-referer'}, 'see-this', 'newline in request header');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
-is($frame->{headers}->{':status'}, 400, 'newline in request header - bad request');
-
-}
+is($frame->{headers}->{':status'}, 400,
+	'newline in request header - bad request');
 
 # invalid header name as seen with underscore should not lead to ignoring rest
 
@@ -984,9 +979,6 @@ is($frame->{headers}->{'x-referer'}, 'se
 
 # other invalid header name characters as seen with ':'
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 $s = Test::Nginx::HTTP2->new();
 $sid = $s->new_stream({ headers => [
 	{ name => ':method', value => 'GET', mode => 0 },
@@ -1024,8 +1016,6 @@ is($frame->{headers}->{':status'}, 400, 
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 is($frame->{headers}->{':status'}, 400, 'control in header name');
 
-}
-
 # header name with underscore - underscores_in_headers on
 
 $s = Test::Nginx::HTTP2->new(port(8086));
diff --git a/h2_http2.t b/h2_http2.t
--- a/h2_http2.t
+++ b/h2_http2.t
@@ -24,7 +24,8 @@ select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
 my $t = Test::Nginx->new()->has(qw/http http_ssl http_v2 socket_ssl_alpn/)
-	->has_daemon('openssl');
+	->has_daemon('openssl')
+	->plan(11);
 
 $t->write_file_expand('nginx.conf', <<'EOF');
 
@@ -108,7 +109,7 @@ foreach my $name ('localhost') {
 }
 
 $t->write_file('index.html', '');
-$t->try_run('no http2')->plan(11);
+$t->try_run();
 
 ###############################################################################
 
diff --git a/h2_proxy_protocol.t b/h2_proxy_protocol.t
--- a/h2_proxy_protocol.t
+++ b/h2_proxy_protocol.t
@@ -71,12 +71,7 @@ is($frame->{headers}->{'x-pp'}, '192.0.2
 
 # invalid PROXY protocol string
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.1');
-
 $proxy = 'BOGUS TCP4 192.0.2.1 192.0.2.2 1234 5678' . CRLF;
 ok(!http($proxy), 'PROXY invalid protocol');
 
-}
-
 ###############################################################################
diff --git a/http_headers_multi.t b/http_headers_multi.t
--- a/http_headers_multi.t
+++ b/http_headers_multi.t
@@ -144,15 +144,9 @@ like(get('/', map { "X-Forwarded-For: $_
 	qr/X-Forwarded-For: foo, bar, bazz/, 'multi $http_x_forwarded_for');
 like(get('/', 'Cookie: foo=1', 'Cookie: bar=2', 'Cookie: bazz=3'),
 	qr/X-Cookie: foo=1; bar=2; bazz=3/, 'multi $http_cookie');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/', 'Foo: foo', 'Foo: bar', 'Foo: bazz'),
 	qr/X-Foo: foo, bar, bazz/, 'multi $http_foo');
 
-}
-
 # request cookies, $cookie_*
 
 my $r = get('/', 'Cookie: foo=1', 'Cookie: bar=2', 'Cookie: bazz=3');
@@ -167,27 +161,16 @@ like($r, qr/X-Cookie-Bazz: 3/, '$cookie_
 
 like($r, qr/X-Sent-CC: foo, bar, bazz/, 'multi $sent_http_cache_control');
 like($r, qr/X-Sent-Link: foo, bar, bazz/, 'multi $sent_http_link');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like($r, qr/X-Sent-Foo: foo, bar, bazz/, 'multi $sent_http_foo');
 
-}
-
 # upstream response headers, $upstream_http_*
 
 $r = get('/u');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like($r, qr/X-Upstream-Set-Cookie: foo=1, bar=2, bazz=3/,
 	'multi $upstream_http_set_cookie');
 like($r, qr/X-Upstream-Bar: foo, bar, bazz/, 'multi $upstream_http_bar');
 
-}
-
 # upstream response cookies, $upstream_cookie_*
 
 like($r, qr/X-Upstream-Cookie-Foo: 1/, '$upstream_cookie_foo');
@@ -196,14 +179,9 @@ like($r, qr/X-Upstream-Cookie-Bazz: 3/, 
 
 # response trailers, $sent_trailer_*
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/t'), qr/X-Sent-Trailer-Foo: foo, bar, bazz/,
 	'multi $sent_trailer_foo');
 
-}
-
 # various variables for request headers:
 #
 # $http_host, $http_user_agent, $http_referer
@@ -216,19 +194,12 @@ like(get('/t'), qr/X-Sent-Trailer-Foo: f
 like(get('/v'), qr/X-HTTP-Host: localhost/, '$http_host');
 like(get('/v', 'Host: foo', 'Host: bar'),
 	qr/400 Bad/, 'duplicate host rejected');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/v', 'User-Agent: foo', 'User-Agent: bar'),
 	qr/X-User-Agent: foo, bar/, 'multi $http_user_agent (invalid)');
 like(get('/v', 'Referer: foo', 'Referer: bar'),
 	qr/X-Referer: foo, bar/, 'multi $http_referer (invalid)');
 like(get('/v', 'Via: foo', 'Via: bar', 'Via: bazz'),
 	qr/X-Via: foo, bar, bazz/, 'multi $http_via');
-
-}
-
 like(get('/v', 'Cookie: foo', 'Cookie: bar', 'Cookie: bazz'),
 	qr/X-Cookie: foo; bar; bazz/, 'multi $http_cookie');
 like(get('/v', 'X-Forwarded-For: foo', 'X-Forwarded-For: bar',
@@ -246,15 +217,9 @@ like(get('/v', 'Content-Length: 0', 'Con
 
 like(get('/v', 'Content-Type: foo'),
 	qr/X-Content-Type: foo/, '$content_type');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/v', 'Content-Type: foo', 'Content-Type: bar'),
 	qr/X-Content-Type: foo, bar/, 'multi $content_type (invalid)');
 
-}
-
 like(http("GET /v HTTP/1.0" . CRLF . CRLF),
 	qr/X-Host: localhost/, '$host from server_name');
 like(http("GET /v HTTP/1.0" . CRLF . "Host: foo" . CRLF . CRLF),
diff --git a/http_request.t b/http_request.t
--- a/http_request.t
+++ b/http_request.t
@@ -91,10 +91,6 @@ like(http(CRLF . "GET / HTTP/1.0" . CRLF
 	'empty line ignored');
 like(http(LF . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 200 /,
 	'empty line with just LF ignored');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(http(CR . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
 	'empty line with just CR rejected');
 like(http(CRLF . CRLF . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
@@ -104,8 +100,6 @@ like(http(LF . LF . "GET / HTTP/1.0" . C
 like(http(CR . CR . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
 	'multiple CRs rejected');
 
-}
-
 # method
 
 like(http("FOO / HTTP/1.0" . CRLF . CRLF), qr/ 200 /, 'method');
diff --git a/image_filter_finalize.t b/image_filter_finalize.t
--- a/image_filter_finalize.t
+++ b/image_filter_finalize.t
@@ -142,10 +142,4 @@ http_get('/slow');
 http_get('/t3');
 like(http_get('/time.log'), qr!/t3:.*, [1-9]\.!, 'upstream response time');
 
-# "aio_write" is used to produce the following alert on some platforms:
-# "readv() failed (9: Bad file descriptor) while reading upstream"
-
-$t->todo_alerts() if $t->read_file('nginx.conf') =~ /aio_write on/
-	and $t->read_file('nginx.conf') =~ /aio threads/;
-
 ###############################################################################
diff --git a/mail_max_commands.t b/mail_max_commands.t
--- a/mail_max_commands.t
+++ b/mail_max_commands.t
@@ -60,7 +60,7 @@ mail {
 
 EOF
 
-$t->try_run('no max_commands')->plan(18);
+$t->run()->plan(18);
 
 ###############################################################################
 
diff --git a/mail_smtp.t b/mail_smtp.t
--- a/mail_smtp.t
+++ b/mail_smtp.t
@@ -297,15 +297,9 @@ my $s = Test::Nginx::SMTP->new();
 	. 'RSET');
 
 $s->read();
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 $s->ok('pipelined long rcpt to');
 $s->ok('pipelined long rset');
 
-}
-
 # Connection must stay even if error returned to rcpt to command
 
 $s = Test::Nginx::SMTP->new();
diff --git a/perl.t b/perl.t
--- a/perl.t
+++ b/perl.t
@@ -153,9 +153,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xfoo: foo/, 'perl header_in unknown');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'X-Foo: foo' . CRLF
@@ -163,8 +160,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xfoo: foo, bar/, 'perl header_in unknown2');
 
-}
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'Cookie: foo' . CRLF
@@ -191,9 +186,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xff: foo1, foo2/, 'perl header_in xff2');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'Connection: close' . CRLF
@@ -207,8 +199,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/connection: close, foo/, 'perl header_in connection2');
 
-}
-
 # headers_out content-length tests with range filter
 
 like(http_get('/range'), qr/Content-Length: 42.*^x{42}$/ms,
diff --git a/proxy_available.t b/proxy_available.t
--- a/proxy_available.t
+++ b/proxy_available.t
@@ -76,13 +76,8 @@ IO::Select->new($s)->can_read(3);
 
 $t->reload();
 
-TODO: {
-local $TODO = 'not yet' if $^O eq 'linux' and !$t->has_version('1.23.1');
-
 like(http_end($s), qr/AND-THIS/, 'zero available - buffered');
 
-}
-
 $s = http_get('/unbuffered', start => 1);
 IO::Select->new($s)->can_read(3);
 
@@ -90,8 +85,6 @@ IO::Select->new($s)->can_read(3);
 
 like(http_end($s), qr/AND-THIS/, 'zero available - unbuffered');
 
-$t->todo_alerts() if $^O eq 'linux' and !$t->has_version('1.23.1');
-
 ###############################################################################
 
 sub http_daemon {
diff --git a/proxy_cache_control.t b/proxy_cache_control.t
--- a/proxy_cache_control.t
+++ b/proxy_cache_control.t
@@ -194,15 +194,10 @@ like(get('/cache-control'), qr/HIT/, 'ca
 like(get('/x-accel-expires'), qr/HIT/, 'x-accel-expires');
 like(get('/x-accel-expires-at'), qr/EXPIRED/, 'x-accel-expires at');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 # the second header to disable cache is duplicate and ignored
 
 like(get('/x-accel-expires-duplicate'), qr/HIT/, 'x-accel-expires duplicate');
 
-}
-
 # with cache headers ignored, the response will be fresh
 
 like(get('/ignore'), qr/MISS/, 'cache headers ignored');
@@ -211,15 +206,8 @@ like(get('/ignore'), qr/MISS/, 'cache he
 
 like(get('/cache-control-before-expires'), qr/HIT/,
 	'cache-control before expires');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/cache-control-after-expires'), qr/HIT/,
 	'cache-control after expires');
-
-}
-
 like(get('/cache-control-no-cache-before-expires'), qr/MISS/,
 	'cache-control no-cache before expires');
 like(get('/cache-control-no-cache-after-expires'), qr/MISS/,
@@ -228,14 +216,7 @@ like(get('/cache-control-no-cache-after-
 # X-Accel-Expires is preferred over both Cache-Control and Expires
 
 like(get('/x-accel-expires-before'), qr/HIT/, 'x-accel-expires before');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/x-accel-expires-after'), qr/HIT/, 'x-accel-expires after');
-
-}
-
 like(get('/x-accel-expires-0-before'), qr/MISS/, 'x-accel-expires 0 before');
 like(get('/x-accel-expires-0-after'), qr/MISS/, 'x-accel-expires 0 after');
 
@@ -250,15 +231,9 @@ like(get('/cache-control-no-cache-multi'
 
 like(get('/extension-before-x-accel-expires'),
 	qr/STALE/, 'cache-control extensions before x-accel-expires');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/extension-after-x-accel-expires'),
 	qr/STALE/, 'cache-control extensions after x-accel-expires');
 
-}
-
 # Set-Cookie is considered when caching with Cache-Control
 
 like(get('/set-cookie'), qr/MISS/, 'set-cookie not cached');
diff --git a/proxy_cache_use_stale.t b/proxy_cache_use_stale.t
--- a/proxy_cache_use_stale.t
+++ b/proxy_cache_use_stale.t
@@ -247,11 +247,6 @@ like($r, qr/STALE.*^(SEE-THIS){1024}$/ms
 $r = read_all(http_get('/ssi.html', start => 1));
 like($r, qr/^xxx (SEE-THIS){1024} xxx$/ms, 's-w-r - not blocked in subrequest');
 
-# "aio_write" is used to produce "open socket ... left in connection" alerts.
-
-$t->todo_alerts() if $t->read_file('nginx.conf') =~ /aio_write on/
-        and $t->read_file('nginx.conf') =~ /aio threads/ and $^O eq 'freebsd';
-
 # due to the missing content_handler inheritance in a cloned subrequest,
 # this used to access a static file in the update request
 
diff --git a/proxy_cache_vary.t b/proxy_cache_vary.t
--- a/proxy_cache_vary.t
+++ b/proxy_cache_vary.t
@@ -266,14 +266,8 @@ like(get('/', 'bar,foo'), qr/HIT/ms, 'no
 
 like(get('/multi', 'foo'), qr/MISS/ms, 'multi first');
 like(get('/multi', 'foo'), qr/HIT/ms, 'multi second');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/multi', 'bar'), qr/MISS/ms, 'multi other');
 
-}
-
 # keep c->body_start when Vary changes (ticket #2029)
 
 # before 1.19.3, this prevented updating c->body_start of a main key
diff --git a/proxy_duplicate_headers.t b/proxy_duplicate_headers.t
--- a/proxy_duplicate_headers.t
+++ b/proxy_duplicate_headers.t
@@ -57,9 +57,6 @@ EOF
 
 like(http_get('/'), qr/200 OK/, 'normal');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/invalid-length'), qr/502 Bad/, 'invalid length');
 like(http_get('/duplicate-length'), qr/502 Bad/, 'duplicate length');
 like(http_get('/unknown-transfer-encoding'), qr/502 Bad/,
@@ -74,8 +71,6 @@ like(http_get('/transfer-encoding-and-le
 like(http_get('/duplicate-expires'), qr/Expires: foo(?!.*bar)/s,
 	'duplicate expires ignored');
 
-}
-
 ###############################################################################
 
 sub http_daemon {
diff --git a/proxy_intercept_errors.t b/proxy_intercept_errors.t
--- a/proxy_intercept_errors.t
+++ b/proxy_intercept_errors.t
@@ -94,12 +94,7 @@ like(http_get('/auth'), qr/401.*WWW-Auth
 # make sure multiple WWW-Authenticate headers are returned
 # along with intercepted response (ticket #485)
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/auth-multi'), qr/401.*WWW-Authenticate: foo.*bar.*intercept/s,
 	'intercepted 401 multi');
 
-}
-
 ###############################################################################
diff --git a/proxy_protocol2_tlv.t b/proxy_protocol2_tlv.t
--- a/proxy_protocol2_tlv.t
+++ b/proxy_protocol2_tlv.t
@@ -23,7 +23,7 @@ use Test::Nginx;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/http map/)
+my $t = Test::Nginx->new()->has(qw/http map/)->plan(14)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -80,7 +80,7 @@ http {
 EOF
 
 $t->write_file('t1', 'SEE-THIS');
-$t->try_run('no proxy_protocol tlv')->plan(14);
+$t->run();
 
 ###############################################################################
 
diff --git a/quic_retry.t b/quic_retry.t
--- a/quic_retry.t
+++ b/quic_retry.t
@@ -110,9 +110,6 @@ is($frame->{error}, 11, 'retry token inv
 
 # connection with retry token, corrupted
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.2');
-
 substr($retry_token, 32) ^= "\xff";
 $s = Test::Nginx::HTTP3->new(8980, token => $retry_token, probe => 1);
 $frames = $s->read(all => [{ type => 'CONNECTION_CLOSE' }]);
@@ -120,16 +117,11 @@ substr($retry_token, 32) ^= "\xff";
 ($frame) = grep { $_->{type} eq "CONNECTION_CLOSE" } @$frames;
 is($frame->{error}, 11, 'retry token decrypt error');
 
-}
-
 # resending client Initial packets after receiving a Retry packet
 # to simulate server Initial packet loss triggering its retransmit,
 # used to create extra nginx connections before 8f7e6d8c061e,
 # caught by CRYPTO stream mismatch among server Initial packets
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.3');
-
 $s = new_connection_resend();
 $sid = $s->new_stream();
 
@@ -141,8 +133,6 @@ eval {
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 is($frame->{headers}->{':status'}, 403, 'resend initial');
 
-}
-
 ###############################################################################
 
 # expanded handshake version to send repetitive Initial packets
diff --git a/range_clearing.t b/range_clearing.t
--- a/range_clearing.t
+++ b/range_clearing.t
@@ -62,8 +62,6 @@ EOF
 
 ###############################################################################
 
-local $TODO = 'not yet' unless $t->has_version('1.23.1');
-
 like(http_get_range('/', 'Range: bytes=0-4'),
 	qr/ 206 (?!.*stub)/s, 'content range cleared - range request');
 like(http_get_range('/', 'Range: bytes=0-2,4-'),
diff --git a/scgi.t b/scgi.t
--- a/scgi.t
+++ b/scgi.t
@@ -81,9 +81,6 @@ like(http_get('/var?b=127.0.0.1:' . port
 	'scgi with variables');
 like(http_get('/var?b=u'), qr/SEE-THIS/, 'scgi with variables to upstream');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r = http(<<EOF);
 GET / HTTP/1.0
 Host: localhost
@@ -106,8 +103,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'scgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {
diff --git a/ssl_certificate.t b/ssl_certificate.t
--- a/ssl_certificate.t
+++ b/ssl_certificate.t
@@ -179,7 +179,6 @@ like(get('default', 8080, $s), qr/defaul
 
 TODO: {
 # ticket key name mismatch prevents session resumption
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
 local $TODO = 'no SSL_session_key, old IO::Socket::SSL'
 	if $IO::Socket::SSL::VERSION < 1.965;
 
diff --git a/ssl_session_ticket_key.t b/ssl_session_ticket_key.t
--- a/ssl_session_ticket_key.t
+++ b/ssl_session_ticket_key.t
@@ -90,8 +90,6 @@ foreach my $name ('localhost') {
 #
 # with a single worker process it is only the 2nd test that fails
 
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
-
 my $key = get_ticket_key_name();
 
 select undef, undef, undef, 0.5;
diff --git a/stream_proxy_protocol2_tlv.t b/stream_proxy_protocol2_tlv.t
--- a/stream_proxy_protocol2_tlv.t
+++ b/stream_proxy_protocol2_tlv.t
@@ -24,7 +24,7 @@ use Test::Nginx::Stream qw/ stream /;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/stream stream_return map/)
+my $t = Test::Nginx->new()->has(qw/stream stream_return map/)->plan(14)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -63,7 +63,7 @@ stream {
 
 EOF
 
-$t->try_run('no proxy_protocol tlv')->plan(14);
+$t->run();
 
 ###############################################################################
 
@@ -86,9 +86,6 @@ like($r, qr/x:\x0d?$/m, 'non-existent');
 
 # big proxy protocol header with TLVs
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.3');
-
 my $sub = pp2_create_tlv(0x21, "TLSv1.2");
 $sub .= pp2_create_tlv(0x22, "example.com");
 $sub .= pp2_create_tlv(0x23, "AES256-SHA");
@@ -107,8 +104,6 @@ like($r, qr/ssl-sig-alg:SHA1\x0d?$/m, 'S
 like($r, qr/ssl-key-alg:RSA512\x0d?$/m, 'SSL_KEY_ALG');
 like($r, qr/ssl-binary:true/, 'SSL_BINARY');
 
-}
-
 ###############################################################################
 
 sub pp_get {
diff --git a/stream_ssl_certificate.t b/stream_ssl_certificate.t
--- a/stream_ssl_certificate.t
+++ b/stream_ssl_certificate.t
@@ -156,7 +156,6 @@ like(get('default', 8080, $s), qr/defaul
 
 TODO: {
 # ticket key name mismatch prevents session resumption
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
 local $TODO = 'no SSL_session_key, old IO::Socket::SSL'
 	if $IO::Socket::SSL::VERSION < 1.965;
 
diff --git a/uwsgi.t b/uwsgi.t
--- a/uwsgi.t
+++ b/uwsgi.t
@@ -102,9 +102,6 @@ like(http_get('/var?b=127.0.0.1:' . port
 	'uwsgi with variables');
 like(http_get('/var?b=u'), qr/SEE-THIS/, 'uwsgi with variables to upstream');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r = http(<<EOF);
 GET / HTTP/1.0
 Host: localhost
@@ -127,8 +124,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'uwsgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {



From mdounin at mdounin.ru  Thu Jun  6 17:07:32 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Thu, 06 Jun 2024 20:07:32 +0300
Subject: [nginx-tests] Tests: reworked HTTP/2 tests to use "http2 on".
Message-ID: <hg.11463d379570.1717693652.-4628363803976424338@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/11463d379570
branches:  
changeset: 1986:11463d379570
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 05:08:02 2024 +0300
description:
Tests: reworked HTTP/2 tests to use "http2 on".

diffstat:

 grpc.t                                |   7 ++-----
 grpc_next_upstream.t                  |  12 ++++++------
 grpc_pass.t                           |  10 ++++------
 grpc_request_buffering.t              |   8 +++-----
 grpc_ssl.t                            |  10 ++++------
 h2.t                                  |  18 ++++++++----------
 h2_absolute_redirect.t                |   9 +++------
 h2_auth_request.t                     |   8 +++-----
 h2_error_page.t                       |   7 ++-----
 h2_fastcgi_request_buffering.t        |   8 +++-----
 h2_headers.t                          |  21 +++++++++++----------
 h2_keepalive.t                        |  13 ++++++-------
 h2_limit_conn.t                       |   9 +++------
 h2_limit_req.t                        |   9 +++------
 h2_priority.t                         |   7 ++-----
 h2_proxy_cache.t                      |   9 +++------
 h2_proxy_max_temp_file_size.t         |   9 +++------
 h2_proxy_protocol.t                   |   9 +++------
 h2_proxy_request_buffering.t          |   8 +++-----
 h2_proxy_request_buffering_redirect.t |   7 ++-----
 h2_proxy_request_buffering_ssl.t      |   8 +++-----
 h2_proxy_ssl.t                        |   9 +++------
 h2_request_body.t                     |   8 ++------
 h2_request_body_extra.t               |   8 ++------
 h2_request_body_preread.t             |  15 +++++++--------
 h2_server_tokens.t                    |   8 +++-----
 h2_ssl_verify_client.t                |   7 +++----
 h2_trailers.t                         |   9 +++------
 h2_variables.t                        |   8 +++-----
 h3_server_name.t                      |   8 +++-----
 proxy_ssl_conf_command.t              |   9 +++------
 worker_shutdown_timeout_h2.t          |  10 +++-------
 32 files changed, 115 insertions(+), 190 deletions(-)

diffs (1088 lines):

diff --git a/grpc.t b/grpc.t
--- a/grpc.t
+++ b/grpc.t
@@ -44,9 +44,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 128k;
         large_client_header_buffers 4 32k;
 
@@ -90,11 +91,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_next_upstream.t b/grpc_next_upstream.t
--- a/grpc_next_upstream.t
+++ b/grpc_next_upstream.t
@@ -69,9 +69,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
+        listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 404;
         }
@@ -91,9 +93,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 200 "TEST-OK-IF-YOU-SEE-THIS\n";
         }
@@ -106,11 +110,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_pass.t b/grpc_pass.t
--- a/grpc_pass.t
+++ b/grpc_pass.t
@@ -63,10 +63,12 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
-        listen       127.0.0.1:8082 http2 ssl;
+        listen       127.0.0.1:8081;
+        listen       127.0.0.1:8082 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate_key localhost.key;
         ssl_certificate localhost.crt;
 
@@ -98,11 +100,7 @@ foreach my $name ('localhost') {
 
 $t->run_daemon(\&dns_daemon, port(8982), $t);
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run()->plan(5);
-open STDERR, ">&", \*OLDERR;
 
 $t->waitforfile($t->testdir . '/' . port(8982));
 
diff --git a/grpc_request_buffering.t b/grpc_request_buffering.t
--- a/grpc_request_buffering.t
+++ b/grpc_request_buffering.t
@@ -38,10 +38,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location /mirror { }
 
         location / {
@@ -64,11 +66,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/grpc_ssl.t b/grpc_ssl.t
--- a/grpc_ssl.t
+++ b/grpc_ssl.t
@@ -44,7 +44,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8081 http2 ssl;
+        listen       127.0.0.1:8081 ssl;
         server_name  localhost;
 
         ssl_certificate_key localhost.key;
@@ -53,6 +53,7 @@ http {
         ssl_verify_client optional;
         ssl_client_certificate client.crt;
 
+        http2 on;
         http2_body_preread_size 128k;
 
         location / {
@@ -62,9 +63,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 128k;
 
         location / {
@@ -129,11 +131,7 @@ sleep 1 if $^O eq 'MSWin32';
 
 $t->write_file('password', 'client');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2.t b/h2.t
--- a/h2.t
+++ b/h2.t
@@ -40,8 +40,10 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
+
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
@@ -88,26 +90,26 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
         return 200   first;
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost2;
         return 200   second;
     }
 
     server {
-        listen       127.0.0.1:8083 http2;
+        listen       127.0.0.1:8083;
         server_name  localhost;
 
         http2_max_concurrent_streams 1;
     }
 
     server {
-        listen       127.0.0.1:8086 http2;
+        listen       127.0.0.1:8086;
         server_name  localhost;
 
         send_timeout 1s;
@@ -115,7 +117,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8087 http2;
+        listen       127.0.0.1:8087;
         server_name  localhost;
 
         client_header_timeout 1s;
@@ -132,11 +134,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 # file size is slightly beyond initial window size: 2**16 + 80 bytes
 
diff --git a/h2_absolute_redirect.t b/h2_absolute_redirect.t
--- a/h2_absolute_redirect.t
+++ b/h2_absolute_redirect.t
@@ -36,10 +36,11 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
     absolute_redirect off;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  on;
 
         absolute_redirect on;
@@ -75,7 +76,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  off;
 
         location / { }
@@ -107,11 +108,7 @@ EOF
 mkdir($t->testdir() . '/dir');
 mkdir($t->testdir() . '/dir sp');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run()->plan(23);
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_auth_request.t b/h2_auth_request.t
--- a/h2_auth_request.t
+++ b/h2_auth_request.t
@@ -39,10 +39,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             return 200;
         }
@@ -66,11 +68,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_error_page.t b/h2_error_page.t
--- a/h2_error_page.t
+++ b/h2_error_page.t
@@ -37,9 +37,10 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
         lingering_close off;
 
         error_page 400 = /close;
@@ -54,11 +55,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_fastcgi_request_buffering.t b/h2_fastcgi_request_buffering.t
--- a/h2_fastcgi_request_buffering.t
+++ b/h2_fastcgi_request_buffering.t
@@ -38,9 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             fastcgi_request_buffering off;
             fastcgi_pass 127.0.0.1:8081;
@@ -52,11 +54,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_headers.t b/h2_headers.t
--- a/h2_headers.t
+++ b/h2_headers.t
@@ -37,11 +37,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
-        listen       127.0.0.1:8082 http2 sndbuf=128;
+        listen       127.0.0.1:8082 sndbuf=128;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 2 64k;
 
         location / {
@@ -89,31 +90,35 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8084 http2;
+        listen       127.0.0.1:8084;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 4 512;
     }
 
     server {
-        listen       127.0.0.1:8085 http2;
+        listen       127.0.0.1:8085;
         server_name  localhost;
 
+        http2 on;
         large_client_header_buffers 1 512;
     }
 
     server {
-        listen       127.0.0.1:8086 http2;
+        listen       127.0.0.1:8086;
         server_name  localhost;
 
+        http2 on;
         underscores_in_headers on;
         add_header X-Sent-Foo $http_x_foo always;
     }
 
     server {
-        listen       127.0.0.1:8087 http2;
+        listen       127.0.0.1:8087;
         server_name  localhost;
 
+        http2 on;
         ignore_invalid_headers off;
         add_header X-Sent-Foo $http_x_foo always;
     }
@@ -123,11 +128,7 @@ EOF
 
 $t->run_daemon(\&http_daemon);
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 $t->waitforsocket('127.0.0.1:' . port(8083));
 
diff --git a/h2_keepalive.t b/h2_keepalive.t
--- a/h2_keepalive.t
+++ b/h2_keepalive.t
@@ -39,27 +39,30 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2 sndbuf=1m;
+        listen       127.0.0.1:8080 sndbuf=1m;
         server_name  localhost;
 
+        http2 on;
         keepalive_requests 2;
 
         location / { }
     }
 
     server {
-        listen       127.0.0.1:8081 http2;
+        listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         keepalive_timeout 0;
 
         location / { }
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
         keepalive_time 1s;
 
         add_header X-Conn $connection_requests:$connection_time;
@@ -73,11 +76,7 @@ EOF
 $t->write_file('index.html', 'SEE-THAT' x 50000);
 $t->write_file('t.html', 'SEE-THAT');
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_limit_conn.t b/h2_limit_conn.t
--- a/h2_limit_conn.t
+++ b/h2_limit_conn.t
@@ -39,9 +39,11 @@ http {
     limit_conn_zone  $binary_remote_addr  zone=conn:1m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /t.html {
             limit_conn conn 1;
         }
@@ -51,12 +53,7 @@ http {
 EOF
 
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_limit_req.t b/h2_limit_req.t
--- a/h2_limit_req.t
+++ b/h2_limit_req.t
@@ -41,10 +41,12 @@ http {
     limit_req_zone   $binary_remote_addr  zone=req:1m rate=1r/s;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location / { }
         location /limit_req {
             limit_req  zone=req burst=2;
@@ -64,12 +66,7 @@ EOF
 
 $t->write_file('index.html', '');
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_priority.t b/h2_priority.t
--- a/h2_priority.t
+++ b/h2_priority.t
@@ -37,18 +37,15 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
+        http2 on;
     }
 }
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 # file size is slightly beyond initial window size: 2**16 + 80 bytes
 
diff --git a/h2_proxy_cache.t b/h2_proxy_cache.t
--- a/h2_proxy_cache.t
+++ b/h2_proxy_cache.t
@@ -39,10 +39,12 @@ http {
     proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
+
         location /cache {
             proxy_pass http://127.0.0.1:8081/;
             proxy_cache NAME;
@@ -68,12 +70,7 @@ EOF
 
 $t->write_file('t.html', 'SEE-THIS');
 $t->write_file('slow.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_max_temp_file_size.t b/h2_proxy_max_temp_file_size.t
--- a/h2_proxy_max_temp_file_size.t
+++ b/h2_proxy_max_temp_file_size.t
@@ -38,9 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         proxy_buffer_size 4k;
         proxy_buffers 8 4k;
 
@@ -66,12 +68,7 @@ http {
 EOF
 
 $t->write_file('1', 'X' x (1024 * 1024));
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_protocol.t b/h2_proxy_protocol.t
--- a/h2_proxy_protocol.t
+++ b/h2_proxy_protocol.t
@@ -39,9 +39,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 proxy_protocol http2;
+        listen       127.0.0.1:8080 proxy_protocol;
         server_name  localhost;
 
+        http2 on;
+
         location /pp {
             set_real_ip_from 127.0.0.1/32;
             real_ip_header proxy_protocol;
@@ -54,12 +56,7 @@ http {
 EOF
 
 $t->write_file('t.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering.t b/h2_proxy_request_buffering.t
--- a/h2_proxy_request_buffering.t
+++ b/h2_proxy_request_buffering.t
@@ -40,10 +40,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_request_buffering off;
             proxy_pass http://127.0.0.1:8081/;
@@ -65,11 +67,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering_redirect.t b/h2_proxy_request_buffering_redirect.t
--- a/h2_proxy_request_buffering_redirect.t
+++ b/h2_proxy_request_buffering_redirect.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         proxy_http_version 1.1;
 
         location / {
@@ -68,11 +69,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_request_buffering_ssl.t b/h2_proxy_request_buffering_ssl.t
--- a/h2_proxy_request_buffering_ssl.t
+++ b/h2_proxy_request_buffering_ssl.t
@@ -41,9 +41,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_request_buffering off;
             proxy_pass https://127.0.0.1:8082;
@@ -98,11 +100,7 @@ foreach my $name ('localhost') {
 		or die "Can't create certificate for $name: $!\n";
 }
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_proxy_ssl.t b/h2_proxy_ssl.t
--- a/h2_proxy_ssl.t
+++ b/h2_proxy_ssl.t
@@ -39,10 +39,12 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate_key localhost.key;
         ssl_certificate localhost.crt;
 
@@ -74,12 +76,7 @@ foreach my $name ('localhost') {
 }
 
 $t->write_file('index.html', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body.t b/h2_request_body.t
--- a/h2_request_body.t
+++ b/h2_request_body.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         error_page 400 /proxy2/t.html;
 
         location / {
@@ -77,12 +78,7 @@ EOF
 $t->write_file('index.html', '');
 $t->write_file('t.html', 'SEE-THIS');
 $t->write_file('slow.html', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body_extra.t b/h2_request_body_extra.t
--- a/h2_request_body_extra.t
+++ b/h2_request_body_extra.t
@@ -38,10 +38,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         client_header_buffer_size 1k;
         client_body_buffer_size 2k;
 
@@ -88,12 +89,7 @@ http {
 EOF
 
 $t->plan(50);
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_request_body_preread.t b/h2_request_body_preread.t
--- a/h2_request_body_preread.t
+++ b/h2_request_body_preread.t
@@ -40,10 +40,11 @@ http {
     limit_req_zone   $binary_remote_addr  zone=req:1m rate=20r/m;
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         listen       127.0.0.1:8081;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 10;
 
         location /t { }
@@ -59,9 +60,10 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8082 http2;
+        listen       127.0.0.1:8082;
         server_name  localhost;
 
+        http2 on;
         http2_body_preread_size 0;
 
         location / {
@@ -76,9 +78,11 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8083 http2;
+        listen       127.0.0.1:8083;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             add_header X-Body $request_body;
             proxy_pass http://127.0.0.1:8081/t;
@@ -89,12 +93,7 @@ http {
 EOF
 
 $t->write_file('t', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_server_tokens.t b/h2_server_tokens.t
--- a/h2_server_tokens.t
+++ b/h2_server_tokens.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /200 {
             return 200;
         }
@@ -88,11 +90,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_ssl_verify_client.t b/h2_ssl_verify_client.t
--- a/h2_ssl_verify_client.t
+++ b/h2_ssl_verify_client.t
@@ -38,6 +38,7 @@ events {
 http {
     %%TEST_GLOBALS_HTTP%%
 
+    http2 on;
     ssl_certificate_key localhost.key;
     ssl_certificate localhost.crt;
 
@@ -46,7 +47,7 @@ http {
     add_header X-Verify $ssl_client_verify;
 
     server {
-        listen       127.0.0.1:8080 ssl http2;
+        listen       127.0.0.1:8080 ssl;
         server_name  localhost;
 
         ssl_client_certificate client.crt;
@@ -55,7 +56,7 @@ http {
     }
 
     server {
-        listen       127.0.0.1:8080 ssl http2;
+        listen       127.0.0.1:8080 ssl;
         server_name  example.com;
 
         location / { }
@@ -84,9 +85,7 @@ foreach my $name ('localhost', 'client')
 
 $t->write_file('t', 'SEE-THIS');
 
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 my $s = get_ssl_socket();
 plan(skip_all => 'no alpn') unless $s->alpn_selected();
diff --git a/h2_trailers.t b/h2_trailers.t
--- a/h2_trailers.t
+++ b/h2_trailers.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             add_trailer X-Var $host;
         }
@@ -60,12 +62,7 @@ EOF
 $t->write_file('index.html', 'SEE-THIS');
 $t->write_file('empty', '');
 $t->write_file('continuation', 'SEE-THIS');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h2_variables.t b/h2_variables.t
--- a/h2_variables.t
+++ b/h2_variables.t
@@ -37,9 +37,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location /h2 {
             return 200 $http2;
         }
@@ -60,11 +62,7 @@ http {
 
 EOF
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/h3_server_name.t b/h3_server_name.t
--- a/h3_server_name.t
+++ b/h3_server_name.t
@@ -44,10 +44,12 @@ http {
     ssl_certificate localhost.crt;
 
     server {
-        listen       127.0.0.1:8443 ssl http2;
+        listen       127.0.0.1:8443 ssl;
         listen       127.0.0.1:%%PORT_8980_UDP%% quic;
         server_name  ~^(?P<name>.+)\.example\.com$;
 
+        http2 on;
+
         location / {
             return 200 $name;
         }
@@ -74,11 +76,7 @@ foreach my $name ('localhost') {
 		or die "Can't create certificate for $name: $!\n";
 }
 
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/proxy_ssl_conf_command.t b/proxy_ssl_conf_command.t
--- a/proxy_ssl_conf_command.t
+++ b/proxy_ssl_conf_command.t
@@ -72,9 +72,11 @@ http {
 
     server {
         listen       127.0.0.1:8081 ssl;
-        listen       127.0.0.1:8082 ssl http2;
+        listen       127.0.0.1:8082 ssl;
         server_name  localhost;
 
+        http2 on;
+
         ssl_certificate localhost.crt;
         ssl_certificate_key localhost.key;
         ssl_verify_client optional_no_ca;
@@ -106,12 +108,7 @@ foreach my $name ('localhost', 'override
 }
 
 $t->write_file('index.html', '');
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
 
 ###############################################################################
 
diff --git a/worker_shutdown_timeout_h2.t b/worker_shutdown_timeout_h2.t
--- a/worker_shutdown_timeout_h2.t
+++ b/worker_shutdown_timeout_h2.t
@@ -39,9 +39,11 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     server {
-        listen       127.0.0.1:8080 http2;
+        listen       127.0.0.1:8080;
         server_name  localhost;
 
+        http2 on;
+
         location / {
             proxy_pass http://127.0.0.1:8081;
             proxy_read_timeout 5s;
@@ -51,13 +53,7 @@ http {
 EOF
 
 $t->run_daemon(\&http_silent_daemon);
-
-# suppress deprecation warning
-
-open OLDERR, ">&", \*STDERR; close STDERR;
 $t->run();
-open STDERR, ">&", \*OLDERR;
-
 $t->waitforsocket('127.0.0.1:' . port(8081));
 
 ###############################################################################


From mdounin at mdounin.ru  Thu Jun  6 17:07:32 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Thu, 06 Jun 2024 20:07:32 +0300
Subject: [nginx-tests] Tests: removed TODO and try_run() checks for legac...
Message-ID: <hg.a095b971fbcc.1717693652.-4628363803976424338@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/a095b971fbcc
branches:  
changeset: 1987:a095b971fbcc
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 18:38:01 2024 +0300
description:
Tests: removed TODO and try_run() checks for legacy versions.

For h2_http2.t, try_run() is preserved to ensure that deprecation
warnings for "listen ... http2" are suppressed, yet plan() is reported
before try_run(), so failure to start will be properly reported.

diffstat:

 auth_request.t               |   5 -----
 autoindex_win32.t            |   5 -----
 body_chunked.t               |  10 ----------
 dav_utf8.t                   |   2 --
 fastcgi_header_params.t      |   5 -----
 h2.t                         |   5 -----
 h2_error_page.t              |   6 ------
 h2_headers.t                 |  14 ++------------
 h2_http2.t                   |   5 +++--
 h2_proxy_protocol.t          |   5 -----
 http_headers_multi.t         |  35 -----------------------------------
 http_request.t               |   6 ------
 image_filter_finalize.t      |   6 ------
 mail_max_commands.t          |   2 +-
 mail_smtp.t                  |   6 ------
 perl.t                       |  10 ----------
 proxy_available.t            |   7 -------
 proxy_cache_control.t        |  25 -------------------------
 proxy_cache_use_stale.t      |   5 -----
 proxy_cache_vary.t           |   6 ------
 proxy_duplicate_headers.t    |   5 -----
 proxy_intercept_errors.t     |   5 -----
 proxy_protocol2_tlv.t        |   4 ++--
 quic_retry.t                 |  10 ----------
 range_clearing.t             |   2 --
 scgi.t                       |   5 -----
 ssl_certificate.t            |   1 -
 ssl_session_ticket_key.t     |   2 --
 stream_proxy_protocol2_tlv.t |   9 ++-------
 stream_ssl_certificate.t     |   1 -
 uwsgi.t                      |   5 -----
 31 files changed, 10 insertions(+), 209 deletions(-)

diffs (793 lines):

diff --git a/auth_request.t b/auth_request.t
--- a/auth_request.t
+++ b/auth_request.t
@@ -203,14 +203,9 @@ like(http_post_big('/proxy-double'), qr/
 
 # Multiple WWW-Authenticate headers (ticket #485).
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/proxy-multi-auth'), qr/WWW-Authenticate: foo.*bar/s,
 	'multiple www-authenticate headers');
 
-}
-
 SKIP: {
 	eval { require FCGI; };
 	skip 'FCGI not installed', 2 if $@;
diff --git a/autoindex_win32.t b/autoindex_win32.t
--- a/autoindex_win32.t
+++ b/autoindex_win32.t
@@ -76,9 +76,6 @@ my $r = http_get('/');
 like($r, qr!href="test-file"!ms, 'file');
 like($r, qr!href="test-dir/"!ms, 'directory');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 like($r, qr!href="test-file-(%d0%bc%d0%b8){3}"!msi, 'utf file link');
 like($r, qr!test-file-(\xd0\xbc\xd0\xb8){3}</a>!ms, 'utf file name');
 
@@ -92,8 +89,6 @@ like($r, qr!Index of /test-dir-(\xd0\xbc
 like($r, qr!href="test-subfile-(%d0%bc%d0%b8){3}"!msi, 'utf subdir link');
 like($r, qr!test-subfile-(\xd0\xbc\xd0\xb8){3}</a>!msi, 'utf subdir name');
 
-}
-
 ###############################################################################
 
 sub win32_mkdir {
diff --git a/body_chunked.t b/body_chunked.t
--- a/body_chunked.t
+++ b/body_chunked.t
@@ -179,9 +179,6 @@ like(
 	qr/ 200 /, 'chunk extensions'
 );
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -194,8 +191,6 @@ like(
 	qr/ 413 /, 'too many chunk extensions'
 );
 
-}
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -208,9 +203,6 @@ like(
 	qr/ 200 /, 'trailers'
 );
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(
 	http(
 		'GET /large HTTP/1.1' . CRLF
@@ -223,8 +215,6 @@ like(
 	qr/ 413 /, 'too many trailers'
 );
 
-}
-
 # proxy_next_upstream
 
 like(http_get_body('/next', '0123456789'),
diff --git a/dav_utf8.t b/dav_utf8.t
--- a/dav_utf8.t
+++ b/dav_utf8.t
@@ -57,8 +57,6 @@ EOF
 
 ###############################################################################
 
-local $TODO = 'not yet' if $^O eq 'MSWin32' and !$t->has_version('1.23.4');
-
 my $d = $t->testdir();
 my $r;
 
diff --git a/fastcgi_header_params.t b/fastcgi_header_params.t
--- a/fastcgi_header_params.t
+++ b/fastcgi_header_params.t
@@ -59,9 +59,6 @@ EOF
 like(http_get_headers('/'), qr/SEE-THIS/,
 	'fastcgi request with many ignored headers');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r;
 
 $r = http(<<EOF);
@@ -88,8 +85,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'fastcgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {
diff --git a/h2.t b/h2.t
--- a/h2.t
+++ b/h2.t
@@ -1014,15 +1014,10 @@ is($frame->{headers}->{':status'}, 200, 
 
 # invalid connection preface
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.1');
-
 like(http('x' x 16), qr/400 Bad Request/, 'invalid preface');
 like(http('PRI * HTTP/2.0' . CRLF . CRLF . 'x' x 8), qr/400 Bad Request/,
 	'invalid preface 2');
 
-}
-
 # GOAWAY on SYN_STREAM with even StreamID
 
 $s = Test::Nginx::HTTP2->new();
diff --git a/h2_error_page.t b/h2_error_page.t
--- a/h2_error_page.t
+++ b/h2_error_page.t
@@ -85,19 +85,13 @@ my $s2 = Test::Nginx::HTTP2->new();
 $sid = $s2->new_stream({ method => 'foo' });
 $frames = $s2->read(all => [{ type => 'RST_STREAM' }]);
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 ($frame) = grep { $_->{type} eq "RST_STREAM" } @$frames;
 is($frame->{sid}, $sid, 'error 400 return 444 - invalid header');
 
-}
-
 # while keeping $s1 and $s2, stop nginx; this should result in
 # "open socket ... left in connection ..." alerts if any of these
 # sockets are still open
 
 $t->stop();
-$t->todo_alerts() unless $t->has_version('1.23.4');
 
 ###############################################################################
diff --git a/h2_headers.t b/h2_headers.t
--- a/h2_headers.t
+++ b/h2_headers.t
@@ -959,13 +959,8 @@ ok($frame, 'HPACK table boundary - heade
 
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 isnt($frame->{headers}->{'x-referer'}, 'see-this', 'newline in request header');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
-is($frame->{headers}->{':status'}, 400, 'newline in request header - bad request');
-
-}
+is($frame->{headers}->{':status'}, 400,
+	'newline in request header - bad request');
 
 # invalid header name as seen with underscore should not lead to ignoring rest
 
@@ -984,9 +979,6 @@ is($frame->{headers}->{'x-referer'}, 'se
 
 # other invalid header name characters as seen with ':'
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.4');
-
 $s = Test::Nginx::HTTP2->new();
 $sid = $s->new_stream({ headers => [
 	{ name => ':method', value => 'GET', mode => 0 },
@@ -1024,8 +1016,6 @@ is($frame->{headers}->{':status'}, 400, 
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 is($frame->{headers}->{':status'}, 400, 'control in header name');
 
-}
-
 # header name with underscore - underscores_in_headers on
 
 $s = Test::Nginx::HTTP2->new(port(8086));
diff --git a/h2_http2.t b/h2_http2.t
--- a/h2_http2.t
+++ b/h2_http2.t
@@ -24,7 +24,8 @@ select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
 my $t = Test::Nginx->new()->has(qw/http http_ssl http_v2 socket_ssl_alpn/)
-	->has_daemon('openssl');
+	->has_daemon('openssl')
+	->plan(11);
 
 $t->write_file_expand('nginx.conf', <<'EOF');
 
@@ -108,7 +109,7 @@ foreach my $name ('localhost') {
 }
 
 $t->write_file('index.html', '');
-$t->try_run('no http2')->plan(11);
+$t->try_run();
 
 ###############################################################################
 
diff --git a/h2_proxy_protocol.t b/h2_proxy_protocol.t
--- a/h2_proxy_protocol.t
+++ b/h2_proxy_protocol.t
@@ -71,12 +71,7 @@ is($frame->{headers}->{'x-pp'}, '192.0.2
 
 # invalid PROXY protocol string
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.1');
-
 $proxy = 'BOGUS TCP4 192.0.2.1 192.0.2.2 1234 5678' . CRLF;
 ok(!http($proxy), 'PROXY invalid protocol');
 
-}
-
 ###############################################################################
diff --git a/http_headers_multi.t b/http_headers_multi.t
--- a/http_headers_multi.t
+++ b/http_headers_multi.t
@@ -144,15 +144,9 @@ like(get('/', map { "X-Forwarded-For: $_
 	qr/X-Forwarded-For: foo, bar, bazz/, 'multi $http_x_forwarded_for');
 like(get('/', 'Cookie: foo=1', 'Cookie: bar=2', 'Cookie: bazz=3'),
 	qr/X-Cookie: foo=1; bar=2; bazz=3/, 'multi $http_cookie');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/', 'Foo: foo', 'Foo: bar', 'Foo: bazz'),
 	qr/X-Foo: foo, bar, bazz/, 'multi $http_foo');
 
-}
-
 # request cookies, $cookie_*
 
 my $r = get('/', 'Cookie: foo=1', 'Cookie: bar=2', 'Cookie: bazz=3');
@@ -167,27 +161,16 @@ like($r, qr/X-Cookie-Bazz: 3/, '$cookie_
 
 like($r, qr/X-Sent-CC: foo, bar, bazz/, 'multi $sent_http_cache_control');
 like($r, qr/X-Sent-Link: foo, bar, bazz/, 'multi $sent_http_link');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like($r, qr/X-Sent-Foo: foo, bar, bazz/, 'multi $sent_http_foo');
 
-}
-
 # upstream response headers, $upstream_http_*
 
 $r = get('/u');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like($r, qr/X-Upstream-Set-Cookie: foo=1, bar=2, bazz=3/,
 	'multi $upstream_http_set_cookie');
 like($r, qr/X-Upstream-Bar: foo, bar, bazz/, 'multi $upstream_http_bar');
 
-}
-
 # upstream response cookies, $upstream_cookie_*
 
 like($r, qr/X-Upstream-Cookie-Foo: 1/, '$upstream_cookie_foo');
@@ -196,14 +179,9 @@ like($r, qr/X-Upstream-Cookie-Bazz: 3/, 
 
 # response trailers, $sent_trailer_*
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/t'), qr/X-Sent-Trailer-Foo: foo, bar, bazz/,
 	'multi $sent_trailer_foo');
 
-}
-
 # various variables for request headers:
 #
 # $http_host, $http_user_agent, $http_referer
@@ -216,19 +194,12 @@ like(get('/t'), qr/X-Sent-Trailer-Foo: f
 like(get('/v'), qr/X-HTTP-Host: localhost/, '$http_host');
 like(get('/v', 'Host: foo', 'Host: bar'),
 	qr/400 Bad/, 'duplicate host rejected');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/v', 'User-Agent: foo', 'User-Agent: bar'),
 	qr/X-User-Agent: foo, bar/, 'multi $http_user_agent (invalid)');
 like(get('/v', 'Referer: foo', 'Referer: bar'),
 	qr/X-Referer: foo, bar/, 'multi $http_referer (invalid)');
 like(get('/v', 'Via: foo', 'Via: bar', 'Via: bazz'),
 	qr/X-Via: foo, bar, bazz/, 'multi $http_via');
-
-}
-
 like(get('/v', 'Cookie: foo', 'Cookie: bar', 'Cookie: bazz'),
 	qr/X-Cookie: foo; bar; bazz/, 'multi $http_cookie');
 like(get('/v', 'X-Forwarded-For: foo', 'X-Forwarded-For: bar',
@@ -246,15 +217,9 @@ like(get('/v', 'Content-Length: 0', 'Con
 
 like(get('/v', 'Content-Type: foo'),
 	qr/X-Content-Type: foo/, '$content_type');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/v', 'Content-Type: foo', 'Content-Type: bar'),
 	qr/X-Content-Type: foo, bar/, 'multi $content_type (invalid)');
 
-}
-
 like(http("GET /v HTTP/1.0" . CRLF . CRLF),
 	qr/X-Host: localhost/, '$host from server_name');
 like(http("GET /v HTTP/1.0" . CRLF . "Host: foo" . CRLF . CRLF),
diff --git a/http_request.t b/http_request.t
--- a/http_request.t
+++ b/http_request.t
@@ -91,10 +91,6 @@ like(http(CRLF . "GET / HTTP/1.0" . CRLF
 	'empty line ignored');
 like(http(LF . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 200 /,
 	'empty line with just LF ignored');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 like(http(CR . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
 	'empty line with just CR rejected');
 like(http(CRLF . CRLF . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
@@ -104,8 +100,6 @@ like(http(LF . LF . "GET / HTTP/1.0" . C
 like(http(CR . CR . "GET / HTTP/1.0" . CRLF . CRLF), qr/ 400 /,
 	'multiple CRs rejected');
 
-}
-
 # method
 
 like(http("FOO / HTTP/1.0" . CRLF . CRLF), qr/ 200 /, 'method');
diff --git a/image_filter_finalize.t b/image_filter_finalize.t
--- a/image_filter_finalize.t
+++ b/image_filter_finalize.t
@@ -142,10 +142,4 @@ http_get('/slow');
 http_get('/t3');
 like(http_get('/time.log'), qr!/t3:.*, [1-9]\.!, 'upstream response time');
 
-# "aio_write" is used to produce the following alert on some platforms:
-# "readv() failed (9: Bad file descriptor) while reading upstream"
-
-$t->todo_alerts() if $t->read_file('nginx.conf') =~ /aio_write on/
-	and $t->read_file('nginx.conf') =~ /aio threads/;
-
 ###############################################################################
diff --git a/mail_max_commands.t b/mail_max_commands.t
--- a/mail_max_commands.t
+++ b/mail_max_commands.t
@@ -60,7 +60,7 @@ mail {
 
 EOF
 
-$t->try_run('no max_commands')->plan(18);
+$t->run()->plan(18);
 
 ###############################################################################
 
diff --git a/mail_smtp.t b/mail_smtp.t
--- a/mail_smtp.t
+++ b/mail_smtp.t
@@ -297,15 +297,9 @@ my $s = Test::Nginx::SMTP->new();
 	. 'RSET');
 
 $s->read();
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.5');
-
 $s->ok('pipelined long rcpt to');
 $s->ok('pipelined long rset');
 
-}
-
 # Connection must stay even if error returned to rcpt to command
 
 $s = Test::Nginx::SMTP->new();
diff --git a/perl.t b/perl.t
--- a/perl.t
+++ b/perl.t
@@ -153,9 +153,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xfoo: foo/, 'perl header_in unknown');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'X-Foo: foo' . CRLF
@@ -163,8 +160,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xfoo: foo, bar/, 'perl header_in unknown2');
 
-}
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'Cookie: foo' . CRLF
@@ -191,9 +186,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/xff: foo1, foo2/, 'perl header_in xff2');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http(
 	'GET / HTTP/1.0' . CRLF
 	. 'Connection: close' . CRLF
@@ -207,8 +199,6 @@ like(http(
 	. 'Host: localhost' . CRLF . CRLF
 ), qr/connection: close, foo/, 'perl header_in connection2');
 
-}
-
 # headers_out content-length tests with range filter
 
 like(http_get('/range'), qr/Content-Length: 42.*^x{42}$/ms,
diff --git a/proxy_available.t b/proxy_available.t
--- a/proxy_available.t
+++ b/proxy_available.t
@@ -76,13 +76,8 @@ IO::Select->new($s)->can_read(3);
 
 $t->reload();
 
-TODO: {
-local $TODO = 'not yet' if $^O eq 'linux' and !$t->has_version('1.23.1');
-
 like(http_end($s), qr/AND-THIS/, 'zero available - buffered');
 
-}
-
 $s = http_get('/unbuffered', start => 1);
 IO::Select->new($s)->can_read(3);
 
@@ -90,8 +85,6 @@ IO::Select->new($s)->can_read(3);
 
 like(http_end($s), qr/AND-THIS/, 'zero available - unbuffered');
 
-$t->todo_alerts() if $^O eq 'linux' and !$t->has_version('1.23.1');
-
 ###############################################################################
 
 sub http_daemon {
diff --git a/proxy_cache_control.t b/proxy_cache_control.t
--- a/proxy_cache_control.t
+++ b/proxy_cache_control.t
@@ -194,15 +194,10 @@ like(get('/cache-control'), qr/HIT/, 'ca
 like(get('/x-accel-expires'), qr/HIT/, 'x-accel-expires');
 like(get('/x-accel-expires-at'), qr/EXPIRED/, 'x-accel-expires at');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 # the second header to disable cache is duplicate and ignored
 
 like(get('/x-accel-expires-duplicate'), qr/HIT/, 'x-accel-expires duplicate');
 
-}
-
 # with cache headers ignored, the response will be fresh
 
 like(get('/ignore'), qr/MISS/, 'cache headers ignored');
@@ -211,15 +206,8 @@ like(get('/ignore'), qr/MISS/, 'cache he
 
 like(get('/cache-control-before-expires'), qr/HIT/,
 	'cache-control before expires');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/cache-control-after-expires'), qr/HIT/,
 	'cache-control after expires');
-
-}
-
 like(get('/cache-control-no-cache-before-expires'), qr/MISS/,
 	'cache-control no-cache before expires');
 like(get('/cache-control-no-cache-after-expires'), qr/MISS/,
@@ -228,14 +216,7 @@ like(get('/cache-control-no-cache-after-
 # X-Accel-Expires is preferred over both Cache-Control and Expires
 
 like(get('/x-accel-expires-before'), qr/HIT/, 'x-accel-expires before');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/x-accel-expires-after'), qr/HIT/, 'x-accel-expires after');
-
-}
-
 like(get('/x-accel-expires-0-before'), qr/MISS/, 'x-accel-expires 0 before');
 like(get('/x-accel-expires-0-after'), qr/MISS/, 'x-accel-expires 0 after');
 
@@ -250,15 +231,9 @@ like(get('/cache-control-no-cache-multi'
 
 like(get('/extension-before-x-accel-expires'),
 	qr/STALE/, 'cache-control extensions before x-accel-expires');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/extension-after-x-accel-expires'),
 	qr/STALE/, 'cache-control extensions after x-accel-expires');
 
-}
-
 # Set-Cookie is considered when caching with Cache-Control
 
 like(get('/set-cookie'), qr/MISS/, 'set-cookie not cached');
diff --git a/proxy_cache_use_stale.t b/proxy_cache_use_stale.t
--- a/proxy_cache_use_stale.t
+++ b/proxy_cache_use_stale.t
@@ -247,11 +247,6 @@ like($r, qr/STALE.*^(SEE-THIS){1024}$/ms
 $r = read_all(http_get('/ssi.html', start => 1));
 like($r, qr/^xxx (SEE-THIS){1024} xxx$/ms, 's-w-r - not blocked in subrequest');
 
-# "aio_write" is used to produce "open socket ... left in connection" alerts.
-
-$t->todo_alerts() if $t->read_file('nginx.conf') =~ /aio_write on/
-        and $t->read_file('nginx.conf') =~ /aio threads/ and $^O eq 'freebsd';
-
 # due to the missing content_handler inheritance in a cloned subrequest,
 # this used to access a static file in the update request
 
diff --git a/proxy_cache_vary.t b/proxy_cache_vary.t
--- a/proxy_cache_vary.t
+++ b/proxy_cache_vary.t
@@ -266,14 +266,8 @@ like(get('/', 'bar,foo'), qr/HIT/ms, 'no
 
 like(get('/multi', 'foo'), qr/MISS/ms, 'multi first');
 like(get('/multi', 'foo'), qr/HIT/ms, 'multi second');
-
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(get('/multi', 'bar'), qr/MISS/ms, 'multi other');
 
-}
-
 # keep c->body_start when Vary changes (ticket #2029)
 
 # before 1.19.3, this prevented updating c->body_start of a main key
diff --git a/proxy_duplicate_headers.t b/proxy_duplicate_headers.t
--- a/proxy_duplicate_headers.t
+++ b/proxy_duplicate_headers.t
@@ -57,9 +57,6 @@ EOF
 
 like(http_get('/'), qr/200 OK/, 'normal');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/invalid-length'), qr/502 Bad/, 'invalid length');
 like(http_get('/duplicate-length'), qr/502 Bad/, 'duplicate length');
 like(http_get('/unknown-transfer-encoding'), qr/502 Bad/,
@@ -74,8 +71,6 @@ like(http_get('/transfer-encoding-and-le
 like(http_get('/duplicate-expires'), qr/Expires: foo(?!.*bar)/s,
 	'duplicate expires ignored');
 
-}
-
 ###############################################################################
 
 sub http_daemon {
diff --git a/proxy_intercept_errors.t b/proxy_intercept_errors.t
--- a/proxy_intercept_errors.t
+++ b/proxy_intercept_errors.t
@@ -94,12 +94,7 @@ like(http_get('/auth'), qr/401.*WWW-Auth
 # make sure multiple WWW-Authenticate headers are returned
 # along with intercepted response (ticket #485)
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 like(http_get('/auth-multi'), qr/401.*WWW-Authenticate: foo.*bar.*intercept/s,
 	'intercepted 401 multi');
 
-}
-
 ###############################################################################
diff --git a/proxy_protocol2_tlv.t b/proxy_protocol2_tlv.t
--- a/proxy_protocol2_tlv.t
+++ b/proxy_protocol2_tlv.t
@@ -23,7 +23,7 @@ use Test::Nginx;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/http map/)
+my $t = Test::Nginx->new()->has(qw/http map/)->plan(14)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -80,7 +80,7 @@ http {
 EOF
 
 $t->write_file('t1', 'SEE-THIS');
-$t->try_run('no proxy_protocol tlv')->plan(14);
+$t->run();
 
 ###############################################################################
 
diff --git a/quic_retry.t b/quic_retry.t
--- a/quic_retry.t
+++ b/quic_retry.t
@@ -110,9 +110,6 @@ is($frame->{error}, 11, 'retry token inv
 
 # connection with retry token, corrupted
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.2');
-
 substr($retry_token, 32) ^= "\xff";
 $s = Test::Nginx::HTTP3->new(8980, token => $retry_token, probe => 1);
 $frames = $s->read(all => [{ type => 'CONNECTION_CLOSE' }]);
@@ -120,16 +117,11 @@ substr($retry_token, 32) ^= "\xff";
 ($frame) = grep { $_->{type} eq "CONNECTION_CLOSE" } @$frames;
 is($frame->{error}, 11, 'retry token decrypt error');
 
-}
-
 # resending client Initial packets after receiving a Retry packet
 # to simulate server Initial packet loss triggering its retransmit,
 # used to create extra nginx connections before 8f7e6d8c061e,
 # caught by CRYPTO stream mismatch among server Initial packets
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.25.3');
-
 $s = new_connection_resend();
 $sid = $s->new_stream();
 
@@ -141,8 +133,6 @@ eval {
 ($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
 is($frame->{headers}->{':status'}, 403, 'resend initial');
 
-}
-
 ###############################################################################
 
 # expanded handshake version to send repetitive Initial packets
diff --git a/range_clearing.t b/range_clearing.t
--- a/range_clearing.t
+++ b/range_clearing.t
@@ -62,8 +62,6 @@ EOF
 
 ###############################################################################
 
-local $TODO = 'not yet' unless $t->has_version('1.23.1');
-
 like(http_get_range('/', 'Range: bytes=0-4'),
 	qr/ 206 (?!.*stub)/s, 'content range cleared - range request');
 like(http_get_range('/', 'Range: bytes=0-2,4-'),
diff --git a/scgi.t b/scgi.t
--- a/scgi.t
+++ b/scgi.t
@@ -81,9 +81,6 @@ like(http_get('/var?b=127.0.0.1:' . port
 	'scgi with variables');
 like(http_get('/var?b=u'), qr/SEE-THIS/, 'scgi with variables to upstream');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r = http(<<EOF);
 GET / HTTP/1.0
 Host: localhost
@@ -106,8 +103,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'scgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {
diff --git a/ssl_certificate.t b/ssl_certificate.t
--- a/ssl_certificate.t
+++ b/ssl_certificate.t
@@ -179,7 +179,6 @@ like(get('default', 8080, $s), qr/defaul
 
 TODO: {
 # ticket key name mismatch prevents session resumption
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
 local $TODO = 'no SSL_session_key, old IO::Socket::SSL'
 	if $IO::Socket::SSL::VERSION < 1.965;
 
diff --git a/ssl_session_ticket_key.t b/ssl_session_ticket_key.t
--- a/ssl_session_ticket_key.t
+++ b/ssl_session_ticket_key.t
@@ -90,8 +90,6 @@ foreach my $name ('localhost') {
 #
 # with a single worker process it is only the 2nd test that fails
 
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
-
 my $key = get_ticket_key_name();
 
 select undef, undef, undef, 0.5;
diff --git a/stream_proxy_protocol2_tlv.t b/stream_proxy_protocol2_tlv.t
--- a/stream_proxy_protocol2_tlv.t
+++ b/stream_proxy_protocol2_tlv.t
@@ -24,7 +24,7 @@ use Test::Nginx::Stream qw/ stream /;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/stream stream_return map/)
+my $t = Test::Nginx->new()->has(qw/stream stream_return map/)->plan(14)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -63,7 +63,7 @@ stream {
 
 EOF
 
-$t->try_run('no proxy_protocol tlv')->plan(14);
+$t->run();
 
 ###############################################################################
 
@@ -86,9 +86,6 @@ like($r, qr/x:\x0d?$/m, 'non-existent');
 
 # big proxy protocol header with TLVs
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.3');
-
 my $sub = pp2_create_tlv(0x21, "TLSv1.2");
 $sub .= pp2_create_tlv(0x22, "example.com");
 $sub .= pp2_create_tlv(0x23, "AES256-SHA");
@@ -107,8 +104,6 @@ like($r, qr/ssl-sig-alg:SHA1\x0d?$/m, 'S
 like($r, qr/ssl-key-alg:RSA512\x0d?$/m, 'SSL_KEY_ALG');
 like($r, qr/ssl-binary:true/, 'SSL_BINARY');
 
-}
-
 ###############################################################################
 
 sub pp_get {
diff --git a/stream_ssl_certificate.t b/stream_ssl_certificate.t
--- a/stream_ssl_certificate.t
+++ b/stream_ssl_certificate.t
@@ -156,7 +156,6 @@ like(get('default', 8080, $s), qr/defaul
 
 TODO: {
 # ticket key name mismatch prevents session resumption
-local $TODO = 'not yet' unless $t->has_version('1.23.2');
 local $TODO = 'no SSL_session_key, old IO::Socket::SSL'
 	if $IO::Socket::SSL::VERSION < 1.965;
 
diff --git a/uwsgi.t b/uwsgi.t
--- a/uwsgi.t
+++ b/uwsgi.t
@@ -102,9 +102,6 @@ like(http_get('/var?b=127.0.0.1:' . port
 	'uwsgi with variables');
 like(http_get('/var?b=u'), qr/SEE-THIS/, 'uwsgi with variables to upstream');
 
-TODO: {
-local $TODO = 'not yet' unless $t->has_version('1.23.0');
-
 my $r = http(<<EOF);
 GET / HTTP/1.0
 Host: localhost
@@ -127,8 +124,6 @@ like($r, qr/X-Cookie: foo; bar; bazz/,
 like($r, qr/X-Foo: foo, bar, bazz/,
 	'uwsgi with multiple unknown headers');
 
-}
-
 ###############################################################################
 
 sub http_get_headers {


From mdounin at mdounin.ru  Thu Jun  6 17:08:43 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Thu, 06 Jun 2024 20:08:43 +0300
Subject: [nginx-site] Documented XOAUTH2 and OAUTHBEARER authentication m...
Message-ID: <hg.7b7dbaa7d777.1717693723.-7209860318828295439@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-site/rev/7b7dbaa7d777
branches:  
changeset: 3088:7b7dbaa7d777
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 04 18:35:21 2024 +0300
description:
Documented XOAUTH2 and OAUTHBEARER authentication methods.

diffstat:

 xml/en/docs/mail/ngx_mail_auth_http_module.xml |  14 +++++++++++++-
 xml/en/docs/mail/ngx_mail_imap_module.xml      |  14 +++++++++++++-
 xml/en/docs/mail/ngx_mail_pop3_module.xml      |  14 +++++++++++++-
 xml/en/docs/mail/ngx_mail_smtp_module.xml      |  14 +++++++++++++-
 xml/ru/docs/mail/ngx_mail_auth_http_module.xml |  14 +++++++++++++-
 xml/ru/docs/mail/ngx_mail_imap_module.xml      |  14 +++++++++++++-
 xml/ru/docs/mail/ngx_mail_pop3_module.xml      |  14 +++++++++++++-
 xml/ru/docs/mail/ngx_mail_smtp_module.xml      |  14 +++++++++++++-
 8 files changed, 104 insertions(+), 8 deletions(-)

diffs (248 lines):

diff --git a/xml/en/docs/mail/ngx_mail_auth_http_module.xml b/xml/en/docs/mail/ngx_mail_auth_http_module.xml
--- a/xml/en/docs/mail/ngx_mail_auth_http_module.xml
+++ b/xml/en/docs/mail/ngx_mail_auth_http_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_auth_http_module"
         link="/en/docs/mail/ngx_mail_auth_http_module.html"
         lang="en"
-        rev="11">
+        rev="12">
 
 <section id="directives" name="Directives">
 
@@ -203,6 +203,18 @@ Auth-SMTP-To: RCPT TO: &lt;postmaster at ma
 </para>
 
 <para>
+For the XOAUTH2 and OAUTHBEARER authentication methods (1.27.1),
+the <header>Auth-Error-SASL</header> header
+could be used to return an error response
+in the form of an additional base64-encoded SASL challenge
+(<link
+url="https://developers.google.com/gmail/imap/xoauth2-protocol#error_response"
+>XOAUTH2</link>,
+<link url="https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.2"
+>OAUTHBEARER</link>).
+</para>
+
+<para>
 For the SSL/TLS client connection (1.7.11),
 the <header>Auth-SSL</header> header is added, and
 <header>Auth-SSL-Verify</header> will contain
diff --git a/xml/en/docs/mail/ngx_mail_imap_module.xml b/xml/en/docs/mail/ngx_mail_imap_module.xml
--- a/xml/en/docs/mail/ngx_mail_imap_module.xml
+++ b/xml/en/docs/mail/ngx_mail_imap_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_imap_module"
         link="/en/docs/mail/ngx_mail_imap_module.html"
         lang="en"
-        rev="7">
+        rev="8">
 
 <section id="directives" name="Directives">
 
@@ -47,6 +47,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH=EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH=XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH=OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/en/docs/mail/ngx_mail_pop3_module.xml b/xml/en/docs/mail/ngx_mail_pop3_module.xml
--- a/xml/en/docs/mail/ngx_mail_pop3_module.xml
+++ b/xml/en/docs/mail/ngx_mail_pop3_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_pop3_module"
         link="/en/docs/mail/ngx_mail_pop3_module.html"
         lang="en"
-        rev="5">
+        rev="6">
 
 <section id="directives" name="Directives">
 
@@ -49,6 +49,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/en/docs/mail/ngx_mail_smtp_module.xml b/xml/en/docs/mail/ngx_mail_smtp_module.xml
--- a/xml/en/docs/mail/ngx_mail_smtp_module.xml
+++ b/xml/en/docs/mail/ngx_mail_smtp_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_mail_smtp_module"
         link="/en/docs/mail/ngx_mail_smtp_module.html"
         lang="en"
-        rev="8">
+        rev="9">
 
 <section id="directives" name="Directives">
 
@@ -48,6 +48,18 @@ In order for this method to work, the pa
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 <tag-name><literal>none</literal></tag-name>
 <tag-desc>
 Authentication is not required.
diff --git a/xml/ru/docs/mail/ngx_mail_auth_http_module.xml b/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
--- a/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_auth_http_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_auth_http_module"
         link="/ru/docs/mail/ngx_mail_auth_http_module.html"
         lang="ru"
-        rev="11">
+        rev="12">
 
 <section id="directives" name="?????????">
 
@@ -201,6 +201,18 @@ Auth-SMTP-To: RCPT TO: &lt;postmaster at ma
 </para>
 
 <para>
+??? ??????? ?????????????? XOAUTH2 and OAUTHBEARER (1.27.1)
+? ????????? <header>Auth-Error-SASL</header>
+????? ??????? ?????????? ?? ??????
+? ????? ??????????????? SASL challenge ? base64
+(<link
+url="https://developers.google.com/gmail/imap/xoauth2-protocol#error_response"
+>XOAUTH2</link>,
+<link url="https://datatracker.ietf.org/doc/html/rfc7628#section-3.2.2"
+>OAUTHBEARER</link>).
+</para>
+
+<para>
 ??? ??????????? ?????????? ?? ????????? SSL/TLS (1.7.11)
 ??????????? ????????? <header>Auth-SSL</header>, ? ???? ?????????
 <link doc="ngx_mail_ssl_module.xml" id="ssl_verify_client"/> ????????,
diff --git a/xml/ru/docs/mail/ngx_mail_imap_module.xml b/xml/ru/docs/mail/ngx_mail_imap_module.xml
--- a/xml/ru/docs/mail/ngx_mail_imap_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_imap_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_imap_module"
         link="/ru/docs/mail/ngx_mail_imap_module.html"
         lang="ru"
-        rev="7">
+        rev="8">
 
 <section id="directives" name="?????????">
 
@@ -47,6 +47,18 @@
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH=EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH=XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH=OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/ru/docs/mail/ngx_mail_pop3_module.xml b/xml/ru/docs/mail/ngx_mail_pop3_module.xml
--- a/xml/ru/docs/mail/ngx_mail_pop3_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_pop3_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_pop3_module"
         link="/ru/docs/mail/ngx_mail_pop3_module.html"
         lang="ru"
-        rev="5">
+        rev="6">
 
 <section id="directives" name="?????????">
 
@@ -49,6 +49,18 @@
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 </list>
 </para>
 
diff --git a/xml/ru/docs/mail/ngx_mail_smtp_module.xml b/xml/ru/docs/mail/ngx_mail_smtp_module.xml
--- a/xml/ru/docs/mail/ngx_mail_smtp_module.xml
+++ b/xml/ru/docs/mail/ngx_mail_smtp_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_mail_smtp_module"
         link="/ru/docs/mail/ngx_mail_smtp_module.html"
         lang="ru"
-        rev="8">
+        rev="9">
 
 <section id="directives" name="?????????">
 
@@ -48,6 +48,18 @@ SMTP-????????.
 <link url="https://datatracker.ietf.org/doc/html/rfc4422">AUTH EXTERNAL</link> (1.11.6).
 </tag-desc>
 
+<tag-name><literal>xoauth2</literal></tag-name>
+<tag-desc>
+<link url="https://developers.google.com/gmail/imap/xoauth2-protocol"
+>AUTH XOAUTH2</link> (1.27.1).
+</tag-desc>
+
+<tag-name><literal>oauthbearer</literal></tag-name>
+<tag-desc>
+<link url="https://datatracker.ietf.org/doc/html/rfc7628"
+>AUTH OAUTHBEARER</link> (1.27.1).
+</tag-desc>
+
 <tag-name><literal>none</literal></tag-name>
 <tag-desc>
 ?????????????? ?? ?????????.

From mdounin at mdounin.ru  Sun Jun 16 01:29:27 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 04:29:27 +0300
Subject: [nginx] Version bump.
Message-ID: <hg.22f6716fe23d.1718501367.5824173280855779873@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/22f6716fe23d
branches:  
changeset: 9293:22f6716fe23d
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Sun Jun 16 04:15:43 2024 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff --git a/src/core/nginx.h b/src/core/nginx.h
--- a/src/core/nginx.h
+++ b/src/core/nginx.h
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1027001
-#define NGINX_VERSION      "1.27.1"
+#define nginx_version      1027002
+#define NGINX_VERSION      "1.27.2"
 
 #define NGINX_NAME         "freenginx"
 #define NGINX_VER          NGINX_NAME "/" NGINX_VERSION


From mdounin at mdounin.ru  Sun Jun 16 01:29:27 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 04:29:27 +0300
Subject: [nginx] Resolver: allowed responses with AD bit set.
Message-ID: <hg.ea0eef2dd12c.1718501367.5824173280855779873@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/ea0eef2dd12c
branches:  
changeset: 9294:ea0eef2dd12c
user:      Kirill A. Korinsky <kirill at korins.ky>
date:      Sun Jun 16 04:17:27 2024 +0300
description:
Resolver: allowed responses with AD bit set.

diffstat:

 src/core/ngx_resolver.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/src/core/ngx_resolver.c b/src/core/ngx_resolver.c
--- a/src/core/ngx_resolver.c
+++ b/src/core/ngx_resolver.c
@@ -1774,7 +1774,7 @@ ngx_resolver_process_response(ngx_resolv
                    (response->nar_hi << 8) + response->nar_lo);
 
     /* response to a standard query */
-    if ((flags & 0xf870) != 0x8000 || (trunc && tcp)) {
+    if ((flags & 0xf850) != 0x8000 || (trunc && tcp)) {
         ngx_log_error(r->log_level, r->log, 0,
                       "invalid %s DNS response %ui fl:%04Xi",
                       tcp ? "TCP" : "UDP", ident, flags);


From mdounin at mdounin.ru  Sun Jun 16 03:23:57 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:23:57 +0300
Subject: [PATCH 0 of 4] error_log rate limiting
Message-ID: <patchbomb.1718508237@vm-bsd.mdounin.ru>

Hello!

The following patch series introduces error_log rate limiting,
with the default rate limit set to 1000 messages per second (for each
worker process).

Rate limiting uses "leaky bucket" algorithm (as in the limit_req module)
with burst set to a value proportional to the log message severity,
so more severe messages can be logged even if less severe are already
rate-limited.  That is, rate limiting more or less dynamically adjusts
log level to keep logging rate under the specified limit.

Review and testing appreciated.

-- 
Maxim Dounin



From mdounin at mdounin.ru  Sun Jun 16 03:23:58 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:23:58 +0300
Subject: [PATCH 1 of 4] Core: fixed ENOSPC handling for error logs
In-Reply-To: <patchbomb.1718508237@vm-bsd.mdounin.ru>
References: <patchbomb.1718508237@vm-bsd.mdounin.ru>
Message-ID: <1be15e0f5fe464710dd1.1718508238@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718501944 -10800
#      Sun Jun 16 04:39:04 2024 +0300
# Node ID 1be15e0f5fe464710dd1a2d20cf4a3289149816c
# Parent  ea0eef2dd12c2d41349d63c532e942cf95fc4d7b
Core: fixed ENOSPC handling for error logs.

For each connection a new ngx_log_t structure is created, and saving
anything into disk_full_time field in this structure doesn't affect
other connections.  Fix is to move the disk_full_time field into the
ngx_open_file_t structure.

diff --git a/src/core/ngx_conf_file.c b/src/core/ngx_conf_file.c
--- a/src/core/ngx_conf_file.c
+++ b/src/core/ngx_conf_file.c
@@ -951,6 +951,7 @@ ngx_conf_open_file(ngx_cycle_t *cycle, n
         file->name = *name;
     }
 
+    file->disk_full_time = 0;
     file->flush = NULL;
     file->data = NULL;
 
diff --git a/src/core/ngx_conf_file.h b/src/core/ngx_conf_file.h
--- a/src/core/ngx_conf_file.h
+++ b/src/core/ngx_conf_file.h
@@ -90,6 +90,8 @@ struct ngx_open_file_s {
     ngx_fd_t              fd;
     ngx_str_t             name;
 
+    time_t                disk_full_time;
+
     void                (*flush)(ngx_open_file_t *file, ngx_log_t *log);
     void                 *data;
 };
diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -169,7 +169,7 @@ ngx_log_error_core(ngx_uint_t level, ngx
             goto next;
         }
 
-        if (ngx_time() == log->disk_full_time) {
+        if (ngx_time() == log->file->disk_full_time) {
 
             /*
              * on FreeBSD writing to a full filesystem with enabled softupdates
@@ -183,7 +183,7 @@ ngx_log_error_core(ngx_uint_t level, ngx
         n = ngx_write_fd(log->file->fd, errstr, p - errstr);
 
         if (n == -1 && ngx_errno == NGX_ENOSPC) {
-            log->disk_full_time = ngx_time();
+            log->file->disk_full_time = ngx_time();
         }
 
         if (log->file->fd == ngx_stderr) {
diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h
--- a/src/core/ngx_log.h
+++ b/src/core/ngx_log.h
@@ -53,8 +53,6 @@ struct ngx_log_s {
 
     ngx_atomic_uint_t    connection;
 
-    time_t               disk_full_time;
-
     ngx_log_handler_pt   handler;
     void                *data;
 



From mdounin at mdounin.ru  Sun Jun 16 03:23:59 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:23:59 +0300
Subject: [PATCH 2 of 4] Core: moved ngx_log_set_levels() to a proper position
In-Reply-To: <patchbomb.1718508237@vm-bsd.mdounin.ru>
References: <patchbomb.1718508237@vm-bsd.mdounin.ru>
Message-ID: <70b8d952a8a8398b485b.1718508239@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718501946 -10800
#      Sun Jun 16 04:39:06 2024 +0300
# Node ID 70b8d952a8a8398b485b9bcb14582164d8e7b697
# Parent  1be15e0f5fe464710dd1a2d20cf4a3289149816c
Core: moved ngx_log_set_levels() to a proper position.

Previous order is an artifact from the time before 5254:7ecaa9e4bf1b,
when ngx_log_set_levels() was a non-static function.

No functional changes.

diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -476,69 +476,6 @@ ngx_log_get_file_log(ngx_log_t *head)
 
 
 static char *
-ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
-{
-    ngx_uint_t   i, n, d, found;
-    ngx_str_t   *value;
-
-    if (cf->args->nelts == 2) {
-        log->log_level = NGX_LOG_ERR;
-        return NGX_CONF_OK;
-    }
-
-    value = cf->args->elts;
-
-    for (i = 2; i < cf->args->nelts; i++) {
-        found = 0;
-
-        for (n = 1; n <= NGX_LOG_DEBUG; n++) {
-            if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
-
-                if (log->log_level != 0) {
-                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                                       "duplicate log level \"%V\"",
-                                       &value[i]);
-                    return NGX_CONF_ERROR;
-                }
-
-                log->log_level = n;
-                found = 1;
-                break;
-            }
-        }
-
-        for (n = 0, d = NGX_LOG_DEBUG_FIRST; d <= NGX_LOG_DEBUG_LAST; d <<= 1) {
-            if (ngx_strcmp(value[i].data, debug_levels[n++]) == 0) {
-                if (log->log_level & ~NGX_LOG_DEBUG_ALL) {
-                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                                       "invalid log level \"%V\"",
-                                       &value[i]);
-                    return NGX_CONF_ERROR;
-                }
-
-                log->log_level |= d;
-                found = 1;
-                break;
-            }
-        }
-
-
-        if (!found) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "invalid log level \"%V\"", &value[i]);
-            return NGX_CONF_ERROR;
-        }
-    }
-
-    if (log->log_level == NGX_LOG_DEBUG) {
-        log->log_level = NGX_LOG_DEBUG_ALL;
-    }
-
-    return NGX_CONF_OK;
-}
-
-
-static char *
 ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_log_t  *dummy;
@@ -673,6 +610,69 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 }
 
 
+static char *
+ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
+{
+    ngx_uint_t   i, n, d, found;
+    ngx_str_t   *value;
+
+    if (cf->args->nelts == 2) {
+        log->log_level = NGX_LOG_ERR;
+        return NGX_CONF_OK;
+    }
+
+    value = cf->args->elts;
+
+    for (i = 2; i < cf->args->nelts; i++) {
+        found = 0;
+
+        for (n = 1; n <= NGX_LOG_DEBUG; n++) {
+            if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
+
+                if (log->log_level != 0) {
+                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                       "duplicate log level \"%V\"",
+                                       &value[i]);
+                    return NGX_CONF_ERROR;
+                }
+
+                log->log_level = n;
+                found = 1;
+                break;
+            }
+        }
+
+        for (n = 0, d = NGX_LOG_DEBUG_FIRST; d <= NGX_LOG_DEBUG_LAST; d <<= 1) {
+            if (ngx_strcmp(value[i].data, debug_levels[n++]) == 0) {
+                if (log->log_level & ~NGX_LOG_DEBUG_ALL) {
+                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                       "invalid log level \"%V\"",
+                                       &value[i]);
+                    return NGX_CONF_ERROR;
+                }
+
+                log->log_level |= d;
+                found = 1;
+                break;
+            }
+        }
+
+
+        if (!found) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid log level \"%V\"", &value[i]);
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    if (log->log_level == NGX_LOG_DEBUG) {
+        log->log_level = NGX_LOG_DEBUG_ALL;
+    }
+
+    return NGX_CONF_OK;
+}
+
+
 static void
 ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log)
 {



From mdounin at mdounin.ru  Sun Jun 16 03:24:00 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:24:00 +0300
Subject: [PATCH 3 of 4] Core: simplified log levels matching
In-Reply-To: <patchbomb.1718508237@vm-bsd.mdounin.ru>
References: <patchbomb.1718508237@vm-bsd.mdounin.ru>
Message-ID: <dc678c5dbd83fb80bca6.1718508240@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718501950 -10800
#      Sun Jun 16 04:39:10 2024 +0300
# Node ID dc678c5dbd83fb80bca6a5ce388f5f64d2752a84
# Parent  70b8d952a8a8398b485b9bcb14582164d8e7b697
Core: simplified log levels matching.

diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -613,18 +613,12 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 static char *
 ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
 {
-    ngx_uint_t   i, n, d, found;
+    ngx_uint_t   i, n, d;
     ngx_str_t   *value;
 
-    if (cf->args->nelts == 2) {
-        log->log_level = NGX_LOG_ERR;
-        return NGX_CONF_OK;
-    }
-
     value = cf->args->elts;
 
     for (i = 2; i < cf->args->nelts; i++) {
-        found = 0;
 
         for (n = 1; n <= NGX_LOG_DEBUG; n++) {
             if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
@@ -637,8 +631,7 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
                 }
 
                 log->log_level = n;
-                found = 1;
-                break;
+                goto next;
             }
         }
 
@@ -652,17 +645,21 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
                 }
 
                 log->log_level |= d;
-                found = 1;
-                break;
+                goto next;
             }
         }
 
+        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                           "invalid log level \"%V\"", &value[i]);
+        return NGX_CONF_ERROR;
 
-        if (!found) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "invalid log level \"%V\"", &value[i]);
-            return NGX_CONF_ERROR;
-        }
+    next:
+
+        continue;
+    }
+
+    if (log->log_level == 0) {
+        log->log_level = NGX_LOG_ERR;
     }
 
     if (log->log_level == NGX_LOG_DEBUG) {



From mdounin at mdounin.ru  Sun Jun 16 03:24:01 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:24:01 +0300
Subject: [PATCH 4 of 4] Core: error logging rate limiting
In-Reply-To: <patchbomb.1718508237@vm-bsd.mdounin.ru>
References: <patchbomb.1718508237@vm-bsd.mdounin.ru>
Message-ID: <b96ac0999e5734664189.1718508241@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718501953 -10800
#      Sun Jun 16 04:39:13 2024 +0300
# Node ID b96ac0999e5734664189b16052a2147f546b2303
# Parent  dc678c5dbd83fb80bca6a5ce388f5f64d2752a84
Core: error logging rate limiting.

With this change, error logging to files can be rate-limited with
the "rate=" parameter.  The parameter specifies allowed log messages
rate to a particular file (per worker), in messages per second (m/s).
By default, "rate=1000m/s" is used.

Rate limiting is implemented using the "leaky bucket" method, similarly
to the limit_req module.

Maximum burst size is set to the number of log messages per second
for each severity level, so "error" messages are logged even if the
rate limit is hit by "info" messages (but not vice versa).  When the
limit is reached for a particular level, the "too many log messages,
limiting" message is logged at this level.

If debug logging is enabled, either for the particular log file or for
the particular connection, rate limiting is not used.

diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h
--- a/src/core/ngx_connection.h
+++ b/src/core/ngx_connection.h
@@ -207,6 +207,7 @@ struct ngx_connection_s {
 #define ngx_set_connection_log(c, l)                                         \
                                                                              \
     c->log->file = l->file;                                                  \
+    c->log->limit = l->limit;                                                \
     c->log->next = l->next;                                                  \
     c->log->writer = l->writer;                                              \
     c->log->wdata = l->wdata;                                                \
diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -9,8 +9,9 @@
 #include <ngx_core.h>
 
 
+static ngx_int_t ngx_log_check_rate(ngx_log_t *log, ngx_uint_t level);
 static char *ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
-static char *ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log);
+static char *ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log);
 static void ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log);
 
 
@@ -164,6 +165,12 @@ ngx_log_error_core(ngx_uint_t level, ngx
             break;
         }
 
+        if (log->limit && !debug_connection) {
+            if (ngx_log_check_rate(log, level) == NGX_BUSY) {
+                goto next;
+            }
+        }
+
         if (log->writer) {
             log->writer(log, level, errstr, p - errstr);
             goto next;
@@ -314,6 +321,69 @@ ngx_log_errno(u_char *buf, u_char *last,
 }
 
 
+static ngx_int_t
+ngx_log_check_rate(ngx_log_t *log, ngx_uint_t level)
+{
+    ngx_log_t          temp_log;
+    ngx_int_t          excess, changed, burst;
+    ngx_atomic_int_t   ms;
+    ngx_atomic_uint_t  now, last;
+
+    now = ngx_current_msec;
+
+    last = log->limit->last;
+    excess = log->limit->excess;
+
+    ms = (ngx_atomic_int_t) (now - last);
+
+    if (ms < -60000) {
+        ms = 1;
+
+    } else if (ms < 0) {
+        ms = 0;
+    }
+
+    changed = excess - log->limit->rate * ms / 1000 + 1000;
+
+    if (changed < 0) {
+        changed = 0;
+    }
+
+    burst = (log->log_level - level + 1) * log->limit->rate;
+
+    if (changed > burst) {
+        if (excess <= burst) {
+
+            ngx_atomic_fetch_add(&log->limit->excess, 1000);
+            
+            /* log message to this log only */
+
+            temp_log = *log;
+            temp_log.connection = 0;
+            temp_log.handler = NULL;
+            temp_log.limit = NULL;
+            temp_log.next = NULL;
+
+            ngx_log_error(level, &temp_log, 0,
+                          "too many log messages, limiting");
+        }
+
+        return NGX_BUSY;
+    }
+
+    if (ms > 0
+        && ngx_atomic_cmp_set(&log->limit->last, last, now))
+    {
+        ngx_atomic_fetch_add(&log->limit->excess, changed - excess);
+
+    } else {
+        ngx_atomic_fetch_add(&log->limit->excess, 1000);
+    }
+
+    return NGX_OK;
+}
+
+
 ngx_log_t *
 ngx_log_init(u_char *prefix, u_char *error_log)
 {
@@ -598,7 +668,7 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
         }
     }
 
-    if (ngx_log_set_levels(cf, new_log) != NGX_CONF_OK) {
+    if (ngx_log_set_params(cf, new_log) != NGX_CONF_OK) {
         return NGX_CONF_ERROR;
     }
 
@@ -611,13 +681,17 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 
 
 static char *
-ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
+ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log)
 {
+    size_t       len;
+    ngx_int_t    rate;
     ngx_uint_t   i, n, d;
     ngx_str_t   *value;
 
     value = cf->args->elts;
 
+    rate = 1000;
+
     for (i = 2; i < cf->args->nelts; i++) {
 
         for (n = 1; n <= NGX_LOG_DEBUG; n++) {
@@ -649,8 +723,33 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
             }
         }
 
-        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                           "invalid log level \"%V\"", &value[i]);
+        if (ngx_strncmp(value[i].data, "rate=", 5) == 0) {
+
+            len = value[i].len;
+
+            if (ngx_strncmp(value[i].data + len - 3, "m/s", 3) == 0) {
+                len -= 3;
+            }
+
+            rate = ngx_atoi(value[i].data + 5, len - 5);
+            if (rate < 0) {
+                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                   "invalid rate \"%V\"", &value[i]);
+                return NGX_CONF_ERROR;
+            }
+
+            continue;
+        }
+
+        if (log->log_level) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid parameter \"%V\"", &value[i]);
+
+        } else {
+             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                "invalid log level \"%V\"", &value[i]);
+        }
+
         return NGX_CONF_ERROR;
 
     next:
@@ -666,6 +765,17 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
         log->log_level = NGX_LOG_DEBUG_ALL;
     }
 
+    if (rate > 0
+        && log->log_level < NGX_LOG_DEBUG)
+    {
+        log->limit = ngx_pcalloc(cf->pool, sizeof(ngx_log_limit_t));
+        if (log->limit == NULL) {
+            return NGX_CONF_ERROR;
+        }
+
+        log->limit->rate = rate * 1000;
+    }
+
     return NGX_CONF_OK;
 }
 
diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h
--- a/src/core/ngx_log.h
+++ b/src/core/ngx_log.h
@@ -47,6 +47,13 @@ typedef void (*ngx_log_writer_pt) (ngx_l
     u_char *buf, size_t len);
 
 
+typedef struct {
+    ngx_uint_t           rate;
+    ngx_atomic_t         excess;
+    ngx_atomic_t         last;
+} ngx_log_limit_t;
+
+
 struct ngx_log_s {
     ngx_uint_t           log_level;
     ngx_open_file_t     *file;
@@ -67,6 +74,8 @@ struct ngx_log_s {
 
     char                *action;
 
+    ngx_log_limit_t     *limit;
+
     ngx_log_t           *next;
 };
 



From mdounin at mdounin.ru  Sun Jun 16 03:54:53 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:54:53 +0300
Subject: [PATCH] Documented $r->log_error() logging level
Message-ID: <23b9cbb0c11de936ca3b.1718510093@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718393731 -10800
#      Fri Jun 14 22:35:31 2024 +0300
# Node ID 23b9cbb0c11de936ca3bca0c1c67528f7ba0fd51
# Parent  7b7dbaa7d777cd8cacaebb177d76559f84736b94
Documented $r->log_error() logging level.

diff --git a/xml/en/docs/http/ngx_http_perl_module.xml b/xml/en/docs/http/ngx_http_perl_module.xml
--- a/xml/en/docs/http/ngx_http_perl_module.xml
+++ b/xml/en/docs/http/ngx_http_perl_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_http_perl_module"
         link="/en/docs/http/ngx_http_perl_module.html"
         lang="en"
-        rev="7">
+        rev="8">
 
 <section id="summary">
 
@@ -320,7 +320,8 @@ supports redirections to named locations
 <value>message</value>)</literal></tag-name>
 <tag-desc>
 writes the specified <value>message</value> into the
-<link doc="../ngx_core_module.xml" id="error_log"/>.
+<link doc="../ngx_core_module.xml" id="error_log"/>
+at the <literal>error</literal> level.
 If <value>errno</value> is non-zero, an error code and its description
 will be appended to the message.
 </tag-desc>
diff --git a/xml/ru/docs/http/ngx_http_perl_module.xml b/xml/ru/docs/http/ngx_http_perl_module.xml
--- a/xml/ru/docs/http/ngx_http_perl_module.xml
+++ b/xml/ru/docs/http/ngx_http_perl_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_http_perl_module"
         link="/ru/docs/http/ngx_http_perl_module.html"
         lang="ru"
-        rev="7">
+        rev="8">
 
 <section id="summary">
 
@@ -320,7 +320,8 @@ 1;
 <value>?????????</value>)</literal></tag-name>
 <tag-desc>
 ?????????? ????????? <value>?????????</value> ?
-<link doc="../ngx_core_module.xml" id="error_log"/>.
+<link doc="../ngx_core_module.xml" id="error_log"/>
+?? ?????? <literal>error</literal>.
 ???? <value>???_??????</value> ?????????, ?? ? ????????? ????? ????????
 ??? ?????? ? ?? ????????.
 </tag-desc>

From mdounin at mdounin.ru  Sun Jun 16 03:55:48 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Sun, 16 Jun 2024 06:55:48 +0300
Subject: [PATCH] Documented the "rate" parameter of the "error_log" directive
Message-ID: <27532d42102bb58bff34.1718510148@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718510130 -10800
#      Sun Jun 16 06:55:30 2024 +0300
# Node ID 27532d42102bb58bff34ca3f2b7020a8c1634786
# Parent  23b9cbb0c11de936ca3bca0c1c67528f7ba0fd51
Documented the "rate" parameter of the "error_log" directive.

diff --git a/xml/en/docs/ngx_core_module.xml b/xml/en/docs/ngx_core_module.xml
--- a/xml/en/docs/ngx_core_module.xml
+++ b/xml/en/docs/ngx_core_module.xml
@@ -10,7 +10,7 @@
 <module name="Core functionality"
         link="/en/docs/ngx_core_module.html"
         lang="en"
-        rev="28">
+        rev="29">
 
 <section id="example" name="Example Configuration">
 
@@ -209,7 +209,10 @@ and should not be set directly by the us
 
 
 <directive name="error_log">
-<syntax><value>file</value> [<value>level</value>]</syntax>
+<syntax>
+    <value>file</value>
+    [<value>level</value>]
+    [<literal>rate</literal>=<value>rate</value>]</syntax>
 <default>logs/error.log error</default>
 <context>main</context>
 <context>http</context>
@@ -258,7 +261,20 @@ For <literal>debug</literal> logging to 
 be built with <literal>--with-debug</literal>,
 see ?<link doc="debugging_log.xml"/>?.
 </note>
+</para>
 
+<para>
+The <literal>rate</literal> parameter (1.27.2) specifies
+the maximum allowed logging rate in messages per second (m/s)
+for each worker process.
+By default, <literal>rate=1000m/s</literal> is used.
+Setting <value>rate</value> to 0 disables rate limiting.
+Additionally, rate limiting is not used
+if logging level is set to <literal>debug</literal> or
+debugging log is enabled for the <link id="debug_connection">connection</link>.
+</para>
+
+<para>
 <note>
 The directive can be specified on the
 <literal>stream</literal> level
diff --git a/xml/ru/docs/ngx_core_module.xml b/xml/ru/docs/ngx_core_module.xml
--- a/xml/ru/docs/ngx_core_module.xml
+++ b/xml/ru/docs/ngx_core_module.xml
@@ -10,7 +10,7 @@
 <module name="???????? ????????????????"
         link="/ru/docs/ngx_core_module.html"
         lang="ru"
-        rev="28">
+        rev="29">
 
 <section id="example" name="?????? ????????????">
 
@@ -208,7 +208,10 @@ env OPENSSL_ALLOW_PROXY_CERTS=1;
 
 
 <directive name="error_log">
-<syntax><value>????</value> [<value>???????</value>]</syntax>
+<syntax>
+    <value>????</value>
+    [<value>???????</value>]
+    [<literal>rate</literal>=<value>????????</value>]</syntax>
 <default>logs/error.log error</default>
 <context>main</context>
 <context>http</context>
@@ -256,7 +259,20 @@ env OPENSSL_ALLOW_PROXY_CERTS=1;
 nginx ? <literal>--with-debug</literal>,
 ??. ?<link doc="debugging_log.xml"/>?.
 </note>
+</para>
 
+<para>
+???????? <literal>rate</literal> (1.27.2) ??????
+??????????? ?????????? ???????? ?????? ? ??? ? ?????????? ? ??????? (m/s)
+??? ??????? ???????? ????????.
+?? ????????? ???????????? <literal>rate=1000m/s</literal>.
+????????? <literal>????????</literal> ? 0 ????????? ??????????? ????????.
+????? ????, ??????????? ???????? ?? ????????????,
+???? ??????? ???? ?????????? ? <literal>debug</literal> ???
+?????????? ??? ??????? ??? <link id="debug_connection">??????????</link>.
+</para>
+
+<para>
 <note>
 ????????? ????? ???? ??????? ??
 ?????? <literal>stream</literal>

From hnakamur at gmail.com  Tue Jun 18 22:15:53 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Wed, 19 Jun 2024 07:15:53 +0900
Subject: [PATCH] [nginx-tests] Tests: Add sleep to sub_filter_multi.t for
 TEST_NGINX_UNSAFE=1
Message-ID: <CAN-DUMS8CKM6iVqTUguOyQK2oaCrN0CQLrYSLsNjnNVYZLzskQ@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1718746554 -32400
#      Wed Jun 19 06:35:54 2024 +0900
# Node ID 8d9e1bc9721896618b0bb4095e39be46ca8fc280
# Parent  a095b971fbcc99a77206173f6130d5ff2681c389
Add sleep to sub_filter_multi for NGINX_TEST_UNSAFE=1

Without this fix, sub_filter_multi.t fails like below:
```
$ sudo -u nginx TEST_NGINX_UNSAFE=1
TEST_NGINX_BINARY=../freenginx/objs/nginx prove sub_filter_multi.t
sub_filter_multi.t .. 37/44
#   Failed test 'shortbuf match 1.3'
#   at sub_filter_multi.t line 366.
#                   undef
#     doesn't match '(?^:(+ABCDE){3})'
sub_filter_multi.t .. 42/44
#   Failed test 'shortbuf match 5'
#   at sub_filter_multi.t line 376.
#                   undef
#     doesn't match '(?^:+ABCDE(-*nyABCDE){2})'
# Looks like you failed 2 tests of 44.
sub_filter_multi.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/44 subtests
```

diff -r a095b971fbcc -r 8d9e1bc97218 sub_filter_multi.t
--- a/sub_filter_multi.t        Tue Jun 04 18:38:01 2024 +0300
+++ b/sub_filter_multi.t        Wed Jun 19 06:35:54 2024 +0900
@@ -363,8 +363,8 @@
        qr/(+A){3}/, 'shortbuf match 1.1');
 like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCD' x 3),
        qr/(+ABCD){3}/, 'shortbuf match 1.2');
-like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3),
-       qr/(+ABCDE){3}/, 'shortbuf match 1.3');
+like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3, sleep => 1),
+               qr/(+ABCDE){3}/, 'shortbuf match 1.3');
 like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyzB' x 2),
        qr/(+A-B){2}/, 'shortbuf match 2.1 (multiple replace)');
 like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyz' x 2),
@@ -373,7 +373,7 @@
        qr/(+A*){3}/, 'shortbuf match 3 (1 byte search pattern)');
 like(http_get('/shortbuf/match4?a=' . 'pattABCDEFGHI' x 3),
        qr/(+ABCDEFGHI){3}/, 'shortbuf match 4');
-like(http_get('/shortbuf/match5?a=abpatternyzABCDE' . 'abpatternyABCDE' x 2),
+like(http_get('/shortbuf/match5?a=abpatternyzABCDE' .
'abpatternyABCDE' x 2, sleep => 1),
        qr/+ABCDE(-*nyABCDE){2}/, 'shortbuf match 5');
 }


From mdounin at mdounin.ru  Thu Jun 20 00:45:20 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 20 Jun 2024 03:45:20 +0300
Subject: [PATCH] [nginx-tests] Tests: Add sleep to sub_filter_multi.t for
 TEST_NGINX_UNSAFE=1
In-Reply-To: <CAN-DUMS8CKM6iVqTUguOyQK2oaCrN0CQLrYSLsNjnNVYZLzskQ@mail.gmail.com>
References: <CAN-DUMS8CKM6iVqTUguOyQK2oaCrN0CQLrYSLsNjnNVYZLzskQ@mail.gmail.com>
Message-ID: <ZnN7oMAYD4a6J88J@mdounin.ru>

Hello!

On Wed, Jun 19, 2024 at 07:15:53AM +0900, Hiroaki Nakamura wrote:

> # HG changeset patch
> # User Hiroaki Nakamura <hnakamur at gmail.com>
> # Date 1718746554 -32400
> #      Wed Jun 19 06:35:54 2024 +0900
> # Node ID 8d9e1bc9721896618b0bb4095e39be46ca8fc280
> # Parent  a095b971fbcc99a77206173f6130d5ff2681c389
> Add sleep to sub_filter_multi for NGINX_TEST_UNSAFE=1
> 
> Without this fix, sub_filter_multi.t fails like below:
> ```
> $ sudo -u nginx TEST_NGINX_UNSAFE=1
> TEST_NGINX_BINARY=../freenginx/objs/nginx prove sub_filter_multi.t
> sub_filter_multi.t .. 37/44
> #   Failed test 'shortbuf match 1.3'
> #   at sub_filter_multi.t line 366.
> #                   undef
> #     doesn't match '(?^:(+ABCDE){3})'
> sub_filter_multi.t .. 42/44
> #   Failed test 'shortbuf match 5'
> #   at sub_filter_multi.t line 376.
> #                   undef
> #     doesn't match '(?^:+ABCDE(-*nyABCDE){2})'
> # Looks like you failed 2 tests of 44.
> sub_filter_multi.t .. Dubious, test returned 2 (wstat 512, 0x200)
> Failed 2/44 subtests
> ```
> 
> diff -r a095b971fbcc -r 8d9e1bc97218 sub_filter_multi.t
> --- a/sub_filter_multi.t        Tue Jun 04 18:38:01 2024 +0300
> +++ b/sub_filter_multi.t        Wed Jun 19 06:35:54 2024 +0900
> @@ -363,8 +363,8 @@
>         qr/(+A){3}/, 'shortbuf match 1.1');
>  like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCD' x 3),
>         qr/(+ABCD){3}/, 'shortbuf match 1.2');
> -like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3),
> -       qr/(+ABCDE){3}/, 'shortbuf match 1.3');
> +like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3, sleep => 1),
> +               qr/(+ABCDE){3}/, 'shortbuf match 1.3');
>  like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyzB' x 2),
>         qr/(+A-B){2}/, 'shortbuf match 2.1 (multiple replace)');
>  like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyz' x 2),
> @@ -373,7 +373,7 @@
>         qr/(+A*){3}/, 'shortbuf match 3 (1 byte search pattern)');
>  like(http_get('/shortbuf/match4?a=' . 'pattABCDEFGHI' x 3),
>         qr/(+ABCDEFGHI){3}/, 'shortbuf match 4');
> -like(http_get('/shortbuf/match5?a=abpatternyzABCDE' . 'abpatternyABCDE' x 2),
> +like(http_get('/shortbuf/match5?a=abpatternyzABCDE' .
> 'abpatternyABCDE' x 2, sleep => 1),
>         qr/+ABCDE(-*nyABCDE){2}/, 'shortbuf match 5');
>  }

Thanks for the patch.

Using the "sleep" option looks wrong to me: it is designed to 
introduce a pause before sending the request body, and such a 
pause is not needed in these tests.

Instead, test failures you are seeing seems to be a result of the 
fact that tests in question take a lot of time, and http_get() 
times out while waiting for responses.

Overall, it looks like the tests are suboptimal and very fragile: 
they use "limit_rate 4; limit_rate_after 160;" on the upstream 
server, which results in 1 second response time for each 4 
bytes (over 164 bytes which are sent initially).  For example, the 
"shortbuf match 5" test, currently results in 147 bytes of 
the response headers and uses 46 bytes of the response body, so 
the last response byte is sent after 8 seconds - which is exactly 
the timeout value http_get() uses.  As such, the test is highly 
likely to fail.

Most likely, the test took slightly less time when it was 
initially written due to slightly shorter response headers, and 
therefore used to succeed.  Accordingly, a quick fix would be to 
use a larger limit_rate_after value, such as 165 or 170.

A better fix would be to rewrite all these tests with embedded 
Perl, similarly to how it is done in sub_filter_perl.t, so 
multiple buffers will be generated without any unneeded delays.  
This will make the tests much more robust and much faster.  Not 
sure it worth the effort though, especially given that these tests 
are under TEST_NGINX_UNSAFE and thus not really expected to be run 
automatically.

Below is a patch to bump limit_rate_after to 170, please take a 
look if it works for you:

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718843281 -10800
#      Thu Jun 20 03:28:01 2024 +0300
# Node ID 9ed5047551e7455353d61c3b7e955e959a594050
# Parent  a095b971fbcc99a77206173f6130d5ff2681c389
Tests: adjusted sub_filter_multi.t limit rate settings.

With previous settings some tests with short buffers, which are under
TEST_NGINX_UNSAFE, used to hit 8 seconds timeout in http_get(), leading
to test failures.

Reported by Hiroaki Nakamura,
https://freenginx.org/pipermail/nginx-devel/2024-June/000373.html

diff --git a/sub_filter_multi.t b/sub_filter_multi.t
--- a/sub_filter_multi.t
+++ b/sub_filter_multi.t
@@ -244,7 +244,7 @@ http {
         listen       127.0.0.1:8081;
 
         limit_rate 4;
-        limit_rate_after 160;
+        limit_rate_after 170;
 
         location / {
             return 200 $arg_a;


-- 
Maxim Dounin
http://mdounin.ru/


From hnakamur at gmail.com  Thu Jun 20 03:45:15 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Thu, 20 Jun 2024 12:45:15 +0900
Subject: [PATCH] [nginx-tests] Tests: Add sleep to sub_filter_multi.t for
 TEST_NGINX_UNSAFE=1
In-Reply-To: <ZnN7oMAYD4a6J88J@mdounin.ru>
References: <CAN-DUMS8CKM6iVqTUguOyQK2oaCrN0CQLrYSLsNjnNVYZLzskQ@mail.gmail.com>
 <ZnN7oMAYD4a6J88J@mdounin.ru>
Message-ID: <CAN-DUMTGOtU4m-D+s+rx3uShyxtdn_H-ur24jwHG_2_fs8NJCw@mail.gmail.com>

Hello,

Thank you for your thorough explanation.
I have confirmed that your patch works for me.

Best regards,
Hiroaki Nakamura

2024?6?20?(?) 9:51 Maxim Dounin <mdounin at mdounin.ru>:

>
> Hello!
>
> On Wed, Jun 19, 2024 at 07:15:53AM +0900, Hiroaki Nakamura wrote:
>
> > # HG changeset patch
> > # User Hiroaki Nakamura <hnakamur at gmail.com>
> > # Date 1718746554 -32400
> > #      Wed Jun 19 06:35:54 2024 +0900
> > # Node ID 8d9e1bc9721896618b0bb4095e39be46ca8fc280
> > # Parent  a095b971fbcc99a77206173f6130d5ff2681c389
> > Add sleep to sub_filter_multi for NGINX_TEST_UNSAFE=1
> >
> > Without this fix, sub_filter_multi.t fails like below:
> > ```
> > $ sudo -u nginx TEST_NGINX_UNSAFE=1
> > TEST_NGINX_BINARY=../freenginx/objs/nginx prove sub_filter_multi.t
> > sub_filter_multi.t .. 37/44
> > #   Failed test 'shortbuf match 1.3'
> > #   at sub_filter_multi.t line 366.
> > #                   undef
> > #     doesn't match '(?^:(+ABCDE){3})'
> > sub_filter_multi.t .. 42/44
> > #   Failed test 'shortbuf match 5'
> > #   at sub_filter_multi.t line 376.
> > #                   undef
> > #     doesn't match '(?^:+ABCDE(-*nyABCDE){2})'
> > # Looks like you failed 2 tests of 44.
> > sub_filter_multi.t .. Dubious, test returned 2 (wstat 512, 0x200)
> > Failed 2/44 subtests
> > ```
> >
> > diff -r a095b971fbcc -r 8d9e1bc97218 sub_filter_multi.t
> > --- a/sub_filter_multi.t        Tue Jun 04 18:38:01 2024 +0300
> > +++ b/sub_filter_multi.t        Wed Jun 19 06:35:54 2024 +0900
> > @@ -363,8 +363,8 @@
> >         qr/(+A){3}/, 'shortbuf match 1.1');
> >  like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCD' x 3),
> >         qr/(+ABCD){3}/, 'shortbuf match 1.2');
> > -like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3),
> > -       qr/(+ABCDE){3}/, 'shortbuf match 1.3');
> > +like(http_get('/shortbuf/match1?a=' . 'abpatternyzABCDE' x 3, sleep => 1),
> > +               qr/(+ABCDE){3}/, 'shortbuf match 1.3');
> >  like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyzB' x 2),
> >         qr/(+A-B){2}/, 'shortbuf match 2.1 (multiple replace)');
> >  like(http_get('/shortbuf/match2?a=' . 'abpatternyzAabpaernyz' x 2),
> > @@ -373,7 +373,7 @@
> >         qr/(+A*){3}/, 'shortbuf match 3 (1 byte search pattern)');
> >  like(http_get('/shortbuf/match4?a=' . 'pattABCDEFGHI' x 3),
> >         qr/(+ABCDEFGHI){3}/, 'shortbuf match 4');
> > -like(http_get('/shortbuf/match5?a=abpatternyzABCDE' . 'abpatternyABCDE' x 2),
> > +like(http_get('/shortbuf/match5?a=abpatternyzABCDE' .
> > 'abpatternyABCDE' x 2, sleep => 1),
> >         qr/+ABCDE(-*nyABCDE){2}/, 'shortbuf match 5');
> >  }
>
> Thanks for the patch.
>
> Using the "sleep" option looks wrong to me: it is designed to
> introduce a pause before sending the request body, and such a
> pause is not needed in these tests.
>
> Instead, test failures you are seeing seems to be a result of the
> fact that tests in question take a lot of time, and http_get()
> times out while waiting for responses.
>
> Overall, it looks like the tests are suboptimal and very fragile:
> they use "limit_rate 4; limit_rate_after 160;" on the upstream
> server, which results in 1 second response time for each 4
> bytes (over 164 bytes which are sent initially).  For example, the
> "shortbuf match 5" test, currently results in 147 bytes of
> the response headers and uses 46 bytes of the response body, so
> the last response byte is sent after 8 seconds - which is exactly
> the timeout value http_get() uses.  As such, the test is highly
> likely to fail.
>
> Most likely, the test took slightly less time when it was
> initially written due to slightly shorter response headers, and
> therefore used to succeed.  Accordingly, a quick fix would be to
> use a larger limit_rate_after value, such as 165 or 170.
>
> A better fix would be to rewrite all these tests with embedded
> Perl, similarly to how it is done in sub_filter_perl.t, so
> multiple buffers will be generated without any unneeded delays.
> This will make the tests much more robust and much faster.  Not
> sure it worth the effort though, especially given that these tests
> are under TEST_NGINX_UNSAFE and thus not really expected to be run
> automatically.
>
> Below is a patch to bump limit_rate_after to 170, please take a
> look if it works for you:
>
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1718843281 -10800
> #      Thu Jun 20 03:28:01 2024 +0300
> # Node ID 9ed5047551e7455353d61c3b7e955e959a594050
> # Parent  a095b971fbcc99a77206173f6130d5ff2681c389
> Tests: adjusted sub_filter_multi.t limit rate settings.
>
> With previous settings some tests with short buffers, which are under
> TEST_NGINX_UNSAFE, used to hit 8 seconds timeout in http_get(), leading
> to test failures.
>
> Reported by Hiroaki Nakamura,
> https://freenginx.org/pipermail/nginx-devel/2024-June/000373.html
>
> diff --git a/sub_filter_multi.t b/sub_filter_multi.t
> --- a/sub_filter_multi.t
> +++ b/sub_filter_multi.t
> @@ -244,7 +244,7 @@ http {
>          listen       127.0.0.1:8081;
>
>          limit_rate 4;
> -        limit_rate_after 160;
> +        limit_rate_after 170;
>
>          location / {
>              return 200 $arg_a;
>
>
> --
> Maxim Dounin
> http://mdounin.ru/


From hnakamur at gmail.com  Thu Jun 20 11:39:35 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Thu, 20 Jun 2024 20:39:35 +0900
Subject: [PATCH 0 of 3] [nginx] cache: Update Age response header correctly
Message-ID: <CAN-DUMTqD-iHEhGUbqGiJiXfR3+DKzfbf5y5UaSC0byMr_UBGg@mail.gmail.com>

Hello!

Currently nginx does not update Age response header when it
receives responses from upstreams nor when it sends a cached
response.

This causes the problem that nginx might send an expired
cache because nginx does not consider the initial age
to calculate freshness of caches.

With this patchset, nginx updates Age response header
correctly when it receives responses from upstream and when
it sends a cached response as specified in RFC 9111 [1].

- patch 1: Update Age response header when nginx receives
           a response from upstreams and when it sends a
           cached response.
- patch 2: Save response time and corrected initial age
           to cache file header.
- patch 3: Tests: Update and add test files to patchset
           for convenience.
           Contents of the "*.t" files are to be put into
           nginx-tests repository.

Link: https://www.rfc-editor.org/rfc/rfc9111 [1]

Thanks!
Hiroaki Nakamura


From hnakamur at gmail.com  Thu Jun 20 11:39:43 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Thu, 20 Jun 2024 20:39:43 +0900
Subject: [PATCH 1 of 3] Correctly calculate and set Age header
Message-ID: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1718882801 -32400
#      Thu Jun 20 20:26:41 2024 +0900
# Branch correct_age
# Node ID c81df54e3d0333c26d4296792dc0df767b386f91
# Parent  73929a4f3447d558747623884b5ba281c13332d8
Correctly calculate and set Age header.
Implement the calculation of the Age header as specified in
"RFC 9111: HTTP Caching"
https://www.rfc-editor.org/rfc/rfc9111.html

diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_cache.h Thu Jun 20 20:26:41 2024 +0900
@@ -59,6 +59,8 @@
     size_t                           body_start;
     off_t                            fs_size;
     ngx_msec_t                       lock_time;
+    time_t                           response_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_node_t;


@@ -75,6 +77,8 @@
     time_t                           error_sec;
     time_t                           last_modified;
     time_t                           date;
+    time_t                           response_time;
+    time_t                           corrected_initial_age;

     ngx_str_t                        etag;
     ngx_str_t                        vary;
diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:41 2024 +0900
@@ -971,6 +971,8 @@
     fcn->uniq = 0;
     fcn->body_start = 0;
     fcn->fs_size = 0;
+    fcn->response_time = 0;
+    fcn->corrected_initial_age = 0;

 done:

@@ -980,6 +982,8 @@

     c->uniq = fcn->uniq;
     c->error = fcn->error;
+    c->response_time = fcn->response_time;
+    c->corrected_initial_age = fcn->corrected_initial_age;
     c->node = fcn;

 failed:
@@ -1624,6 +1628,7 @@
 ngx_int_t
 ngx_http_cache_send(ngx_http_request_t *r)
 {
+    time_t             resident_time, current_age;
     ngx_int_t          rc;
     ngx_buf_t         *b;
     ngx_chain_t        out;
@@ -1646,6 +1651,17 @@
         return NGX_HTTP_INTERNAL_SERVER_ERROR;
     }

+    /*
+     * Update age response header.
+     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+     */
+    resident_time = ngx_time() - c->response_time;
+    current_age = c->corrected_initial_age + resident_time;
+    r->headers_out.age_n = current_age;
+    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                   "http file cache send, resp:%O, resident:%d, age:%d",
+                   c->response_time, resident_time, current_age);
+
     rc = ngx_http_send_header(r);

     if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c  Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_header_filter_module.c  Thu Jun 20 20:26:41 2024 +0900
@@ -322,6 +322,10 @@
         len += sizeof("Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT" CRLF) - 1;
     }

+    if (r->headers_out.age_n != -1) {
+        len += sizeof("Age: ") - 1 + NGX_OFF_T_LEN + 2;
+    }
+
     c = r->connection;

     if (r->headers_out.location
@@ -518,6 +522,10 @@
         *b->last++ = CR; *b->last++ = LF;
     }

+    if (r->headers_out.age_n != -1) {
+        b->last = ngx_sprintf(b->last, "Age: %O" CRLF, r->headers_out.age_n);
+    }
+
     if (host.data) {

         p = b->last + sizeof("Location: ") - 1;
diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c       Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_request.c       Thu Jun 20 20:26:41 2024 +0900
@@ -646,6 +646,7 @@
     r->headers_in.keep_alive_n = -1;
     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;

     r->uri_changes = NGX_HTTP_MAX_URI_CHANGES + 1;
     r->subrequests = NGX_HTTP_MAX_SUBREQUESTS + 1;
diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_request.h
--- a/src/http/ngx_http_request.h       Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_request.h       Thu Jun 20 20:26:41 2024 +0900
@@ -290,6 +290,7 @@
     off_t                             content_offset;
     time_t                            date_time;
     time_t                            last_modified_time;
+    off_t                             age_n;
 } ngx_http_headers_out_t;


diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_special_response.c
--- a/src/http/ngx_http_special_response.c      Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_special_response.c      Thu Jun 20 20:26:41 2024 +0900
@@ -581,6 +581,7 @@

     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;
 }


diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:41 2024 +0900
@@ -50,6 +50,8 @@
     ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_test_next(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
+static void ngx_http_upstream_update_age(ngx_http_request_t *r,
+    ngx_http_upstream_t *u, time_t now);
 static ngx_int_t ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_test_connect(ngx_connection_t *c);
@@ -132,6 +134,8 @@
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_process_vary(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
+static ngx_int_t ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_copy_header_line(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t
@@ -319,6 +323,10 @@
                  ngx_http_upstream_copy_header_line,
                  offsetof(ngx_http_headers_out_t, content_encoding), 0 },

+    { ngx_string("Age"),
+                 ngx_http_upstream_process_age, 0,
+                 ngx_http_upstream_ignore_header_line, 0, 0 },
+
     { ngx_null_string, NULL, 0, NULL, 0, 0 }
 };

@@ -499,6 +507,7 @@

     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     return NGX_OK;
 }
@@ -1068,6 +1077,7 @@
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -1549,6 +1559,7 @@
     ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));

     u->start_time = ngx_current_msec;
+    u->request_time = ngx_time();

     u->state->response_time = (ngx_msec_t) -1;
     u->state->connect_time = (ngx_msec_t) -1;
@@ -2008,6 +2019,7 @@
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -2529,6 +2541,8 @@
         return;
     }

+    ngx_http_upstream_update_age(r, u, ngx_time());
+
     ngx_http_upstream_send_response(r, u);
 }

@@ -2615,6 +2629,7 @@
                        "http upstream not modified");

         now = ngx_time();
+        ngx_http_upstream_update_age(r, u, now);

         valid = r->cache->valid_sec;
         updating = r->cache->updating_sec;
@@ -2648,7 +2663,12 @@
             valid = ngx_http_file_cache_valid(u->conf->cache_valid,
                                               u->headers_in.status_n);
             if (valid) {
-                valid = now + valid;
+                ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                               "adjust cache valid_sec:%O, "
+                               "valid:%O, init_age:%d for 304",
+                               now + valid - r->cache->corrected_initial_age,
+                               valid, r->cache->corrected_initial_age);
+                valid = now + valid - r->cache->corrected_initial_age;
             }
         }

@@ -2672,6 +2692,59 @@
 }


+static void
+ngx_http_upstream_update_age(ngx_http_request_t *r, ngx_http_upstream_t *u,
+    time_t now)
+{
+    time_t  response_time, date, apparent_age, response_delay, age_value,
+            corrected_age_value, corrected_initial_age;
+
+    /*
+     * Update age response header.
+     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+     */
+    response_time = now;
+    if (u->headers_in.date != NULL) {
+        date = ngx_parse_http_time(u->headers_in.date->value.data,
+                                   u->headers_in.date->value.len);
+        if (date == NGX_ERROR) {
+            date = now;
+        }
+    } else {
+        date = now;
+    }
+    apparent_age = ngx_max(0, response_time - date);
+
+    response_delay = response_time - u->request_time;
+    age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
+    corrected_age_value = age_value + response_delay;
+
+    corrected_initial_age = ngx_max(apparent_age, corrected_age_value);
+    r->headers_out.age_n = corrected_initial_age;
+
+    ngx_log_debug8(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                   "http upstream set age:%O, req:%O, resp:%O, date:%O, "
+                   "a_age:%O, resp_delay:%O, u_age:%O, c_age:%O",
+                   corrected_initial_age, u->request_time, response_time, date,
+                   apparent_age, response_delay, u->headers_in.age_n,
+                   corrected_age_value);
+
+#if (NGX_HTTP_CACHE)
+    if (r->cache) {
+        r->cache->response_time = response_time;
+        r->cache->corrected_initial_age = corrected_initial_age;
+        if (u->headers_in.adjusting_valid_sec) {
+            r->cache->valid_sec -= corrected_initial_age;
+            ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                           "http upstream adjusted cache "
+                           "valid_sec:%O, init_age:%O",
+                           r->cache->valid_sec, corrected_initial_age);
+        }
+    }
+#endif
+}
+
+
 static ngx_int_t
 ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
     ngx_http_upstream_t *u)
@@ -2747,6 +2820,7 @@
                                                           status);
                         if (valid) {
                             r->cache->valid_sec = ngx_time() + valid;
+                            u->headers_in.adjusting_valid_sec = 1;
                         }
                     }

@@ -2952,6 +3026,7 @@
     r->headers_out.status_line = u->headers_in.status_line;

     r->headers_out.content_length_n = u->headers_in.content_length_n;
+    r->headers_out.age_n = u->headers_in.age_n;

     r->disable_not_modified = !u->cacheable;

@@ -4616,6 +4691,7 @@

                 if (valid) {
                     r->cache->valid_sec = ngx_time() + valid;
+                    u->headers_in.adjusting_valid_sec = 1;
                     r->cache->error = rc;
                 }
             }
@@ -4891,6 +4967,7 @@
         }

         r->cache->valid_sec = ngx_time() + n;
+        u->headers_in.adjusting_valid_sec = 1;
         u->headers_in.expired = 0;
     }

@@ -5053,6 +5130,7 @@

         default:
             r->cache->valid_sec = ngx_time() + n;
+            u->headers_in.adjusting_valid_sec = 1;
             u->headers_in.no_cache = 0;
             u->headers_in.expired = 0;
             return NGX_OK;
@@ -5319,6 +5397,39 @@


 static ngx_int_t
+ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset)
+{
+    ngx_http_upstream_t  *u;
+
+    u = r->upstream;
+
+    if (u->headers_in.age) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "upstream sent duplicate header line: "%V: %V", "
+                      "previous value: "%V: %V"",
+                      &h->key, &h->value,
+                      &u->headers_in.age->key,
+                      &u->headers_in.age->value);
+        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
+    }
+
+    h->next = NULL;
+    u->headers_in.age = h;
+    u->headers_in.age_n = ngx_atoof(h->value.data, h->value.len);
+
+    if (u->headers_in.age_n == NGX_ERROR) {
+        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                      "upstream sent invalid "Age" header: "
+                      ""%V: %V"", &h->key, &h->value);
+        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
 ngx_http_upstream_copy_header_line(ngx_http_request_t *r, ngx_table_elt_t *h,
     ngx_uint_t offset)
 {
diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:41 2024 +0900
@@ -287,14 +287,17 @@

     ngx_table_elt_t                 *cache_control;
     ngx_table_elt_t                 *set_cookie;
+    ngx_table_elt_t                 *age;

     off_t                            content_length_n;
     time_t                           last_modified_time;
+    off_t                            age_n;

     unsigned                         connection_close:1;
     unsigned                         chunked:1;
     unsigned                         no_cache:1;
     unsigned                         expired:1;
+    unsigned                         adjusting_valid_sec:1;
 } ngx_http_upstream_headers_in_t;


@@ -369,6 +372,7 @@
                                          ngx_table_elt_t *h);

     ngx_msec_t                       start_time;
+    time_t                           request_time;

     ngx_http_upstream_state_t       *state;

diff -r 73929a4f3447 -r c81df54e3d03 src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/v2/ngx_http_v2.h Thu Jun 20 20:26:41 2024 +0900
@@ -398,6 +398,7 @@
 #define NGX_HTTP_V2_STATUS_404_INDEX      13
 #define NGX_HTTP_V2_STATUS_500_INDEX      14

+#define NGX_HTTP_V2_AGE_INDEX             21
 #define NGX_HTTP_V2_CONTENT_LENGTH_INDEX  28
 #define NGX_HTTP_V2_CONTENT_TYPE_INDEX    31
 #define NGX_HTTP_V2_DATE_INDEX            33
diff -r 73929a4f3447 -r c81df54e3d03 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c   Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/v2/ngx_http_v2_filter_module.c   Thu Jun 20 20:26:41 2024 +0900
@@ -258,6 +258,10 @@
         len += 1 + ngx_http_v2_literal_size("Wed, 31 Dec 1986 18:00:00 GMT");
     }

+    if (r->headers_out.age_n != -1) {
+        len += 1 + ngx_http_v2_integer_octets(NGX_OFF_T_LEN) + NGX_OFF_T_LEN;
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -552,6 +556,18 @@
         pos = ngx_http_v2_write_value(pos, pos, len, tmp);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                       "http2 output header: "age: %O"",
+                       r->headers_out.age_n);
+
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_AGE_INDEX);
+
+        p = pos;
+        pos = ngx_sprintf(pos + 1, "%O", r->headers_out.age_n);
+        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (pos - p - 1);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
                        "http2 output header: "location: %V"",
diff -r 73929a4f3447 -r c81df54e3d03 src/http/v3/ngx_http_v3_filter_module.c
--- a/src/http/v3/ngx_http_v3_filter_module.c   Thu Jun 20 20:26:24 2024 +0900
+++ b/src/http/v3/ngx_http_v3_filter_module.c   Thu Jun 20 20:26:41 2024 +0900
@@ -13,6 +13,7 @@
 /* static table indices */
 #define NGX_HTTP_V3_HEADER_AUTHORITY                 0
 #define NGX_HTTP_V3_HEADER_PATH_ROOT                 1
+#define NGX_HTTP_V3_HEADER_AGE_ZERO                  2
 #define NGX_HTTP_V3_HEADER_CONTENT_LENGTH_ZERO       4
 #define NGX_HTTP_V3_HEADER_DATE                      6
 #define NGX_HTTP_V3_HEADER_LAST_MODIFIED             10
@@ -213,6 +214,15 @@
                                   sizeof("Mon, 28 Sep 1970 06:00:00 GMT") - 1);
     }

+    if (r->headers_out.age_n > 0) {
+        len += ngx_http_v3_encode_field_lri(NULL, 0,
+                                            NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                            NULL, NGX_OFF_T_LEN);
+    } else if (r->headers_out.age_n == 0) {
+        len += ngx_http_v3_encode_field_ri(NULL, 0,
+                                           NGX_HTTP_V3_HEADER_AGE_ZERO);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -452,6 +462,27 @@
                                               p, n);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
+                       "http3 output header: "age: %O"",
+                       r->headers_out.age_n);
+
+        if (r->headers_out.age_n > 0) {
+            p = ngx_sprintf(b->last, "%O", r->headers_out.age_n);
+            n = p - b->last;
+
+            b->last = (u_char *) ngx_http_v3_encode_field_lri(b->last, 0,
+                                                   NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                                   NULL, n);
+
+            b->last = ngx_sprintf(b->last, "%O", r->headers_out.age_n);
+
+        } else {
+            b->last = (u_char *) ngx_http_v3_encode_field_ri(b->last, 0,
+                                                  NGX_HTTP_V3_HEADER_AGE_ZERO);
+        }
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
                        "http3 output header: "location: %V"",


From hnakamur at gmail.com  Thu Jun 20 11:39:51 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Thu, 20 Jun 2024 20:39:51 +0900
Subject: [PATCH 2 of 3] Save response time and corrected initial age to file
 cache header
Message-ID: <CAN-DUMRnsszDqfpc-fVsrrN19W0_Dmuweqoox-3_A+-Xq_AX+A@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1718882809 -32400
#      Thu Jun 20 20:26:49 2024 +0900
# Branch correct_age
# Node ID 56187d3f24d45b895f6ddb86495cbaf098540107
# Parent  c81df54e3d0333c26d4296792dc0df767b386f91
Save response time and corrected initial age to file cache header.

diff -r c81df54e3d03 -r 56187d3f24d4 src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h Thu Jun 20 20:26:41 2024 +0900
+++ b/src/http/ngx_http_cache.h Thu Jun 20 20:26:49 2024 +0900
@@ -27,7 +27,7 @@
 #define NGX_HTTP_CACHE_ETAG_LEN      128
 #define NGX_HTTP_CACHE_VARY_LEN      128

-#define NGX_HTTP_CACHE_VERSION       5
+#define NGX_HTTP_CACHE_VERSION       6


 typedef struct {
@@ -145,6 +145,8 @@
     u_char                           vary_len;
     u_char                           vary[NGX_HTTP_CACHE_VARY_LEN];
     u_char                           variant[NGX_HTTP_CACHE_KEY_LEN];
+    time_t                           response_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_header_t;


diff -r c81df54e3d03 -r 56187d3f24d4 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:41 2024 +0900
+++ b/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:49 2024 +0900
@@ -627,6 +627,8 @@
     c->body_start = h->body_start;
     c->etag.len = h->etag_len;
     c->etag.data = h->etag;
+    c->response_time = h->response_time;
+    c->corrected_initial_age = h->corrected_initial_age;

     r->cached = 1;

@@ -1330,6 +1332,8 @@
     h->valid_msec = (u_short) c->valid_msec;
     h->header_start = (u_short) c->header_start;
     h->body_start = (u_short) c->body_start;
+    h->response_time = c->response_time;
+    h->corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h->etag_len = (u_char) c->etag.len;
@@ -1594,6 +1598,8 @@
     h.valid_msec = (u_short) c->valid_msec;
     h.header_start = (u_short) c->header_start;
     h.body_start = (u_short) c->body_start;
+    h.response_time = c->response_time;
+    h.corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h.etag_len = (u_char) c->etag.len;


From hnakamur at gmail.com  Thu Jun 20 11:39:57 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Thu, 20 Jun 2024 20:39:57 +0900
Subject: [PATCH 3 of 3] Tests: Update and add tests for Age header
Message-ID: <CAN-DUMSUBKmhKQ3xeB8Vqq_kmO+TzcvuYZPcux5NAYbQG8+_aA@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1718882815 -32400
#      Thu Jun 20 20:26:55 2024 +0900
# Branch correct_age
# Node ID 6306793c01e9688994134e03175aad51def359cf
# Parent  56187d3f24d45b895f6ddb86495cbaf098540107
Tests: Update and add tests for Age header.

diff -r 56187d3f24d4 -r 6306793c01e9 h2_proxy_cache_age.t
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/h2_proxy_cache_age.t      Thu Jun 20 20:26:55 2024 +0900
@@ -0,0 +1,198 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/2 proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP2;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http http_v2 proxy cache/)->plan(8)
+       ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
+    proxy_cache_path %%TESTDIR%%/cache2   keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:8080 http2;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8081;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t.html', 'SEE-THIS');
+
+# suppress deprecation warning
+
+open OLDERR, ">&", *STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", *OLDERR;
+
+###############################################################################
+
+my $s = Test::Nginx::HTTP2->new();
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age updated');
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+# slow origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=6&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 6, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 5, 'slow origin updated');
+
+}
+
+# update age after restart
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+open OLDERR, ">&", *STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", *OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = Test::Nginx::HTTP2->new();
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r 56187d3f24d4 -r 6306793c01e9 h2_ssl_proxy_cache_age.t
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/h2_ssl_proxy_cache_age.t  Thu Jun 20 20:26:55 2024 +0900
@@ -0,0 +1,243 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/2 ssl proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP2;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()
+       ->has(qw/http http_ssl http_v2 proxy cache socket_ssl/)->plan(10)
+       ->has_daemon('openssl');
+
+$t->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache   keys_zone=NAME:1m;
+    proxy_cache_path %%TESTDIR%%/cache2  keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:8080 http2 ssl;
+        server_name  localhost;
+
+        ssl_certificate_key localhost.key;
+        ssl_certificate localhost.crt;
+
+        location / {
+            proxy_pass   http://127.0.0.1:8081;
+            proxy_cache  NAME;
+            proxy_http_version 1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('openssl.conf', <<EOF);
+[ req ]
+default_bits = 2048
+encrypt_key = no
+distinguished_name = req_distinguished_name
+[ req_distinguished_name ]
+EOF
+
+my $d = $t->testdir();
+
+foreach my $name ('localhost') {
+       system('openssl req -x509 -new '
+               . "-config $d/openssl.conf -subj /CN=$name/ "
+               . "-out $d/$name.crt -keyout $d/$name.key "
+               . ">>$d/openssl.out 2>&1") == 0
+               or die "Can't create certificate for $name: $!
";
+}
+
+$t->write_file('t.html', 'SEE-THIS');
+
+open OLDERR, ">&", *STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", *OLDERR;
+
+###############################################################################
+
+my $s = getconn(port(8080));
+ok($s, 'ssl connection');
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age updated');
+
+# slow origin
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=6&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 6, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 5, 'slow origin updated');
+
+}
+
+# update age after restart
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+open OLDERR, ">&", *STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", *OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = getconn(port(8080));
+ok($s, 'ssl connection');
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+sub getconn {
+       my ($port) = @_;
+       my $s;
+
+       eval {
+               my $sock = Test::Nginx::HTTP2::new_socket($port, SSL => 1,
+                       alpn => 'h2');
+               $s = Test::Nginx::HTTP2->new($port, socket => $sock)
+                       if $sock->alpn_selected();
+       };
+
+       return $s if defined $s;
+
+       eval {
+               my $sock = Test::Nginx::HTTP2::new_socket($port, SSL => 1,
+                       npn => 'h2');
+               $s = Test::Nginx::HTTP2->new($port, socket => $sock)
+                       if $sock->next_proto_negotiated();
+       };
+
+       return $s;
+}
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r 56187d3f24d4 -r 6306793c01e9 h3_proxy_cache_age.t
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/h3_proxy_cache_age.t      Thu Jun 20 20:26:55 2024 +0900
@@ -0,0 +1,210 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/3 proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP3;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http http_v3 proxy cryptx/)
+       ->has_daemon('openssl')->plan(8)
+       ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    ssl_certificate_key localhost.key;
+    ssl_certificate localhost.crt;
+
+    log_format test $uri:$status:$request_completion;
+
+    proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:%%PORT_8980_UDP%% quic;
+        server_name  localhost;
+
+        location / {
+            proxy_pass http://127.0.0.1:8081/;
+            proxy_cache NAME;
+            proxy_http_version 1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('openssl.conf', <<EOF);
+[ req ]
+default_bits = 2048
+encrypt_key = no
+distinguished_name = req_distinguished_name
+[ req_distinguished_name ]
+EOF
+
+my $d = $t->testdir();
+
+foreach my $name ('localhost') {
+       system('openssl req -x509 -new '
+               . "-config $d/openssl.conf -subj /CN=$name/ "
+               . "-out $d/$name.crt -keyout $d/$name.key "
+               . ">>$d/openssl.out 2>&1") == 0
+               or die "Can't create certificate for $name: $!
";
+}
+
+my $content = 'SEE-THIS';
+$t->write_file('t.html', $content);
+$t->run();
+
+###############################################################################
+
+my $s = Test::Nginx::HTTP3->new();
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age updated');
+
+# slow origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=6&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 6, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 5, 'slow origin updated');
+
+# update age after restart
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 0, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+open OLDERR, ">&", *STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", *OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = Test::Nginx::HTTP3->new();
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r 56187d3f24d4 -r 6306793c01e9 proxy_cache_age.t
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/proxy_cache_age.t Thu Jun 20 20:26:55 2024 +0900
@@ -0,0 +1,176 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+# (C) Hiroaki Nakamura
+
+# Tests for age in http proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+use Socket qw/ CRLF /;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy cache/)->plan(8)
+       ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path   %%TESTDIR%%/cache  levels=1:2
+                       keys_zone=NAME:1m;
+    proxy_cache_path   %%TESTDIR%%/cache2  levels=1:2
+                       keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       100;
+    }
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8081;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+            add_header parent_date $upstream_http_date;
+            add_header child_msec $msec;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+            add_header origin_date $upstream_http_date;
+            add_header parent_msec $msec;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+
+        location / {
+            add_header Cache-Control $http_x_cache_control;
+            limit_rate $rate;
+            add_header origin_msec $msec;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t.html', 'SEE-THIS');
+$t->write_file('t2.html', 'SEE-THIS');
+$t->write_file('t3.html', 'SEE-THIS');
+
+$t->run();
+
+###############################################################################
+
+# normal origin
+
+wait_until_next_second();
+
+like(get('/t.html', 's-maxage=2'), qr/
Age: 0
/, 'age first');
+
+sleep 2;
+
+like(get('/t.html', 's-maxage=2'), qr/
Age: 2
/, 'age hit');
+
+sleep 1;
+
+like(http_get('/t.html'), qr/
Age: 0
/, 'age updated');
+
+# slow origin
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+wait_until_next_second();
+
+like(get('/t2.html?slow=1', 's-maxage=6'), qr/
Age: 4
/,
+    'slow origin first');
+
+sleep 2;
+
+like(http_get('/t2.html?slow=1'), qr/
Age: 6
/, 'slow origin hit');
+
+sleep 1;
+
+like(http_get('/t2.html?slow=1'), qr/
Age: 5
/, 'slow origin updated');
+
+}
+
+# update age after restart
+
+wait_until_next_second();
+
+like(get('/t3.html', 's-maxage=20'), qr/
Age: 0
/, 'age before restart');
+my $t1 = time();
+
+$t->stop();
+
+$t->run();
+
+my $resident_time = time() - $t1;
+like(http_get('/t3.html'), qr/
Age: $resident_time
/,
+    'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+sub get {
+       my ($url, $extra, %extra) = @_;
+       return http(<<EOF, %extra);
+GET $url HTTP/1.1
+Host: localhost
+Connection: close
+X-Cache-Control: $extra
+
+EOF
+}
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r 56187d3f24d4 -r 6306793c01e9 proxy_cache_use_stale.t
--- a/proxy_cache_use_stale.t   Thu Jun 20 20:26:49 2024 +0900
+++ b/proxy_cache_use_stale.t   Thu Jun 20 20:26:55 2024 +0900
@@ -206,7 +206,8 @@

 $t->write_file('t6.html', 'SEE-THAT');

-my $s = get('/t6.html', 'max-age=1, stale-while-revalidate=2', start => 1);
+# max-age must be 5 here since response delay is 4 seconds.
+my $s = get('/t6.html', 'max-age=5, stale-while-revalidate=2', start => 1);
 select undef, undef, undef, 0.2;
 like(http_get('/t6.html'), qr/UPDATING.*SEE-THIS/s, 's-w-r - updating');
 like(http_end($s), qr/STALE.*SEE-THIS/s, 's-w-r - updating stale');


From mdounin at mdounin.ru  Thu Jun 20 18:13:38 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Thu, 20 Jun 2024 21:13:38 +0300
Subject: [nginx-tests] Tests: adjusted sub_filter_multi.t limit rate sett...
Message-ID: <hg.b5c1c3ef2345.1718907218.8382647654794956330@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/b5c1c3ef2345
branches:  
changeset: 1988:b5c1c3ef2345
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Thu Jun 20 21:12:00 2024 +0300
description:
Tests: adjusted sub_filter_multi.t limit rate settings.

With previous settings some tests with short buffers, which are under
TEST_NGINX_UNSAFE, used to hit 8 seconds timeout in http_get(), leading
to test failures.

Reported by Hiroaki Nakamura,
https://freenginx.org/pipermail/nginx-devel/2024-June/000373.html

diffstat:

 sub_filter_multi.t |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/sub_filter_multi.t b/sub_filter_multi.t
--- a/sub_filter_multi.t
+++ b/sub_filter_multi.t
@@ -244,7 +244,7 @@ http {
         listen       127.0.0.1:8081;
 
         limit_rate 4;
-        limit_rate_after 160;
+        limit_rate_after 170;
 
         location / {
             return 200 $arg_a;


From mdounin at mdounin.ru  Thu Jun 20 18:14:18 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 20 Jun 2024 21:14:18 +0300
Subject: [PATCH] [nginx-tests] Tests: Add sleep to sub_filter_multi.t for
 TEST_NGINX_UNSAFE=1
In-Reply-To: <CAN-DUMTGOtU4m-D+s+rx3uShyxtdn_H-ur24jwHG_2_fs8NJCw@mail.gmail.com>
References: <CAN-DUMS8CKM6iVqTUguOyQK2oaCrN0CQLrYSLsNjnNVYZLzskQ@mail.gmail.com>
 <ZnN7oMAYD4a6J88J@mdounin.ru>
 <CAN-DUMTGOtU4m-D+s+rx3uShyxtdn_H-ur24jwHG_2_fs8NJCw@mail.gmail.com>
Message-ID: <ZnRxesGe6kWn5-5v@mdounin.ru>

Hello!

On Thu, Jun 20, 2024 at 12:45:15PM +0900, Hiroaki Nakamura wrote:

> Thank you for your thorough explanation.
> I have confirmed that your patch works for me.

Committed, thanks for checking.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Fri Jun 21 01:11:00 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Fri, 21 Jun 2024 04:11:00 +0300
Subject: [PATCH] Upstream: fixed proxy_no_cache when caching errors
Message-ID: <0ba8dda4d6833d566b8e.1718932260@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718929130 -10800
#      Fri Jun 21 03:18:50 2024 +0300
# Node ID 0ba8dda4d6833d566b8e429f5f14d3a9ac2a07d5
# Parent  ea0eef2dd12c2d41349d63c532e942cf95fc4d7b
Upstream: fixed proxy_no_cache when caching errors.

Caching errors, notably intercepted errors and internally generated
502/504 errors, as well as handling of cache revalidation with 304,
did not take into account u->conf->no_cache predicates configured.
As a result, an error might be cached even if configuration explicitly
says not to.  Fix is to check u->conf->no_cache in these cases.

To simplify usage in multiple places, checking u->conf->no_cache is now
done in a separate function.  As a minor optimization, u->conf->no_cache
is only checked if u->cacheable is set.

As a side effect, this change also fixes caching errors after
proxy_cache_bypass.  Also, during cache revalidation u->cacheable is
now tested, so 304 responses which disable caching won't extend
cacheability of stored responses.

Additionally, when caching internally generated 502/504 errors
u->cacheable is now explicitly updated from u->headers_in.no_cache and
u->headers_in.expired, restoring the behaviour before 8041:0784ab86ad08
(1.23.0) when an error happens while reading the response headers.

Reported by Kirill A. Korinsky,
https://freenginx.org/pipermail/nginx/2024-April/000082.html

diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -21,6 +21,8 @@ static ngx_int_t ngx_http_upstream_cache
     ngx_http_request_t *r, ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_cache_check_range(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
+static ngx_int_t ngx_http_upstream_no_cache(ngx_http_request_t *r,
+    ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_cache_status(ngx_http_request_t *r,
     ngx_http_variable_value_t *v, uintptr_t data);
 static ngx_int_t ngx_http_upstream_cache_key(ngx_http_request_t *r,
@@ -2632,6 +2634,12 @@ ngx_http_upstream_test_next(ngx_http_req
         updating = r->cache->updating_sec;
         error = r->cache->error_sec;
 
+        if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+            ngx_http_upstream_finalize_request(r, u,
+                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
+            return NGX_OK;
+        }
+
         rc = u->reinit_request(r);
 
         if (rc != NGX_OK) {
@@ -2650,30 +2658,33 @@ ngx_http_upstream_test_next(ngx_http_req
             rc = NGX_HTTP_INTERNAL_SERVER_ERROR;
         }
 
-        if (valid == 0) {
-            valid = r->cache->valid_sec;
-            updating = r->cache->updating_sec;
-            error = r->cache->error_sec;
-        }
-
-        if (valid == 0) {
-            valid = ngx_http_file_cache_valid(u->conf->cache_valid,
-                                              u->headers_in.status_n);
+        if (u->cacheable) {
+
+            if (valid == 0) {
+                valid = r->cache->valid_sec;
+                updating = r->cache->updating_sec;
+                error = r->cache->error_sec;
+            }
+
+            if (valid == 0) {
+                valid = ngx_http_file_cache_valid(u->conf->cache_valid,
+                                                  u->headers_in.status_n);
+                if (valid) {
+                    valid = now + valid;
+                }
+            }
+
             if (valid) {
-                valid = now + valid;
+                r->cache->valid_sec = valid;
+                r->cache->updating_sec = updating;
+                r->cache->error_sec = error;
+
+                r->cache->date = now;
+
+                ngx_http_file_cache_update_header(r);
             }
         }
 
-        if (valid) {
-            r->cache->valid_sec = valid;
-            r->cache->updating_sec = updating;
-            r->cache->error_sec = error;
-
-            r->cache->date = now;
-
-            ngx_http_file_cache_update_header(r);
-        }
-
         ngx_http_upstream_finalize_request(r, u, rc);
         return NGX_OK;
     }
@@ -2745,8 +2756,10 @@ ngx_http_upstream_intercept_errors(ngx_h
 
             if (r->cache) {
 
-                if (u->headers_in.no_cache || u->headers_in.expired) {
-                    u->cacheable = 0;
+                if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+                    ngx_http_upstream_finalize_request(r, u,
+                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
+                    return NGX_OK;
                 }
 
                 if (u->cacheable) {
@@ -3159,29 +3172,8 @@ ngx_http_upstream_send_response(ngx_http
         r->cache->file.fd = NGX_INVALID_FILE;
     }
 
-    switch (ngx_http_test_predicates(r, u->conf->no_cache)) {
-
-    case NGX_ERROR:
+    if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
         ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
-        return;
-
-    case NGX_DECLINED:
-        u->cacheable = 0;
-        break;
-
-    default: /* NGX_OK */
-
-        if (u->cache_status == NGX_HTTP_CACHE_BYPASS) {
-
-            /* create cache if previously bypassed */
-
-            if (ngx_http_file_cache_create(r) != NGX_OK) {
-                ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
-                return;
-            }
-        }
-
-        break;
     }
 
     if (u->cacheable) {
@@ -3372,6 +3364,50 @@ ngx_http_upstream_send_response(ngx_http
 }
 
 
+#if (NGX_HTTP_CACHE)
+
+static ngx_int_t
+ngx_http_upstream_no_cache(ngx_http_request_t *r, ngx_http_upstream_t *u)
+{
+    ngx_int_t  rc;
+
+    if (!u->cacheable) {
+        return NGX_OK;
+    }
+
+    if (u->headers_in.no_cache || u->headers_in.expired) {
+        u->cacheable = 0;
+        return NGX_OK;
+    }
+
+    rc = ngx_http_test_predicates(r, u->conf->no_cache);
+
+    if (rc == NGX_ERROR) {
+        return NGX_ERROR;
+    }
+
+    if (rc == NGX_DECLINED) {
+        u->cacheable = 0;
+        return NGX_OK;
+    }
+
+    /* rc == NGX_OK */
+
+    if (u->cache_status == NGX_HTTP_CACHE_BYPASS) {
+
+        /* create cache if previously bypassed */
+
+        if (ngx_http_file_cache_create(r) != NGX_OK) {
+            return NGX_ERROR;
+        }
+    }
+
+    return NGX_OK;
+}
+
+#endif
+
+
 static void
 ngx_http_upstream_upgrade(ngx_http_request_t *r, ngx_http_upstream_t *u)
 {
@@ -4619,9 +4655,15 @@ ngx_http_upstream_finalize_request(ngx_h
 
     if (r->cache) {
 
-        if (u->cacheable) {
-
-            if (rc == NGX_HTTP_BAD_GATEWAY || rc == NGX_HTTP_GATEWAY_TIME_OUT) {
+        if (rc == NGX_HTTP_BAD_GATEWAY || rc == NGX_HTTP_GATEWAY_TIME_OUT) {
+
+            if (!u->header_sent) {
+                if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+                    u->cacheable = 0;
+                }
+            }
+
+            if (u->cacheable) {
                 time_t  valid;
 
                 valid = ngx_http_file_cache_valid(u->conf->cache_valid, rc);



From mdounin at mdounin.ru  Fri Jun 21 01:14:05 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Fri, 21 Jun 2024 04:14:05 +0300
Subject: [PATCH] Tests: proxy_no_cache tests
In-Reply-To: <0ba8dda4d6833d566b8e.1718932260@vm-bsd.mdounin.ru>
References: <0ba8dda4d6833d566b8e.1718932260@vm-bsd.mdounin.ru>
Message-ID: <69d7d71cee53633a3257.1718932445@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1718932097 -10800
#      Fri Jun 21 04:08:17 2024 +0300
# Node ID 69d7d71cee53633a3257fe7ecf1838a9ca882499
# Parent  b5c1c3ef234570408fdbb79343b40b063bc3b83b
Tests: proxy_no_cache tests.

diff --git a/proxy_cache_bypass.t b/proxy_cache_bypass.t
--- a/proxy_cache_bypass.t
+++ b/proxy_cache_bypass.t
@@ -21,7 +21,7 @@ use Test::Nginx;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(8)
+my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(12)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -65,12 +65,17 @@ http {
 
         location / {
         }
+
+        location /t3 {
+            add_header Transfer-Encoding $arg_bypass;
+        }
     }
 }
 
 EOF
 
 $t->write_file('t', 'SEE-THIS');
+$t->write_file('t3', 'SEE-THIS');
 
 $t->run();
 
@@ -82,6 +87,9 @@ like(http_get('/t'), qr/SEE-THIS/, 'requ
 
 like(http_get('/t'), qr/SEE-THIS/, 'request cached');
 like(http_get('/t?bypass=1'), qr/NOOP/, 'cache bypassed');
+
+unlink $t->testdir() . '/t';
+
 like(http_get('/t'), qr/NOOP/, 'cached after bypass');
 
 # ticket #827, cache item "error" field was not cleared
@@ -93,6 +101,33 @@ like(http_get('/t2'), qr/403 Forbidden/,
 
 like(http_get('/t2'), qr/403 Forbidden/, 'error cached');
 like(http_get('/t2?bypass=1'), qr/NOOP/, 'error cache bypassed');
-like(http_get('/t2'), qr/NOOP/, 'error cached after bypass');
+
+unlink $t->testdir() . '/t2';
+
+like(http_get('/t2'), qr/NOOP/, 'file cached after bypass');
+
+# make sure the error is cached after bypass
+
+like(http_get('/t2?bypass=1'), qr/403 Forbidden/, 'file cache bypassed');
+
+$t->write_file('t2', 'NOOP');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t2'), qr/403 Forbidden/, 'error cached again');
+
+}
+
+# similarly, internal 502/504 is cached after bypass
+
+like(http_get('/t3?bypass=1'), qr/502 Bad/, 'internal 502');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t3'), qr/502 Bad/, 'internal 502 cached');
+
+}
 
 ###############################################################################
diff --git a/proxy_no_cache.t b/proxy_no_cache.t
new file mode 100644
--- /dev/null
+++ b/proxy_no_cache.t
@@ -0,0 +1,174 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+
+# Tests for http proxy cache, proxy_no_cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(16)
+	->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache keys_zone=one:1m;
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location / {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1y;
+            proxy_no_cache $arg_nocache;
+
+            proxy_intercept_errors on;
+            error_page 404 = @fallback;
+        }
+
+        location /t3 {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1y;
+            proxy_no_cache $arg_nocache;
+        }
+
+        location /t4 {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1s;
+            proxy_no_cache $upstream_http_x_no_cache;
+
+            proxy_cache_revalidate on;
+        }
+
+        location @fallback {
+            return 403;
+        }
+
+        add_header X-Cache-Status $upstream_cache_status always;
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+        }
+
+        location /t3 {
+            set $nocache "";
+            if ($arg_expires) {
+                set $nocache "no-cache";
+            }
+            add_header Cache-Control $nocache;
+            add_header Transfer-Encoding invalid;
+        }
+
+        location /t4 {
+            set $nocache "";
+            if ($arg_expires) {
+                set $nocache "no-cache";
+            }
+            add_header Cache-Control $nocache;
+            add_header X-No-Cache $arg_nocache;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t', 'SEE-THIS');
+$t->write_file('t3', 'SEE-THIS');
+$t->write_file('t4', 'SEE-THIS');
+
+$t->run();
+
+###############################################################################
+
+like(http_get('/t?nocache=1'), qr/MISS.*SEE-THIS/s, 'request');
+like(http_get('/t'), qr/MISS.*SEE-THIS/s, 'request not cached');
+like(http_get('/t'), qr/HIT.*SEE-THIS/s, 'request cached');
+
+# proxy_no_cache with intercepted errors,
+# ngx_http_upstream_intercept_errors()
+
+like(http_get('/t2?nocache=1'), qr/403 Forbidden/, 'intercepted error');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t2'), qr/403 Forbidden.*MISS/s, 'intercepted error not cached');
+
+}
+
+like(http_get('/t2'), qr/403 Forbidden.*HIT/s, 'intercepted error cached');
+
+# proxy_no_cache with internal 502/504 errors,
+# ngx_http_upstream_finalize_request()
+
+like(http_get('/t3?nocache=1'), qr/502 Bad/, 'internal 502 error');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t3?expires=1'), qr/502 Bad.*MISS/s,
+	'internal 502 error expires');
+like(http_get('/t3'), qr/502 Bad.*MISS/s, 'internal 502 error not cached');
+
+}
+
+like(http_get('/t3'), qr/502 Bad.*HIT/s, 'internal 502 error cached');
+
+# proxy_no_cache with revalidate and 304,
+# ngx_http_upstream_test_next()
+
+like(http_get('/t4'), qr/MISS.*SEE-THIS/s, 'revalidate');
+like(http_get('/t4'), qr/HIT.*SEE-THIS/s, 'revalidate cached');
+select undef, undef, undef, 2.5;
+like(http_get('/t4?nocache=1'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate nocache');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t4?expires=1'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate expires');
+like(http_get('/t4'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate again');
+
+}
+
+like(http_get('/t4'), qr/HIT.*SEE-THIS/s, 'revalidate again cached');
+
+###############################################################################



From mdounin at mdounin.ru  Sun Jun 23 23:42:20 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 24 Jun 2024 02:42:20 +0300
Subject: [PATCH 1 of 3] Correctly calculate and set Age header
In-Reply-To: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
Message-ID: <Zniy3EFG005hIsXl@mdounin.ru>

Hello!

On Thu, Jun 20, 2024 at 08:39:43PM +0900, Hiroaki Nakamura wrote:

> # HG changeset patch
> # User Hiroaki Nakamura <hnakamur at gmail.com>
> # Date 1718882801 -32400
> #      Thu Jun 20 20:26:41 2024 +0900
> # Branch correct_age
> # Node ID c81df54e3d0333c26d4296792dc0df767b386f91
> # Parent  73929a4f3447d558747623884b5ba281c13332d8
> Correctly calculate and set Age header.
> Implement the calculation of the Age header as specified in
> "RFC 9111: HTTP Caching"
> https://www.rfc-editor.org/rfc/rfc9111.html

Thanks for the patches.

Note that it might be a good idea to update the commit log to 
follow style rules, such as:

: Cache: added calculation of the Age header.
: 
: Implement the calculation of the Age header as specified in
: RFC 9111 "HTTP Caching", https://www.rfc-editor.org/rfc/rfc9111.html

It also might be also a good idea to add additional implementation 
details which might worth explaining, especially given that RFC 
911 does not really specify how the Age header is to be 
calculated, but rather gives some guidelines on possible 
approaches.

See below for some more comments.

> 
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_cache.h
> --- a/src/http/ngx_http_cache.h Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_cache.h Thu Jun 20 20:26:41 2024 +0900
> @@ -59,6 +59,8 @@
>      size_t                           body_start;
>      off_t                            fs_size;
>      ngx_msec_t                       lock_time;
> +    time_t                           response_time;
> +    time_t                           corrected_initial_age;
>  } ngx_http_file_cache_node_t;
> 
> 
> @@ -75,6 +77,8 @@
>      time_t                           error_sec;
>      time_t                           last_modified;
>      time_t                           date;
> +    time_t                           response_time;

Note that the "response_time" field being added seems to be 
exactly equivalent to the exiting "date" field.

> +    time_t                           corrected_initial_age;
> 
>      ngx_str_t                        etag;
>      ngx_str_t                        vary;
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_file_cache.c
> --- a/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_file_cache.c    Thu Jun 20 20:26:41 2024 +0900
> @@ -971,6 +971,8 @@
>      fcn->uniq = 0;
>      fcn->body_start = 0;
>      fcn->fs_size = 0;
> +    fcn->response_time = 0;
> +    fcn->corrected_initial_age = 0;
> 
>  done:
> 
> @@ -980,6 +982,8 @@

Nitpicking: please consider adding

[diff]
showfunc=1

to your Mercurial configuration to ensure that function names are 
shown in diffs, this makes reviews easier.

> 
>      c->uniq = fcn->uniq;
>      c->error = fcn->error;
> +    c->response_time = fcn->response_time;
> +    c->corrected_initial_age = fcn->corrected_initial_age;
>      c->node = fcn;
> 
>  failed:
> @@ -1624,6 +1628,7 @@
>  ngx_int_t
>  ngx_http_cache_send(ngx_http_request_t *r)
>  {
> +    time_t             resident_time, current_age;
>      ngx_int_t          rc;
>      ngx_buf_t         *b;
>      ngx_chain_t        out;
> @@ -1646,6 +1651,17 @@
>          return NGX_HTTP_INTERNAL_SERVER_ERROR;
>      }
> 
> +    /*
> +     * Update age response header.
> +     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
> +     */
> +    resident_time = ngx_time() - c->response_time;
> +    current_age = c->corrected_initial_age + resident_time;
> +    r->headers_out.age_n = current_age;

Note that this uses data from the cache node, which is not 
available if the cache node was loaded from disk, for example, on 
server restart, and this will certainly lead to incorrect results.  
If I'm reading the code correctly, it will result in seconds since 
the Epoch in the Age header on all responses after a restart.

This suggests that either things should be implemented quite 
differently, or this patch needs to be merged with the second one, 
which implements loading relevant information from the cache 
header.

> +    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> +                   "http file cache send, resp:%O, resident:%d, age:%d",
> +                   c->response_time, resident_time, current_age);

Note that there seems to be somewhat too many debug logging in the 
patch.

Please also note that using "%d" for time_t is wrong, as time_t 
size might be different from the size of int.  For time_t, "%T" 
should be used instead.

See ngx_sprintf() comment in src/core/ngx_string.c for the full 
list of supported formats and corresponding types.

> +
>      rc = ngx_http_send_header(r);
> 
>      if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_header_filter_module.c
> --- a/src/http/ngx_http_header_filter_module.c  Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_header_filter_module.c  Thu Jun 20 20:26:41 2024 +0900
> @@ -322,6 +322,10 @@
>          len += sizeof("Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT" CRLF) - 1;
>      }
> 
> +    if (r->headers_out.age_n != -1) {
> +        len += sizeof("Age: ") - 1 + NGX_OFF_T_LEN + 2;
> +    }
> +
>      c = r->connection;
> 
>      if (r->headers_out.location
> @@ -518,6 +522,10 @@
>          *b->last++ = CR; *b->last++ = LF;
>      }
> 
> +    if (r->headers_out.age_n != -1) {
> +        b->last = ngx_sprintf(b->last, "Age: %O" CRLF, r->headers_out.age_n);
> +    }
> +
>      if (host.data) {
> 
>          p = b->last + sizeof("Location: ") - 1;
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_request.c
> --- a/src/http/ngx_http_request.c       Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_request.c       Thu Jun 20 20:26:41 2024 +0900
> @@ -646,6 +646,7 @@
>      r->headers_in.keep_alive_n = -1;
>      r->headers_out.content_length_n = -1;
>      r->headers_out.last_modified_time = -1;
> +    r->headers_out.age_n = -1;
> 
>      r->uri_changes = NGX_HTTP_MAX_URI_CHANGES + 1;
>      r->subrequests = NGX_HTTP_MAX_SUBREQUESTS + 1;
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_request.h
> --- a/src/http/ngx_http_request.h       Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_request.h       Thu Jun 20 20:26:41 2024 +0900
> @@ -290,6 +290,7 @@
>      off_t                             content_offset;
>      time_t                            date_time;
>      time_t                            last_modified_time;
> +    off_t                             age_n;

Using off_t here looks wrong, as off_t is a type for file offsets, 
and not for dates.

(Not sure it needs to be here at all though, a better solution 
might be to rewrite/add the header within the upstream module.)

>  } ngx_http_headers_out_t;
> 
> 
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_special_response.c
> --- a/src/http/ngx_http_special_response.c      Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_special_response.c      Thu Jun 20 20:26:41 2024 +0900
> @@ -581,6 +581,7 @@
> 
>      r->headers_out.content_length_n = -1;
>      r->headers_out.last_modified_time = -1;
> +    r->headers_out.age_n = -1;
>  }
> 
> 
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.c
> --- a/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:41 2024 +0900
> @@ -50,6 +50,8 @@
>      ngx_http_upstream_t *u);
>  static ngx_int_t ngx_http_upstream_test_next(ngx_http_request_t *r,
>      ngx_http_upstream_t *u);
> +static void ngx_http_upstream_update_age(ngx_http_request_t *r,
> +    ngx_http_upstream_t *u, time_t now);
>  static ngx_int_t ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
>      ngx_http_upstream_t *u);
>  static ngx_int_t ngx_http_upstream_test_connect(ngx_connection_t *c);
> @@ -132,6 +134,8 @@
>      ngx_table_elt_t *h, ngx_uint_t offset);
>  static ngx_int_t ngx_http_upstream_process_vary(ngx_http_request_t *r,
>      ngx_table_elt_t *h, ngx_uint_t offset);
> +static ngx_int_t ngx_http_upstream_process_age(ngx_http_request_t *r,
> +    ngx_table_elt_t *h, ngx_uint_t offset);
>  static ngx_int_t ngx_http_upstream_copy_header_line(ngx_http_request_t *r,
>      ngx_table_elt_t *h, ngx_uint_t offset);
>  static ngx_int_t
> @@ -319,6 +323,10 @@
>                   ngx_http_upstream_copy_header_line,
>                   offsetof(ngx_http_headers_out_t, content_encoding), 0 },
> 
> +    { ngx_string("Age"),
> +                 ngx_http_upstream_process_age, 0,
> +                 ngx_http_upstream_ignore_header_line, 0, 0 },
> +
>      { ngx_null_string, NULL, 0, NULL, 0, 0 }
>  };
> 
> @@ -499,6 +507,7 @@
> 
>      u->headers_in.content_length_n = -1;
>      u->headers_in.last_modified_time = -1;
> +    u->headers_in.age_n = -1;
> 
>      return NGX_OK;
>  }
> @@ -1068,6 +1077,7 @@
>      ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
>      u->headers_in.content_length_n = -1;
>      u->headers_in.last_modified_time = -1;
> +    u->headers_in.age_n = -1;
> 
>      if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
>                        sizeof(ngx_table_elt_t))
> @@ -1549,6 +1559,7 @@
>      ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));
> 
>      u->start_time = ngx_current_msec;
> +    u->request_time = ngx_time();

Adding a duplicate start time shouldn't be needed.

> 
>      u->state->response_time = (ngx_msec_t) -1;
>      u->state->connect_time = (ngx_msec_t) -1;
> @@ -2008,6 +2019,7 @@
>      ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
>      u->headers_in.content_length_n = -1;
>      u->headers_in.last_modified_time = -1;
> +    u->headers_in.age_n = -1;
> 
>      if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
>                        sizeof(ngx_table_elt_t))
> @@ -2529,6 +2541,8 @@
>          return;
>      }
> 
> +    ngx_http_upstream_update_age(r, u, ngx_time());

Note that the "ngx_time()" as an argument looks unneeded, it 
should be better to obtain the time in the function itself.

Also, from semantic point of view this probably should be in 
ngx_http_upstream_process_headers(), and not a dedicated function 
call.

> +
>      ngx_http_upstream_send_response(r, u);
>  }
> 
> @@ -2615,6 +2629,7 @@
>                         "http upstream not modified");
> 
>          now = ngx_time();
> +        ngx_http_upstream_update_age(r, u, now);
> 
>          valid = r->cache->valid_sec;
>          updating = r->cache->updating_sec;
> @@ -2648,7 +2663,12 @@
>              valid = ngx_http_file_cache_valid(u->conf->cache_valid,
>                                                u->headers_in.status_n);
>              if (valid) {
> -                valid = now + valid;
> +                ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> +                               "adjust cache valid_sec:%O, "
> +                               "valid:%O, init_age:%d for 304",
> +                               now + valid - r->cache->corrected_initial_age,
> +                               valid, r->cache->corrected_initial_age);
> +                valid = now + valid - r->cache->corrected_initial_age;

Not sure if Age calculations should apply to caching time as 
calculated with proxy_cache_valid directives, but if it does, it 
should apply consistently, in all cases where proxy_cache_valid 
times are used.  At least ngx_http_upstream_intercept_errors() 
case seems to be ignored in the patch.

Also, if Age applies, there should be a way to ignore it, 
similarly to how Cache-Control can be ignored with the 
proxy_ignore_headers directive.

>              }
>          }
> 
> @@ -2672,6 +2692,59 @@
>  }
> 
> 
> +static void
> +ngx_http_upstream_update_age(ngx_http_request_t *r, ngx_http_upstream_t *u,
> +    time_t now)
> +{
> +    time_t  response_time, date, apparent_age, response_delay, age_value,
> +            corrected_age_value, corrected_initial_age;
> +
> +    /*
> +     * Update age response header.
> +     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
> +     */
> +    response_time = now;
> +    if (u->headers_in.date != NULL) {
> +        date = ngx_parse_http_time(u->headers_in.date->value.data,
> +                                   u->headers_in.date->value.len);
> +        if (date == NGX_ERROR) {
> +            date = now;
> +        }
> +    } else {
> +        date = now;
> +    }
> +    apparent_age = ngx_max(0, response_time - date);
> +
> +    response_delay = response_time - u->request_time;
> +    age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
> +    corrected_age_value = age_value + response_delay;
> +
> +    corrected_initial_age = ngx_max(apparent_age, corrected_age_value);
> +    r->headers_out.age_n = corrected_initial_age;

Note that this approach, as described in RFC 9111 as a possible 
way to calculate the Age header, implies that time on both proxy 
server and the origin server must be "reasonably well 
synchronized", which is not really the case in many practical 
situations.

If the time on the origin server is mostly arbitrary, for example, 
in the past, apparent_age might be very large, leading to a very 
large corrected_initial_age as well, preventing caching.

As such, I would rather consider a more robust algorithm instead.

> +
> +    ngx_log_debug8(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
> +                   "http upstream set age:%O, req:%O, resp:%O, date:%O, "
> +                   "a_age:%O, resp_delay:%O, u_age:%O, c_age:%O",
> +                   corrected_initial_age, u->request_time, response_time, date,
> +                   apparent_age, response_delay, u->headers_in.age_n,
> +                   corrected_age_value);
> +
> +#if (NGX_HTTP_CACHE)
> +    if (r->cache) {
> +        r->cache->response_time = response_time;
> +        r->cache->corrected_initial_age = corrected_initial_age;
> +        if (u->headers_in.adjusting_valid_sec) {
> +            r->cache->valid_sec -= corrected_initial_age;

The "adjusting_valid_sec" logic looks unclear (probably at least 
needs a better name) and fragile.  If I'm reading the code 
correctly, it will do the wrong thing at least if the upstream 
server returns something like:

Cache-Control: max-age=60
X-Accel-Expires: @<time>

since u->headers_in.adjusting_valid_sec won't be cleared.

> +            ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
> +                           "http upstream adjusted cache "
> +                           "valid_sec:%O, init_age:%O",
> +                           r->cache->valid_sec, corrected_initial_age);
> +        }
> +    }
> +#endif

If I'm reading the code correctly, this will add the Age 
header to responses which are not cached.  This somewhat 
contradicts to what RFC 9111 says in the description of the Age 
header (https://www.rfc-editor.org/rfc/rfc9111.html#section-5.1-6):

: The presence of an Age header field implies that the response was 
: not generated or validated by the origin server for this request.

Please also note that in many cases [free]nginx is expected to be 
seen as the origin server, even if it caches some upstream server 
responses according to the configuration (in many cases, ignoring 
Cache-Control or Expires headers).  In this case using the 
Age header even on cached responses might not be desired.  It 
would be great to dig further into possible use cases and make 
sure all are properly covered.

> +}
> +
> +
>  static ngx_int_t
>  ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
>      ngx_http_upstream_t *u)
> @@ -2747,6 +2820,7 @@
>                                                            status);
>                          if (valid) {
>                              r->cache->valid_sec = ngx_time() + valid;
> +                            u->headers_in.adjusting_valid_sec = 1;
>                          }
>                      }
> 
> @@ -2952,6 +3026,7 @@
>      r->headers_out.status_line = u->headers_in.status_line;
> 
>      r->headers_out.content_length_n = u->headers_in.content_length_n;
> +    r->headers_out.age_n = u->headers_in.age_n;
> 
>      r->disable_not_modified = !u->cacheable;
> 
> @@ -4616,6 +4691,7 @@
> 
>                  if (valid) {
>                      r->cache->valid_sec = ngx_time() + valid;
> +                    u->headers_in.adjusting_valid_sec = 1;
>                      r->cache->error = rc;
>                  }
>              }
> @@ -4891,6 +4967,7 @@
>          }
> 
>          r->cache->valid_sec = ngx_time() + n;
> +        u->headers_in.adjusting_valid_sec = 1;
>          u->headers_in.expired = 0;
>      }
> 
> @@ -5053,6 +5130,7 @@
> 
>          default:
>              r->cache->valid_sec = ngx_time() + n;
> +            u->headers_in.adjusting_valid_sec = 1;
>              u->headers_in.no_cache = 0;
>              u->headers_in.expired = 0;
>              return NGX_OK;
> @@ -5319,6 +5397,39 @@
> 
> 
>  static ngx_int_t
> +ngx_http_upstream_process_age(ngx_http_request_t *r,
> +    ngx_table_elt_t *h, ngx_uint_t offset)
> +{
> +    ngx_http_upstream_t  *u;
> +
> +    u = r->upstream;
> +
> +    if (u->headers_in.age) {
> +        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> +                      "upstream sent duplicate header line: "%V: %V", "
> +                      "previous value: "%V: %V"",
> +                      &h->key, &h->value,
> +                      &u->headers_in.age->key,
> +                      &u->headers_in.age->value);
> +        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> +    }
> +
> +    h->next = NULL;
> +    u->headers_in.age = h;
> +    u->headers_in.age_n = ngx_atoof(h->value.data, h->value.len);
> +
> +    if (u->headers_in.age_n == NGX_ERROR) {
> +        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> +                      "upstream sent invalid "Age" header: "
> +                      ""%V: %V"", &h->key, &h->value);
> +        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> +    }

Note that returning errors here violates RFC 9111 SHOULD 
recommendations, notably 
(https://www.rfc-editor.org/rfc/rfc9111.html#section-5.1):

: Although it is defined as a singleton header field, a cache 
: encountering a message with a list-based Age field value SHOULD 
: use the first member of the field value, discarding subsequent 
: ones.

: If the field value (after discarding additional members, as per 
: above) is invalid (e.g., it contains something other than a 
: non-negative integer), a cache SHOULD ignore the field.

Unless there are good reasons, a better solution might be to 
report warnings and follow the recommendations instead.

> +
> +    return NGX_OK;
> +}
> +
> +
> +static ngx_int_t
>  ngx_http_upstream_copy_header_line(ngx_http_request_t *r, ngx_table_elt_t *h,
>      ngx_uint_t offset)
>  {
> diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.h
> --- a/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:24 2024 +0900
> +++ b/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:41 2024 +0900
> @@ -287,14 +287,17 @@
> 
>      ngx_table_elt_t                 *cache_control;
>      ngx_table_elt_t                 *set_cookie;
> +    ngx_table_elt_t                 *age;
> 
>      off_t                            content_length_n;
>      time_t                           last_modified_time;
> +    off_t                            age_n;

See above, "off_t" looks wrong.

[...]

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/


From hnakamur at gmail.com  Tue Jun 25 13:47:55 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Tue, 25 Jun 2024 22:47:55 +0900
Subject: [PATCH 1 of 3] Correctly calculate and set Age header
In-Reply-To: <Zniy3EFG005hIsXl@mdounin.ru>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
 <Zniy3EFG005hIsXl@mdounin.ru>
Message-ID: <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>

Hello,

Thank you for your detailed review!

2024?6?24?(?) 9:02 Maxim Dounin <mdounin at mdounin.ru>:
>
> Hello!
>
> On Thu, Jun 20, 2024 at 08:39:43PM +0900, Hiroaki Nakamura wrote:

> Note that it might be a good idea to update the commit log to
> follow style rules, such as:
>
> : Cache: added calculation of the Age header.
> :
> : Implement the calculation of the Age header as specified in
> : RFC 9111 "HTTP Caching", https://www.rfc-editor.org/rfc/rfc9111.html

I have updated the commit message.

> > @@ -75,6 +77,8 @@
> >      time_t                           error_sec;
> >      time_t                           last_modified;
> >      time_t                           date;
> > +    time_t                           response_time;
>
> Note that the "response_time" field being added seems to be
> exactly equivalent to the exiting "date" field.

Yes, the "response_time" is equivalent to the "date" field when
receiving a response from the upstream.
However the 'date" is updated when sending a cached response.
We need to keep the cache creation time.
I have renamed "response_time" to "creation_time" to make it clearer.

> [diff]
> showfunc=1

I have added this config to my ~/.hgrc.

> Note that this uses data from the cache node, which is not
> available if the cache node was loaded from disk, for example, on
> server restart, and this will certainly lead to incorrect results.
> If I'm reading the code correctly, it will result in seconds since
> the Epoch in the Age header on all responses after a restart.
>
> This suggests that either things should be implemented quite
> differently, or this patch needs to be merged with the second one,
> which implements loading relevant information from the cache
> header.

I have merged this patch with the second one.

> > +    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> > +                   "http file cache send, resp:%O, resident:%d, age:%d",
> > +                   c->response_time, resident_time, current_age);

> Please also note that using "%d" for time_t is wrong, as time_t
> size might be different from the size of int.  For time_t, "%T"
> should be used instead.

I have changed to use "%T".

> > diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_request.h
> > --- a/src/http/ngx_http_request.h       Thu Jun 20 20:26:24 2024 +0900
> > +++ b/src/http/ngx_http_request.h       Thu Jun 20 20:26:41 2024 +0900
> > @@ -290,6 +290,7 @@
> >      off_t                             content_offset;
> >      time_t                            date_time;
> >      time_t                            last_modified_time;
> > +    off_t                             age_n;
>
> Using off_t here looks wrong, as off_t is a type for file offsets,
> and not for dates.

I have changed the type of age_n to time_t.

> > diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.c
> > --- a/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:24 2024 +0900
> > +++ b/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:41 2024 +0900
> > @@ -1549,6 +1559,7 @@
> >      ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));
> >
> >      u->start_time = ngx_current_msec;
> > +    u->request_time = ngx_time();
>
> Adding a duplicate start time shouldn't be needed.

It is not a duplicate since u->request_time is real clock time as opposed to
u->start_time is monotonic clock.

We need a real clock time of response_time since we calculate
the difference from the upstream date value.

Also we use the calculation response_time - request_time
when the upstream date is not present or not well synchronized (see below),
so we also need request_time to be a real clock time.

> > @@ -2529,6 +2541,8 @@
> >          return;
> >      }
> >
> > +    ngx_http_upstream_update_age(r, u, ngx_time());
>
> Note that the "ngx_time()" as an argument looks unneeded, it
> should be better to obtain the time in the function itself.

I have changed not to pass ngx_time() as an argument.

> Also, from semantic point of view this probably should be in
> ngx_http_upstream_process_headers(), and not a dedicated function
> call.

I have moved the code for
         * resident_time = now - response_time;
         * current_age = corrected_initial_age + resident_time;
from ngx_http_cache_send to ngx_http_upstream_process_headers.

As for ngx_http_upstream_update_age, I tried to move it into
ngx_http_upstream_process_headers(), but I could not make tests pass.
So I keep ngx_http_upstream_update_age to be called from
ngx_http_upstream_process_header and ngx_http_upstream_test_next.

> > @@ -2648,7 +2663,12 @@
> >              valid = ngx_http_file_cache_valid(u->conf->cache_valid,
> >                                                u->headers_in.status_n);
> >              if (valid) {
> > -                valid = now + valid;
> > +                ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> > +                               "adjust cache valid_sec:%O, "
> > +                               "valid:%O, init_age:%d for 304",
> > +                               now + valid - r->cache->corrected_initial_age,
> > +                               valid, r->cache->corrected_initial_age);
> > +                valid = now + valid - r->cache->corrected_initial_age;
>
> Not sure if Age calculations should apply to caching time as
> calculated with proxy_cache_valid directives, but if it does, it
> should apply consistently, in all cases where proxy_cache_valid
> times are used.  At least ngx_http_upstream_intercept_errors()
> case seems to be ignored in the patch.

I have updated ngx_http_upstream_intercept_errors as well.

> Also, if Age applies, there should be a way to ignore it,
> similarly to how Cache-Control can be ignored with the
> proxy_ignore_headers directive.

I do not know how to do this.
Maybe add
    { ngx_string("Age"), NGX_HTTP_UPSTREAM_IGN_AGE },
to ngx_http_upstream_ignore_headers_masks?

> > @@ -2672,6 +2692,59 @@
> >  }
> >
> >
> > +static void
> > +ngx_http_upstream_update_age(ngx_http_request_t *r, ngx_http_upstream_t *u,
> > +    time_t now)
> > +{
> > +    time_t  response_time, date, apparent_age, response_delay, age_value,
> > +            corrected_age_value, corrected_initial_age;
> > +
> > +    /*
> > +     * Update age response header.
> > +     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
> > +     */
> > +    response_time = now;
> > +    if (u->headers_in.date != NULL) {
> > +        date = ngx_parse_http_time(u->headers_in.date->value.data,
> > +                                   u->headers_in.date->value.len);
> > +        if (date == NGX_ERROR) {
> > +            date = now;
> > +        }
> > +    } else {
> > +        date = now;
> > +    }
> > +    apparent_age = ngx_max(0, response_time - date);
> > +
> > +    response_delay = response_time - u->request_time;
> > +    age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
> > +    corrected_age_value = age_value + response_delay;
> > +
> > +    corrected_initial_age = ngx_max(apparent_age, corrected_age_value);
> > +    r->headers_out.age_n = corrected_initial_age;
>
> Note that this approach, as described in RFC 9111 as a possible
> way to calculate the Age header, implies that time on both proxy
> server and the origin server must be "reasonably well
> synchronized", which is not really the case in many practical
> situations.
>
> If the time on the origin server is mostly arbitrary, for example,
> in the past, apparent_age might be very large, leading to a very
> large corrected_initial_age as well, preventing caching.
>
> As such, I would rather consider a more robust algorithm instead.

Yes, if the upstream date is too old, then apparant_age becomes too big.

How about using the following calculation?

if upstream date is between request_time and response_time, we assume
the upstream's clock is reasonably well synchronized and use

corrected_initial_age = age_value + (response_time - date);

if upstream date is not present or not between request_time
and response_time, we do not rely on the upstream date and use

corrected_initial_age = age_value + (response_time - u->request_time);

When sending a cached response, we calculate the Age
as specified in RFC 9111:

resident_time = now - response_time;
current_age = corrected_initial_age + resident_time;

where response_time is cache_creation_time.

I have updated my patchset to use the above calculation.

> > +#if (NGX_HTTP_CACHE)
> > +    if (r->cache) {
> > +        r->cache->response_time = response_time;
> > +        r->cache->corrected_initial_age = corrected_initial_age;
> > +        if (u->headers_in.adjusting_valid_sec) {
> > +            r->cache->valid_sec -= corrected_initial_age;
>
> The "adjusting_valid_sec" logic looks unclear (probably at least
> needs a better name) and fragile.  If I'm reading the code
> correctly, it will do the wrong thing at least if the upstream
> server returns something like:
>
> Cache-Control: max-age=60
> X-Accel-Expires: @<time>
>
> since u->headers_in.adjusting_valid_sec won't be cleared.

I found that we must ignore Expires when Cache-Control with max-age or s-maxage
is set according to https://www.rfc-editor.org/rfc/rfc9111#name-expires:

"If a response includes a Cache-Control header field with the max-age directive
(Section 5.2.2.1), a recipient MUST ignore the Expires header field. Likewise,
if a response includes the s-maxage directive (Section 5.2.2.10), a shared cache
recipient MUST ignore the Expires header field."

I have renamed adjusting_valid_sec to relative_freshness and added guard for
Expires and X-Accel-Expires: @<time> like below:

    /*
     * https://www.rfc-editor.org/rfc/rfc9111#name-expires
     *
     * If a response includes a Cache-Control header field with the max-age
     * directive (Section 5.2.2.1), a recipient MUST ignore the Expires
     * header field. Likewise, if a response includes the s-maxage directive
     * (Section 5.2.2.10), a shared cache recipient MUST ignore the Expires
     * header field.
     */
    if (!u->headers_in.relative_freshness) {
        r->cache->valid_sec = expires;
    }

> > +            ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
> > +                           "http upstream adjusted cache "
> > +                           "valid_sec:%O, init_age:%O",
> > +                           r->cache->valid_sec, corrected_initial_age);
> > +        }
> > +    }
> > +#endif
>
> If I'm reading the code correctly, this will add the Age
> header to responses which are not cached.  This somewhat
> contradicts to what RFC 9111 says in the description of the Age
> header (https://www.rfc-editor.org/rfc/rfc9111.html#section-5.1-6):
>
> : The presence of an Age header field implies that the response was
> : not generated or validated by the origin server for this request.
>
> Please also note that in many cases [free]nginx is expected to be
> seen as the origin server, even if it caches some upstream server
> responses according to the configuration (in many cases, ignoring
> Cache-Control or Expires headers).  In this case using the
> Age header even on cached responses might not be desired.  It
> would be great to dig further into possible use cases and make
> sure all are properly covered.

I have changed to not add Age if the calculated value is zero and also
updated tests.

> > @@ -5319,6 +5397,39 @@
> >
> >
> >  static ngx_int_t
> > +ngx_http_upstream_process_age(ngx_http_request_t *r,
> > +    ngx_table_elt_t *h, ngx_uint_t offset)
> > +{
> > +    ngx_http_upstream_t  *u;
> > +
> > +    u = r->upstream;
> > +
> > +    if (u->headers_in.age) {
> > +        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> > +                      "upstream sent duplicate header line: "%V: %V", "
> > +                      "previous value: "%V: %V"",
> > +                      &h->key, &h->value,
> > +                      &u->headers_in.age->key,
> > +                      &u->headers_in.age->value);
> > +        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> > +    }
> > +
> > +    h->next = NULL;
> > +    u->headers_in.age = h;
> > +    u->headers_in.age_n = ngx_atoof(h->value.data, h->value.len);
> > +
> > +    if (u->headers_in.age_n == NGX_ERROR) {
> > +        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> > +                      "upstream sent invalid "Age" header: "
> > +                      ""%V: %V"", &h->key, &h->value);
> > +        return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> > +    }
>
> Note that returning errors here violates RFC 9111 SHOULD
> recommendations, notably
> (https://www.rfc-editor.org/rfc/rfc9111.html#section-5.1):
>
> : Although it is defined as a singleton header field, a cache
> : encountering a message with a list-based Age field value SHOULD
> : use the first member of the field value, discarding subsequent
> : ones.
>
> : If the field value (after discarding additional members, as per
> : above) is invalid (e.g., it contains something other than a
> : non-negative integer), a cache SHOULD ignore the field.
>
> Unless there are good reasons, a better solution might be to
> report warnings and follow the recommendations instead.

I have changed the invalid Age value and duplicated Age headers
to report warnings.

> > diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.h
> > --- a/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:24 2024 +0900
> > +++ b/src/http/ngx_http_upstream.h      Thu Jun 20 20:26:41 2024 +0900
> > @@ -287,14 +287,17 @@
> >
> >      ngx_table_elt_t                 *cache_control;
> >      ngx_table_elt_t                 *set_cookie;
> > +    ngx_table_elt_t                 *age;
> >
> >      off_t                            content_length_n;
> >      time_t                           last_modified_time;
> > +    off_t                            age_n;
>
> See above, "off_t" looks wrong.

I have changed it to time_t.

I am sending updated two patches:

* [PATCH 1 of 2] Cache: added calculation of the Age header
* [PATCH 2 of 2] Tests: Update and add tests for Age header

Could you take another look?

Thank you,
Hiroaki Nakamura


From hnakamur at gmail.com  Tue Jun 25 13:48:49 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Tue, 25 Jun 2024 22:48:49 +0900
Subject: [PATCH 1 of 2] Cache: added calculation of the Age header
In-Reply-To: <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
 <Zniy3EFG005hIsXl@mdounin.ru>
 <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
Message-ID: <CAN-DUMSdQT8eh0GF4qzwV2sK81RS0X_+0FneqzO=VvAG_AC8ig@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1719322447 -32400
#      Tue Jun 25 22:34:07 2024 +0900
# Node ID a1e4c426f1063c7a2ab30e60df7e1756ae7d8918
# Parent  6e76116ca337e24cf0d807e042e40c96344cd22e
Cache: added calculation of the Age header.
* Implement the modified calculation of the Age header as specified in
  RFC 9111 "HTTP Caching", https://www.rfc-editor.org/rfc/rfc9111.html

  Calculation initial age as specified in RFC 9111:

  apparent_age = max(0, response_time - date_value);

  response_delay = response_time - request_time;
  corrected_age_value = age_value + response_delay;

  corrected_initial_age = max(apparent_age, corrected_age_value);

  However, if the upstream date is too old, then apparant_age
  becomes too big.

  Therefore, we use the following calculation:

  if upstream date is between request_time and response_time, we assume
  the upstream's clock is reasonably well synchronized and use

  corrected_initial_age = age_value + (response_time - date);

  if upstream date is not present or not between request_time
  and response_time, we do not rely on the upstream date and use

  corrected_initial_age = age_value + (response_time - u->request_time);

  When sending a cached response, we calculate the Age
  as specified in RFC 9111:

  resident_time = now - response_time;
  current_age = corrected_initial_age + resident_time;

  where response_time is cache_creation_time.

* Also ignore Expires and X-Accel-Expires when Cache-Control max-age
  or s-maxage is set as specified in
  https://www.rfc-editor.org/rfc/rfc9111#name-expires:

  "If a response includes a Cache-Control header field with the
  max-age directive (Section 5.2.2.1), a recipient MUST ignore
  the Expires header field. Likewise, if a response includes the
  s-maxage directive (Section 5.2.2.10), a shared cache recipient
  MUST ignore the Expires header field."

diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_cache.h Tue Jun 25 22:34:07 2024 +0900
@@ -27,7 +27,7 @@
 #define NGX_HTTP_CACHE_ETAG_LEN      128
 #define NGX_HTTP_CACHE_VARY_LEN      128

-#define NGX_HTTP_CACHE_VERSION       5
+#define NGX_HTTP_CACHE_VERSION       6


 typedef struct {
@@ -59,6 +59,8 @@ typedef struct {
     size_t                           body_start;
     off_t                            fs_size;
     ngx_msec_t                       lock_time;
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_node_t;


@@ -75,6 +77,8 @@ struct ngx_http_cache_s {
     time_t                           error_sec;
     time_t                           last_modified;
     time_t                           date;
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;

     ngx_str_t                        etag;
     ngx_str_t                        vary;
@@ -141,6 +145,8 @@ typedef struct {
     u_char                           vary_len;
     u_char                           vary[NGX_HTTP_CACHE_VARY_LEN];
     u_char                           variant[NGX_HTTP_CACHE_KEY_LEN];
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_header_t;


diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_file_cache.c Tue Jun 25 22:34:07 2024 +0900
@@ -627,6 +627,8 @@ ngx_http_file_cache_read(ngx_http_reques
     c->body_start = h->body_start;
     c->etag.len = h->etag_len;
     c->etag.data = h->etag;
+    c->creation_time = h->creation_time;
+    c->corrected_initial_age = h->corrected_initial_age;

     r->cached = 1;

@@ -971,6 +973,8 @@ renew:
     fcn->uniq = 0;
     fcn->body_start = 0;
     fcn->fs_size = 0;
+    fcn->creation_time = 0;
+    fcn->corrected_initial_age = 0;

 done:

@@ -980,6 +984,8 @@ done:

     c->uniq = fcn->uniq;
     c->error = fcn->error;
+    c->creation_time = fcn->creation_time;
+    c->corrected_initial_age = fcn->corrected_initial_age;
     c->node = fcn;

 failed:
@@ -1326,6 +1332,8 @@ ngx_http_file_cache_set_header(ngx_http_
     h->valid_msec = (u_short) c->valid_msec;
     h->header_start = (u_short) c->header_start;
     h->body_start = (u_short) c->body_start;
+    h->creation_time = c->creation_time;
+    h->corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h->etag_len = (u_char) c->etag.len;
@@ -1590,6 +1598,8 @@ ngx_http_file_cache_update_header(ngx_ht
     h.valid_msec = (u_short) c->valid_msec;
     h.header_start = (u_short) c->header_start;
     h.body_start = (u_short) c->body_start;
+    h.creation_time = c->creation_time;
+    h.corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h.etag_len = (u_char) c->etag.len;
diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_header_filter_module.c Tue Jun 25 22:34:07 2024 +0900
@@ -322,6 +322,10 @@ ngx_http_header_filter(ngx_http_request_
         len += sizeof("Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT" CRLF) - 1;
     }

+    if (r->headers_out.age_n != -1) {
+        len += sizeof("Age: ") - 1 + NGX_TIME_T_LEN + 2;
+    }
+
     c = r->connection;

     if (r->headers_out.location
@@ -518,6 +522,10 @@ ngx_http_header_filter(ngx_http_request_
         *b->last++ = CR; *b->last++ = LF;
     }

+    if (r->headers_out.age_n != -1) {
+        b->last = ngx_sprintf(b->last, "Age: %T" CRLF, r->headers_out.age_n);
+    }
+
     if (host.data) {

         p = b->last + sizeof("Location: ") - 1;
diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_request.c Tue Jun 25 22:34:07 2024 +0900
@@ -646,6 +646,7 @@ ngx_http_alloc_request(ngx_connection_t
     r->headers_in.keep_alive_n = -1;
     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;

     r->uri_changes = NGX_HTTP_MAX_URI_CHANGES + 1;
     r->subrequests = NGX_HTTP_MAX_SUBREQUESTS + 1;
diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_request.h
--- a/src/http/ngx_http_request.h Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_request.h Tue Jun 25 22:34:07 2024 +0900
@@ -291,6 +291,7 @@ typedef struct {
     off_t                             content_offset;
     time_t                            date_time;
     time_t                            last_modified_time;
+    time_t                            age_n;
 } ngx_http_headers_out_t;


diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_special_response.c
--- a/src/http/ngx_http_special_response.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_special_response.c Tue Jun 25 22:34:07 2024 +0900
@@ -581,6 +581,7 @@ ngx_http_clean_header(ngx_http_request_t

     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;
 }


diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_upstream.c Tue Jun 25 22:34:07 2024 +0900
@@ -52,6 +52,8 @@ static void ngx_http_upstream_process_he
     ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_test_next(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
+static void ngx_http_upstream_update_age(ngx_http_request_t *r,
+    ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_test_connect(ngx_connection_t *c);
@@ -134,6 +136,8 @@ static ngx_int_t
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_process_vary(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
+static ngx_int_t ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_copy_header_line(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t
@@ -321,6 +325,10 @@ static ngx_http_upstream_header_t  ngx_h
                  ngx_http_upstream_copy_header_line,
                  offsetof(ngx_http_headers_out_t, content_encoding), 0 },

+    { ngx_string("Age"),
+                 ngx_http_upstream_process_age, 0,
+                 ngx_http_upstream_ignore_header_line, 0, 0 },
+
     { ngx_null_string, NULL, 0, NULL, 0, 0 }
 };

@@ -505,6 +513,7 @@ ngx_http_upstream_create(ngx_http_reques

     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     return NGX_OK;
 }
@@ -959,6 +968,7 @@ ngx_http_upstream_cache(ngx_http_request
         c->updating_sec = 0;
         c->error_sec = 0;

+        u->headers_in.relative_freshness = 0;
         u->buffer.start = NULL;
         u->cache_status = NGX_HTTP_CACHE_EXPIRED;

@@ -1074,6 +1084,7 @@ ngx_http_upstream_cache_send(ngx_http_re
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -1555,6 +1566,7 @@ ngx_http_upstream_connect(ngx_http_reque
     ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));

     u->start_time = ngx_current_msec;
+    u->request_time = ngx_time();

     u->state->response_time = (ngx_msec_t) -1;
     u->state->connect_time = (ngx_msec_t) -1;
@@ -2014,6 +2026,7 @@ ngx_http_upstream_reinit(ngx_http_reques
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -2541,6 +2554,8 @@ ngx_http_upstream_process_header(ngx_htt
         return;
     }

+    ngx_http_upstream_update_age(r, u);
+
     ngx_http_upstream_send_response(r, u);
 }

@@ -2627,6 +2642,7 @@ ngx_http_upstream_test_next(ngx_http_req
                        "http upstream not modified");

         now = ngx_time();
+        ngx_http_upstream_update_age(r, u);

         valid = r->cache->valid_sec;
         updating = r->cache->updating_sec;
@@ -2660,7 +2676,12 @@ ngx_http_upstream_test_next(ngx_http_req
             valid = ngx_http_file_cache_valid(u->conf->cache_valid,
                                               u->headers_in.status_n);
             if (valid) {
-                valid = now + valid;
+                ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                               "adjust cache valid_sec:%T, "
+                               "valid:%T, init_age:%d for 304",
+                               now + valid - r->cache->corrected_initial_age,
+                               valid, r->cache->corrected_initial_age);
+                valid = now + valid - r->cache->corrected_initial_age;
             }
         }

@@ -2684,6 +2705,77 @@ ngx_http_upstream_test_next(ngx_http_req
 }


+static void
+ngx_http_upstream_update_age(ngx_http_request_t *r, ngx_http_upstream_t *u)
+{
+    time_t  response_time, age_value, date, corrected_initial_age;
+
+    /*
+     * Update age response header.
+     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+     *
+     * apparent_age = max(0, response_time - date_value);
+     *
+     * response_delay = response_time - request_time;
+     * corrected_age_value = age_value + response_delay;
+     *
+     * corrected_initial_age = max(apparent_age, corrected_age_value);
+     *
+     * However, if the upstream date is too old, then apparant_age
+     * becomes too big.
+     *
+     * Therefore, we use the following calculation:
+     *
+     * if upstream date is between request_time and response_time, we assume
+     * the upstream's clock is reasonably well synchronized and use
+     *
+     * corrected_initial_age = age_value + (response_time - date);
+     *
+     * if upstream date is not present or not between request_time
+     * and response_time, we do not rely on the upstream date and use
+     *
+     * corrected_initial_age = age_value + (response_time - u->request_time);
+     */
+    response_time = ngx_time();
+
+    age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
+
+    if (u->headers_in.date != NULL) {
+        date = ngx_parse_http_time(u->headers_in.date->value.data,
+                                   u->headers_in.date->value.len);
+    } else {
+        date = NGX_ERROR;
+    }
+
+    if (date >= u->request_time && date <= response_time) {
+        corrected_initial_age = age_value + (response_time - date);
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                       "http upstream set age:%T, upstream date ok",
+                       corrected_initial_age);
+    } else {
+        corrected_initial_age = age_value + (response_time - u->request_time);
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                       "http upstream set age:%T, upstream date not ok",
+                       corrected_initial_age);
+    }
+
+    r->headers_out.age_n = corrected_initial_age ? corrected_initial_age : -1;
+
+#if (NGX_HTTP_CACHE)
+    if (r->cache) {
+        r->cache->creation_time = response_time;
+        r->cache->corrected_initial_age = corrected_initial_age;
+        if (u->headers_in.relative_freshness) {
+            r->cache->valid_sec -= corrected_initial_age;
+            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                           "http upstream adjusted cache valid_sec:%T",
+                           r->cache->valid_sec);
+        }
+    }
+#endif
+}
+
+
 static ngx_int_t
 ngx_http_upstream_intercept_errors(ngx_http_request_t *r,
     ngx_http_upstream_t *u)
@@ -2759,6 +2851,7 @@ ngx_http_upstream_intercept_errors(ngx_h
                                                           status);
                         if (valid) {
                             r->cache->valid_sec = ngx_time() + valid;
+                            u->headers_in.relative_freshness = 1;
                         }
                     }

@@ -2964,6 +3057,31 @@ ngx_http_upstream_process_headers(ngx_ht
     r->headers_out.status_line = u->headers_in.status_line;

     r->headers_out.content_length_n = u->headers_in.content_length_n;
+    r->headers_out.age_n = u->headers_in.age_n;
+
+#if (NGX_HTTP_CACHE)
+    if (r->cache) {
+        time_t             resident_time, current_age;
+        ngx_http_cache_t  *c;
+
+        c = r->cache;
+
+        /*
+         * Update age response header.
+         * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+         *
+         * resident_time = now - response_time;
+         * current_age = corrected_initial_age + resident_time;
+         */
+        resident_time = ngx_time() - c->creation_time;
+        current_age = c->corrected_initial_age + resident_time;
+        r->headers_out.age_n = current_age ? current_age : -1;
+        ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                       "http upstream process headers, created:%T, "
+                       "resident:%T, age:%T",
+                       c->creation_time, resident_time, current_age);
+    }
+#endif

     r->disable_not_modified = !u->cacheable;

@@ -3196,6 +3314,7 @@ ngx_http_upstream_send_response(ngx_http
                                               u->headers_in.status_n);
             if (valid) {
                 r->cache->valid_sec = now + valid;
+                u->headers_in.relative_freshness = 1;
             }
         }

@@ -4628,6 +4747,7 @@ ngx_http_upstream_finalize_request(ngx_h

                 if (valid) {
                     r->cache->valid_sec = ngx_time() + valid;
+                    u->headers_in.relative_freshness = 1;
                     r->cache->error = rc;
                 }
             }
@@ -4903,6 +5023,7 @@ ngx_http_upstream_process_cache_control(
         }

         r->cache->valid_sec = ngx_time() + n;
+        u->headers_in.relative_freshness = 1;
         u->headers_in.expired = 0;
     }

@@ -5005,7 +5126,18 @@ ngx_http_upstream_process_expires(ngx_ht
         return NGX_OK;
     }

-    r->cache->valid_sec = expires;
+    /*
+     * https://www.rfc-editor.org/rfc/rfc9111#name-expires
+     *
+     * If a response includes a Cache-Control header field with the max-age
+     * directive (Section 5.2.2.1), a recipient MUST ignore the Expires
+     * header field. Likewise, if a response includes the s-maxage directive
+     * (Section 5.2.2.10), a shared cache recipient MUST ignore the Expires
+     * header field.
+     */
+    if (!u->headers_in.relative_freshness) {
+        r->cache->valid_sec = expires;
+    }
     }
 #endif

@@ -5065,6 +5197,7 @@ ngx_http_upstream_process_accel_expires(

         default:
             r->cache->valid_sec = ngx_time() + n;
+            u->headers_in.relative_freshness = 1;
             u->headers_in.no_cache = 0;
             u->headers_in.expired = 0;
             return NGX_OK;
@@ -5077,7 +5210,18 @@ ngx_http_upstream_process_accel_expires(
     n = ngx_atoi(p, len);

     if (n != NGX_ERROR) {
-        r->cache->valid_sec = n;
+        /*
+        * https://www.rfc-editor.org/rfc/rfc9111#name-expires
+        *
+        * If a response includes a Cache-Control header field with the max-age
+        * directive (Section 5.2.2.1), a recipient MUST ignore the Expires
+        * header field. Likewise, if a response includes the s-maxage directive
+        * (Section 5.2.2.10), a shared cache recipient MUST ignore the Expires
+        * header field.
+        */
+        if (!u->headers_in.relative_freshness) {
+            r->cache->valid_sec = n;
+        }
         u->headers_in.no_cache = 0;
         u->headers_in.expired = 0;
     }
@@ -5331,6 +5475,38 @@ ngx_http_upstream_process_vary(ngx_http_


 static ngx_int_t
+ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset)
+{
+    ngx_http_upstream_t  *u;
+
+    u = r->upstream;
+
+    if (u->headers_in.age) {
+        ngx_log_error(NGX_LOG_WARN, r->connection->log, 0,
+                      "ignore duplicate age header from upstream: \"%V: %V\", "
+                      "previous value: \"%V: %V\"",
+                      &h->key, &h->value,
+                      &u->headers_in.age->key,
+                      &u->headers_in.age->value);
+        return NGX_OK;
+    }
+
+    h->next = NULL;
+    u->headers_in.age = h;
+    u->headers_in.age_n = ngx_atoof(h->value.data, h->value.len);
+
+    if (u->headers_in.age_n == NGX_ERROR) {
+        ngx_log_error(NGX_LOG_WARN, r->connection->log, 0,
+                      "ignore invalid \"Age\" header from upstream: "
+                      "\"%V: %V\"", &h->key, &h->value);
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
 ngx_http_upstream_copy_header_line(ngx_http_request_t *r, ngx_table_elt_t *h,
     ngx_uint_t offset)
 {
diff -r 6e76116ca337 -r a1e4c426f106 src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/ngx_http_upstream.h Tue Jun 25 22:34:07 2024 +0900
@@ -287,14 +287,17 @@ typedef struct {

     ngx_table_elt_t                 *cache_control;
     ngx_table_elt_t                 *set_cookie;
+    ngx_table_elt_t                 *age;

     off_t                            content_length_n;
     time_t                           last_modified_time;
+    time_t                           age_n;

     unsigned                         connection_close:1;
     unsigned                         chunked:1;
     unsigned                         no_cache:1;
     unsigned                         expired:1;
+    unsigned                         relative_freshness:1;
 } ngx_http_upstream_headers_in_t;


@@ -369,6 +372,7 @@ struct ngx_http_upstream_s {
                                          ngx_table_elt_t *h);

     ngx_msec_t                       start_time;
+    time_t                           request_time;

     ngx_http_upstream_state_t       *state;

diff -r 6e76116ca337 -r a1e4c426f106 src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/v2/ngx_http_v2.h Tue Jun 25 22:34:07 2024 +0900
@@ -398,6 +398,7 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt
 #define NGX_HTTP_V2_STATUS_404_INDEX      13
 #define NGX_HTTP_V2_STATUS_500_INDEX      14

+#define NGX_HTTP_V2_AGE_INDEX             21
 #define NGX_HTTP_V2_CONTENT_LENGTH_INDEX  28
 #define NGX_HTTP_V2_CONTENT_TYPE_INDEX    31
 #define NGX_HTTP_V2_DATE_INDEX            33
diff -r 6e76116ca337 -r a1e4c426f106 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/v2/ngx_http_v2_filter_module.c Tue Jun 25 22:34:07 2024 +0900
@@ -258,6 +258,10 @@ ngx_http_v2_header_filter(ngx_http_reque
         len += 1 + ngx_http_v2_literal_size("Wed, 31 Dec 1986 18:00:00 GMT");
     }

+    if (r->headers_out.age_n != -1) {
+        len += 1 + ngx_http_v2_integer_octets(NGX_TIME_T_LEN) + NGX_TIME_T_LEN;
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -552,6 +556,18 @@ ngx_http_v2_header_filter(ngx_http_reque
         pos = ngx_http_v2_write_value(pos, pos, len, tmp);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                       "http2 output header: \"age: %T\"",
+                       r->headers_out.age_n);
+
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_AGE_INDEX);
+
+        p = pos;
+        pos = ngx_sprintf(pos + 1, "%T", r->headers_out.age_n);
+        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (pos - p - 1);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
                        "http2 output header: \"location: %V\"",
diff -r 6e76116ca337 -r a1e4c426f106 src/http/v3/ngx_http_v3_filter_module.c
--- a/src/http/v3/ngx_http_v3_filter_module.c Tue Jun 25 22:33:57 2024 +0900
+++ b/src/http/v3/ngx_http_v3_filter_module.c Tue Jun 25 22:34:07 2024 +0900
@@ -13,6 +13,7 @@
 /* static table indices */
 #define NGX_HTTP_V3_HEADER_AUTHORITY                 0
 #define NGX_HTTP_V3_HEADER_PATH_ROOT                 1
+#define NGX_HTTP_V3_HEADER_AGE_ZERO                  2
 #define NGX_HTTP_V3_HEADER_CONTENT_LENGTH_ZERO       4
 #define NGX_HTTP_V3_HEADER_DATE                      6
 #define NGX_HTTP_V3_HEADER_LAST_MODIFIED             10
@@ -213,6 +214,15 @@ ngx_http_v3_header_filter(ngx_http_reque
                                   sizeof("Mon, 28 Sep 1970 06:00:00 GMT") - 1);
     }

+    if (r->headers_out.age_n > 0) {
+        len += ngx_http_v3_encode_field_lri(NULL, 0,
+                                            NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                            NULL, NGX_TIME_T_LEN);
+    } else if (r->headers_out.age_n == 0) {
+        len += ngx_http_v3_encode_field_ri(NULL, 0,
+                                           NGX_HTTP_V3_HEADER_AGE_ZERO);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -452,6 +462,27 @@ ngx_http_v3_header_filter(ngx_http_reque
                                               p, n);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
+                       "http3 output header: \"age: %T\"",
+                       r->headers_out.age_n);
+
+        if (r->headers_out.age_n > 0) {
+            p = ngx_sprintf(b->last, "%T", r->headers_out.age_n);
+            n = p - b->last;
+
+            b->last = (u_char *) ngx_http_v3_encode_field_lri(b->last, 0,
+                                                   NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                                   NULL, n);
+
+            b->last = ngx_sprintf(b->last, "%T", r->headers_out.age_n);
+
+        } else {
+            b->last = (u_char *) ngx_http_v3_encode_field_ri(b->last, 0,
+                                                  NGX_HTTP_V3_HEADER_AGE_ZERO);
+        }
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
                        "http3 output header: \"location: %V\"",


From hnakamur at gmail.com  Tue Jun 25 13:50:43 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Tue, 25 Jun 2024 22:50:43 +0900
Subject: [PATCH 2 of 2] Tests: Update and add tests for Age header
In-Reply-To: <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
 <Zniy3EFG005hIsXl@mdounin.ru>
 <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
Message-ID: <CAN-DUMRFr-RanHosEW-MDCZ_7vc97hzfdDpeBiLGOeGOeqL4CQ@mail.gmail.com>

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1719322456 -32400
#      Tue Jun 25 22:34:16 2024 +0900
# Node ID 758a6c4c1babb9d60d6f507ee409152eeb51cbc5
# Parent  a1e4c426f1063c7a2ab30e60df7e1756ae7d8918
Tests: Update and add tests for Age header.

diff -r a1e4c426f106 -r 758a6c4c1bab h2_proxy_cache_age.t
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/h2_proxy_cache_age.t Tue Jun 25 22:34:16 2024 +0900
@@ -0,0 +1,203 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/2 proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP2;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http http_v2 proxy cache/)->plan(8)
+ ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
+    proxy_cache_path %%TESTDIR%%/cache2   keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:8080 http2;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8080.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8081;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8081.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8082.log debug;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t.html', 'SEE-THIS');
+
+# suppress deprecation warning
+
+open OLDERR, ">&", \*STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", \*OLDERR;
+
+###############################################################################
+
+my $s = Test::Nginx::HTTP2->new();
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age updated');
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+# slow origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=4&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin updated');
+
+}
+
+# update age after restart
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+select undef, undef, undef, 1.0;
+
+open OLDERR, ">&", \*STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", \*OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = Test::Nginx::HTTP2->new();
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r a1e4c426f106 -r 758a6c4c1bab h2_ssl_proxy_cache_age.t
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/h2_ssl_proxy_cache_age.t Tue Jun 25 22:34:16 2024 +0900
@@ -0,0 +1,248 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/2 ssl proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP2;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()
+ ->has(qw/http http_ssl http_v2 proxy cache socket_ssl/)->plan(10)
+ ->has_daemon('openssl');
+
+$t->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache   keys_zone=NAME:1m;
+    proxy_cache_path %%TESTDIR%%/cache2  keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:8080 http2 ssl;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8080.log debug;
+
+        ssl_certificate_key localhost.key;
+        ssl_certificate localhost.crt;
+
+        location / {
+            proxy_pass   http://127.0.0.1:8081;
+            proxy_cache  NAME;
+            proxy_http_version 1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8081.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8082.log debug;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('openssl.conf', <<EOF);
+[ req ]
+default_bits = 2048
+encrypt_key = no
+distinguished_name = req_distinguished_name
+[ req_distinguished_name ]
+EOF
+
+my $d = $t->testdir();
+
+foreach my $name ('localhost') {
+ system('openssl req -x509 -new '
+ . "-config $d/openssl.conf -subj /CN=$name/ "
+ . "-out $d/$name.crt -keyout $d/$name.key "
+ . ">>$d/openssl.out 2>&1") == 0
+ or die "Can't create certificate for $name: $!\n";
+}
+
+$t->write_file('t.html', 'SEE-THIS');
+
+open OLDERR, ">&", \*STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", \*OLDERR;
+
+###############################################################################
+
+my $s = getconn(port(8080));
+ok($s, 'ssl connection');
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age updated');
+
+# slow origin
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=4&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin updated');
+
+}
+
+# update age after restart
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+select undef, undef, undef, 1.0;
+
+open OLDERR, ">&", \*STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", \*OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = getconn(port(8080));
+ok($s, 'ssl connection');
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+sub getconn {
+ my ($port) = @_;
+ my $s;
+
+ eval {
+ my $sock = Test::Nginx::HTTP2::new_socket($port, SSL => 1,
+ alpn => 'h2');
+ $s = Test::Nginx::HTTP2->new($port, socket => $sock)
+ if $sock->alpn_selected();
+ };
+
+ return $s if defined $s;
+
+ eval {
+ my $sock = Test::Nginx::HTTP2::new_socket($port, SSL => 1,
+ npn => 'h2');
+ $s = Test::Nginx::HTTP2->new($port, socket => $sock)
+ if $sock->next_proto_negotiated();
+ };
+
+ return $s;
+}
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r a1e4c426f106 -r 758a6c4c1bab h3_proxy_cache_age.t
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/h3_proxy_cache_age.t Tue Jun 25 22:34:16 2024 +0900
@@ -0,0 +1,215 @@
+#!/usr/bin/perl
+
+# (C) Sergey Kandaurov
+# (C) Nginx, Inc.
+# (C) Hiroaki Nakamura
+
+# Tests for age in HTTP/3 proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+use Test::Nginx::HTTP3;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http http_v3 proxy cryptx/)
+ ->has_daemon('openssl')->plan(8)
+ ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    ssl_certificate_key localhost.key;
+    ssl_certificate localhost.crt;
+
+    log_format test $uri:$status:$request_completion;
+
+    proxy_cache_path %%TESTDIR%%/cache    keys_zone=NAME:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       90;
+    }
+
+    server {
+        listen       127.0.0.1:%%PORT_8980_UDP%% quic;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8080.log debug;
+
+        location / {
+            proxy_pass http://127.0.0.1:8081/;
+            proxy_cache NAME;
+            proxy_http_version 1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8081.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8082.log debug;
+
+        location / {
+            add_header Cache-Control s-maxage=$arg_ttl;
+            limit_rate $rate;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('openssl.conf', <<EOF);
+[ req ]
+default_bits = 2048
+encrypt_key = no
+distinguished_name = req_distinguished_name
+[ req_distinguished_name ]
+EOF
+
+my $d = $t->testdir();
+
+foreach my $name ('localhost') {
+ system('openssl req -x509 -new '
+ . "-config $d/openssl.conf -subj /CN=$name/ "
+ . "-out $d/$name.crt -keyout $d/$name.key "
+ . ">>$d/openssl.out 2>&1") == 0
+ or die "Can't create certificate for $name: $!\n";
+}
+
+my $content = 'SEE-THIS';
+$t->write_file('t.html', $content);
+$t->run();
+
+###############################################################################
+
+my $s = Test::Nginx::HTTP3->new();
+
+my ($path, $sid, $frames, $frame, $t1, $resident_time);
+
+# normal origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=2';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'age hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age updated');
+
+# slow origin
+
+wait_until_next_second();
+
+$path = '/t.html?ttl=4&slow=1';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin first');
+
+select undef, undef, undef, 2.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 4, 'slow origin hit');
+
+select undef, undef, undef, 1.0;
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, 2, 'slow origin updated');
+
+# update age after restart
+
+$path = '/t.html?ttl=20';
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, undef, 'age before restart');
+$t1 = time();
+
+$t->stop();
+
+select undef, undef, undef, 1.0;
+
+open OLDERR, ">&", \*STDERR; close STDERR;
+$t->run();
+open STDERR, ">&", \*OLDERR;
+
+$resident_time = time() - $t1;
+
+$s = Test::Nginx::HTTP3->new();
+
+$sid = $s->new_stream({ path => $path });
+$frames = $s->read(all => [{ sid => $sid, fin => 1 }]);
+($frame) = grep { $_->{type} eq "HEADERS" } @$frames;
+is($frame->{headers}->{'age'}, $resident_time, 'age after restart');
+
+$t->stop();
+
+###############################################################################
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r a1e4c426f106 -r 758a6c4c1bab proxy_cache_age.t
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/proxy_cache_age.t Tue Jun 25 22:34:16 2024 +0900
@@ -0,0 +1,183 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+# (C) Hiroaki Nakamura
+
+# Tests for age in http proxy cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+use Socket qw/ CRLF /;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+use POSIX qw/ ceil /;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy cache/)->plan(8)
+ ->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path   %%TESTDIR%%/cache  levels=1:2
+                       keys_zone=NAME:1m;
+    proxy_cache_path   %%TESTDIR%%/cache2  levels=1:2
+                       keys_zone=NAME2:1m;
+
+    map $arg_slow $rate {
+        default 8k;
+        1       100;
+    }
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8080.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8081;
+            proxy_cache            NAME;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+            add_header parent_date $upstream_http_date;
+            add_header child_msec $msec;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8081.log debug;
+
+        location / {
+            proxy_pass             http://127.0.0.1:8082;
+            proxy_cache            NAME2;
+            proxy_http_version     1.1;
+            proxy_cache_revalidate on;
+            add_header origin_date $upstream_http_date;
+            add_header parent_msec $msec;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8082;
+        server_name  localhost;
+        error_log    %%TESTDIR%%/error8082.log debug;
+
+        location / {
+            add_header Cache-Control $http_x_cache_control;
+            limit_rate $rate;
+            add_header origin_msec $msec;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t.html', 'SEE-THIS');
+$t->write_file('t2.html', 'SEE-THIS');
+$t->write_file('t3.html', 'SEE-THIS');
+
+$t->run();
+
+###############################################################################
+
+# normal origin
+
+wait_until_next_second();
+
+unlike(get('/t.html', 's-maxage=2'), qr/\r\nAge: /, 'age first');
+
+sleep 2;
+
+like(get('/t.html', 's-maxage=2'), qr/\r\nAge: 2\r\n/, 'age hit');
+
+sleep 1;
+
+unlike(http_get('/t.html'), qr/\r\nAge: /, 'age updated');
+
+# slow origin
+
+SKIP: {
+skip 'no exec on win32', 3 if $^O eq 'MSWin32';
+
+wait_until_next_second();
+
+like(get('/t2.html?slow=1', 's-maxage=4'), qr/\r\nAge: 2\r\n/,
+    'slow origin first');
+
+sleep 2;
+
+like(http_get('/t2.html?slow=1'), qr/\r\nAge: 4\r\n/, 'slow origin hit');
+
+sleep 1;
+
+like(http_get('/t2.html?slow=1'), qr/\r\nAge: 2\r\n/, 'slow origin updated');
+
+}
+
+# update age after restart
+
+wait_until_next_second();
+
+unlike(get('/t3.html', 's-maxage=20'), qr/\r\nAge: /, 'age before restart');
+my $t1 = time();
+
+$t->stop();
+
+sleep 1;
+
+$t->run();
+
+my $resident_time = time() - $t1;
+like(http_get('/t3.html'), qr/\r\nAge: $resident_time\r\n/,
+    'age after restart');
+
+# is(1, 0, 'intentional error');
+
+$t->stop();
+
+###############################################################################
+
+sub get {
+ my ($url, $extra, %extra) = @_;
+ return http(<<EOF, %extra);
+GET $url HTTP/1.1
+Host: localhost
+Connection: close
+X-Cache-Control: $extra
+
+EOF
+}
+
+# Wait until the next second boundary.
+# Calling this before sending a request increases the likelihood that the
+# timestamp value does not cross into the next second while sending a request
+# and receiving a response.
+sub wait_until_next_second {
+    my $now = time();
+    my $next_second = ceil($now);
+    my $sleep = $next_second - $now;
+    select undef, undef, undef, $sleep;
+}
+
+###############################################################################
diff -r a1e4c426f106 -r 758a6c4c1bab proxy_cache_use_stale.t
--- a/proxy_cache_use_stale.t Tue Jun 25 22:34:07 2024 +0900
+++ b/proxy_cache_use_stale.t Tue Jun 25 22:34:16 2024 +0900
@@ -206,7 +206,8 @@ like(http_get('/regexp.html'), qr/HIT/,

 $t->write_file('t6.html', 'SEE-THAT');

-my $s = get('/t6.html', 'max-age=1, stale-while-revalidate=2', start => 1);
+# max-age must be 5 here since response delay is 4 seconds.
+my $s = get('/t6.html', 'max-age=5, stale-while-revalidate=2', start => 1);
 select undef, undef, undef, 0.2;
 like(http_get('/t6.html'), qr/UPDATING.*SEE-THIS/s, 's-w-r - updating');
 like(http_end($s), qr/STALE.*SEE-THIS/s, 's-w-r - updating stale');


From mdounin at mdounin.ru  Tue Jun 25 19:03:03 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 22:03:03 +0300
Subject: [nginx] Upstream: fixed proxy_no_cache when caching errors.
Message-ID: <hg.c5623963c29e.1719342183.659210111136320968@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/c5623963c29e
branches:  
changeset: 9295:c5623963c29e
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 21:44:50 2024 +0300
description:
Upstream: fixed proxy_no_cache when caching errors.

Caching errors, notably intercepted errors and internally generated
502/504 errors, as well as handling of cache revalidation with 304,
did not take into account u->conf->no_cache predicates configured.
As a result, an error might be cached even if configuration explicitly
says not to.  Fix is to check u->conf->no_cache in these cases.

To simplify usage in multiple places, checking u->conf->no_cache is now
done in a separate function.  As a minor optimization, u->conf->no_cache
is only checked if u->cacheable is set.

As a side effect, this change also fixes caching errors after
proxy_cache_bypass.  Also, during cache revalidation u->cacheable is
now tested, so 304 responses which disable caching won't extend
cacheability of stored responses.

Additionally, when caching internally generated 502/504 errors
u->cacheable is now explicitly updated from u->headers_in.no_cache and
u->headers_in.expired, restoring the behaviour before 8041:0784ab86ad08
(1.23.0) when an error happens while reading the response headers.

Reported by Kirill A. Korinsky,
https://freenginx.org/pipermail/nginx/2024-April/000082.html

diffstat:

 src/http/ngx_http_upstream.c |  136 ++++++++++++++++++++++++++++--------------
 1 files changed, 89 insertions(+), 47 deletions(-)

diffs (193 lines):

diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -21,6 +21,8 @@ static ngx_int_t ngx_http_upstream_cache
     ngx_http_request_t *r, ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_cache_check_range(ngx_http_request_t *r,
     ngx_http_upstream_t *u);
+static ngx_int_t ngx_http_upstream_no_cache(ngx_http_request_t *r,
+    ngx_http_upstream_t *u);
 static ngx_int_t ngx_http_upstream_cache_status(ngx_http_request_t *r,
     ngx_http_variable_value_t *v, uintptr_t data);
 static ngx_int_t ngx_http_upstream_cache_key(ngx_http_request_t *r,
@@ -2632,6 +2634,12 @@ ngx_http_upstream_test_next(ngx_http_req
         updating = r->cache->updating_sec;
         error = r->cache->error_sec;
 
+        if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+            ngx_http_upstream_finalize_request(r, u,
+                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
+            return NGX_OK;
+        }
+
         rc = u->reinit_request(r);
 
         if (rc != NGX_OK) {
@@ -2650,30 +2658,33 @@ ngx_http_upstream_test_next(ngx_http_req
             rc = NGX_HTTP_INTERNAL_SERVER_ERROR;
         }
 
-        if (valid == 0) {
-            valid = r->cache->valid_sec;
-            updating = r->cache->updating_sec;
-            error = r->cache->error_sec;
-        }
-
-        if (valid == 0) {
-            valid = ngx_http_file_cache_valid(u->conf->cache_valid,
-                                              u->headers_in.status_n);
+        if (u->cacheable) {
+
+            if (valid == 0) {
+                valid = r->cache->valid_sec;
+                updating = r->cache->updating_sec;
+                error = r->cache->error_sec;
+            }
+
+            if (valid == 0) {
+                valid = ngx_http_file_cache_valid(u->conf->cache_valid,
+                                                  u->headers_in.status_n);
+                if (valid) {
+                    valid = now + valid;
+                }
+            }
+
             if (valid) {
-                valid = now + valid;
+                r->cache->valid_sec = valid;
+                r->cache->updating_sec = updating;
+                r->cache->error_sec = error;
+
+                r->cache->date = now;
+
+                ngx_http_file_cache_update_header(r);
             }
         }
 
-        if (valid) {
-            r->cache->valid_sec = valid;
-            r->cache->updating_sec = updating;
-            r->cache->error_sec = error;
-
-            r->cache->date = now;
-
-            ngx_http_file_cache_update_header(r);
-        }
-
         ngx_http_upstream_finalize_request(r, u, rc);
         return NGX_OK;
     }
@@ -2745,8 +2756,10 @@ ngx_http_upstream_intercept_errors(ngx_h
 
             if (r->cache) {
 
-                if (u->headers_in.no_cache || u->headers_in.expired) {
-                    u->cacheable = 0;
+                if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+                    ngx_http_upstream_finalize_request(r, u,
+                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
+                    return NGX_OK;
                 }
 
                 if (u->cacheable) {
@@ -3159,29 +3172,8 @@ ngx_http_upstream_send_response(ngx_http
         r->cache->file.fd = NGX_INVALID_FILE;
     }
 
-    switch (ngx_http_test_predicates(r, u->conf->no_cache)) {
-
-    case NGX_ERROR:
+    if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
         ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
-        return;
-
-    case NGX_DECLINED:
-        u->cacheable = 0;
-        break;
-
-    default: /* NGX_OK */
-
-        if (u->cache_status == NGX_HTTP_CACHE_BYPASS) {
-
-            /* create cache if previously bypassed */
-
-            if (ngx_http_file_cache_create(r) != NGX_OK) {
-                ngx_http_upstream_finalize_request(r, u, NGX_ERROR);
-                return;
-            }
-        }
-
-        break;
     }
 
     if (u->cacheable) {
@@ -3372,6 +3364,50 @@ ngx_http_upstream_send_response(ngx_http
 }
 
 
+#if (NGX_HTTP_CACHE)
+
+static ngx_int_t
+ngx_http_upstream_no_cache(ngx_http_request_t *r, ngx_http_upstream_t *u)
+{
+    ngx_int_t  rc;
+
+    if (!u->cacheable) {
+        return NGX_OK;
+    }
+
+    if (u->headers_in.no_cache || u->headers_in.expired) {
+        u->cacheable = 0;
+        return NGX_OK;
+    }
+
+    rc = ngx_http_test_predicates(r, u->conf->no_cache);
+
+    if (rc == NGX_ERROR) {
+        return NGX_ERROR;
+    }
+
+    if (rc == NGX_DECLINED) {
+        u->cacheable = 0;
+        return NGX_OK;
+    }
+
+    /* rc == NGX_OK */
+
+    if (u->cache_status == NGX_HTTP_CACHE_BYPASS) {
+
+        /* create cache if previously bypassed */
+
+        if (ngx_http_file_cache_create(r) != NGX_OK) {
+            return NGX_ERROR;
+        }
+    }
+
+    return NGX_OK;
+}
+
+#endif
+
+
 static void
 ngx_http_upstream_upgrade(ngx_http_request_t *r, ngx_http_upstream_t *u)
 {
@@ -4619,9 +4655,15 @@ ngx_http_upstream_finalize_request(ngx_h
 
     if (r->cache) {
 
-        if (u->cacheable) {
-
-            if (rc == NGX_HTTP_BAD_GATEWAY || rc == NGX_HTTP_GATEWAY_TIME_OUT) {
+        if (rc == NGX_HTTP_BAD_GATEWAY || rc == NGX_HTTP_GATEWAY_TIME_OUT) {
+
+            if (!u->header_sent) {
+                if (ngx_http_upstream_no_cache(r, u) != NGX_OK) {
+                    u->cacheable = 0;
+                }
+            }
+
+            if (u->cacheable) {
                 time_t  valid;
 
                 valid = ngx_http_file_cache_valid(u->conf->cache_valid, rc);


From mdounin at mdounin.ru  Tue Jun 25 19:03:24 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 22:03:24 +0300
Subject: [nginx-tests] Tests: proxy_no_cache tests.
Message-ID: <hg.bf027a972ccf.1719342204.-1720797922433582400@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-tests/rev/bf027a972ccf
branches:  
changeset: 1989:bf027a972ccf
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 21:46:50 2024 +0300
description:
Tests: proxy_no_cache tests.

diffstat:

 proxy_cache_bypass.t |   39 ++++++++++-
 proxy_no_cache.t     |  174 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 211 insertions(+), 2 deletions(-)

diffs (254 lines):

diff --git a/proxy_cache_bypass.t b/proxy_cache_bypass.t
--- a/proxy_cache_bypass.t
+++ b/proxy_cache_bypass.t
@@ -21,7 +21,7 @@ use Test::Nginx;
 select STDERR; $| = 1;
 select STDOUT; $| = 1;
 
-my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(8)
+my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(12)
 	->write_file_expand('nginx.conf', <<'EOF');
 
 %%TEST_GLOBALS%%
@@ -65,12 +65,17 @@ http {
 
         location / {
         }
+
+        location /t3 {
+            add_header Transfer-Encoding $arg_bypass;
+        }
     }
 }
 
 EOF
 
 $t->write_file('t', 'SEE-THIS');
+$t->write_file('t3', 'SEE-THIS');
 
 $t->run();
 
@@ -82,6 +87,9 @@ like(http_get('/t'), qr/SEE-THIS/, 'requ
 
 like(http_get('/t'), qr/SEE-THIS/, 'request cached');
 like(http_get('/t?bypass=1'), qr/NOOP/, 'cache bypassed');
+
+unlink $t->testdir() . '/t';
+
 like(http_get('/t'), qr/NOOP/, 'cached after bypass');
 
 # ticket #827, cache item "error" field was not cleared
@@ -93,6 +101,33 @@ like(http_get('/t2'), qr/403 Forbidden/,
 
 like(http_get('/t2'), qr/403 Forbidden/, 'error cached');
 like(http_get('/t2?bypass=1'), qr/NOOP/, 'error cache bypassed');
-like(http_get('/t2'), qr/NOOP/, 'error cached after bypass');
+
+unlink $t->testdir() . '/t2';
+
+like(http_get('/t2'), qr/NOOP/, 'file cached after bypass');
+
+# make sure the error is cached after bypass
+
+like(http_get('/t2?bypass=1'), qr/403 Forbidden/, 'file cache bypassed');
+
+$t->write_file('t2', 'NOOP');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t2'), qr/403 Forbidden/, 'error cached again');
+
+}
+
+# similarly, internal 502/504 is cached after bypass
+
+like(http_get('/t3?bypass=1'), qr/502 Bad/, 'internal 502');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t3'), qr/502 Bad/, 'internal 502 cached');
+
+}
 
 ###############################################################################
diff --git a/proxy_no_cache.t b/proxy_no_cache.t
new file mode 100644
--- /dev/null
+++ b/proxy_no_cache.t
@@ -0,0 +1,174 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+
+# Tests for http proxy cache, proxy_no_cache.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy cache rewrite/)->plan(16)
+	->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    proxy_cache_path %%TESTDIR%%/cache keys_zone=one:1m;
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location / {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1y;
+            proxy_no_cache $arg_nocache;
+
+            proxy_intercept_errors on;
+            error_page 404 = @fallback;
+        }
+
+        location /t3 {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1y;
+            proxy_no_cache $arg_nocache;
+        }
+
+        location /t4 {
+            proxy_pass http://127.0.0.1:8081;
+
+            proxy_cache one;
+            proxy_cache_key $uri;
+            proxy_cache_valid any 1s;
+            proxy_no_cache $upstream_http_x_no_cache;
+
+            proxy_cache_revalidate on;
+        }
+
+        location @fallback {
+            return 403;
+        }
+
+        add_header X-Cache-Status $upstream_cache_status always;
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+
+        location / {
+        }
+
+        location /t3 {
+            set $nocache "";
+            if ($arg_expires) {
+                set $nocache "no-cache";
+            }
+            add_header Cache-Control $nocache;
+            add_header Transfer-Encoding invalid;
+        }
+
+        location /t4 {
+            set $nocache "";
+            if ($arg_expires) {
+                set $nocache "no-cache";
+            }
+            add_header Cache-Control $nocache;
+            add_header X-No-Cache $arg_nocache;
+        }
+    }
+}
+
+EOF
+
+$t->write_file('t', 'SEE-THIS');
+$t->write_file('t3', 'SEE-THIS');
+$t->write_file('t4', 'SEE-THIS');
+
+$t->run();
+
+###############################################################################
+
+like(http_get('/t?nocache=1'), qr/MISS.*SEE-THIS/s, 'request');
+like(http_get('/t'), qr/MISS.*SEE-THIS/s, 'request not cached');
+like(http_get('/t'), qr/HIT.*SEE-THIS/s, 'request cached');
+
+# proxy_no_cache with intercepted errors,
+# ngx_http_upstream_intercept_errors()
+
+like(http_get('/t2?nocache=1'), qr/403 Forbidden/, 'intercepted error');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t2'), qr/403 Forbidden.*MISS/s, 'intercepted error not cached');
+
+}
+
+like(http_get('/t2'), qr/403 Forbidden.*HIT/s, 'intercepted error cached');
+
+# proxy_no_cache with internal 502/504 errors,
+# ngx_http_upstream_finalize_request()
+
+like(http_get('/t3?nocache=1'), qr/502 Bad/, 'internal 502 error');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t3?expires=1'), qr/502 Bad.*MISS/s,
+	'internal 502 error expires');
+like(http_get('/t3'), qr/502 Bad.*MISS/s, 'internal 502 error not cached');
+
+}
+
+like(http_get('/t3'), qr/502 Bad.*HIT/s, 'internal 502 error cached');
+
+# proxy_no_cache with revalidate and 304,
+# ngx_http_upstream_test_next()
+
+like(http_get('/t4'), qr/MISS.*SEE-THIS/s, 'revalidate');
+like(http_get('/t4'), qr/HIT.*SEE-THIS/s, 'revalidate cached');
+select undef, undef, undef, 2.5;
+like(http_get('/t4?nocache=1'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate nocache');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/t4?expires=1'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate expires');
+like(http_get('/t4'), qr/REVALIDATED.*SEE-THIS/s,
+	'revalidate again');
+
+}
+
+like(http_get('/t4'), qr/HIT.*SEE-THIS/s, 'revalidate again cached');
+
+###############################################################################


From mdounin at mdounin.ru  Tue Jun 25 20:31:21 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 23:31:21 +0300
Subject: [nginx] Core: fixed ENOSPC handling for error logs.
Message-ID: <hg.af5b47569cb2.1719347481.7427339081749964242@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/af5b47569cb2
branches:  
changeset: 9296:af5b47569cb2
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 22:57:57 2024 +0300
description:
Core: fixed ENOSPC handling for error logs.

For each connection a new ngx_log_t structure is created, and saving
anything into disk_full_time field in this structure doesn't affect
other connections.  Fix is to move the disk_full_time field into the
ngx_open_file_t structure.

diffstat:

 src/core/ngx_conf_file.c |  1 +
 src/core/ngx_conf_file.h |  2 ++
 src/core/ngx_log.c       |  4 ++--
 src/core/ngx_log.h       |  2 --
 4 files changed, 5 insertions(+), 4 deletions(-)

diffs (56 lines):

diff --git a/src/core/ngx_conf_file.c b/src/core/ngx_conf_file.c
--- a/src/core/ngx_conf_file.c
+++ b/src/core/ngx_conf_file.c
@@ -951,6 +951,7 @@ ngx_conf_open_file(ngx_cycle_t *cycle, n
         file->name = *name;
     }
 
+    file->disk_full_time = 0;
     file->flush = NULL;
     file->data = NULL;
 
diff --git a/src/core/ngx_conf_file.h b/src/core/ngx_conf_file.h
--- a/src/core/ngx_conf_file.h
+++ b/src/core/ngx_conf_file.h
@@ -90,6 +90,8 @@ struct ngx_open_file_s {
     ngx_fd_t              fd;
     ngx_str_t             name;
 
+    time_t                disk_full_time;
+
     void                (*flush)(ngx_open_file_t *file, ngx_log_t *log);
     void                 *data;
 };
diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -169,7 +169,7 @@ ngx_log_error_core(ngx_uint_t level, ngx
             goto next;
         }
 
-        if (ngx_time() == log->disk_full_time) {
+        if (ngx_time() == log->file->disk_full_time) {
 
             /*
              * on FreeBSD writing to a full filesystem with enabled softupdates
@@ -183,7 +183,7 @@ ngx_log_error_core(ngx_uint_t level, ngx
         n = ngx_write_fd(log->file->fd, errstr, p - errstr);
 
         if (n == -1 && ngx_errno == NGX_ENOSPC) {
-            log->disk_full_time = ngx_time();
+            log->file->disk_full_time = ngx_time();
         }
 
         if (log->file->fd == ngx_stderr) {
diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h
--- a/src/core/ngx_log.h
+++ b/src/core/ngx_log.h
@@ -53,8 +53,6 @@ struct ngx_log_s {
 
     ngx_atomic_uint_t    connection;
 
-    time_t               disk_full_time;
-
     ngx_log_handler_pt   handler;
     void                *data;
 


From mdounin at mdounin.ru  Tue Jun 25 20:31:21 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 23:31:21 +0300
Subject: [nginx] Core: moved ngx_log_set_levels() to a proper position.
Message-ID: <hg.e1f15d47d102.1719347481.7427339081749964242@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/e1f15d47d102
branches:  
changeset: 9297:e1f15d47d102
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 22:58:11 2024 +0300
description:
Core: moved ngx_log_set_levels() to a proper position.

Previous order is an artifact from the time before 5254:7ecaa9e4bf1b,
when ngx_log_set_levels() was a non-static function.

No functional changes.

diffstat:

 src/core/ngx_log.c |  126 ++++++++++++++++++++++++++--------------------------
 1 files changed, 63 insertions(+), 63 deletions(-)

diffs (143 lines):

diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -476,69 +476,6 @@ ngx_log_get_file_log(ngx_log_t *head)
 
 
 static char *
-ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
-{
-    ngx_uint_t   i, n, d, found;
-    ngx_str_t   *value;
-
-    if (cf->args->nelts == 2) {
-        log->log_level = NGX_LOG_ERR;
-        return NGX_CONF_OK;
-    }
-
-    value = cf->args->elts;
-
-    for (i = 2; i < cf->args->nelts; i++) {
-        found = 0;
-
-        for (n = 1; n <= NGX_LOG_DEBUG; n++) {
-            if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
-
-                if (log->log_level != 0) {
-                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                                       "duplicate log level \"%V\"",
-                                       &value[i]);
-                    return NGX_CONF_ERROR;
-                }
-
-                log->log_level = n;
-                found = 1;
-                break;
-            }
-        }
-
-        for (n = 0, d = NGX_LOG_DEBUG_FIRST; d <= NGX_LOG_DEBUG_LAST; d <<= 1) {
-            if (ngx_strcmp(value[i].data, debug_levels[n++]) == 0) {
-                if (log->log_level & ~NGX_LOG_DEBUG_ALL) {
-                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                                       "invalid log level \"%V\"",
-                                       &value[i]);
-                    return NGX_CONF_ERROR;
-                }
-
-                log->log_level |= d;
-                found = 1;
-                break;
-            }
-        }
-
-
-        if (!found) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "invalid log level \"%V\"", &value[i]);
-            return NGX_CONF_ERROR;
-        }
-    }
-
-    if (log->log_level == NGX_LOG_DEBUG) {
-        log->log_level = NGX_LOG_DEBUG_ALL;
-    }
-
-    return NGX_CONF_OK;
-}
-
-
-static char *
 ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_log_t  *dummy;
@@ -673,6 +610,69 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 }
 
 
+static char *
+ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
+{
+    ngx_uint_t   i, n, d, found;
+    ngx_str_t   *value;
+
+    if (cf->args->nelts == 2) {
+        log->log_level = NGX_LOG_ERR;
+        return NGX_CONF_OK;
+    }
+
+    value = cf->args->elts;
+
+    for (i = 2; i < cf->args->nelts; i++) {
+        found = 0;
+
+        for (n = 1; n <= NGX_LOG_DEBUG; n++) {
+            if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
+
+                if (log->log_level != 0) {
+                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                       "duplicate log level \"%V\"",
+                                       &value[i]);
+                    return NGX_CONF_ERROR;
+                }
+
+                log->log_level = n;
+                found = 1;
+                break;
+            }
+        }
+
+        for (n = 0, d = NGX_LOG_DEBUG_FIRST; d <= NGX_LOG_DEBUG_LAST; d <<= 1) {
+            if (ngx_strcmp(value[i].data, debug_levels[n++]) == 0) {
+                if (log->log_level & ~NGX_LOG_DEBUG_ALL) {
+                    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                       "invalid log level \"%V\"",
+                                       &value[i]);
+                    return NGX_CONF_ERROR;
+                }
+
+                log->log_level |= d;
+                found = 1;
+                break;
+            }
+        }
+
+
+        if (!found) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid log level \"%V\"", &value[i]);
+            return NGX_CONF_ERROR;
+        }
+    }
+
+    if (log->log_level == NGX_LOG_DEBUG) {
+        log->log_level = NGX_LOG_DEBUG_ALL;
+    }
+
+    return NGX_CONF_OK;
+}
+
+
 static void
 ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log)
 {


From mdounin at mdounin.ru  Tue Jun 25 20:31:21 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 23:31:21 +0300
Subject: [nginx] Core: simplified log levels matching.
Message-ID: <hg.14770557be17.1719347481.7427339081749964242@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/14770557be17
branches:  
changeset: 9298:14770557be17
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 22:58:36 2024 +0300
description:
Core: simplified log levels matching.

diffstat:

 src/core/ngx_log.c |  29 +++++++++++++----------------
 1 files changed, 13 insertions(+), 16 deletions(-)

diffs (62 lines):

diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -613,18 +613,12 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 static char *
 ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
 {
-    ngx_uint_t   i, n, d, found;
+    ngx_uint_t   i, n, d;
     ngx_str_t   *value;
 
-    if (cf->args->nelts == 2) {
-        log->log_level = NGX_LOG_ERR;
-        return NGX_CONF_OK;
-    }
-
     value = cf->args->elts;
 
     for (i = 2; i < cf->args->nelts; i++) {
-        found = 0;
 
         for (n = 1; n <= NGX_LOG_DEBUG; n++) {
             if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) {
@@ -637,8 +631,7 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
                 }
 
                 log->log_level = n;
-                found = 1;
-                break;
+                goto next;
             }
         }
 
@@ -652,17 +645,21 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
                 }
 
                 log->log_level |= d;
-                found = 1;
-                break;
+                goto next;
             }
         }
 
+        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                           "invalid log level \"%V\"", &value[i]);
+        return NGX_CONF_ERROR;
 
-        if (!found) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                               "invalid log level \"%V\"", &value[i]);
-            return NGX_CONF_ERROR;
-        }
+    next:
+
+        continue;
+    }
+
+    if (log->log_level == 0) {
+        log->log_level = NGX_LOG_ERR;
     }
 
     if (log->log_level == NGX_LOG_DEBUG) {


From mdounin at mdounin.ru  Tue Jun 25 20:31:22 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Tue, 25 Jun 2024 23:31:22 +0300
Subject: [nginx] Core: error logging rate limiting.
Message-ID: <hg.2706b60dc225.1719347482.7427339081749964242@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx/rev/2706b60dc225
branches:  
changeset: 9299:2706b60dc225
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Jun 25 22:58:56 2024 +0300
description:
Core: error logging rate limiting.

With this change, error logging to files can be rate-limited with
the "rate=" parameter.  The parameter specifies allowed log messages
rate to a particular file (per worker), in messages per second (m/s).
By default, "rate=1000m/s" is used.

Rate limiting is implemented using the "leaky bucket" method, similarly
to the limit_req module.

Maximum burst size is set to the number of log messages per second
for each severity level, so "error" messages are logged even if the
rate limit is hit by "info" messages (but not vice versa).  When the
limit is reached for a particular level, the "too many log messages,
limiting" message is logged at this level.

If debug logging is enabled, either for the particular log file or for
the particular connection, rate limiting is not used.

diffstat:

 src/core/ngx_connection.h |    1 +
 src/core/ngx_log.c        |  120 ++++++++++++++++++++++++++++++++++++++++++++-
 src/core/ngx_log.h        |    9 +++
 3 files changed, 125 insertions(+), 5 deletions(-)

diffs (216 lines):

diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h
--- a/src/core/ngx_connection.h
+++ b/src/core/ngx_connection.h
@@ -207,6 +207,7 @@ struct ngx_connection_s {
 #define ngx_set_connection_log(c, l)                                         \
                                                                              \
     c->log->file = l->file;                                                  \
+    c->log->limit = l->limit;                                                \
     c->log->next = l->next;                                                  \
     c->log->writer = l->writer;                                              \
     c->log->wdata = l->wdata;                                                \
diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c
--- a/src/core/ngx_log.c
+++ b/src/core/ngx_log.c
@@ -9,8 +9,9 @@
 #include <ngx_core.h>
 
 
+static ngx_int_t ngx_log_check_rate(ngx_log_t *log, ngx_uint_t level);
 static char *ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
-static char *ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log);
+static char *ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log);
 static void ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log);
 
 
@@ -164,6 +165,12 @@ ngx_log_error_core(ngx_uint_t level, ngx
             break;
         }
 
+        if (log->limit && !debug_connection) {
+            if (ngx_log_check_rate(log, level) == NGX_BUSY) {
+                goto next;
+            }
+        }
+
         if (log->writer) {
             log->writer(log, level, errstr, p - errstr);
             goto next;
@@ -314,6 +321,69 @@ ngx_log_errno(u_char *buf, u_char *last,
 }
 
 
+static ngx_int_t
+ngx_log_check_rate(ngx_log_t *log, ngx_uint_t level)
+{
+    ngx_log_t          temp_log;
+    ngx_int_t          excess, changed, burst;
+    ngx_atomic_int_t   ms;
+    ngx_atomic_uint_t  now, last;
+
+    now = ngx_current_msec;
+
+    last = log->limit->last;
+    excess = log->limit->excess;
+
+    ms = (ngx_atomic_int_t) (now - last);
+
+    if (ms < -60000) {
+        ms = 1;
+
+    } else if (ms < 0) {
+        ms = 0;
+    }
+
+    changed = excess - log->limit->rate * ms / 1000 + 1000;
+
+    if (changed < 0) {
+        changed = 0;
+    }
+
+    burst = (log->log_level - level + 1) * log->limit->rate;
+
+    if (changed > burst) {
+        if (excess <= burst) {
+
+            ngx_atomic_fetch_add(&log->limit->excess, 1000);
+
+            /* log message to this log only */
+
+            temp_log = *log;
+            temp_log.connection = 0;
+            temp_log.handler = NULL;
+            temp_log.limit = NULL;
+            temp_log.next = NULL;
+
+            ngx_log_error(level, &temp_log, 0,
+                          "too many log messages, limiting");
+        }
+
+        return NGX_BUSY;
+    }
+
+    if (ms > 0
+        && ngx_atomic_cmp_set(&log->limit->last, last, now))
+    {
+        ngx_atomic_fetch_add(&log->limit->excess, changed - excess);
+
+    } else {
+        ngx_atomic_fetch_add(&log->limit->excess, 1000);
+    }
+
+    return NGX_OK;
+}
+
+
 ngx_log_t *
 ngx_log_init(u_char *prefix, u_char *error_log)
 {
@@ -598,7 +668,7 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
         }
     }
 
-    if (ngx_log_set_levels(cf, new_log) != NGX_CONF_OK) {
+    if (ngx_log_set_params(cf, new_log) != NGX_CONF_OK) {
         return NGX_CONF_ERROR;
     }
 
@@ -611,13 +681,17 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_
 
 
 static char *
-ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log)
+ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log)
 {
+    size_t       len;
+    ngx_int_t    rate;
     ngx_uint_t   i, n, d;
     ngx_str_t   *value;
 
     value = cf->args->elts;
 
+    rate = 1000;
+
     for (i = 2; i < cf->args->nelts; i++) {
 
         for (n = 1; n <= NGX_LOG_DEBUG; n++) {
@@ -649,8 +723,33 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
             }
         }
 
-        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
-                           "invalid log level \"%V\"", &value[i]);
+        if (ngx_strncmp(value[i].data, "rate=", 5) == 0) {
+
+            len = value[i].len;
+
+            if (ngx_strncmp(value[i].data + len - 3, "m/s", 3) == 0) {
+                len -= 3;
+            }
+
+            rate = ngx_atoi(value[i].data + 5, len - 5);
+            if (rate < 0) {
+                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                   "invalid rate \"%V\"", &value[i]);
+                return NGX_CONF_ERROR;
+            }
+
+            continue;
+        }
+
+        if (log->log_level) {
+            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                               "invalid parameter \"%V\"", &value[i]);
+
+        } else {
+             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+                                "invalid log level \"%V\"", &value[i]);
+        }
+
         return NGX_CONF_ERROR;
 
     next:
@@ -666,6 +765,17 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_l
         log->log_level = NGX_LOG_DEBUG_ALL;
     }
 
+    if (rate > 0
+        && log->log_level < NGX_LOG_DEBUG)
+    {
+        log->limit = ngx_pcalloc(cf->pool, sizeof(ngx_log_limit_t));
+        if (log->limit == NULL) {
+            return NGX_CONF_ERROR;
+        }
+
+        log->limit->rate = rate * 1000;
+    }
+
     return NGX_CONF_OK;
 }
 
diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h
--- a/src/core/ngx_log.h
+++ b/src/core/ngx_log.h
@@ -47,6 +47,13 @@ typedef void (*ngx_log_writer_pt) (ngx_l
     u_char *buf, size_t len);
 
 
+typedef struct {
+    ngx_uint_t           rate;
+    ngx_atomic_t         excess;
+    ngx_atomic_t         last;
+} ngx_log_limit_t;
+
+
 struct ngx_log_s {
     ngx_uint_t           log_level;
     ngx_open_file_t     *file;
@@ -67,6 +74,8 @@ struct ngx_log_s {
 
     char                *action;
 
+    ngx_log_limit_t     *limit;
+
     ngx_log_t           *next;
 };
 


From mdounin at mdounin.ru  Wed Jun 26 00:50:16 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Wed, 26 Jun 2024 03:50:16 +0300
Subject: [PATCH] SSI: fixed incorrect or duplicate stub output
Message-ID: <d78ecfffb8cd81355471.1719363016@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1719362615 -10800
#      Wed Jun 26 03:43:35 2024 +0300
# Node ID d78ecfffb8cd813554716e8d9d29f2007bd94422
# Parent  2706b60dc2255a36677884be43a4367fef081c40
SSI: fixed incorrect or duplicate stub output.

Following 3518:eb3aaf8bd2a9 (0.8.37), r->request_output is only set
if there are data in the first buffer sent in the subrequest.  As a
result, following the change mentioned this flag cannot be used to
prevent duplicate ngx_http_ssi_stub_output() calls, since it is not
set if there was already some output, but the first buffer was empty.

Still, when there are multiple subrequests, even an empty subrequest
response might be delayed by the postpone filter, leading to a second
call of ngx_http_ssi_stub_output() during finalization from
ngx_http_writer() the subreqest buffers are released by the postpone
filter.  Since r->request_output is not set after the first call, this
resulted in duplicate stub output.

Additionally, checking only the first buffer might be wrong in some
unusual cases.  For example, the first buffer might be empty if
$r->flush() is called before printing any data in the embedded Perl
module.

Depending on the postpone_output value and corresponding sizes, this
issue can result in either duplicate or unexpected stub output, or
"zero size buf in writer" alerts.

Following 8124:f5515e727656 (1.23.4), it became slightly easier to
reproduce the issue, as empty static files and empty cache items now
result in a response with an empty buffer.  Before the change, an empty
proxied response can be used to reproduce the issue.

Fix is check all buffers and set r->request_output if any non-empty
buffers are sent.  This ensures that all unusual cases of non-empty
responses are covered, and also that r->request_output will be set
after the first stub output, preventing duplicate output.

Reported by Jan Gassen.

diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c
+++ b/src/http/ngx_http_copy_filter_module.c
@@ -83,6 +83,7 @@ static ngx_int_t
 ngx_http_copy_filter(ngx_http_request_t *r, ngx_chain_t *in)
 {
     ngx_int_t                     rc;
+    ngx_chain_t                  *cl;
     ngx_connection_t             *c;
     ngx_output_chain_ctx_t       *ctx;
     ngx_http_core_loc_conf_t     *clcf;
@@ -132,9 +133,14 @@ ngx_http_copy_filter(ngx_http_request_t 
             ctx->thread_handler = ngx_http_copy_thread_handler;
         }
 #endif
+    }
 
-        if (in && in->buf && ngx_buf_size(in->buf)) {
-            r->request_output = 1;
+    if (!r->request_output) {
+        for (cl = in; cl; cl = cl->next) {
+            if (ngx_buf_size(cl->buf)) {
+                r->request_output = 1;
+                break;
+            }
         }
     }
 



From mdounin at mdounin.ru  Wed Jun 26 00:52:20 2024
From: mdounin at mdounin.ru (=?utf-8?q?Maxim_Dounin?=)
Date: Wed, 26 Jun 2024 03:52:20 +0300
Subject: [PATCH] Tests: SSI include stub tests
In-Reply-To: <d78ecfffb8cd81355471.1719363016@vm-bsd.mdounin.ru>
References: <d78ecfffb8cd81355471.1719363016@vm-bsd.mdounin.ru>
Message-ID: <302308d4f924d5c68da9.1719363140@vm-bsd.mdounin.ru>

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1719363008 -10800
#      Wed Jun 26 03:50:08 2024 +0300
# Node ID 302308d4f924d5c68da9ffa05ad87007b48aa4e8
# Parent  bf027a972ccf8d6ff141f85e9a41615009e8005b
Tests: SSI include stub tests.

diff --git a/ssi_stub.t b/ssi_stub.t
new file mode 100644
--- /dev/null
+++ b/ssi_stub.t
@@ -0,0 +1,108 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+
+# Tests for nginx ssi module, stub output.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy ssi/)->plan(4)
+	->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+    %%TEST_GLOBALS_HTTP%%
+
+    server {
+        listen       127.0.0.1:8080;
+        server_name  localhost;
+
+        location / {
+            ssi on;
+        }
+
+        location = /empty {
+            # static
+        }
+
+        location = /error404 {
+            # static 404
+        }
+
+        location = /proxy404 {
+            proxy_pass http://127.0.0.1:8081;
+            proxy_intercept_errors on;
+            error_page 404 /error404;
+        }
+
+        location = /not_empty {
+            proxy_pass http://127.0.0.1:8081;
+        }
+    }
+
+    server {
+        listen       127.0.0.1:8081;
+        server_name  localhost;
+    }
+}
+
+EOF
+
+$t->write_file('empty.html',
+	'<!--#block name="fallback" -->fallback<!--#endblock -->' .
+	':<!--#include virtual="/empty" stub="fallback" -->:');
+
+$t->write_file('error.html',
+	'<!--#block name="fallback" -->fallback<!--#endblock -->' .
+	':<!--#include virtual="/error404" stub="fallback" -->:');
+
+$t->write_file('proxy_error.html',
+	'<!--#block name="fallback" -->fallback<!--#endblock -->' .
+	':<!--#include virtual="/proxy404" stub="fallback" -->:');
+
+$t->write_file('postponed.html',
+	'<!--#block name="fallback" -->fallback<!--#endblock -->' .
+	':<!--#include virtual="/not_empty" -->' .
+	':<!--#include virtual="/empty" stub="fallback" -->:');
+
+$t->write_file('empty', '');
+$t->write_file('not_empty', 'not empty');
+
+$t->run();
+
+###############################################################################
+
+like(http_get('/empty.html'), qr/:fallback:/, 'ssi stub empty');
+like(http_get('/error.html'), qr/:fallback:/, 'ssi stub error');
+like(http_get('/proxy_error.html'), qr/:fallback:/, 'ssi stub proxied error');
+
+TODO: {
+local $TODO = 'not yet' unless $t->has_version('1.27.2');
+
+like(http_get('/postponed.html'), qr/:not empty:fallback:/s,
+	'ssi stub postponed');
+
+}
+
+###############################################################################



From hnakamur at gmail.com  Wed Jun 26 11:44:56 2024
From: hnakamur at gmail.com (Hiroaki Nakamura)
Date: Wed, 26 Jun 2024 20:44:56 +0900
Subject: [PATCH 1 of 3] Correctly calculate and set Age header
In-Reply-To: <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
 <Zniy3EFG005hIsXl@mdounin.ru>
 <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
Message-ID: <CAN-DUMT3q-wzyJRucGfEtXMse-RzozsRhe3jURVNW1iLHLhavA@mail.gmail.com>

Hello,

* I have deleted ngx_http_upstream_update_age and changed to
   update age header in ngx_http_upstream_process_headers.
* I have changed to use ngx_atotm for converting age header
  string to time_t.

Thank you,
Hiroaki Nakamura

# HG changeset patch
# User Hiroaki Nakamura <hnakamur at gmail.com>
# Date 1719401770 -32400
#      Wed Jun 26 20:36:10 2024 +0900
# Node ID 57facb7deaca2a323e71cfee28a087534b3ed1c5
# Parent  ed8a5f82e23aa8bd0a4b3ab56d414e12145717ab
Cache: added calculation of the Age header.
* Implement the modified calculation of the Age header as specified in
  RFC 9111 "HTTP Caching", https://www.rfc-editor.org/rfc/rfc9111.html

  Calculation initial age as specified in RFC 9111:

  apparent_age = max(0, response_time - date_value);

  response_delay = response_time - request_time;
  corrected_age_value = age_value + response_delay;

  corrected_initial_age = max(apparent_age, corrected_age_value);

  However, if the upstream date is too old, then apparant_age
  becomes too big.

  Therefore, we use the following calculation:

  if upstream date is between request_time and response_time, we assume
  the upstream's clock is reasonably well synchronized and use

  corrected_initial_age = age_value + (response_time - date);

  if upstream date is not present or not between request_time
  and response_time, we do not rely on the upstream date and use

  corrected_initial_age = age_value + (response_time - u->request_time);

  When sending a cached response, we calculate the Age
  as specified in RFC 9111:

  resident_time = now - response_time;
  current_age = corrected_initial_age + resident_time;

  where response_time is cache_creation_time.

* Also ignore Expires and X-Accel-Expires when Cache-Control max-age
  or s-maxage is set as specified in
  https://www.rfc-editor.org/rfc/rfc9111#name-expires:

  "If a response includes a Cache-Control header field with the
  max-age directive (Section 5.2.2.1), a recipient MUST ignore
  the Expires header field. Likewise, if a response includes the
  s-maxage directive (Section 5.2.2.10), a shared cache recipient
  MUST ignore the Expires header field."

diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_cache.h
--- a/src/http/ngx_http_cache.h Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_cache.h Wed Jun 26 20:36:10 2024 +0900
@@ -27,7 +27,7 @@
 #define NGX_HTTP_CACHE_ETAG_LEN      128
 #define NGX_HTTP_CACHE_VARY_LEN      128

-#define NGX_HTTP_CACHE_VERSION       5
+#define NGX_HTTP_CACHE_VERSION       6


 typedef struct {
@@ -59,6 +59,8 @@ typedef struct {
     size_t                           body_start;
     off_t                            fs_size;
     ngx_msec_t                       lock_time;
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_node_t;


@@ -75,6 +77,8 @@ struct ngx_http_cache_s {
     time_t                           error_sec;
     time_t                           last_modified;
     time_t                           date;
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;

     ngx_str_t                        etag;
     ngx_str_t                        vary;
@@ -141,6 +145,8 @@ typedef struct {
     u_char                           vary_len;
     u_char                           vary[NGX_HTTP_CACHE_VARY_LEN];
     u_char                           variant[NGX_HTTP_CACHE_KEY_LEN];
+    time_t                           creation_time;
+    time_t                           corrected_initial_age;
 } ngx_http_file_cache_header_t;


diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_file_cache.c Wed Jun 26 20:36:10 2024 +0900
@@ -627,6 +627,8 @@ ngx_http_file_cache_read(ngx_http_reques
     c->body_start = h->body_start;
     c->etag.len = h->etag_len;
     c->etag.data = h->etag;
+    c->creation_time = h->creation_time;
+    c->corrected_initial_age = h->corrected_initial_age;

     r->cached = 1;

@@ -971,6 +973,8 @@ renew:
     fcn->uniq = 0;
     fcn->body_start = 0;
     fcn->fs_size = 0;
+    fcn->creation_time = 0;
+    fcn->corrected_initial_age = 0;

 done:

@@ -980,6 +984,8 @@ done:

     c->uniq = fcn->uniq;
     c->error = fcn->error;
+    c->creation_time = fcn->creation_time;
+    c->corrected_initial_age = fcn->corrected_initial_age;
     c->node = fcn;

 failed:
@@ -1326,6 +1332,8 @@ ngx_http_file_cache_set_header(ngx_http_
     h->valid_msec = (u_short) c->valid_msec;
     h->header_start = (u_short) c->header_start;
     h->body_start = (u_short) c->body_start;
+    h->creation_time = c->creation_time;
+    h->corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h->etag_len = (u_char) c->etag.len;
@@ -1590,6 +1598,8 @@ ngx_http_file_cache_update_header(ngx_ht
     h.valid_msec = (u_short) c->valid_msec;
     h.header_start = (u_short) c->header_start;
     h.body_start = (u_short) c->body_start;
+    h.creation_time = c->creation_time;
+    h.corrected_initial_age = c->corrected_initial_age;

     if (c->etag.len <= NGX_HTTP_CACHE_ETAG_LEN) {
         h.etag_len = (u_char) c->etag.len;
diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_header_filter_module.c
--- a/src/http/ngx_http_header_filter_module.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_header_filter_module.c Wed Jun 26 20:36:10 2024 +0900
@@ -322,6 +322,10 @@ ngx_http_header_filter(ngx_http_request_
         len += sizeof("Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT" CRLF) - 1;
     }

+    if (r->headers_out.age_n != -1) {
+        len += sizeof("Age: ") - 1 + NGX_TIME_T_LEN + 2;
+    }
+
     c = r->connection;

     if (r->headers_out.location
@@ -518,6 +522,10 @@ ngx_http_header_filter(ngx_http_request_
         *b->last++ = CR; *b->last++ = LF;
     }

+    if (r->headers_out.age_n != -1) {
+        b->last = ngx_sprintf(b->last, "Age: %T" CRLF, r->headers_out.age_n);
+    }
+
     if (host.data) {

         p = b->last + sizeof("Location: ") - 1;
diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_request.c
--- a/src/http/ngx_http_request.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_request.c Wed Jun 26 20:36:10 2024 +0900
@@ -646,6 +646,7 @@ ngx_http_alloc_request(ngx_connection_t
     r->headers_in.keep_alive_n = -1;
     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;

     r->uri_changes = NGX_HTTP_MAX_URI_CHANGES + 1;
     r->subrequests = NGX_HTTP_MAX_SUBREQUESTS + 1;
diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_request.h
--- a/src/http/ngx_http_request.h Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_request.h Wed Jun 26 20:36:10 2024 +0900
@@ -291,6 +291,7 @@ typedef struct {
     off_t                             content_offset;
     time_t                            date_time;
     time_t                            last_modified_time;
+    time_t                            age_n;
 } ngx_http_headers_out_t;


diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_special_response.c
--- a/src/http/ngx_http_special_response.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_special_response.c Wed Jun 26 20:36:10 2024 +0900
@@ -581,6 +581,7 @@ ngx_http_clean_header(ngx_http_request_t

     r->headers_out.content_length_n = -1;
     r->headers_out.last_modified_time = -1;
+    r->headers_out.age_n = -1;
 }


diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_upstream.c Wed Jun 26 20:36:10 2024 +0900
@@ -136,6 +136,8 @@ static ngx_int_t
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_process_vary(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
+static ngx_int_t ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t ngx_http_upstream_copy_header_line(ngx_http_request_t *r,
     ngx_table_elt_t *h, ngx_uint_t offset);
 static ngx_int_t
@@ -323,6 +325,10 @@ static ngx_http_upstream_header_t  ngx_h
                  ngx_http_upstream_copy_header_line,
                  offsetof(ngx_http_headers_out_t, content_encoding), 0 },

+    { ngx_string("Age"),
+                 ngx_http_upstream_process_age, 0,
+                 ngx_http_upstream_ignore_header_line, 0, 0 },
+
     { ngx_null_string, NULL, 0, NULL, 0, 0 }
 };

@@ -507,6 +513,7 @@ ngx_http_upstream_create(ngx_http_reques

     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     return NGX_OK;
 }
@@ -961,6 +968,7 @@ ngx_http_upstream_cache(ngx_http_request
         c->updating_sec = 0;
         c->error_sec = 0;

+        u->headers_in.relative_freshness = 0;
         u->buffer.start = NULL;
         u->cache_status = NGX_HTTP_CACHE_EXPIRED;

@@ -1057,6 +1065,7 @@ ngx_http_upstream_cache_get(ngx_http_req
 static ngx_int_t
 ngx_http_upstream_cache_send(ngx_http_request_t *r, ngx_http_upstream_t *u)
 {
+    time_t             date_when_revalidated, age_when_revalidated;
     ngx_int_t          rc;
     ngx_http_cache_t  *c;

@@ -1068,6 +1077,9 @@ ngx_http_upstream_cache_send(ngx_http_re
         return ngx_http_cache_send(r);
     }

+    date_when_revalidated = u->headers_in.date_when_revalidated;
+    age_when_revalidated = u->headers_in.age_n;
+
     /* TODO: cache stack */

     u->buffer = *c->buf;
@@ -1076,6 +1088,7 @@ ngx_http_upstream_cache_send(ngx_http_re
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -1094,6 +1107,11 @@ ngx_http_upstream_cache_send(ngx_http_re
     rc = u->process_header(r);

     if (rc == NGX_OK) {
+        if (u->cache_status == NGX_HTTP_CACHE_REVALIDATED) {
+            /* use date and age from upstream when revalidated */
+            u->headers_in.date_when_revalidated = date_when_revalidated;
+            u->headers_in.age_n = age_when_revalidated;
+        }

         rc = ngx_http_upstream_process_headers(r, u);

@@ -1557,6 +1575,7 @@ ngx_http_upstream_connect(ngx_http_reque
     ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));

     u->start_time = ngx_current_msec;
+    u->request_time = ngx_time();

     u->state->response_time = (ngx_msec_t) -1;
     u->state->connect_time = (ngx_msec_t) -1;
@@ -2016,6 +2035,7 @@ ngx_http_upstream_reinit(ngx_http_reques
     ngx_memzero(&u->headers_in, sizeof(ngx_http_upstream_headers_in_t));
     u->headers_in.content_length_n = -1;
     u->headers_in.last_modified_time = -1;
+    u->headers_in.age_n = -1;

     if (ngx_list_init(&u->headers_in.headers, r->pool, 8,
                       sizeof(ngx_table_elt_t))
@@ -2648,6 +2668,10 @@ ngx_http_upstream_test_next(ngx_http_req
         }

         u->cache_status = NGX_HTTP_CACHE_REVALIDATED;
+        u->headers_in.date_when_revalidated = u->headers_in.date == NULL ? -1
+                                              : ngx_parse_http_time(
+
u->headers_in.date->value.data,
+
u->headers_in.date->value.len);
         rc = ngx_http_upstream_cache_send(r, u);

         if (rc == NGX_DONE) {
@@ -2670,7 +2694,12 @@ ngx_http_upstream_test_next(ngx_http_req
                 valid = ngx_http_file_cache_valid(u->conf->cache_valid,
                                                   u->headers_in.status_n);
                 if (valid) {
-                    valid = now + valid;
+                    time_t age = r->cache->corrected_initial_age;
+                    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                                   "adjust cache valid_sec:%T, "
+                                   "valid:%T, init_age:%d for 304",
+                                   now + valid - age, valid, age);
+                    valid = now + valid - age;
                 }
             }

@@ -2772,6 +2801,7 @@ ngx_http_upstream_intercept_errors(ngx_h
                                                           status);
                         if (valid) {
                             r->cache->valid_sec = ngx_time() + valid;
+                            u->headers_in.relative_freshness = 1;
                         }
                     }

@@ -2977,6 +3007,109 @@ ngx_http_upstream_process_headers(ngx_ht
     r->headers_out.status_line = u->headers_in.status_line;

     r->headers_out.content_length_n = u->headers_in.content_length_n;
+    r->headers_out.age_n = u->headers_in.age_n;
+
+#if (NGX_HTTP_CACHE)
+    if (r->cache) {
+        ngx_http_cache_t  *c;
+
+        c = r->cache;
+
+        if (u->cache_status == NGX_HTTP_CACHE_HIT
+            || u->cache_status == NGX_HTTP_CACHE_STALE)
+        {
+            time_t  resident_time, current_age;
+
+            /*
+             * Update age response header.
+             * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+             *
+             * resident_time = now - response_time;
+             * current_age = corrected_initial_age + resident_time;
+             */
+            resident_time = ngx_time() - c->creation_time;
+            current_age = c->corrected_initial_age + resident_time;
+            r->headers_out.age_n = current_age ? current_age : -1;
+            ngx_log_debug4(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                           "http upstream process headers, cache_status:%d,"
+                           " created:%T, resident:%T, age:%T",
+                           u->cache_status, c->creation_time, resident_time,
+                           current_age);
+
+        } else if (u->cache_status == NGX_HTTP_CACHE_MISS
+                   || u->cache_status == NGX_HTTP_CACHE_REVALIDATED)
+        {
+            time_t  response_time, age_value, date, corrected_initial_age;
+
+            /*
+             * Update age response header.
+             * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
+             *
+             * apparent_age = max(0, response_time - date_value);
+             *
+             * response_delay = response_time - request_time;
+             * corrected_age_value = age_value + response_delay;
+             *
+             * corrected_initial_age = max(apparent_age, corrected_age_value);
+             *
+             * However, if the upstream date is too old, then apparant_age
+             * becomes too big.
+             *
+             * Therefore, we use the following calculation:
+             *
+             * if upstream date is between request_time and response_time, we
+             * assume the upstream's clock is reasonably well synchronized and
+             * use
+             *
+             * corrected_initial_age = age_value + (response_time - date);
+             *
+             * if upstream date is not present or not between request_time
+             * and response_time, we do not rely on the upstream date and use
+             *
+             * corrected_initial_age = age_value
+             *                         + (response_time - u->request_time);
+             */
+            response_time = ngx_time();
+
+            age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
+
+            if (u->cache_status == NGX_HTTP_CACHE_REVALIDATED) {
+                date = u->headers_in.date_when_revalidated;
+            } else {
+                date = u->headers_in.date == NULL ? -1
+                       : ngx_parse_http_time(u->headers_in.date->value.data,
+                                             u->headers_in.date->value.len);
+            }
+
+            if (date >= u->request_time && date <= response_time) {
+                corrected_initial_age = age_value + (response_time - date);
+                ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                               "http upstream process headers,
cache_status:%d,"
+                               " age:%T, upstream date ok",
+                               u->cache_status, corrected_initial_age);
+            } else {
+                corrected_initial_age = age_value +
+                                        (response_time - u->request_time);
+                ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                               "http upstream process headers,
cache_status:%d,"
+                               " age:%T, upstream date not ok",
+                               u->cache_status, corrected_initial_age);
+            }
+
+            r->headers_out.age_n = corrected_initial_age
+                                   ? corrected_initial_age : -1;
+
+            c->creation_time = response_time;
+            c->corrected_initial_age = corrected_initial_age;
+            if (u->headers_in.relative_freshness) {
+                c->valid_sec -= corrected_initial_age;
+                ngx_log_debug1(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
+                               "http upstream adjusted cache valid_sec:%T",
+                               c->valid_sec);
+            }
+        }
+    }
+#endif

     r->disable_not_modified = !u->cacheable;

@@ -3188,6 +3321,7 @@ ngx_http_upstream_send_response(ngx_http
                                               u->headers_in.status_n);
             if (valid) {
                 r->cache->valid_sec = now + valid;
+                u->headers_in.relative_freshness = 1;
             }
         }

@@ -4670,6 +4804,7 @@ ngx_http_upstream_finalize_request(ngx_h

                 if (valid) {
                     r->cache->valid_sec = ngx_time() + valid;
+                    u->headers_in.relative_freshness = 1;
                     r->cache->error = rc;
                 }
             }
@@ -4945,6 +5080,7 @@ ngx_http_upstream_process_cache_control(
         }

         r->cache->valid_sec = ngx_time() + n;
+        u->headers_in.relative_freshness = 1;
         u->headers_in.expired = 0;
     }

@@ -5047,7 +5183,18 @@ ngx_http_upstream_process_expires(ngx_ht
         return NGX_OK;
     }

-    r->cache->valid_sec = expires;
+    /*
+     * https://www.rfc-editor.org/rfc/rfc9111#name-expires
+     *
+     * If a response includes a Cache-Control header field with the max-age
+     * directive (Section 5.2.2.1), a recipient MUST ignore the Expires
+     * header field. Likewise, if a response includes the s-maxage directive
+     * (Section 5.2.2.10), a shared cache recipient MUST ignore the Expires
+     * header field.
+     */
+    if (!u->headers_in.relative_freshness) {
+        r->cache->valid_sec = expires;
+    }
     }
 #endif

@@ -5107,6 +5254,7 @@ ngx_http_upstream_process_accel_expires(

         default:
             r->cache->valid_sec = ngx_time() + n;
+            u->headers_in.relative_freshness = 1;
             u->headers_in.no_cache = 0;
             u->headers_in.expired = 0;
             return NGX_OK;
@@ -5119,7 +5267,18 @@ ngx_http_upstream_process_accel_expires(
     n = ngx_atoi(p, len);

     if (n != NGX_ERROR) {
-        r->cache->valid_sec = n;
+        /*
+        * https://www.rfc-editor.org/rfc/rfc9111#name-expires
+        *
+        * If a response includes a Cache-Control header field with the max-age
+        * directive (Section 5.2.2.1), a recipient MUST ignore the Expires
+        * header field. Likewise, if a response includes the s-maxage directive
+        * (Section 5.2.2.10), a shared cache recipient MUST ignore the Expires
+        * header field.
+        */
+        if (!u->headers_in.relative_freshness) {
+            r->cache->valid_sec = n;
+        }
         u->headers_in.no_cache = 0;
         u->headers_in.expired = 0;
     }
@@ -5373,6 +5532,38 @@ ngx_http_upstream_process_vary(ngx_http_


 static ngx_int_t
+ngx_http_upstream_process_age(ngx_http_request_t *r,
+    ngx_table_elt_t *h, ngx_uint_t offset)
+{
+    ngx_http_upstream_t  *u;
+
+    u = r->upstream;
+
+    if (u->headers_in.age) {
+        ngx_log_error(NGX_LOG_WARN, r->connection->log, 0,
+                      "ignore duplicate age header from upstream: \"%V: %V\", "
+                      "previous value: \"%V: %V\"",
+                      &h->key, &h->value,
+                      &u->headers_in.age->key,
+                      &u->headers_in.age->value);
+        return NGX_OK;
+    }
+
+    h->next = NULL;
+    u->headers_in.age = h;
+    u->headers_in.age_n = ngx_atotm(h->value.data, h->value.len);
+
+    if (u->headers_in.age_n == NGX_ERROR) {
+        ngx_log_error(NGX_LOG_WARN, r->connection->log, 0,
+                      "ignore invalid \"Age\" header from upstream: "
+                      "\"%V: %V\"", &h->key, &h->value);
+    }
+
+    return NGX_OK;
+}
+
+
+static ngx_int_t
 ngx_http_upstream_copy_header_line(ngx_http_request_t *r, ngx_table_elt_t *h,
     ngx_uint_t offset)
 {
diff -r ed8a5f82e23a -r 57facb7deaca src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/ngx_http_upstream.h Wed Jun 26 20:36:10 2024 +0900
@@ -287,14 +287,18 @@ typedef struct {

     ngx_table_elt_t                 *cache_control;
     ngx_table_elt_t                 *set_cookie;
+    ngx_table_elt_t                 *age;

     off_t                            content_length_n;
     time_t                           last_modified_time;
+    time_t                           age_n;
+    time_t                           date_when_revalidated;

     unsigned                         connection_close:1;
     unsigned                         chunked:1;
     unsigned                         no_cache:1;
     unsigned                         expired:1;
+    unsigned                         relative_freshness:1;
 } ngx_http_upstream_headers_in_t;


@@ -369,6 +373,7 @@ struct ngx_http_upstream_s {
                                          ngx_table_elt_t *h);

     ngx_msec_t                       start_time;
+    time_t                           request_time;

     ngx_http_upstream_state_t       *state;

diff -r ed8a5f82e23a -r 57facb7deaca src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/v2/ngx_http_v2.h Wed Jun 26 20:36:10 2024 +0900
@@ -398,6 +398,7 @@ ngx_int_t ngx_http_v2_table_size(ngx_htt
 #define NGX_HTTP_V2_STATUS_404_INDEX      13
 #define NGX_HTTP_V2_STATUS_500_INDEX      14

+#define NGX_HTTP_V2_AGE_INDEX             21
 #define NGX_HTTP_V2_CONTENT_LENGTH_INDEX  28
 #define NGX_HTTP_V2_CONTENT_TYPE_INDEX    31
 #define NGX_HTTP_V2_DATE_INDEX            33
diff -r ed8a5f82e23a -r 57facb7deaca src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/v2/ngx_http_v2_filter_module.c Wed Jun 26 20:36:10 2024 +0900
@@ -258,6 +258,10 @@ ngx_http_v2_header_filter(ngx_http_reque
         len += 1 + ngx_http_v2_literal_size("Wed, 31 Dec 1986 18:00:00 GMT");
     }

+    if (r->headers_out.age_n != -1) {
+        len += 1 + ngx_http_v2_integer_octets(NGX_TIME_T_LEN) + NGX_TIME_T_LEN;
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -552,6 +556,18 @@ ngx_http_v2_header_filter(ngx_http_reque
         pos = ngx_http_v2_write_value(pos, pos, len, tmp);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
+                       "http2 output header: \"age: %T\"",
+                       r->headers_out.age_n);
+
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_AGE_INDEX);
+
+        p = pos;
+        pos = ngx_sprintf(pos + 1, "%T", r->headers_out.age_n);
+        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (pos - p - 1);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, fc->log, 0,
                        "http2 output header: \"location: %V\"",
diff -r ed8a5f82e23a -r 57facb7deaca src/http/v3/ngx_http_v3_filter_module.c
--- a/src/http/v3/ngx_http_v3_filter_module.c Wed Jun 26 20:36:03 2024 +0900
+++ b/src/http/v3/ngx_http_v3_filter_module.c Wed Jun 26 20:36:10 2024 +0900
@@ -13,6 +13,7 @@
 /* static table indices */
 #define NGX_HTTP_V3_HEADER_AUTHORITY                 0
 #define NGX_HTTP_V3_HEADER_PATH_ROOT                 1
+#define NGX_HTTP_V3_HEADER_AGE_ZERO                  2
 #define NGX_HTTP_V3_HEADER_CONTENT_LENGTH_ZERO       4
 #define NGX_HTTP_V3_HEADER_DATE                      6
 #define NGX_HTTP_V3_HEADER_LAST_MODIFIED             10
@@ -213,6 +214,15 @@ ngx_http_v3_header_filter(ngx_http_reque
                                   sizeof("Mon, 28 Sep 1970 06:00:00 GMT") - 1);
     }

+    if (r->headers_out.age_n > 0) {
+        len += ngx_http_v3_encode_field_lri(NULL, 0,
+                                            NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                            NULL, NGX_TIME_T_LEN);
+    } else if (r->headers_out.age_n == 0) {
+        len += ngx_http_v3_encode_field_ri(NULL, 0,
+                                           NGX_HTTP_V3_HEADER_AGE_ZERO);
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {

         if (r->headers_out.location->value.data[0] == '/'
@@ -452,6 +462,27 @@ ngx_http_v3_header_filter(ngx_http_reque
                                               p, n);
     }

+    if (r->headers_out.age_n != -1) {
+        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
+                       "http3 output header: \"age: %T\"",
+                       r->headers_out.age_n);
+
+        if (r->headers_out.age_n > 0) {
+            p = ngx_sprintf(b->last, "%T", r->headers_out.age_n);
+            n = p - b->last;
+
+            b->last = (u_char *) ngx_http_v3_encode_field_lri(b->last, 0,
+                                                   NGX_HTTP_V3_HEADER_AGE_ZERO,
+                                                   NULL, n);
+
+            b->last = ngx_sprintf(b->last, "%T", r->headers_out.age_n);

+
+        } else {
+            b->last = (u_char *) ngx_http_v3_encode_field_ri(b->last, 0,
+                                                  NGX_HTTP_V3_HEADER_AGE_ZERO);
+        }
+    }
+
     if (r->headers_out.location && r->headers_out.location->value.len) {
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
                        "http3 output header: \"location: %V\"",


From mdounin at mdounin.ru  Thu Jun 27 00:22:35 2024
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Thu, 27 Jun 2024 03:22:35 +0300
Subject: [PATCH 1 of 3] Correctly calculate and set Age header
In-Reply-To: <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
References: <CAN-DUMQwyKd07fkbpa5osUnAH0sJvdwT=QgO6aGtCpqmn2iBNw@mail.gmail.com>
 <Zniy3EFG005hIsXl@mdounin.ru>
 <CAN-DUMQzHifqXq4Uc+gRQqNG4pKuNcz=Zwz_+K+Srx-QKWW34Q@mail.gmail.com>
Message-ID: <Znywy92srHcefS69@mdounin.ru>

Hello!

On Tue, Jun 25, 2024 at 10:47:55PM +0900, Hiroaki Nakamura wrote:

> 2024?6?24?(?) 9:02 Maxim Dounin <mdounin at mdounin.ru>:
> >
> > Hello!
> >
> > On Thu, Jun 20, 2024 at 08:39:43PM +0900, Hiroaki Nakamura wrote:
> 
> > Note that it might be a good idea to update the commit log to
> > follow style rules, such as:
> >
> > : Cache: added calculation of the Age header.
> > :
> > : Implement the calculation of the Age header as specified in
> > : RFC 9111 "HTTP Caching", https://www.rfc-editor.org/rfc/rfc9111.html
> 
> I have updated the commit message.

The commit message certainly needs more work.  Still, you may want 
to postpone it, as well as code modifications, before various 
conceptual questions are cleaned up.

> > > @@ -75,6 +77,8 @@
> > >      time_t                           error_sec;
> > >      time_t                           last_modified;
> > >      time_t                           date;
> > > +    time_t                           response_time;
> >
> > Note that the "response_time" field being added seems to be
> > exactly equivalent to the exiting "date" field.
> 
> Yes, the "response_time" is equivalent to the "date" field when
> receiving a response from the upstream.
> However the 'date" is updated when sending a cached response.
> We need to keep the cache creation time.
> I have renamed "response_time" to "creation_time" to make it clearer.

The "date" field is set when the cache item is created, and 
can be updated on cache item revalidation (with 
"proxy_cache_revalidate on").  It is not updated when sending a 
cached response.

[...]

> > > +    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> > > +                   "http file cache send, resp:%O, resident:%d, age:%d",
> > > +                   c->response_time, resident_time, current_age);
> 
> > Please also note that using "%d" for time_t is wrong, as time_t
> > size might be different from the size of int.  For time_t, "%T"
> > should be used instead.
> 
> I have changed to use "%T".

Note that "too many debug logging" comment still applies.

[...]

> > > diff -r 73929a4f3447 -r c81df54e3d03 src/http/ngx_http_upstream.c
> > > --- a/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:24 2024 +0900
> > > +++ b/src/http/ngx_http_upstream.c      Thu Jun 20 20:26:41 2024 +0900
> > > @@ -1549,6 +1559,7 @@
> > >      ngx_memzero(u->state, sizeof(ngx_http_upstream_state_t));
> > >
> > >      u->start_time = ngx_current_msec;
> > > +    u->request_time = ngx_time();
> >
> > Adding a duplicate start time shouldn't be needed.
> 
> It is not a duplicate since u->request_time is real clock time as opposed to
> u->start_time is monotonic clock.
> 
> We need a real clock time of response_time since we calculate
> the difference from the upstream date value.
> 
> Also we use the calculation response_time - request_time
> when the upstream date is not present or not well synchronized (see below),
> so we also need request_time to be a real clock time.

Actually, monotonic clock is much better suited for time 
difference calculation, and that's exactly the reason why it is 
used for u->start_time.  And, since at any particular moment you 
know u->start_time, current monotonic time and current wall clock 
time, it is trivial to calculate wall clock start time if needed 
(but I doubt it's actually needed, see below).

[...]

> > > @@ -2648,7 +2663,12 @@
> > >              valid = ngx_http_file_cache_valid(u->conf->cache_valid,
> > >                                                u->headers_in.status_n);
> > >              if (valid) {
> > > -                valid = now + valid;
> > > +                ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> > > +                               "adjust cache valid_sec:%O, "
> > > +                               "valid:%O, init_age:%d for 304",
> > > +                               now + valid - r->cache->corrected_initial_age,
> > > +                               valid, r->cache->corrected_initial_age);
> > > +                valid = now + valid - r->cache->corrected_initial_age;
> >
> > Not sure if Age calculations should apply to caching time as
> > calculated with proxy_cache_valid directives, but if it does, it
> > should apply consistently, in all cases where proxy_cache_valid
> > times are used.  At least ngx_http_upstream_intercept_errors()
> > case seems to be ignored in the patch.
> 
> I have updated ngx_http_upstream_intercept_errors as well.
> 
> > Also, if Age applies, there should be a way to ignore it,
> > similarly to how Cache-Control can be ignored with the
> > proxy_ignore_headers directive.
> 
> I do not know how to do this.
> Maybe add
>     { ngx_string("Age"), NGX_HTTP_UPSTREAM_IGN_AGE },
> to ngx_http_upstream_ignore_headers_masks?

If at all, yes, this probably should be implemented like this.

Still, for now I'm no convinced that Age needs to be taken into 
account for proxy_cache_valid.

For example, consider a setup with two caching servers: an edge 
cache, a large central cache, and an origin server.  Assuming a 
response without any Cache-Control headers, something like

    proxy_cache_valid 200 1h;

on the edge cache, and

    proxy_cache_valid 200 1d;

on the large cache.

With the suggested patch there will be Age header generated for 
responses returned by the large cache, effectively preventing 
caching on the edge cache.

While such (obviously suboptimal and not desired) behaviour can be 
prevented by using "proxy_ignore_headers Age;" (assuming it is 
implemented), this would require extra otherwise unneeded 
configuration.  Further, this will prevent proper use of the Age 
header for Cache-Control-based caching for other requests.

A better approach might be to limit the Age header to 
Cache-Control-based caching only.  Not sure though, and it 
certainly needs more analysis.

> > > @@ -2672,6 +2692,59 @@
> > >  }
> > >
> > >
> > > +static void
> > > +ngx_http_upstream_update_age(ngx_http_request_t *r, ngx_http_upstream_t *u,
> > > +    time_t now)
> > > +{
> > > +    time_t  response_time, date, apparent_age, response_delay, age_value,
> > > +            corrected_age_value, corrected_initial_age;
> > > +
> > > +    /*
> > > +     * Update age response header.
> > > +     * https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-age
> > > +     */
> > > +    response_time = now;
> > > +    if (u->headers_in.date != NULL) {
> > > +        date = ngx_parse_http_time(u->headers_in.date->value.data,
> > > +                                   u->headers_in.date->value.len);
> > > +        if (date == NGX_ERROR) {
> > > +            date = now;
> > > +        }
> > > +    } else {
> > > +        date = now;
> > > +    }
> > > +    apparent_age = ngx_max(0, response_time - date);
> > > +
> > > +    response_delay = response_time - u->request_time;
> > > +    age_value = u->headers_in.age_n != -1 ? u->headers_in.age_n : 0;
> > > +    corrected_age_value = age_value + response_delay;
> > > +
> > > +    corrected_initial_age = ngx_max(apparent_age, corrected_age_value);
> > > +    r->headers_out.age_n = corrected_initial_age;
> >
> > Note that this approach, as described in RFC 9111 as a possible
> > way to calculate the Age header, implies that time on both proxy
> > server and the origin server must be "reasonably well
> > synchronized", which is not really the case in many practical
> > situations.
> >
> > If the time on the origin server is mostly arbitrary, for example,
> > in the past, apparent_age might be very large, leading to a very
> > large corrected_initial_age as well, preventing caching.
> >
> > As such, I would rather consider a more robust algorithm instead.
> 
> Yes, if the upstream date is too old, then apparant_age becomes too big.
> 
> How about using the following calculation?
> 
> if upstream date is between request_time and response_time, we assume
> the upstream's clock is reasonably well synchronized and use
> 
> corrected_initial_age = age_value + (response_time - date);
> 
> if upstream date is not present or not between request_time
> and response_time, we do not rely on the upstream date and use
> 
> corrected_initial_age = age_value + (response_time - u->request_time);
> 
> When sending a cached response, we calculate the Age
> as specified in RFC 9111:
> 
> resident_time = now - response_time;
> current_age = corrected_initial_age + resident_time;
> 
> where response_time is cache_creation_time.
> 
> I have updated my patchset to use the above calculation.

This looks slightly more reasonable.  At least, this won't result 
in obviously bogus age values if clocks are not synchronized.

Still, a simpler and mostly equivalent solution might be to avoid 
the whole "corrected_initial_age" concept, and use just Age as is, 
combined with the time passed since the cache item was 
created/revalidated.  The time in question is when the response 
headers were received, and usually a good approximation of the 
time when the response was actually generated.

> > > +#if (NGX_HTTP_CACHE)
> > > +    if (r->cache) {
> > > +        r->cache->response_time = response_time;
> > > +        r->cache->corrected_initial_age = corrected_initial_age;
> > > +        if (u->headers_in.adjusting_valid_sec) {
> > > +            r->cache->valid_sec -= corrected_initial_age;
> >
> > The "adjusting_valid_sec" logic looks unclear (probably at least
> > needs a better name) and fragile.  If I'm reading the code
> > correctly, it will do the wrong thing at least if the upstream
> > server returns something like:
> >
> > Cache-Control: max-age=60
> > X-Accel-Expires: @<time>
> >
> > since u->headers_in.adjusting_valid_sec won't be cleared.
> 
> I found that we must ignore Expires when Cache-Control with max-age or s-maxage
> is set according to https://www.rfc-editor.org/rfc/rfc9111#name-expires:
> 
> "If a response includes a Cache-Control header field with the max-age directive
> (Section 5.2.2.1), a recipient MUST ignore the Expires header field. Likewise,
> if a response includes the s-maxage directive (Section 5.2.2.10), a shared cache
> recipient MUST ignore the Expires header field."

The Expires header and X-Accel-Expires header are different 
headers.  The X-Accel-Expires header is a non-standard header 
which takes precedence over both Expires and Cache-Control.

[...]

> > > +            ngx_log_debug2(NGX_LOG_DEBUG_HTTP, u->peer.connection->log, 0,
> > > +                           "http upstream adjusted cache "
> > > +                           "valid_sec:%O, init_age:%O",
> > > +                           r->cache->valid_sec, corrected_initial_age);
> > > +        }
> > > +    }
> > > +#endif
> >
> > If I'm reading the code correctly, this will add the Age
> > header to responses which are not cached.  This somewhat
> > contradicts to what RFC 9111 says in the description of the Age
> > header (https://www.rfc-editor.org/rfc/rfc9111.html#section-5.1-6):
> >
> > : The presence of an Age header field implies that the response was
> > : not generated or validated by the origin server for this request.
> >
> > Please also note that in many cases [free]nginx is expected to be
> > seen as the origin server, even if it caches some upstream server
> > responses according to the configuration (in many cases, ignoring
> > Cache-Control or Expires headers).  In this case using the
> > Age header even on cached responses might not be desired.  It
> > would be great to dig further into possible use cases and make
> > sure all are properly covered.
> 
> I have changed to not add Age if the calculated value is zero and also
> updated tests.

Note that this somewhat contradicts special handling of "Age: 0" 
in HTTP/3 part of your patch.

Also, if I'm reading the patch correctly, as currently implemented 
the code might result in non-zero Age even for directly returned 
non-cached responses.

Also, "Age: 0" might legitimately appear for cached responses 
during the first second after the response was cached.  Not sure 
if it needs to be returned, but given the about RFC quote it might 
be usable for some.

Note well that the "would be great to dig further" comment still 
applies.

[...]

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/


From mdounin at mdounin.ru  Sat Jun 29 00:09:56 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Sat, 29 Jun 2024 03:09:56 +0300
Subject: [nginx-site] Documented $r->log_error() logging level.
Message-ID: <hg.23b9cbb0c11d.1719619796.5844030028749548060@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-site/rev/23b9cbb0c11d
branches:  
changeset: 3089:23b9cbb0c11d
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Jun 14 22:35:31 2024 +0300
description:
Documented $r->log_error() logging level.

diffstat:

 xml/en/docs/http/ngx_http_perl_module.xml |  5 +++--
 xml/ru/docs/http/ngx_http_perl_module.xml |  5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diffs (44 lines):

diff --git a/xml/en/docs/http/ngx_http_perl_module.xml b/xml/en/docs/http/ngx_http_perl_module.xml
--- a/xml/en/docs/http/ngx_http_perl_module.xml
+++ b/xml/en/docs/http/ngx_http_perl_module.xml
@@ -10,7 +10,7 @@
 <module name="Module ngx_http_perl_module"
         link="/en/docs/http/ngx_http_perl_module.html"
         lang="en"
-        rev="7">
+        rev="8">
 
 <section id="summary">
 
@@ -320,7 +320,8 @@ supports redirections to named locations
 <value>message</value>)</literal></tag-name>
 <tag-desc>
 writes the specified <value>message</value> into the
-<link doc="../ngx_core_module.xml" id="error_log"/>.
+<link doc="../ngx_core_module.xml" id="error_log"/>
+at the <literal>error</literal> level.
 If <value>errno</value> is non-zero, an error code and its description
 will be appended to the message.
 </tag-desc>
diff --git a/xml/ru/docs/http/ngx_http_perl_module.xml b/xml/ru/docs/http/ngx_http_perl_module.xml
--- a/xml/ru/docs/http/ngx_http_perl_module.xml
+++ b/xml/ru/docs/http/ngx_http_perl_module.xml
@@ -10,7 +10,7 @@
 <module name="?????? ngx_http_perl_module"
         link="/ru/docs/http/ngx_http_perl_module.html"
         lang="ru"
-        rev="7">
+        rev="8">
 
 <section id="summary">
 
@@ -320,7 +320,8 @@ 1;
 <value>?????????</value>)</literal></tag-name>
 <tag-desc>
 ?????????? ????????? <value>?????????</value> ?
-<link doc="../ngx_core_module.xml" id="error_log"/>.
+<link doc="../ngx_core_module.xml" id="error_log"/>
+?? ?????? <literal>error</literal>.
 ???? <value>???_??????</value> ?????????, ?? ? ????????? ????? ????????
 ??? ?????? ? ?? ????????.
 </tag-desc>

From mdounin at mdounin.ru  Sat Jun 29 00:09:56 2024
From: mdounin at mdounin.ru (=?iso-8859-1?q?Maxim_Dounin?=)
Date: Sat, 29 Jun 2024 03:09:56 +0300
Subject: [nginx-site] Documented the "rate" parameter of the "error_log" ...
Message-ID: <hg.27532d42102b.1719619796.5844030028749548060@ns3.mdounin.ru>

details:   http://freenginx.org/hg/nginx-site/rev/27532d42102b
branches:  
changeset: 3090:27532d42102b
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Sun Jun 16 06:55:30 2024 +0300
description:
Documented the "rate" parameter of the "error_log" directive.

diffstat:

 xml/en/docs/ngx_core_module.xml |  20 ++++++++++++++++++--
 xml/ru/docs/ngx_core_module.xml |  20 ++++++++++++++++++--
 2 files changed, 36 insertions(+), 4 deletions(-)

diffs (90 lines):

diff --git a/xml/en/docs/ngx_core_module.xml b/xml/en/docs/ngx_core_module.xml
--- a/xml/en/docs/ngx_core_module.xml
+++ b/xml/en/docs/ngx_core_module.xml
@@ -10,7 +10,7 @@
 <module name="Core functionality"
         link="/en/docs/ngx_core_module.html"
         lang="en"
-        rev="28">
+        rev="29">
 
 <section id="example" name="Example Configuration">
 
@@ -209,7 +209,10 @@ and should not be set directly by the us
 
 
 <directive name="error_log">
-<syntax><value>file</value> [<value>level</value>]</syntax>
+<syntax>
+    <value>file</value>
+    [<value>level</value>]
+    [<literal>rate</literal>=<value>rate</value>]</syntax>
 <default>logs/error.log error</default>
 <context>main</context>
 <context>http</context>
@@ -258,7 +261,20 @@ For <literal>debug</literal> logging to 
 be built with <literal>--with-debug</literal>,
 see ?<link doc="debugging_log.xml"/>?.
 </note>
+</para>
 
+<para>
+The <literal>rate</literal> parameter (1.27.2) specifies
+the maximum allowed logging rate in messages per second (m/s)
+for each worker process.
+By default, <literal>rate=1000m/s</literal> is used.
+Setting <value>rate</value> to 0 disables rate limiting.
+Additionally, rate limiting is not used
+if logging level is set to <literal>debug</literal> or
+debugging log is enabled for the <link id="debug_connection">connection</link>.
+</para>
+
+<para>
 <note>
 The directive can be specified on the
 <literal>stream</literal> level
diff --git a/xml/ru/docs/ngx_core_module.xml b/xml/ru/docs/ngx_core_module.xml
--- a/xml/ru/docs/ngx_core_module.xml
+++ b/xml/ru/docs/ngx_core_module.xml
@@ -10,7 +10,7 @@
 <module name="???????? ????????????????"
         link="/ru/docs/ngx_core_module.html"
         lang="ru"
-        rev="28">
+        rev="29">
 
 <section id="example" name="?????? ????????????">
 
@@ -208,7 +208,10 @@ env OPENSSL_ALLOW_PROXY_CERTS=1;
 
 
 <directive name="error_log">
-<syntax><value>????</value> [<value>???????</value>]</syntax>
+<syntax>
+    <value>????</value>
+    [<value>???????</value>]
+    [<literal>rate</literal>=<value>????????</value>]</syntax>
 <default>logs/error.log error</default>
 <context>main</context>
 <context>http</context>
@@ -256,7 +259,20 @@ env OPENSSL_ALLOW_PROXY_CERTS=1;
 nginx ? <literal>--with-debug</literal>,
 ??. ?<link doc="debugging_log.xml"/>?.
 </note>
+</para>
 
+<para>
+???????? <literal>rate</literal> (1.27.2) ??????
+??????????? ?????????? ???????? ?????? ? ??? ? ?????????? ? ??????? (m/s)
+??? ??????? ???????? ????????.
+?? ????????? ???????????? <literal>rate=1000m/s</literal>.
+????????? <literal>????????</literal> ? 0 ????????? ??????????? ????????.
+????? ????, ??????????? ???????? ?? ????????????,
+???? ??????? ???? ?????????? ? <literal>debug</literal> ???
+?????????? ??? ??????? ??? <link id="debug_connection">??????????</link>.
+</para>
+
+<para>
 <note>
 ????????? ????? ???? ??????? ??
 ?????? <literal>stream</literal>