[PATCH] Added the "proxy_upstream_allow_duplicate_chunked" configuration directive
Maxim Dounin
mdounin at mdounin.ru
Mon Aug 18 02:46:40 UTC 2025
Hello!
On Mon, Aug 18, 2025 at 12:46:06AM +0300, Бекасов Геннадий wrote:
> # HG changeset patch
> # User Gennady Bekasov <becasov at yandex.ru>
> # Date 1755456060 -10800
> # Sun Aug 17 21:41:00 2025 +0300
> # Node ID 2a14d182adce88e0ea7bb7ca3bd01e93ee8eee90
> # Parent bdfd605f661eea3d272caf1bd5d85e7c539394ca
> Added the "proxy_upstream_allow_duplicate_chunked" configuration
> directive.
> diff -r bdfd605f661e -r 2a14d182adce
> src/http/modules/ngx_http_proxy_module.c
> --- a/src/http/modules/ngx_http_proxy_module.c Wed Jul 23 21:53:19
> 2025 +0300
> +++ b/src/http/modules/ngx_http_proxy_module.c Sun Aug 17 21:41:00
> 2025 +0300
> @@ -367,6 +367,13 @@
> offsetof(ngx_http_proxy_loc_conf_t,
> upstream.ignore_client_abort),
> NULL },
>
> + { ngx_string("proxy_upstream_allow_duplicate_chunked"),
> +
> NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
> + ngx_conf_set_flag_slot,
> + NGX_HTTP_LOC_CONF_OFFSET,
> + offsetof(ngx_http_proxy_loc_conf_t,
> upstream.upstream_allow_duplicate_chunked),
> + NULL },
> +
I don't think that "proxy_upstream_allow_duplicate_chunked" is a
good name for the directive, just "proxy_allow_duplicate_chunked"
would be enough.
> { ngx_string("proxy_bind"),
> NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_T
> AKE12,
> ngx_http_upstream_bind_set_slot,
> @@ -3364,6 +3371,7 @@
> conf->upstream.request_buffering = NGX_CONF_UNSET;
> conf->upstream.ignore_client_abort = NGX_CONF_UNSET;
> conf->upstream.force_ranges = NGX_CONF_UNSET;
> + conf->upstream.upstream_allow_duplicate_chunked = NGX_CONF_UNSET;
>
> conf->upstream.local = NGX_CONF_UNSET_PTR;
> conf->upstream.socket_keepalive = NGX_CONF_UNSET;
> @@ -3494,6 +3502,9 @@
> ngx_conf_merge_value(conf->upstream.force_ranges,
> prev->upstream.force_ranges, 0);
>
> +
> ngx_conf_merge_value(conf->upstream.upstream_allow_duplicate_chunked,
> +
> prev->upstream.upstream_allow_duplicate_chunked, 0);
> +
> ngx_conf_merge_ptr_value(conf->upstream.local,
> prev->upstream.local, NULL);
>
> diff -r bdfd605f661e -r 2a14d182adce src/http/ngx_http_upstream.c
> --- a/src/http/ngx_http_upstream.c Wed Jul 23 21:53:19 2025 +0300
> +++ b/src/http/ngx_http_upstream.c Sun Aug 17 21:41:00 2025 +0300
> @@ -5300,7 +5300,11 @@
> &h->key, &h->value,
> &u->headers_in.transfer_encoding->key,
> &u->headers_in.transfer_encoding->value);
> - return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> + if (!u->conf->upstream_allow_duplicate_chunked) {
> + return NGX_HTTP_UPSTREAM_INVALID_HEADER;
> + }
> + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
> + "insecure data exchange due to pass duplicate
> \"Transfer-Encoding\" header as enabled by
> upstream_allow_duplicate_chunked parameter");
> }
>
> if (u->headers_in.content_length) {
That's certainly too chatty: there is a message at the "error"
level, and adding another one at "alert", which is expected to be
used for serious issues, such as unexpected system or library
errors, or internal bugs, is certainly not the way to go.
At most there should be just one "warn" here with
"proxy_allow_duplicate_chunked on" in the configuration. But
given that it is "off" by default, and needs to be explicitly
enabled, it might be good enough to not log it at all.
> diff -r bdfd605f661e -r 2a14d182adce src/http/ngx_http_upstream.h
> --- a/src/http/ngx_http_upstream.h Wed Jul 23 21:53:19 2025 +0300
> +++ b/src/http/ngx_http_upstream.h Sun Aug 17 21:41:00 2025 +0300
> @@ -177,6 +177,7 @@
> ngx_flag_t request_buffering;
> ngx_flag_t pass_request_headers;
> ngx_flag_t pass_request_body;
> + ngx_flag_t upstream_allow_duplicate_chunked;
>
> ngx_flag_t ignore_client_abort;
> ngx_flag_t intercept_errors;
I would rather put it somewhere after the next block of flags,
which looks more related.
Patch with the above comments incorporated (and more):
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1755481476 -10800
# Mon Aug 18 04:44:36 2025 +0300
# Node ID 0983c4766d832640556a27df8855cd707dd80e7b
# Parent 96ce0adabccbb0cad7a0150598b933ddd6026070
Proxy: added the "proxy_allow_duplicate_chunked" directive.
This directive allows to accept duplicate "Transfer-Encoding: chunked"
header lines. These are invalid, and rejected since 8033:2bf7792c262e
(1.23.0), yet it turns out there are quite a few homegrown proxies, notably
Java-based ones, which emit such duplicate headers.
The "proxy_allow_duplicate_chunked" directive makes it possible to enable
compatibility with such proxies by ignoring duplicate "Transfer-Encoding:
chunked" headers instead of rejecting them.
Prodded by Gennady Bekasov,
https://github.com/freenginx/nginx/issues/11
diff --git a/src/http/modules/ngx_http_fastcgi_module.c b/src/http/modules/ngx_http_fastcgi_module.c
--- a/src/http/modules/ngx_http_fastcgi_module.c
+++ b/src/http/modules/ngx_http_fastcgi_module.c
@@ -2948,10 +2948,10 @@ ngx_http_fastcgi_create_loc_conf(ngx_con
conf->upstream.intercept_errors = NGX_CONF_UNSET;
- /* "fastcgi_cyclic_temp_file" is disabled */
+ /* the hardcoded values */
conf->upstream.cyclic_temp_file = 0;
-
conf->upstream.change_buffering = 1;
+ conf->upstream.duplicate_chunked = 0;
conf->catch_stderr = NGX_CONF_UNSET_PTR;
diff --git a/src/http/modules/ngx_http_grpc_module.c b/src/http/modules/ngx_http_grpc_module.c
--- a/src/http/modules/ngx_http_grpc_module.c
+++ b/src/http/modules/ngx_http_grpc_module.c
@@ -4431,6 +4431,7 @@ ngx_http_grpc_create_loc_conf(ngx_conf_t
conf->upstream.pass_request_headers = 1;
conf->upstream.pass_request_body = 1;
conf->upstream.force_ranges = 0;
+ conf->upstream.duplicate_chunked = 0;
conf->upstream.pass_trailers = 1;
conf->upstream.preserve_output = 1;
diff --git a/src/http/modules/ngx_http_memcached_module.c b/src/http/modules/ngx_http_memcached_module.c
--- a/src/http/modules/ngx_http_memcached_module.c
+++ b/src/http/modules/ngx_http_memcached_module.c
@@ -627,6 +627,7 @@ ngx_http_memcached_create_loc_conf(ngx_c
conf->upstream.pass_request_headers = 0;
conf->upstream.pass_request_body = 0;
conf->upstream.force_ranges = 1;
+ conf->upstream.duplicate_chunked = 0;
conf->index = NGX_CONF_UNSET;
conf->gzip_flag = NGX_CONF_UNSET_UINT;
diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c
+++ b/src/http/modules/ngx_http_proxy_module.c
@@ -695,6 +695,13 @@ static ngx_command_t ngx_http_proxy_com
offsetof(ngx_http_proxy_loc_conf_t, http09),
NULL },
+ { ngx_string("proxy_allow_duplicate_chunked"),
+ NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
+ ngx_conf_set_flag_slot,
+ NGX_HTTP_LOC_CONF_OFFSET,
+ offsetof(ngx_http_proxy_loc_conf_t, upstream.duplicate_chunked),
+ NULL },
+
#if (NGX_HTTP_SSL)
{ ngx_string("proxy_ssl_session_reuse"),
@@ -3392,6 +3399,7 @@ ngx_http_proxy_create_loc_conf(ngx_conf_
conf->upstream.request_buffering = NGX_CONF_UNSET;
conf->upstream.ignore_client_abort = NGX_CONF_UNSET;
conf->upstream.force_ranges = NGX_CONF_UNSET;
+ conf->upstream.duplicate_chunked = NGX_CONF_UNSET;
conf->upstream.local = NGX_CONF_UNSET_PTR;
conf->upstream.socket_keepalive = NGX_CONF_UNSET;
@@ -3444,9 +3452,8 @@ ngx_http_proxy_create_loc_conf(ngx_conf_
conf->ssl_conf_commands = NGX_CONF_UNSET_PTR;
#endif
- /* "proxy_cyclic_temp_file" is disabled */
+ /* the hardcoded values */
conf->upstream.cyclic_temp_file = 0;
-
conf->upstream.change_buffering = 1;
conf->headers_source = NGX_CONF_UNSET_PTR;
@@ -3523,6 +3530,9 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t
ngx_conf_merge_value(conf->upstream.force_ranges,
prev->upstream.force_ranges, 0);
+ ngx_conf_merge_value(conf->upstream.duplicate_chunked,
+ prev->upstream.duplicate_chunked, 0);
+
ngx_conf_merge_ptr_value(conf->upstream.local,
prev->upstream.local, NULL);
diff --git a/src/http/modules/ngx_http_scgi_module.c b/src/http/modules/ngx_http_scgi_module.c
--- a/src/http/modules/ngx_http_scgi_module.c
+++ b/src/http/modules/ngx_http_scgi_module.c
@@ -1334,10 +1334,10 @@ ngx_http_scgi_create_loc_conf(ngx_conf_t
conf->upstream.intercept_errors = NGX_CONF_UNSET;
- /* "scgi_cyclic_temp_file" is disabled */
+ /* the hardcoded values */
conf->upstream.cyclic_temp_file = 0;
-
conf->upstream.change_buffering = 1;
+ conf->upstream.duplicate_chunked = 0;
ngx_str_set(&conf->upstream.module, "scgi");
diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c
--- a/src/http/modules/ngx_http_uwsgi_module.c
+++ b/src/http/modules/ngx_http_uwsgi_module.c
@@ -1578,10 +1578,10 @@ ngx_http_uwsgi_create_loc_conf(ngx_conf_
conf->ssl_conf_commands = NGX_CONF_UNSET_PTR;
#endif
- /* "uwsgi_cyclic_temp_file" is disabled */
+ /* the hardcoded values */
conf->upstream.cyclic_temp_file = 0;
-
conf->upstream.change_buffering = 1;
+ conf->upstream.duplicate_chunked = 0;
ngx_str_set(&conf->upstream.module, "uwsgi");
diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -5292,7 +5292,7 @@ ngx_http_upstream_process_transfer_encod
u = r->upstream;
- if (u->headers_in.transfer_encoding) {
+ if (u->headers_in.transfer_encoding && !u->conf->duplicate_chunked) {
ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
"upstream sent duplicate header line: \"%V: %V\", "
"previous value: \"%V: %V\"",
diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h
--- a/src/http/ngx_http_upstream.h
+++ b/src/http/ngx_http_upstream.h
@@ -182,6 +182,7 @@ typedef struct {
ngx_flag_t intercept_errors;
ngx_flag_t cyclic_temp_file;
ngx_flag_t force_ranges;
+ ngx_flag_t duplicate_chunked;
ngx_path_t *temp_path;
And tests:
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1755485120 -10800
# Mon Aug 18 05:45:20 2025 +0300
# Node ID 29e233ff3ad706c93210884b99cbc80840537bbc
# Parent b579440daf23d27ff4a72d33e01370d0eac101cf
Tests: tests for the "proxy_allow_duplicate_chunked" directive.
diff --git a/proxy_duplicate_chunked.t b/proxy_duplicate_chunked.t
new file mode 100644
--- /dev/null
+++ b/proxy_duplicate_chunked.t
@@ -0,0 +1,121 @@
+#!/usr/bin/perl
+
+# (C) Maxim Dounin
+
+# Test for http backend returning response with duplicate "Transfer-Encoding:
+# chunked" headers and the "proxy_allow_duplicate_chunked" directive.
+
+###############################################################################
+
+use warnings;
+use strict;
+
+use Test::More;
+use Socket qw/ CRLF /;
+
+BEGIN { use FindBin; chdir($FindBin::Bin); }
+
+use lib 'lib';
+use Test::Nginx;
+
+###############################################################################
+
+select STDERR; $| = 1;
+select STDOUT; $| = 1;
+
+my $t = Test::Nginx->new()->has(qw/http proxy/);
+
+$t->write_file_expand('nginx.conf', <<'EOF');
+
+%%TEST_GLOBALS%%
+
+daemon off;
+
+events {
+}
+
+http {
+ %%TEST_GLOBALS_HTTP%%
+
+ server {
+ listen 127.0.0.1:8080;
+ server_name localhost;
+
+ location / {
+ proxy_pass http://127.0.0.1:8081;
+ proxy_read_timeout 1s;
+ }
+
+ location /allow/ {
+ proxy_pass http://127.0.0.1:8081;
+ proxy_read_timeout 1s;
+ proxy_allow_duplicate_chunked on;
+ }
+ }
+}
+
+EOF
+
+$t->run_daemon(\&http_daemon);
+$t->try_run('no proxy_allow_duplicate_chunked')->plan(3);
+$t->waitforsocket('127.0.0.1:' . port(8081));
+
+###############################################################################
+
+like(http_get('/'), qr/200 OK/, 'normal');
+
+like(http_get('/duplicate-chunked'), qr/502 Bad/,
+ 'duplicate transfer encoding');
+
+like(http_get('/allow/duplicate-chunked'), qr/200 OK/,
+ 'duplicate transfer encoding allowed');
+
+###############################################################################
+
+sub http_daemon {
+ my $server = IO::Socket::INET->new(
+ Proto => 'tcp',
+ LocalAddr => '127.0.0.1:' . port(8081),
+ Listen => 5,
+ Reuse => 1
+ )
+ or die "Can't create listening socket: $!\n";
+
+ local $SIG{PIPE} = 'IGNORE';
+
+ while (my $client = $server->accept()) {
+ $client->autoflush(1);
+
+ my $headers = '';
+ my $uri = '';
+
+ while (<$client>) {
+ $headers .= $_;
+ last if (/^\x0d?\x0a?$/);
+ }
+
+ $uri = $1 if $headers =~ /^\S+\s+([^ ]+)\s+HTTP/i;
+
+ if ($uri eq '/') {
+
+ print $client
+ 'HTTP/1.1 200 OK' . CRLF .
+ 'Connection: close' . CRLF .
+ 'Content-Length: 0' . CRLF . CRLF;
+
+ } elsif ($uri =~ m/duplicate-chunked/) {
+
+ print $client
+ 'HTTP/1.1 200 OK' . CRLF .
+ 'Connection: close' . CRLF .
+ 'Transfer-Encoding: chunked' . CRLF .
+ 'Transfer-Encoding: chunked' . CRLF . CRLF .
+ '0' . CRLF . CRLF;
+
+ }
+
+ close $client;
+ }
+}
+
+###############################################################################
--
Maxim Dounin
http://mdounin.ru/
More information about the nginx-devel
mailing list