[nginx] SSL: logging levels of various errors reported with tlsf...

[Maxim Dounin]

details:   http://freenginx.org/hg/nginx/rev/5166febfac40
branches:  
changeset: 9507:5166febfac40
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Thu Apr 30 07:21:42 2026 +0300
description:
SSL: logging levels of various errors reported with tlsfuzzer.

The following errors were observed during tlsfuzzer runs with OpenSSL
3.0.20, and indicate invalid data got from the client:

SSL_do_handshake() failed (SSL: error:0A000104:SSL routines::invalid ccs message)
SSL_read() failed (SSL: error:0A0000B6:SSL routines::not on record boundary)

And the following error was observed during tlsfuzzer runs with OpenSSL
3.4.5:

SSL_do_handshake() failed (SSL: error:0A000156:SSL routines::required compression algorithm missing)

Accordingly, the SSL_R_INVALID_CCS_MESSAGE ("invalid ccs message"),
SSL_R_NOT_ON_RECORD_BOUNDARY ("not on record boundary"), and
SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING ("required compression
algorithm missing") errors are now logged at the "info" level.

Also, starting with OpenSSL 3.2.0, many errors observed in previous
versions are additionally followed by the SSL_R_RECORD_LAYER_FAILURE
("record layer failure") error, for example:

SSL_do_handshake() failed (SSL: error:0A000119:SSL routines::decryption failed or bad record mac error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:0A000119:SSL routines::decryption failed or bad record mac error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A000092:SSL routines::data length too long error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A0001BB:SSL routines::bad record type error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:0A0001BB:SSL routines::bad record type error:0A000139:SSL routines::record layer failure)

The SSL_R_RECORD_LAYER_FAILURE ("record layer failure") error, however,
might be generated for a number of reasons, including memory allocation
failures, and therefore it is wrong to log all such errors at the "info"
level.  Instead, we now try to fallback to ERR_peek_error() in order to
decide which logging level should be used.

With these changes, no additional errors were observed with OpenSSL 3.0.20,
OpenSSL 3.2.6, OpenSSL 3.3.7, OpenSSL 3.4.5, OpenSSL 3.5.6, OpenSSL 3.6.2,
and OpenSSL 4.0.0.

diffstat:

 src/event/ngx_event_openssl.c |  25 +++++++++++++++++++++++++
 1 files changed, 25 insertions(+), 0 deletions(-)

diffs (56 lines):

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -4048,6 +4048,22 @@ ngx_ssl_connection_error(ngx_connection_
 
         n = ERR_GET_REASON(ERR_peek_last_error());
 
+#ifdef SSL_R_RECORD_LAYER_FAILURE
+
+        if (n == SSL_R_RECORD_LAYER_FAILURE
+            && ERR_GET_LIB(ERR_peek_error()) == ERR_LIB_SSL)
+        {
+            /*
+             * OpenSSL 3.2.0+ returns SSL_R_RECORD_LAYER_FAILURE in the
+             * error queue after many different errors, including memory
+             * allocation failures, so fallback to ERR_peek_error()
+             */
+
+            n = ERR_GET_REASON(ERR_peek_error());
+        }
+
+#endif
+
             /* handshake failures */
         if (n == SSL_R_BAD_CHANGE_CIPHER_SPEC                        /*  103 */
 #ifdef SSL_R_NO_SUITABLE_KEY_SHARE
@@ -4101,6 +4117,9 @@ ngx_ssl_connection_error(ngx_connection_
 #ifdef SSL_R_NO_CIPHERS_PASSED
             || n == SSL_R_NO_CIPHERS_PASSED                          /*  182 */
 #endif
+#ifdef SSL_R_NOT_ON_RECORD_BOUNDARY
+            || n == SSL_R_NOT_ON_RECORD_BOUNDARY                     /*  182 */
+#endif
             || n == SSL_R_NO_CIPHERS_SPECIFIED                       /*  183 */
 #ifdef SSL_R_BAD_CIPHER
             || n == SSL_R_BAD_CIPHER                                 /*  186 */
@@ -4146,6 +4165,9 @@ ngx_ssl_connection_error(ngx_connection_
             || n == SSL_R_MISSING_KEY_SHARE                          /*  258 */
 #endif
             || n == SSL_R_UNSUPPORTED_PROTOCOL                       /*  258 */
+#ifdef SSL_R_INVALID_CCS_MESSAGE
+            || n == SSL_R_INVALID_CCS_MESSAGE                        /*  260 */
+#endif
 #ifdef SSL_R_NO_SHARED_GROUP
             || n == SSL_R_NO_SHARED_GROUP                            /*  266 */
 #endif
@@ -4184,6 +4206,9 @@ ngx_ssl_connection_error(ngx_connection_
 #ifdef SSL_R_UNSAFE_LEGACY_RENEGOTIATION_DISABLED
             || n == SSL_R_UNSAFE_LEGACY_RENEGOTIATION_DISABLED       /*  338 */
 #endif
+#ifdef SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING
+            || n == SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING     /*  342 */
+#endif
 #ifdef SSL_R_SCSV_RECEIVED_WHEN_RENEGOTIATING
             || n == SSL_R_SCSV_RECEIVED_WHEN_RENEGOTIATING           /*  345 */
 #endif