[PATCH 1 of 2] SSL: logging levels of various errors reported with tlsfuzzer

[Maxim Dounin]

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1776912197 -10800
#      Thu Apr 23 05:43:17 2026 +0300
# Node ID b49a45f7121b199c9e5fce6e3a1359fd6deb1943
# Parent  5158f263a08ebf7711392744296a1c12261924e3
SSL: logging levels of various errors reported with tlsfuzzer.

The following errors were observed during tlsfuzzer runs with OpenSSL
3.0.20, and indicate invalid data got from the client:

SSL_do_handshake() failed (SSL: error:0A000104:SSL routines::invalid ccs message)
SSL_read() failed (SSL: error:0A0000B6:SSL routines::not on record boundary)

And the following error was observed during tlsfuzzer runs with OpenSSL
3.4.5:

SSL_do_handshake() failed (SSL: error:0A000156:SSL routines::required compression algorithm missing)

Accordingly, the SSL_R_INVALID_CCS_MESSAGE ("invalid ccs message"),
SSL_R_NOT_ON_RECORD_BOUNDARY ("not on record boundary"), and
SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING ("required compression
algorithm missing") errors are now logged at the "info" level.

Also, starting with OpenSSL 3.2.0, many errors observed in previous
versions are additionally followed by the SSL_R_RECORD_LAYER_FAILURE
("record layer failure") error, for example:

SSL_do_handshake() failed (SSL: error:0A000119:SSL routines::decryption failed or bad record mac error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:0A000119:SSL routines::decryption failed or bad record mac error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A000092:SSL routines::data length too long error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number error:0A000139:SSL routines::record layer failure)
SSL_do_handshake() failed (SSL: error:0A0001BB:SSL routines::bad record type error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:0A0001BB:SSL routines::bad record type error:0A000139:SSL routines::record layer failure)

The SSL_R_RECORD_LAYER_FAILURE ("record layer failure") error, however,
might be generated for a number of reasons, including memory allocation
failures, and therefore it is wrong to log all such errors at the "info"
level.  Instead, we now try to fallback to ERR_peek_error() in order to
decide which logging level should be used.

With these changes, no additional errors were observed with OpenSSL 3.0.20,
OpenSSL 3.2.6, OpenSSL 3.3.7, OpenSSL 3.4.5, OpenSSL 3.5.6, OpenSSL 3.6.2,
and OpenSSL 4.0.0.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -4048,6 +4048,22 @@ ngx_ssl_connection_error(ngx_connection_
 
         n = ERR_GET_REASON(ERR_peek_last_error());
 
+#ifdef SSL_R_RECORD_LAYER_FAILURE
+
+        if (n == SSL_R_RECORD_LAYER_FAILURE
+            && ERR_GET_LIB(ERR_peek_error()) == ERR_LIB_SSL)
+        {
+            /*
+             * OpenSSL 3.2.0+ returns SSL_R_RECORD_LAYER_FAILURE in the
+             * error queue after many different errors, including memory
+             * allocation failures, so fallback to ERR_peek_error()
+             */
+
+            n = ERR_GET_REASON(ERR_peek_error());
+        }
+
+#endif
+
             /* handshake failures */
         if (n == SSL_R_BAD_CHANGE_CIPHER_SPEC                        /*  103 */
 #ifdef SSL_R_NO_SUITABLE_KEY_SHARE
@@ -4101,6 +4117,9 @@ ngx_ssl_connection_error(ngx_connection_
 #ifdef SSL_R_NO_CIPHERS_PASSED
             || n == SSL_R_NO_CIPHERS_PASSED                          /*  182 */
 #endif
+#ifdef SSL_R_NOT_ON_RECORD_BOUNDARY
+            || n == SSL_R_NOT_ON_RECORD_BOUNDARY                     /*  182 */
+#endif
             || n == SSL_R_NO_CIPHERS_SPECIFIED                       /*  183 */
 #ifdef SSL_R_BAD_CIPHER
             || n == SSL_R_BAD_CIPHER                                 /*  186 */
@@ -4146,6 +4165,9 @@ ngx_ssl_connection_error(ngx_connection_
             || n == SSL_R_MISSING_KEY_SHARE                          /*  258 */
 #endif
             || n == SSL_R_UNSUPPORTED_PROTOCOL                       /*  258 */
+#ifdef SSL_R_INVALID_CCS_MESSAGE
+            || n == SSL_R_INVALID_CCS_MESSAGE                        /*  260 */
+#endif
 #ifdef SSL_R_NO_SHARED_GROUP
             || n == SSL_R_NO_SHARED_GROUP                            /*  266 */
 #endif
@@ -4184,6 +4206,9 @@ ngx_ssl_connection_error(ngx_connection_
 #ifdef SSL_R_UNSAFE_LEGACY_RENEGOTIATION_DISABLED
             || n == SSL_R_UNSAFE_LEGACY_RENEGOTIATION_DISABLED       /*  338 */
 #endif
+#ifdef SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING
+            || n == SSL_R_REQUIRED_COMPRESSION_ALGORITHM_MISSING     /*  342 */
+#endif
 #ifdef SSL_R_SCSV_RECEIVED_WHEN_RENEGOTIATING
             || n == SSL_R_SCSV_RECEIVED_WHEN_RENEGOTIATING           /*  345 */
 #endif