Vulnerability: Denial of service in Netty The SslHandler in Netty before 3.9.2 allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via a crafted SSLv2Hello message.
Vulnerability ID: GHSA-9959-6p3m-wxpc
CWE: CWE-119
Severity score: 0
Project: https://github.com/netty/netty
Published: 2020-06-30
File: src/main/java/org/jboss/netty/handler/ssl/SslHandler.java
Commit message: Fix a bug where SslHandler does not handle SSLv2Hello correctly

Motivation:

When a SSLv2Hello message is received, SSLEngine expects the application buffer size to be more than 30KB which is larger than what SslBufferPool can provide.  SSLEngine will always return with BUFFER_OVERFLOW status, blocking the SSL session from continuing the handshake.

Modifications:

When SSLEngine.getSession().getApplicationBufferSize() returns a value larger than what SslBufferPool provides, allocate a temporary heap buffer.

Result:

SSLv2Hello is handled correctly.
Commit author: Trustin Lee

Vulnerable version (before fix):
 private ChannelBuffer unwrap(                     // always contain at least one record in decode().  Therefore, if SSLEngine.unwrap() returns                     // BUFFER_OVERFLOW, it is always resolved by retrying after emptying the application buffer.                     for (;;) {                         try {                             result = engine.unwrap(nioInNetBuf, nioOutAppBuf);                             switch (result.getStatus()) {                                 case CLOSED:                                     // notify about the CLOSED state of the SSLEngine. See #137 private ChannelBuffer unwrap(                              break;                         } finally {                             nioOutAppBuf.flip();                              // Sync the offset of the inbound buffer.                             nettyInNetBuf.readerIndex(                                     nettyInNetBufStartOffset + nioInNetBuf.position() - nioInNetBufStartOffset);                              // Copy the unwrapped data into a smaller buffer.                             if (nioOutAppBuf.hasRemaining()) {                                 if (nettyOutAppBuf == null) {                                     ChannelBufferFactory factory = ctx.getChannel().getConfig().getBufferFactory();                                     nettyOutAppBuf = factory.getBuffer(initialNettyOutAppBufCapacity);                                 }                                 nettyOutAppBuf.writeBytes(nioOutAppBuf);                             }                             nioOutAppBuf.clear();                         }                     } 

Fixed version (after fix):
 private ChannelBuffer unwrap(                     // always contain at least one record in decode().  Therefore, if SSLEngine.unwrap() returns                     // BUFFER_OVERFLOW, it is always resolved by retrying after emptying the application buffer.                     for (;;) {                         final int outAppBufSize = engine.getSession().getApplicationBufferSize();                         final ByteBuffer outAppBuf;                         if (nioOutAppBuf.capacity() < outAppBufSize) {                             // SSLEngine wants a buffer larger than what the pool can provide.                             // Allocate a temporary heap buffer.                             outAppBuf = ByteBuffer.allocate(outAppBufSize);                         } else {                             outAppBuf = nioOutAppBuf;                         }                          try {                             result = engine.unwrap(nioInNetBuf, outAppBuf);                             switch (result.getStatus()) {                                 case CLOSED:                                     // notify about the CLOSED state of the SSLEngine. See #137 private ChannelBuffer unwrap(                              break;                         } finally {                             outAppBuf.flip();                              // Sync the offset of the inbound buffer.                             nettyInNetBuf.readerIndex(                                     nettyInNetBufStartOffset + nioInNetBuf.position() - nioInNetBufStartOffset);                              // Copy the unwrapped data into a smaller buffer.                             if (outAppBuf.hasRemaining()) {                                 if (nettyOutAppBuf == null) {                                     ChannelBufferFactory factory = ctx.getChannel().getConfig().getBufferFactory();                                     nettyOutAppBuf = factory.getBuffer(initialNettyOutAppBufCapacity);                                 }                                 nettyOutAppBuf.writeBytes(outAppBuf);                             }                             outAppBuf.clear();                         }                     } 

Patch:
patch: @@ -1268,8 +1268,18 @@ private ChannelBuffer unwrap(
                     // always contain at least one record in decode().  Therefore, if SSLEngine.unwrap() returns
                     // BUFFER_OVERFLOW, it is always resolved by retrying after emptying the application buffer.
                     for (;;) {
+                        final int outAppBufSize = engine.getSession().getApplicationBufferSize();
+                        final ByteBuffer outAppBuf;
+                        if (nioOutAppBuf.capacity() < outAppBufSize) {
+                            // SSLEngine wants a buffer larger than what the pool can provide.
+                            // Allocate a temporary heap buffer.
+                            outAppBuf = ByteBuffer.allocate(outAppBufSize);
+                        } else {
+                            outAppBuf = nioOutAppBuf;
+                        }
+
                         try {
-                            result = engine.unwrap(nioInNetBuf, nioOutAppBuf);
+                            result = engine.unwrap(nioInNetBuf, outAppBuf);
                             switch (result.getStatus()) {
                                 case CLOSED:
                                     // notify about the CLOSED state of the SSLEngine. See #137
@@ -1283,21 +1293,21 @@ private ChannelBuffer unwrap(
 
                             break;
                         } finally {
-                            nioOutAppBuf.flip();
+                            outAppBuf.flip();
 
                             // Sync the offset of the inbound buffer.
                             nettyInNetBuf.readerIndex(
                                     nettyInNetBufStartOffset + nioInNetBuf.position() - nioInNetBufStartOffset);
 
                             // Copy the unwrapped data into a smaller buffer.
-                            if (nioOutAppBuf.hasRemaining()) {
+                            if (outAppBuf.hasRemaining()) {
                                 if (nettyOutAppBuf == null) {
                                     ChannelBufferFactory factory = ctx.getChannel().getConfig().getBufferFactory();
                                     nettyOutAppBuf = factory.getBuffer(initialNettyOutAppBufCapacity);
                                 }
-                                nettyOutAppBuf.writeBytes(nioOutAppBuf);
+                                nettyOutAppBuf.writeBytes(outAppBuf);
                             }
-                            nioOutAppBuf.clear();
+                            outAppBuf.clear();
                         }
                     }
path: src/main/java/org/jboss/netty/handler/ssl/SslHandler.java