CVE-2023-28638 (published 2023-03-27T21:15Z)

Snappier is a high performance C# implementation of the Snappy compression algorithm. This is a buffer overrun vulnerability that can affect any user of Snappier 1.1.0. In this release, much of the code was rewritten to use byte references rather than pointers to pinned buffers. This change generally improves performance and reduces workload on the garbage collector. However, when the garbage collector performs compaction and rearranges memory, it must update any byte references on the stack to refer to the updated location. The .NET garbage collector can only update these byte references if they still point within the buffer or to a point one byte past the end of the buffer. If they point outside this area, the buffer itself may be moved while the byte reference stays the same. There are several places in 1.1.0 where byte references very briefly point outside the valid areas of buffers. These are at locations in the code being used for buffer range checks. While the invalid references are never dereferenced directly, if a GC compaction were to occur during the brief window when they are on the stack then it could invalidate the buffer range check and allow other operations to overrun the buffer.  This should be very difficult for an attacker to trigger intentionally. It would require a repetitive bulk attack with the hope that a GC compaction would occur at precisely the right moment during one of the requests. However, one of the range checks with this problem is a check based on input data in the decompression buffer, meaning malformed input data could be used to increase the chance of success. Note that any resulting buffer overrun is likely to cause access to protected memory, which will then cause an exception and the process to be terminated. Therefore, the most likely result of an attack is a denial of service. This issue has been patched in release 1.1.1. Users are advised to upgrade. Users unable to upgrade may pin buffers to a fixed location before using them for compression or decompression to mitigate some, but not all, of these cases. At least one temporary decompression buffer is internal to the library and never pinned.\n

CVSS v3 base score: 5.9, MEDIUM (CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:H)

Weakness information:
cwe_id: CWE-119
cwe_name: Improper Restriction of Operations within the Bounds of a Memory Buffer
description: The product performs operations on a memory buffer, but it reads from or writes to a memory location outside the buffer's intended boundary. This may result in read or write operations on unexpected memory locations that could be linked to other variables, data structures, or internal program data.
extended_description: 
            Certain languages allow direct addressing of memory locations and do not automatically ensure that these locations are valid for the memory buffer that is being referenced. This can cause read or write operations to be performed on memory locations that may be associated with other variables, data structures, or internal program data.
         
         

Fix information:
repo_url: https://github.com/brantburnett/Snappier
commit_details: repo_url: https://github.com/brantburnett/Snappier
author: Brant Burnett
author_date: 2023-03-26 18:20:20-04:00
msg: Allow ref byte to point just past the end of spans (#73)

Motivation
----------
I was under the impression that `ref byte` should never point past the
end of a `Span<byte>` or `ReadOnlySpan<byte>` because then GC couldn't
recognize the pointer and adjust it during GC moves. However, this there
is a specific exception that allows `ref byte` to point precisely one
byte past the end and still be recognized to allow pointer arithmetic
scenarios like the ones in this algorithm.

Modifications
-------------
Where there is more complex logic (compared to the reference C++
implementation) to allow buffer end pointers to point to the last byte
in the buffer simplify this logic and point one byte past the end of the
buffer.

Also ensure that comparison operations that ensure a certain length
don't involve an intermediate pointer that moves off the beginning of
the buffer.

Results
-------
We're now back to closer to the reference C++ implementation but still
have memory safety.
repository: repo_url: https://github.com/brantburnett/Snappier
repo_name: brantburnett/Snappier
description: High performance C# implementation of the Snappy compression algorithm
repo_language: C#
owner: brantburnett
file_changes: filename: FindMatchLength.cs
change_type: ModificationType.MODIFY
diff: @@ -64,7 +64,7 @@ namespace Snappier.Benchmarks
 
             ref byte s1 = ref _array[0];
             ref byte s2 = ref Unsafe.Add(ref s1, 12);
-            ref byte s2Limit = ref Unsafe.Add(ref s1, _array.Length - 1);
+            ref byte s2Limit = ref Unsafe.Add(ref s1, _array.Length);
 
             return SnappyCompressor.FindMatchLength(ref s1, ref s2, ref s2Limit, ref data);
         }

programming_language: C#
filename: SnappyCompressorTests.cs
change_type: ModificationType.MODIFY
diff: @@ -92,7 +92,7 @@ namespace Snappier.Tests.Internal
             ref byte s2 = ref Unsafe.Add(ref s1, s1String.Length);
 
             var result =
-                SnappyCompressor.FindMatchLength(ref s1, ref s2, ref Unsafe.Add(ref s2, length - 1), ref data);
+                SnappyCompressor.FindMatchLength(ref s1, ref s2, ref Unsafe.Add(ref s2, length), ref data);
 
             Assert.Equal(result.matchLength < 8, result.matchLengthLessThan8);
             Assert.Equal(expectedResult, result.matchLength);

programming_language: Haskell
filename: SnappyCompressor.cs
change_type: ModificationType.MODIFY
diff: @@ -125,8 +125,7 @@ namespace Snappier.Internal
                 uint mask = (uint)(2 * (tableSpan.Length - 1));
 
                 ref byte inputStart = ref Unsafe.AsRef(in input[0]);
-                // Last byte of the input, not one byte past the end, to avoid issues on GC moves
-                ref byte inputEnd = ref Unsafe.Add(ref inputStart, input.Length - 1);
+                ref byte inputEnd = ref Unsafe.Add(ref inputStart, input.Length);
                 ref byte ip = ref inputStart;
 
                 ref byte op = ref output[0];
@@ -134,7 +133,7 @@ namespace Snappier.Internal
 
                 if (input.Length >= Constants.InputMarginBytes)
                 {
-                    ref byte ipLimit = ref Unsafe.Subtract(ref inputEnd, Constants.InputMarginBytes - 1);
+                    ref byte ipLimit = ref Unsafe.Subtract(ref inputEnd, Constants.InputMarginBytes);
 
                     for (uint preload = Helpers.UnsafeReadUInt32(ref Unsafe.Add(ref ip, 1));;)
                     {
@@ -288,7 +287,7 @@ namespace Snappier.Internal
                         // Step 2: A 4-byte match has been found.  We'll later see if more
                         // than 4 bytes match.  But, prior to the match, input
                         // bytes [next_emit, ip) are unmatched.  Emit them as "literal bytes."
-                        Debug.Assert(!Unsafe.IsAddressGreaterThan(ref Unsafe.Add(ref nextEmit, 16), ref Unsafe.Add(ref inputEnd, 1)));
+                        Debug.Assert(!Unsafe.IsAddressGreaterThan(ref Unsafe.Add(ref nextEmit, 16), ref inputEnd));
                         op = ref EmitLiteralFast(ref op, ref nextEmit, (uint) Unsafe.ByteOffset(ref nextEmit, ref ip));
 
                         // Step 3: Call EmitCopy, and then see if another EmitCopy could
@@ -350,9 +349,9 @@ namespace Snappier.Internal
 
                 emit_remainder:
                 // Emit the remaining bytes as a literal
-                if (!Unsafe.IsAddressGreaterThan(ref ip, ref inputEnd))
+                if (Unsafe.IsAddressLessThan(ref ip, ref inputEnd))
                 {
-                    op = ref EmitLiteralSlow(ref op, ref ip, (uint) Unsafe.ByteOffset(ref ip, ref inputEnd) + 1);
+                    op = ref EmitLiteralSlow(ref op, ref ip, (uint) Unsafe.ByteOffset(ref ip, ref inputEnd));
                 }
 
                 return (int) Unsafe.ByteOffset(ref output[0], ref op);
@@ -490,28 +489,23 @@ namespace Snappier.Internal
         /// Find the largest n such that
         ///
         ///   s1[0,n-1] == s2[0,n-1]
-        ///   and n &lt;= (s2_limit - s2 + 1).
+        ///   and n &lt;= (s2_limit - s2).
         ///
         /// Return (n, n &lt; 8).
         /// Reads up to and including *s2_limit but not beyond.
-        /// Does not read *(s1 + (s2_limit - s2 + 1)) or beyond.
-        /// Requires that s2_limit+1 &gt;= s2.
+        /// Does not read *(s1 + (s2_limit - s2)) or beyond.
+        /// Requires that s2_limit &gt;= s2.
         ///
         /// In addition populate *data with the next 5 bytes from the end of the match.
         /// This is only done if 8 bytes are available (s2_limit - s2 &gt;= 8). The point is
         /// that on some arch's this can be done faster in this routine than subsequent
         /// loading from s2 + n.
         /// </summary>
-        /// <remarks>
-        /// The reference implementation has s2Limit as one byte past the end of the input,
-        /// but this implementation has it at the end of the input. This ensures that it always
-        /// points within the array in case GC moves the array.
-        /// </remarks>
         [MethodImpl(MethodImplOptions.AggressiveInlining)]
         internal static (int matchLength, bool matchLengthLessThan8) FindMatchLength(
             ref byte s1, ref byte s2, ref byte s2Limit, ref ulong data)
         {
-            Debug.Assert(!Unsafe.IsAddressLessThan(ref Unsafe.Add(ref s2Limit, 1), ref s2));
+            Debug.Assert(!Unsafe.IsAddressLessThan(ref s2Limit, ref s2));
 
             if (BitConverter.IsLittleEndian && IntPtr.Size == 8)
             {
@@ -521,14 +515,14 @@ namespace Snappier.Internal
 
             int matched = 0;
 
-            while (!Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 3))
+            while (Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)4
                    && Helpers.UnsafeReadUInt32(ref s2) == Helpers.UnsafeReadUInt32(ref Unsafe.Add(ref s1, matched)))
             {
                 s2 = ref Unsafe.Add(ref s2, 4);
                 matched += 4;
             }
 
-            if (BitConverter.IsLittleEndian && !Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 3)))
+            if (BitConverter.IsLittleEndian && Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)4)
             {
                 uint x = Helpers.UnsafeReadUInt32(ref s2) ^ Helpers.UnsafeReadUInt32(ref Unsafe.Add(ref s1, matched));
                 int matchingBits = Helpers.FindLsbSetNonZero(x);
@@ -537,14 +531,14 @@ namespace Snappier.Internal
             }
             else
             {
-                while (!Unsafe.IsAddressGreaterThan(ref s2, ref s2Limit) && Unsafe.Add(ref s1, matched) == s2)
+                while (Unsafe.IsAddressLessThan(ref s2, ref s2Limit) && Unsafe.Add(ref s1, matched) == s2)
                 {
                     s2 = ref Unsafe.Add(ref s2, 1);
                     ++matched;
                 }
             }
 
-            if (!Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 7)))
+            if (Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)8)
             {
                 data = Helpers.UnsafeReadUInt64(ref s2);
             }
@@ -562,7 +556,7 @@ namespace Snappier.Internal
             // immediately.  As an optimization though, it is useful.  It creates some not
             // uncommon code paths that determine, without extra effort, whether the match
             // length is less than 8.
-            if (!Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 15)))
+            if (Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)16)
             {
                 ulong a1 = Helpers.UnsafeReadUInt64(ref s1);
                 ulong a2 = Helpers.UnsafeReadUInt64(ref s2);
@@ -590,7 +584,7 @@ namespace Snappier.Internal
             // time until we find a 64-bit block that doesn't match; then we find
             // the first non-matching bit and use that to calculate the total
             // length of the match.
-            while (!Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 15)))
+            while (Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)16)
             {
                 ulong a1 = Helpers.UnsafeReadUInt64(ref Unsafe.Add(ref s1, matched));
                 ulong a2 = Helpers.UnsafeReadUInt64(ref s2);
@@ -615,7 +609,7 @@ namespace Snappier.Internal
                 }
             }
 
-            while (!Unsafe.IsAddressGreaterThan(ref s2, ref s2Limit))
+            while (Unsafe.IsAddressLessThan(ref s2, ref s2Limit))
             {
                 if (Unsafe.Add(ref s1, matched) == s2)
                 {
@@ -624,7 +618,7 @@ namespace Snappier.Internal
                 }
                 else
                 {
-                    if (!Unsafe.IsAddressGreaterThan(ref s2, ref Unsafe.Subtract(ref s2Limit, 7)))
+                    if (Unsafe.ByteOffset(ref s2, ref s2Limit) >= (nint)8)
                     {
                         data = Helpers.UnsafeReadUInt64(ref s2);
                     }

programming_language: C#
filename: SnappyDecompressor.cs
change_type: ModificationType.MODIFY
diff: @@ -186,17 +186,11 @@ namespace Snappier.Internal
             unchecked
             {
                 ref byte input = ref Unsafe.AsRef(in inputSpan[0]);
-
-                // The reference Snappy implementation uses inputEnd as a pointer one byte past the end of the buffer.
-                // However, this is not safe when using ref locals. The ref must point to somewhere within the array
-                // so that GC can adjust the ref if the memory is moved.
-                ref byte inputEnd = ref Unsafe.Add(ref input, inputSpan.Length - 1);
+                ref byte inputEnd = ref Unsafe.Add(ref input, inputSpan.Length);
 
                 // Track the point in the input before which input is guaranteed to have at least Constants.MaxTagLength bytes left
-                ref byte inputLimitMinMaxTagLength = ref Unsafe.Subtract(ref inputEnd, Math.Min(inputSpan.Length, Constants.MaximumTagLength - 1) - 1);
+                ref byte inputLimitMinMaxTagLength = ref Unsafe.Subtract(ref inputEnd, Math.Min(inputSpan.Length, Constants.MaximumTagLength - 1));
 
-                // We always allocate buffer with at least one extra byte on the end, so bufferEnd doesn't have the same
-                // restrictions as inputEnd.
                 ref byte buffer = ref _lookbackBuffer.Span[0];
                 ref byte bufferEnd = ref Unsafe.Add(ref buffer, _lookbackBuffer.Length);
                 ref byte op = ref Unsafe.Add(ref buffer, _lookbackPosition);
@@ -239,9 +233,9 @@ namespace Snappier.Internal
                     {
                         // Data has been moved to the scratch buffer
                         input = ref scratch;
-                        inputEnd = ref Unsafe.Add(ref input, newScratchLength - 1);
+                        inputEnd = ref Unsafe.Add(ref input, newScratchLength);
                         inputLimitMinMaxTagLength = ref Unsafe.Subtract(ref inputEnd,
-                            Math.Min(newScratchLength, Constants.MaximumTagLength - 1) - 1);
+                            Math.Min(newScratchLength, Constants.MaximumTagLength - 1));
                     }
                 }
 
@@ -256,7 +250,7 @@ namespace Snappier.Internal
                     {
                         nint literalLength = unchecked((c >> 2) + 1);
 
-                        if (TryFastAppend(ref op, ref bufferEnd, in input, Unsafe.ByteOffset(ref input, ref inputEnd) + 1, literalLength))
+                        if (TryFastAppend(ref op, ref bufferEnd, in input, Unsafe.ByteOffset(ref input, ref inputEnd), literalLength))
                         {
                             Debug.Assert(literalLength < 61);
                             op = ref Unsafe.Add(ref op, literalLength);
@@ -280,7 +274,7 @@ namespace Snappier.Internal
                             input = ref Unsafe.Add(ref input, literalLengthLength);
                         }
 
-                        nint inputRemaining = Unsafe.ByteOffset(ref input, ref inputEnd) + 1;
+                        nint inputRemaining = Unsafe.ByteOffset(ref input, ref inputEnd);
                         if (inputRemaining < literalLength)
                         {
                             Append(ref op, ref bufferEnd, in input, inputRemaining);
@@ -306,9 +300,9 @@ namespace Snappier.Internal
                                 {
                                     // Data has been moved to the scratch buffer
                                     input = ref scratch;
-                                    inputEnd = ref Unsafe.Add(ref input, newScratchLength - 1);
+                                    inputEnd = ref Unsafe.Add(ref input, newScratchLength);
                                     inputLimitMinMaxTagLength = ref Unsafe.Subtract(ref inputEnd,
-                                        Math.Min(newScratchLength, Constants.MaximumTagLength - 1) - 1);
+                                        Math.Min(newScratchLength, Constants.MaximumTagLength - 1));
 
                                 }
                             }
@@ -367,9 +361,9 @@ namespace Snappier.Internal
                             {
                                 // Data has been moved to the scratch buffer
                                 input = ref scratch;
-                                inputEnd = ref Unsafe.Add(ref input, newScratchLength - 1);
+                                inputEnd = ref Unsafe.Add(ref input, newScratchLength);
                                 inputLimitMinMaxTagLength = ref Unsafe.Subtract(ref inputEnd,
-                                    Math.Min(newScratchLength, Constants.MaximumTagLength - 1) - 1);
+                                    Math.Min(newScratchLength, Constants.MaximumTagLength - 1));
                             }
                         }
 
@@ -415,7 +409,7 @@ namespace Snappier.Internal
                         (int) literalLengthLength) + 1;
                 }
 
-                nint inputRemaining = Unsafe.ByteOffset(ref input, ref inputEnd) + 1;
+                nint inputRemaining = Unsafe.ByteOffset(ref input, ref inputEnd);
                 if (inputRemaining < literalLength)
                 {
                     Append(ref op, ref bufferEnd, in input, inputRemaining);
@@ -468,7 +462,7 @@ namespace Snappier.Internal
         {
             Debug.Assert(_scratchLength > 0);
 
-            if (Unsafe.IsAddressGreaterThan(ref input, ref inputEnd))
+            if (!Unsafe.IsAddressLessThan(ref input, ref inputEnd))
             {
                 return 0;
             }
@@ -477,7 +471,7 @@ namespace Snappier.Internal
             uint entry = Constants.CharTable[scratch];
             uint needed = (entry >> 11) + 1; // +1 byte for 'c'
 
-            uint toCopy = Math.Min((uint)Unsafe.ByteOffset(ref input, ref inputEnd) + 1, needed - _scratchLength);
+            uint toCopy = Math.Min((uint)Unsafe.ByteOffset(ref input, ref inputEnd), needed - _scratchLength);
             Unsafe.CopyBlockUnaligned(ref Unsafe.Add(ref scratch, _scratchLength), ref input, toCopy);
 
             _scratchLength += toCopy;
@@ -502,7 +496,7 @@ namespace Snappier.Internal
         // always have some extra bytes on the end so we don't risk buffer overruns.
         private uint RefillTag(ref byte input, ref byte inputEnd, ref byte scratch)
         {
-            if (Unsafe.IsAddressGreaterThan(ref input, ref inputEnd))
+            if (!Unsafe.IsAddressLessThan(ref input, ref inputEnd))
             {
                 return uint.MaxValue;
             }
@@ -511,7 +505,7 @@ namespace Snappier.Internal
             uint entry = Constants.CharTable[input];
             uint needed = (entry >> 11) + 1; // +1 byte for 'c'
 
-            uint inputLength = (uint)Unsafe.ByteOffset(ref input, ref inputEnd) + 1;
+            uint inputLength = (uint)Unsafe.ByteOffset(ref input, ref inputEnd);
             if (inputLength < needed)
             {
                 // Data is insufficient, copy to scratch
@@ -555,11 +549,8 @@ namespace Snappier.Internal
                         ArrayPool<byte>.Shared.Return(_lookbackBufferArray);
                     }
 
-                    // Always pad the lookback buffer with an extra byte that we don't use. This allows a "ref byte" reference past
-                    // the end of the perceived buffer that still points within the array. This is a requirement so that GC can recognize
-                    // the "ref byte" points within the array and adjust it if the array is moved.
-                    _lookbackBufferArray = ArrayPool<byte>.Shared.Rent(value.GetValueOrDefault() + 1);
-                    _lookbackBuffer = _lookbackBufferArray.AsMemory(0, _lookbackBufferArray.Length - 1);
+                    _lookbackBufferArray = ArrayPool<byte>.Shared.Rent(value.GetValueOrDefault());
+                    _lookbackBuffer = _lookbackBufferArray.AsMemory(0, _lookbackBufferArray.Length);
                 }
             }
         }
@@ -595,7 +586,7 @@ namespace Snappier.Internal
         }
 
         [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        private void Append(ref byte op, ref byte bufferEnd, in byte input, nint length)
+        private static void Append(ref byte op, ref byte bufferEnd, in byte input, nint length)
         {
             if (length > Unsafe.ByteOffset(ref op, ref bufferEnd))
             {
@@ -606,7 +597,7 @@ namespace Snappier.Internal
         }
 
         [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        private bool TryFastAppend(ref byte op, ref byte bufferEnd, in byte input, nint available, nint length)
+        private static bool TryFastAppend(ref byte op, ref byte bufferEnd, in byte input, nint available, nint length)
         {
             if (length <= 16 && available >= 16 + Constants.MaximumTagLength &&
                 Unsafe.ByteOffset(ref op, ref bufferEnd) >= (nint) 16)
@@ -619,10 +610,13 @@ namespace Snappier.Internal
         }
 
         [MethodImpl(MethodImplOptions.AggressiveInlining)]
-        private void AppendFromSelf(ref byte op, ref byte buffer, ref byte bufferEnd, uint copyOffset, nint length)
+        private static void AppendFromSelf(ref byte op, ref byte buffer, ref byte bufferEnd, uint copyOffset, nint length)
         {
-            ref byte source = ref Unsafe.Subtract(ref op, copyOffset);
-            if (!Unsafe.IsAddressLessThan(ref source, ref op) || Unsafe.IsAddressLessThan(ref source, ref buffer))
+            // ToInt64() ensures that this logic works correctly on x86 (with a slight perf hit on x86, though). This is because
+            // nint is only 32-bit on x86, so casting uint copyOffset to an nint for the comparison can result in a negative number with some
+            // forms of illegal data. This would then bypass the exception and cause unsafe memory access. Performing the comparison
+            // as a long ensures we have enough bits to not lose data. On 64-bit platforms this is effectively a no-op.
+            if (copyOffset == 0 || Unsafe.ByteOffset(ref buffer, ref op).ToInt64() < copyOffset)
             {
                 ThrowHelper.ThrowInvalidDataException("Invalid copy offset");
             }
@@ -632,6 +626,7 @@ namespace Snappier.Internal
                 ThrowHelper.ThrowInvalidDataException("Data too long");
             }
 
+            ref byte source = ref Unsafe.Subtract(ref op, copyOffset);
             CopyHelpers.IncrementalCopy(ref source, ref op,
                 ref Unsafe.Add(ref op, length), ref bufferEnd);
         }

programming_language: C#