Prefer using Stream to byte[]
Tags: csharp, dotnet, performance
Published: 2021-10-17 14:55:47

---
title: Prefer using Stream to byte[]
published: true
description: A short example of how byte[] misuse may hurt memory consumption
tags: csharp, dotnet, performance
canonical_url: https://wkalmar.github.io/post/prefer-stream-to-byte-array/
//cover_image: https://direct_url_to_image.jpg
---
When working with files there are often both APIs operating `byte[]` and `Stream` so quite often people chose `byte[]` counterpart as it requires less ceremony or just intuitively more clear.

You may think of this conclusion as far-fetched but I've decided to write about it after reviewing and refactoring some real-world production code. So you may find this simple trick neglected in your codebase as some other simple things I've mentioned in my [previous articles](https://wkalmar.github.io/post/batch-processing-with-enumeratefiles/).

Let's look at the example as simple as calculating file hash. In spite of its simplicity [some people believe](https://stackoverflow.com/a/2978737/11306392) that the only way to do it is to read the entire file into memory.

Experienced readers may have already foreseen a problem with such an approach. Let's see do some benchmarking on 900MB file to see how the problem manifests and how we can circumvent it.

The baseline will be the naive solution of calculating hash from `byte[]` source

``` csharp
public static Guid ComputeHash(byte[] data)
{
    using HashAlgorithm algorithm = MD5.Create();
    byte[] bytes = algorithm.ComputeHash(data);
    return new Guid(bytes);
}
```

So following the advice from the title of the article we'll add another method that will accept `Stream` convert it to byte array and calculate hash
``` csharp
public async static Task<Guid> ComputeHash(Stream stream, CancellationToken token)
{
    var contents = await ConvertToBytes(stream, token);
    return ComputeHash(contents);
}

private static async Task<byte[]> ConvertToBytes(Stream stream, CancellationToken token)
{
    using var ms = new MemoryStream();
    await stream.CopyToAsync(ms, token);
    return ms.ToArray();
}
```

However, calculating hash from `byte[]` is not the only option. There's also an overload that accepts `Stream`. Let's use it.

``` csharp
public static Guid ComputeStream(Stream stream)
{
    using HashAlgorithm algorithm = MD5.Create();
    byte[] bytes = algorithm.ComputeHash(stream);
    stream.Seek(0, SeekOrigin.Begin);
    return new Guid(bytes);
}
```

Results are quite telling

<img src="https://wkalmar.github.io/9/1.PNG" width="600"/>

So what happened here?
While we tried to blindly follow the advice in the article it didn't help. The key takeaway from these figures is that using `Stream` allows us to process files in chunks instead of loading them into memory naively. While you may not notice this on small files but as soon as you have to deal with large files loading them into memory at once becomes quite costly.

Most of .NET methods that work with `byte[]` already exhibit `Stream` counterpart so it shouldn't be a problem to use it. When you provide your own API you should consider supplying a method that operates with `Stream` in a robust batch-by-batch fashion.

Let's use the following code checking two streams for equality as another example
``` csharp
private const int bufferSize = 2048;

public static bool IsEqual(this Stream stream, Stream otherStream)
{
    if (stream is null) return false;

    if (otherStream is null) return false;

    if (stream.Length != otherStream.Length) return false;

    if (stream == otherStream) return true;

    byte[] buffer = new byte[bufferSize];
    byte[] otherBuffer = new byte[bufferSize];

    while (stream.Read(buffer, 0, buffer.Length) > 0)
    {
        otherStream.Read(otherBuffer, 0, otherBuffer.Length);

        if (!otherBuffer.SequenceEqual(buffer))
        {
            stream.Seek(0, SeekOrigin.Begin);
            otherStream.Seek(0, SeekOrigin.Begin);
            return false;
        }
    }

    stream.Seek(0, SeekOrigin.Begin);
    otherStream.Seek(0, SeekOrigin.Begin);

    return true;
}
```

Here instead of loading two potentially big files into memory, we compare them using chunks of 2KB. Once chunks are different we exit.

## Conclusion
`Stream` APIs allow batch-by-batch processing which allows us to reduce memory consumption on big files. While on a first glance `Stream` API may seem as requiring more ceremony it's definitely a useful tool in one's toolbox.