Optimising Image Processing
Tags: showdev, csharp, optimization, programming
Published: 2018-07-11 11:32:08

---
title: Optimising Image Processing
published: true
description: Reducing memory consumption and time taken by 80%
tags: #showdev, #csharp, #optimization, #programming
canonical_url: https://indy.codes/optimising-image-processing
cover_image: https://thepracticaldev.s3.amazonaws.com/i/7ulo5bhw2oltjs2aorio.jpg
---

Reducing memory consumption and time taken by 80%

## Contents
1. [Investigating old code](#1)
2. [Streaming is winning](#2)
3. [Pool your streams](#3)
4. [Goodbye System.Drawing, you will not be missed](#4)
5. [In the interest of transparency](#5)
6. [TLDR - tell me what to do](#6)


<h2 id="1">Investigating old code</h2>

In my [previous article][previous-article], we covered how we found and solved a fixed cost of using `PutObject` on the AWS S3 .NET client; it was also mentioned that we transform the image before it is uploaded to AWS S3 - this is a process that is executed at least 400,000 times a day. With that in mind I decided it was worth exploring the entire code path. As always you can find all the code talked about [here][article-repo], and we will be using [this image][image-link] in our testing. Let us take a look at the existing implementation. Before we can transform an image we need to get a hold of it:-

```csharp
private static byte[] GetImageFromUrl(string url)
{
    byte[] data;
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Timeout = 10000;
    request.ReadWriteTimeout = 10000;

    using (var response = request.GetResponse())
    using (var responseStream = response.GetResponseStream())
    using (var memoryStream = new MemoryStream())
    {
        int count;
        do
        {
            var buf = new byte[1024];
            count = responseStream.Read(buf, 0, 1024);
            memoryStream.Write(buf, 0, count);
        } while (responseStream.CanRead && count > 0);

        data = memoryStream.ToArray();
    }

    return data;
}
```

Do you see that `.ToArray()`? It is *evil*. I do not have the statistics to hand but I know for a fact I have seen images larger than four megabytes come through this code path. That does not bode well - a quick peak at [the implementation][to-array] shows us why:-

```csharp
public virtual byte[] ToArray()
{
    BCLDebug.Perf(_exposable, "MemoryStream::GetBuffer will let you avoid a copy.");
    byte[] copy = new byte[_length - _origin];
    Buffer.InternalBlockCopy(_buffer, _origin, copy, 0, _length - _origin);
    return copy;
}
```

The LOH threshold is `85,000` bytes this would mean any images larger than the threshold would go straight onto the LOH - this is a problem, I covered why it is a problem in my [previous article][explain-loh]. At this point curiosity got the better of me, I really wanted to know the size of images coming into this service.

The service processes hundreds of thousands images per day, all I need is a quick sample and idea of the size and shape of incoming data. We have a job that runs at 14:00 every day to pull the latest images from one of our customers.

Hitting their service manually returns `14,630` images to be processed. Spinning up a quick console app to perform a `HEAD` request and get the `Content-Length` header:-

```
Total Images: 14,630
Images Above LOH Threshold: 14,276
Average Image Size: 253,818 bytes
Largest Image Size: 693,842 bytes
Smallest Image Size: 10,370 bytes
Standard Deviation of Sizes: 101,184
```

Yikes, that is a *huge* standard deviation and 97.58% of the images are above the LOH threshold (`85,000` bytes). Now that we know the spread of image sizes, we can resume looking at the rest of the current implementation:-

```csharp
public void Transform(string url)
{
    var bytes = GetImageFromUrl(url);

    if (CanCreateImageFrom(bytes))
    {
        using (var stream = new MemoryStream(bytes))
        using (var originalImage = Image.FromStream(stream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);

            // upload scaledImage to AWS S3 in production, in the test harness write to disk
            using (var fileStream = File.Create(@"..\..\v1.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}

private static bool CanCreateImageFrom(byte[] bytes)
{
    try
    {
        using (var stream = new MemoryStream(bytes))
        {
            Image.FromStream(stream);
        }
    }
    catch (ArgumentException)
    {
        return false;
    }

    return true;
}
```

I have suspicions about the two `MemoryStream`'s knocking about but dotTrace can confirm or deny my suspicions. Running `V1` with the sample image one hundred times:-

|                       |V1 
|-                      |-
|Took (ms)              |10,297
|Allocated (kb)         |851,894
|Peak Working Set (kb)  |96,276
|Gen 0 collections      |184
|Gen 1 collections      |101
|Gen 2 collections      |101
|                       |-
|dotTrace Total RAM (MB)|901
|dotTrace SOH (MB)      |410
|dotTrace LOH (MB)      |491

Using dotTrace we can see where the biggest costs are:-

![image-01]

Now that we have established a baseline - lets get cracking!

<h2 id="2">Streaming is winning</h2>

If we inline the `GetImageFromUrl` and remove the `CanCreateImageFrom` - which is not needed because we check the `Content-Type` earlier in the code path, we can directly operate on the incoming stream.

```csharp
public class ImageTransformerV2 : IImageTransformer
{
    public void Transform(string url)
    {
        var request = WebRequest.CreateHttp(url);
        request.Timeout = 10000;
        request.ReadWriteTimeout = 10000;

        using (var response = request.GetResponse())
        using (var responseStream = response.GetResponseStream())
        using (var originalImage = Image.FromStream(responseStream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);

            // upload scaledImage to AWS S3 in production, in the test harness write to disk
            using (var fileStream = File.Create(@"..\..\v2.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}
```

Stats for `V2`:-

|                       |V1     |V2     |% 
|-                      |-      |-      |-
|Took (ms)              |10,297 |7,844  |-23.82%
|Allocated (kb)         |851,894|527,246|-38.10%
|Peak Working Set (kb)  |96,276 |69,436 |-27.87%
|Gen 0 collections      |184    |100    |-45.65%
|Gen 1 collections      |101    |100    |-00.99%
|Gen 2 collections      |101    |100    |-00.99%
|                       |-      |       |
|dotTrace Total RAM (MB)|901    |550    |-38.95%
|dotTrace SOH (MB)      |410    |162    |-60.48%
|dotTrace LOH (MB)      |491    |388    |-20.97%

Awesome, a few minor tweaks and all the metrics have dropped across the board.

<h2 id="3">Pool your streams</h2>

Using dotTrace again, we can see the next biggest cost:-

![image-02]

Something is happening inside of the constructor of `GPStream` that is costing us dearly. Luckily dotTrace can show us the decompiled source which saves us a trip to Reference Sources:-

```csharp
internal GPStream(Stream stream)
{
    if (!stream.CanSeek)
    {
        byte[] buffer = new byte[256];
        int offset = 0;
        int num;

        do
        {
            if (buffer.Length < offset + 256)
            {
                byte[] numArray = new byte[buffer.Length * 2];
                Array.Copy((Array) buffer, (Array) numArray, buffer.Length);
                buffer = numArray;
            }
            num = stream.Read(buffer, offset, 256);
            offset += num;
        } while (num != 0);

        this.dataStream = (Stream) new MemoryStream(buffer);
    }
    else
        this.dataStream = stream;
}
```

You know when you ask yourself a question out loud and you can answer it straight away?


> "What?! I can't seek on the incoming HTTP stream?"

No, you can't. Makes total sense when you think about it and because the incoming `Stream` is not seekable `GPStream` has to take a copy of it.

Okay, first thing - can we move the cost from *framework* code to *our* code? It is not pretty but something like this works:-

```csharp
public void Transform(string url)
{
    var request = WebRequest.CreateHttp(url);
    request.Timeout = 10000;
    request.ReadWriteTimeout = 10000;

    var memoryStream = new MemoryStream();

    using (var response = request.GetResponse())
    using (var responseStream = response.GetResponseStream())
    {
        responseStream.CopyTo(memoryStream);
    }

    using (var originalImage = Image.FromStream(memoryStream))
    using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
    using (var graphics = Graphics.FromImage(scaledImage))
    {
        ImageHelper.TransformImage(graphics, scaledImage, originalImage);

        // upload scaledImage to AWS S3 in production, in the test harness write to disk

        using (var fileStream = File.Create(@"..\..\v2.jpg"))
        {
            scaledImage.Save(fileStream, ImageFormat.Jpeg);
        }
    }
}
```

dotTrace shows us that we have indeed moved the cost from framework code into our code:-

![image-04]

Now we *could* use our old friend `System.Buffers` and just rent a `byte[]` then pass that to the constructor of the `MemoryStream`. That would work, except two things. Firstly, `Content-Length` is [not guaranteed][content-length] to be set. Secondly, a few weeks earlier I was poking around the [.NET driver for Cassandra][cassandra] when I came across `Microsoft.IO.RecyclableMemoryStream` and it felt like it was *exactly* what I needed here. If you want to learn more about `RecyclableMemoryStream` Ben Watson has a [great post][rms] on what it is and its various use cases. Jumping straight into `V3`:-

```csharp
public class ImageTransformerV3 : IImageTransformer
{
    private readonly RecyclableMemoryStreamManager _streamManager;

    public ImageTransformerV3()
    {
        _streamManager = new RecyclableMemoryStreamManager();
    }

    public void Transform(string url)
    {
        var request = WebRequest.CreateHttp(url);
        request.Timeout = 10000;
        request.ReadWriteTimeout = 10000;
        request.AllowReadStreamBuffering = false;
        request.AllowWriteStreamBuffering = false;

        MemoryStream borrowedStream;

        using (var response = request.GetResponse())
        {
            if (response.ContentLength == -1) // Means that content length is NOT sent back by the third party server
            {
                borrowedStream = _streamManager.GetStream(url); // then we let the stream manager had this 
            }
            else
            {
                borrowedStream = _streamManager.GetStream(url, (int)response.ContentLength); // otherwise we borrow a stream with the exact size
            }

            int bufferSize;

            if (response.ContentLength == -1 || response.ContentLength > 81920)
            {
                bufferSize = 81920;
            }
            else
            {
                bufferSize = (int) response.ContentLength;
            }

            // close the http response stream asap, we only need the contents, we don't need to keep it open
            using (var responseStream = response.GetResponseStream())
            {
                responseStream.CopyTo(borrowedStream, bufferSize);
            }
        }

        using (borrowedStream)
        using (var originalImage = Image.FromStream(borrowedStream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);
            // upload scaledImage to AWS S3 in production, in the test harness write to disk

            using (var fileStream = File.Create(@"..\..\v3.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}
```

One of my favourite things about `RecyclableMemoryStream` is that it is a drop-in replacement for `MemoryStream`. The `ArrayPool` from `System.Buffers` requires you to `Rent` and then `Return`. Whereas `RecyclableMemoryStream` handles everything for you as it implements `IDisposable`. Anyway, enough of my admiration for `RecyclableMemoryStream`; stats for `V3`:-

|                       |V1     |V2     |V3     |% (V3 vs. V1)
|-                      |-      |-      |-      |-
|Took (ms)              |10,297 |7,844  |7,688  |-25.33%
|Allocated (kb)         |851,894|527,246|125,739|-85.24%
|Peak Working Set (kb)  |96,276 |69,436 |71,140 |-26.10%
|Gen 0 collections      |184    |100    |29     |-84.23%
|Gen 1 collections      |101    |100    |2      |-98.01%
|Gen 2 collections      |101    |100    |1      |-99.00%
|-                                    
|dotTrace Total RAM (MB)|901    |550    |152    |-83.12%
|dotTrace SOH (MB)      |410    |162    |150    |-63.41%
|dotTrace LOH (MB)      |491    |388    |1.6    |-99.67%

That is an *incredible* improvement from `V1`. This is what `V3` looks like in dotTrace:-

![image-05]

<h2 id="4">Goodbye System.Drawing, you will not be missed</h2>

My knowledge on `System.Drawing` is pretty sketchy but an afternoon reading about it leads me to the conclusion that if you can avoid using `System.Drawing` then you are better off. Whilst searching for an alternative to `System.Drawing`, I came across [this article][image-stream-overload] written by Omar Shahine. 
Huh, I never considered the overloads. Takes two seconds to try this so we might as well; v3 with `useEmbeddedColorManagement` disabled and `validateImageData` turned off these are the stats:-

|                       |V1     |V2     |V3     |% (V3 vs. V1)
|-                      |-      |-      |-      |-
|Took (ms)              |10,297 |7,844  |7,563  |-26.55%
|Allocated (kb)         |851,894|527,246|128,166|-84.95%
|Peak Working Set (kb)  |96,276 |69,436 |53,688 |-44.23%
|Gen 0 collections      |184    |100    |30     |-83.69%
|Gen 1 collections      |101    |100    |2      |-98.01%
|Gen 2 collections      |101    |100    |1      |-99.00%
|-                                    
|dotTrace Total RAM (MB)|901    |550    |154    |-82.90%
|dotTrace SOH (MB)      |410    |162    |153    |-62.68%
|dotTrace LOH (MB)      |491    |388    |1.6    |-99.67%

A minor increase in some areas but a noticeable drop in `Peak Working Set` - great! A slightly more [recent article][system-drawing-alts] by Bertrand Le Roy goes through the various alternatives to `System.Drawing`. Thankfully, there is a nice little chart at the bottom that shows performance in the context of time taken.

According to that article [PhotoSauce.MagicScaler][photo-sauce] is pretty magic - word of warning this library is Windows only. I wonder how magic it really is? Spinning `V4` up:-

```csharp
public class ImageTransformerV4 : IImageTransformer
{
    public void Transform(string url)
    {
        // truncated for brevity 

        MagicImageProcessor.EnableSimd = false;
        MagicImageProcessor.EnablePlanarPipeline = true;

        using (borrowedStream)
        {
            // upload scaledImage to AWS S3 in production, in the test harness write to disk

            using (var fileStream = File.Create(@"..\..\v4.jpg"))
            {
                MagicImageProcessor.ProcessImage(borrowedStream, fileStream, new ProcessImageSettings()
                {
                    Width = 320,
                    Height = 240,
                    ResizeMode = CropScaleMode.Max,
                    SaveFormat = FileFormat.Jpeg,
                    JpegQuality = 70,
                    HybridMode = HybridScaleMode.Turbo
                });
            }
        }
    }
}
```

And the stats:-

|                       |V1     |V2     |V3     |V4     |% (V4 vs. V1)
|-                      |-      |-      |-      |-      |-
|Took (ms)              |10,297 |7,844  |7,563  |1,672  |-83.76%
|Allocated (kb)         |851,894|527,246|128,166|135,876|-84.05%
|Peak Working Set (kb)  |96,276 |69,436 |53,688 |35,596 |-63.02%
|Gen 0 collections      |184    |100    |30     |32     |-82.60%
|Gen 1 collections      |101    |100    |2      |2      |-98.01%
|Gen 2 collections      |101    |100    |1      |1      |-99.00%
|-                                      
|dotTrace Total RAM (MB)|901    |550    |154    |165    |-81.68%
|dotTrace SOH (MB)      |410    |162    |153    |162    |-60.48%
|dotTrace LOH (MB)      |491    |388    |1.6    |1.6    |-99.67%

An insane drop in time taken and a healthy drop in `Peak Working Set`! 

That *is* magic.

<h2 id="5">In the interest of transparency</h2>

Pun not intended - just before this article was about to be published I saw this interesting tweet:-

{% twitter 1012632551486205952 %}

Naturally I was interested, spinning up `V5` gave me similar results to `V4` except a noticeable increase in time taken, peak working set, and LOH allocations. After a conversation on their [Gitter][gitter-im] I learnt that they are:-

1. Still working on [improving their JPEG decoder][imagesharp-jpeg-ticket] - this is relevant because we process other image formats too
2. Aware that their [resizing algorithm][imagesharp-resize-ticket] is not as optimal as it could be - this is relevant because the main part of the transformation is resizing
3. Happy to be featured in this article (which says a lot)

<h2 id="6">TLDR - tell me what to do</h2>

If your image transformation process is mostly resizing and you are hosted on Windows then look into using `PhotoSauce.MagicScaler` with `Microsoft.IO.RecyclableMemoryStream` you will see a sixty to ninety nine percent reduction in various performance related metrics!

Find me [Twitter][me-twitter], [LinkedIn][me-linkedin], or [GitHub][me-github].


[me-twitter]: https://twitter.com/indy_singh_uk
[me-github]: https://github.com/indy-singh
[me-linkedin]: https://uk.linkedin.com/in/indy-singh-uk
[imagesharp-jpeg-ticket]: https://github.com/SixLabors/ImageSharp/issues/601
[imagesharp-resize-ticket]: https://github.com/SixLabors/ImageSharp/issues/642
[photo-sauce]: https://www.nuget.org/packages/PhotoSauce.MagicScaler
[system-drawing-alts]: https://blogs.msdn.microsoft.com/dotnet/2017/01/19/net-core-image-processing/
[image-stream-overload]: https://blogs.msdn.microsoft.com/omars/2004/03/29/system-drawing-image-performance/
[previous-article]: https://dev.to/indy_singh_uk/its-not-your-code-vol-i-17n0
[explain-loh]: https://dev.to/indy_singh_uk/its-not-your-code-vol-i-17n0#2
[to-array]: https://referencesource.microsoft.com/#mscorlib/system/io/memorystream.cs,550
[article-repo]: https://github.com/indy-singh/OptimiseImageProcessing
[image-link]: https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_1MB.jpg
[image-01]: https://thepracticaldev.s3.amazonaws.com/i/7ulo5bhw2oltjs2aorio.jpg
[image-02]: https://thepracticaldev.s3.amazonaws.com/i/bzqur2yor8fs9ms0yo5q.jpg
[image-04]: https://thepracticaldev.s3.amazonaws.com/i/iwf9vije0zwdhota4rds.jpg
[image-05]: https://thepracticaldev.s3.amazonaws.com/i/w6nyssdrikkdfw61ot8b.jpg
[rms]: http://www.philosophicalgeek.com/2015/02/06/announcing-microsoft-io-recycablememorystream/
[cassandra]: https://github.com/datastax/csharp-driver/blob/d8bb178b06c4c075fdeebae44edff8972d6a5a76/src/Cassandra/TcpSocket.cs#L437
[content-length]: https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebresponse.contentlength?view=netframework-4.7.2#remarks
[gitter-im]: https://gitter.im/ImageSharp/General?at=5b3635af7da8cd7c8c786c53