Easily loading lots of data in parallel over HTTP, using Dataflow in .NET Core
Tags: csharp
Published: 2019-06-16 12:41:01

---
title: Easily loading lots of data in parallel over HTTP, using Dataflow in .NET Core
published: true
tags: csharp
canonical_url: https://alistairevans.co.uk/2019/06/16/easily-loading-lots-of-data-in-parallel-over-http-using-dataflow-in-net-core/
---

![](https://thepracticaldev.s3.amazonaws.com/i/0tbfb62dbqv823q799v8.PNG)

I recently had a requirement to load a large amount of data into an application, as fast as possible.

The data in question was about 100,000 transactions, stored line-by-line in a file, that needed to be sent over HTTP to a web application, that would process it and load it into a database.

This is actually pretty easy in .NET, and super efficient using async/await:

{%gist https://gist.github.com/alistairjevans/bddd44f5c59845cb8c87042e5dcbc5b1 %}

Run that through, and I get a time of **133 seconds** ; this isn’t too bad right? Around 750 records per second.

But I feel like I can definitely make this better. For one thing, my environment doesn’t look exactly look like the diagram above. It’s a scaled production environment, so looks more like this:

![](https://thepracticaldev.s3.amazonaws.com/i/46ti1zwxvbz46c19kg0j.PNG)

I’ve got lots of resources that I’m not using right now, because I’m only sending one request at a time, so what I want to do is start loading the data in parallel.

Let’s look at a convenient way of doing this, using the [System.Threading.Tasks.Dataflow](https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library) package, which is available for .NET Framework 4.5+ and .NET Core.

The Dataflow components provide various ways of doing asynchronous processing, but here I’m going to use the [ActionBlock](https://docs.microsoft.com/en-us/dotnet/api/system.threading.tasks.dataflow.actionblock-1), which allows me to post messages that are subsequently processed by a Task, in a callback. More importantly, it let’s me process messages in parallel.

Let’s look at the code for my new `StreamDataInParallel` method:

{%gist  https://gist.github.com/alistairjevans/d7b8167994a5b6fca5716ce0c3905806 %}

The great thing about Dataflow is that in only about 18 lines of code, I’ve got parallel processing of data, pushing HTTP requests to a server at a rate of my choice (controlled by the `maxParallel` parameter).

Also, with the combination of the `SendAsync` method and specifying a `BoundedCapacity`, it means I’m only reading from my file when there are slots available in the buffer, so my memory consumption stays low.

I’ve run this a few times, increasing the number of parallel requests each time, and the results are below:

> Sadly, I wasn’t able to run the benchmarking tests on the production environment (for what I hope are obvious reasons), so I’m running all this locally; the number of parallel requests I can scale to is way higher in production, but it’s all just a factor of total available cores and database server performance.

| Value of `maxParallel` | Average Records/Second |
| ---------------------- | ---------------------- |
| 1 | 750 |
| 2 | 1293 |
| 3 | 1785 |
| 4 | 2150 |
| 5 | 2500 |
| 6 | 2777 |
| 7 | 2941 |
| 8 | 3125 |

![](https://alistairevans.files.wordpress.com/2019/06/image-3.png?w=1100)

With 8 parallel requests, we get over 3000 records/second, with a time of **32 seconds** to load our 100,000 records.

You’ll notice that the speed does start to plateau (or at least I get diminishing returns); this will happen when we start to hit database contention (typically the main throttling factor, depending on your workload).

I’d suggest that you choose a sensible limit for how many requests you have going so you don’t accidentally denial-of-service your environment; we’ve got to assume that there’s other stuff going on at the same time.

Anyway, in conclusion, Dataflow has got loads of applications, this is just one of them that I took advantage of for my problem. So that’s it, go forth and load data faster!