Moving From NodeJs To Go – Doing More, Faster For Less
Tags: go, node, showdev, architecture
Published: 2020-09-03 16:55:07

*this was originally posted SimilarWeb's engineering blog:*
https://similarweb.engineering/moving-from-nodejs-to-go-doing-more-faster-for-less/


### **Motivation and Introduction**
<span style="font-weight: 400;">As an engineer in the Data Factory group at SimilarWeb you quickly learn that providing our clients with accurate data and insights is our top priority. Doing so often requires processing GBs and even TBs of data in a short timeframe.</span>

<span style="font-weight: 400;">This is a challenge I enjoy taking on, and why I’m excited to tell you about the time when we optimized one of our most demanding services. By doing so we increased it’s processing speed&nbsp; </span><b>x28</b><span style="font-weight: 400;"> and&nbsp; decreased resource usage by </span><b>x10</b><span style="font-weight: 400;">!</span>
### ![](https://media.giphy.com/media/l3mZliqXpqkVkkQYE/giphy.gif)
### 
### <b>The Challenge</b>
<span style="font-weight: 400;">Make ~150M calls to an external API to fetch the data and process it in order to have it ready for use by our data analysis team, all within a couple of days.</span>

<span style="font-weight: 400;">If you’re in any way familiar with big data processing you must be thinking </span><i><span style="font-weight: 400;">“Processing 150M results in a distributed way using Spark/MapReduce shouldn’t take too long. Nick, that’s not a challenge”. </span></i><span style="font-weight: 400;">I mean..</span><i><span style="font-weight: 400;">. </span></i><span style="font-weight: 400;">you’re not wrong.</span>

<span style="font-weight: 400;">The challenge is fetching the data from this API as quickly as possible for it to be processed in time. Given the limitations we have with that API. </span><b>1000 Requests/sec</b>

&nbsp;
### <b>What we started with</b>
<span style="font-weight: 400;">Initially we wanted to verify the relevancy of the data to our needs.</span>

<span style="font-weight: 400;">To avoid committing a lot of engineering time to a task that might not pan out, we decided on using some of our existing code to make all these API calls, as a test run. We were able to use that code because it was already being used for continuous processing of similar looking data on a smaller scale.</span>

<span style="font-weight: 400;">Said existing code was a NodeJs service running in docker (</span><i><span style="font-weight: 400;">node:10-slim </span></i><span style="font-weight: 400;">image), orchestrated by Nomad and given 256Mhz of CPU and 128Mb of memory.</span>

<span style="font-weight: 400;">In order to get to around 500 Request/sec we deployed 700 containers, those services could process these requests in an astonishing rate of 1 Request/sec per instance. </span><b>&nbsp;Meaning 179,200Mhz&nbsp; (179.2Ghz) CPU and 89,600Mb (89.6Gb) memory is reserved for this process ALONE on our cluster! </b><span style="font-weight: 400;">(that and we’re barely hitting the half mark of the allowed rate for this API)</span>

<span style="font-weight: 400;">Now. I’m willing to acknowledge this service is not optimized for speed, </span><i><span style="font-weight: 400;">or maybe at all</span></i><span style="font-weight: 400;">, but it worked great for our ongoing use case.&nbsp;</span>

<span style="font-weight: 400;">Some notes regarding the lack of optimization for the curious folk.</span>

 	

- <span style="font-weight: 400;">Input messages were read from Amazon SQS one by one.</span>

 	

- <span style="font-weight: 400;">Output records were written to Kinesis Firehose one by one.</span>

 	

- <span style="font-weight: 400;">Barely any asynchronous code (remind me, why did we use Node here if we’re not utilizing its strengths?)</span>


![](https://media.giphy.com/media/aMMWn9Ogf3UOI/giphy.gif)

&nbsp;
### <b>Optimizing the NodeJs service</b>
<span style="font-weight: 400;">Knowing I’m </span><del><span style="font-weight: 400;">not</span></del><span style="font-weight: 400;"> a NodeJs expert, I started this adventure by putting my thinking cap on and getting my best rubber ducky to debug whatever code I might produce and started optimizing.&nbsp;</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/Image-from-iOS-300x225.jpg)

<span style="font-weight: 400;">First things first, the simple fixes. Increasing the number of messages pulled from SQS at a time, from 1 to 10. I did it to save some time on the network overhead of each call. Batch puts to Kinesis Firehose were another easy fix that saves tons of time on network overhead, because in this case you can batch up to 500 messages at a time.</span>

<span style="font-weight: 400;">With the simple fixes out of the way I had to get some proper asynchronicity into this service. This is where my thinking cap came into play and I started looking into a way to process the 10 messages I was now getting on every pull from SQS.</span>

<span style="font-weight: 400;">In my search for an efficient way to work asynchronously I came across the `workerpool` package. And by looking at the documentation it seemed just perfect, I mean look at the features:</span>

 	

- <span style="font-weight: 400;">Easy to use (</span><i><span style="font-weight: 400;">you love to hear it</span></i><span style="font-weight: 400;">)</span>

 	

- <span style="font-weight: 400;">Runs in the browser and on node.js (</span><i><span style="font-weight: 400;">NodeJs</span></i> <b>✓</b><span style="font-weight: 400;"> )</span>

 	

- <span style="font-weight: 400;">Dynamically offload functions to a worker (</span><i><span style="font-weight: 400;">exactly what I’m looking for</span></i><span style="font-weight: 400;">)</span>

 	

- <span style="font-weight: 400;">Cancel running tasks (</span><i><span style="font-weight: 400;">great in case of a premature shutdown</span></i><span style="font-weight: 400;">)</span>

 	

- <span style="font-weight: 400;">Set a timeout on tasks (</span><i><span style="font-weight: 400;">good to put up boundaries</span></i><span style="font-weight: 400;">)</span>

 	

- <span style="font-weight: 400;">Handles crashed workers (</span><i><span style="font-weight: 400;">shit happens</span></i><span style="font-weight: 400;">)</span>

 	

- <span style="font-weight: 400;">Small: 5 kB minified and gzipped (</span><i><span style="font-weight: 400;">*chef’s kiss*</span></i><span style="font-weight: 400;">)</span>


[<span style="font-weight: 400;">https://www.npmjs.com/package/workerpool</span>](https://www.npmjs.com/package/workerpool)

<span style="font-weight: 400;">Told you **it’s perfect!**</span>

&nbsp;

{% details *Foreshadowing warning* (click to expand) %} 
*Remember to read the small print, or in this case the next paragraph and the font is regular size.*
**<em>In Node.js everything runs in parallel, except your code. What this means is that all I/O code that you write in Node.js is non-blocking, while (conversely) all non-I/O code that you write in Node.js is blocking.</em>**
*Did I mention that I’m no NodeJs expert?*
{% enddetails %}

&nbsp;

<span style="font-weight: 400;">The problem I encountered with this package though, was a </span><i><span style="font-weight: 400;">DataCloneError</span></i><span style="font-weight: 400;"> whenever I attempted to use functions from other files in the project. Basically those functions went unrecognized by the workers and could not be serialized.</span>

<span style="font-weight: 400;">NEXT!</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/staring-guy.jpg)

<span style="font-weight: 400;">Why use a fancy package to do it for me when I can start the number of workers I need and reuse them?</span>

&nbsp;
## <b>What I ended up doing</b>
<span style="font-weight: 400;">In each instance I start, let’s say 5 worker threads (</span>[<span style="font-weight: 400;">https://nodejs.org/api/worker_threads.html</span>](https://nodejs.org/api/worker_threads.html)<span style="font-weight: 400;">), each thread runs in a loop as long as there are messages to process. Each thread reads 10 messages and processes them asynchronously by offloading the processing to another function.&nbsp; The threads run said function asynchronously for each message and then block while waiting for them to finish. Meaning we now have 50 messages being processed asynchronously, </span><b>up from 1!</b>

&nbsp;
## <b>One more thing</b>
<span style="font-weight: 400;">If you read the foreshadowing warning a few paragraphs ago you might have guessed about the main problem with using NodeJs for this workload.</span>

<span style="font-weight: 400;">NodeJs is great for IO, which is what we have here, right?</span>

1. Get messages from SQS. IO
2. Getting the data from the external API. IO
3. Gzip compressing the data so it's small enough to be sent over Kinesis Firehose, especially in batches. **NOT IO**
4. Base64 encoding the data to be sure it's of a valid format for transfer through Kinesis Firehose. **NOT IO**
5. Sending the resulting data to Kinesis Firehose. IO
 	

<span style="font-weight: 400;">Whelp…&nbsp; ⅖ steps are not IO and can only be optimized by horizontally scaling the service.</span>

<span style="font-weight: 400;">All my efforts were rewarded though. With just under 2 Requests/sec per instance processing time. </span><b>2x faster than before. ~300 containers down from ~700.</b>

&nbsp;

<span style="font-weight: 400;">But.&nbsp;</span>

<span style="font-weight: 400;">Some wise words from a python core contributor Raymond Hettinger came to mind.</span>

<figure>
  <img src="https://similarweb.engineering/wp-content/uploads/2020/08/Untitled.jpg" alt="there must be a better way" style="width:100%">
  <figcaption>Fooled ya! No python will slither it’s way into this post :)</figcaption>
</figure>

##### 
&nbsp;
&nbsp;
### <b>Rewriting in Go</b>
<span style="font-weight: 400;">Another language I’m familiar with and am actually more fluent in is </span><b>Go</b><span style="font-weight: 400;">.</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/gointro.jpg)

<span style="font-weight: 400;">It’s a language with a great standard library, amazing community and most importantly for this specific case, Go can handle non IO tasks </span><b>efficiently</b><span style="font-weight: 400;">.</span>

<span style="font-weight: 400;">After you’re done reading you might want to check it out here:</span>

[<span style="font-weight: 400;">https://golang.org/</span>](https://golang.org/)

<span style="font-weight: 400;">There’s a nice tour you can take in order to discover all the amazing features this language has. And when you’re done you can play around with Go right there in the playground, no installation or download required.</span>

&nbsp;

<span style="font-weight: 400;">Back to business.&nbsp;</span>

<span style="font-weight: 400;">This service, just like our original service runs in docker (</span><i><span style="font-weight: 400;">golang:1.13.4-alpine </span></i><span style="font-weight: 400;">image), is orchestrated by Nomad and given 1000Mhz of CPU and 512Mb of memory.</span>
#### 
#### <b>Some basic Go terminologies</b>
<span style="font-weight: 400;">**Goroutine** - </span><i><span style="font-weight: 400;">A goroutine is a lightweight thread managed by the Go runtime</span></i><span style="font-weight: 400;">. You can start a goroutine simply by using the `go` keyword. e.g.&nbsp; `go SomeFunction()` </span>[<span style="font-weight: 400;">https://tour.golang.org/concurrency/1</span>](https://tour.golang.org/concurrency/1)

<span style="font-weight: 400;">**Channels** - Channels are a typed conduit through which you can send and receive values with the channel operator, &lt;-&nbsp;&nbsp;</span><i><span style="font-weight: 400;">By default, sends and receives block until the other side is ready. This allows goroutines to synchronize without explicit locks or condition variables</span></i><span style="font-weight: 400;">. This is the behavior of an </span><b>Unbuffered</b><span style="font-weight: 400;"> Channel.&nbsp;&nbsp;</span>[<span style="font-weight: 400;">https://tour.golang.org/concurrency/2</span>](https://tour.golang.org/concurrency/2)

*<strong>Buffered </strong>*<span style="font-weight: 400;">**Channels** - This type of channel has an internal buffer of a predefined size, meaning messages can be put in it and be received in a later time. Or in other words, </span><i><span style="font-weight: 400;">sends to a buffered channel block only when the buffer is full. Receives block when the buffer is empty</span></i><span style="font-weight: 400;">.&nbsp;</span>[<span style="font-weight: 400;">https://tour.golang.org/concurrency/3</span>](https://tour.golang.org/concurrency/3)

&nbsp;
#### <b>Initial Design</b>
<span style="font-weight: 400;">Using these basic built in features we can achieve greatness. Goroutines for concurrency, channels for synchronization and communication between goroutines.</span>

<span style="font-weight: 400;">Before we get to the design schematic please refer to the legend.</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/legend.jpg)

&nbsp;

<span style="font-weight: 400;">The initial design schematic.</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/ColdDb-initial.jpg)

<span style="font-weight: 400;">What we have here is a single goroutine reading messages from SQS, 10 at a time and putting them into </span><b>A,</b><span style="font-weight: 400;"> an unbuffered channel. Listening to that channel is a single goroutine who’s job consists of parsing every message into something we can use and passing it on into </span><b>B</b><span style="font-weight: 400;">, another unbuffered channel. Reading from it are multiple goroutines that will process all incoming messages.</span>

<span style="font-weight: 400;">At this point any available goroutine will pull a message from </span><b>B</b><span style="font-weight: 400;">, process it and put the result into </span><b>C</b><span style="font-weight: 400;">, a buffered channel that has an internal buffer size matching the batch size we use when putting messages into Firehose. Due to message size constraints in this case I limited the batch size to 50.</span>

<span style="font-weight: 400;">On the other hand, if all message processing goroutines are busy processing messages, the parsing goroutine will block on send to </span><b>B </b><span style="font-weight: 400;">(no one is ready to receive from it), which in turn will lead to the message receiving goroutine to block on send to </span><b>A</b><span style="font-weight: 400;"> for that same reason.&nbsp;</span>

<span style="font-weight: 400;">SQS message reading and message parsing goroutines will unblock as soon as any processing goroutine will be ready to receive a message from </span><b>B.</b>

<span style="font-weight: 400;">This redesign and rewrite resulted in a significant improvement over our initial NodeJs service. We got to ~700 Requests/sec (up from 500) with 25 instances (down from ~300). Or around </span><b>28 Requests/sec per instance</b>

<span style="font-weight: 400;">If you’re still following along, this translates to:</span>

 	

- <span style="font-weight: 400;">More than x14 improvement in per instance processing speed.&nbsp;</span>

 	

- <span style="font-weight: 400;">Total allocation of 25Ghz (down from 179.2Ghz) CPU and 12.5Gb (Down from 89.6Gb) memory, that is more than x7 decrease in CPU and memory allocation compared to our original service.</span>


<span style="font-weight: 400;">Now this is fast! And as a bonus to the performance boost we are saving some serious dough ($$$) with the reduction in resource allocation.</span>

<span style="font-weight: 400;">Finally! I was done and could move on to my next task for that iteration.</span>

![](https://media.giphy.com/media/RKGUifTpiUSj0F9WaE/giphy.gif)

<span style="font-weight: 400;">Or so I thought.&nbsp;</span>

&nbsp;

<span style="font-weight: 400;">On second glance the way this service works reminded me of something I heard about a while back. </span><b>The pipeline pattern</b><span style="font-weight: 400;"> in Go. </span><i><span style="font-weight: 400;">There's no formal definition of a pipeline in Go; it's just one of many kinds of concurrent programs. Informally, a pipeline is a series of stages connected by channels, where each stage is a group of goroutines running the same function. </span></i><span style="font-weight: 400;">(</span><span style="font-weight: 400;">[http](https://blog.golang.org/pipelines)</span>[s://blog.golang.org/pipelines](https://blog.golang.org/pipelines))

<span style="font-weight: 400;">Sounds a lot like what is shown in the design schematic, but we’re not there yet, not exactly. We do have a series of stages connected by channels. But only one of our stages is a </span><b>group </b><span style="font-weight: 400;">of goroutines, other than that we have single goroutines doing the work.&nbsp;</span>

#### 
#### <b>Pipeline pattern, the right way</b>
<span style="font-weight: 400;">In addition to using groups of goroutines over single goroutines in most stages, I decided on switching to buffered channels. The reason I switched to buffered channels was because I wanted to save time on overhead for each stage.</span>

<span style="font-weight: 400;">For example: if all processing goroutines are busy, we used to have the parsing goroutine block and wait for someone to be ready to receive a message from it. Meaning the parsing work will only be done when someone is ready to process the message. Using a buffered channel we can “pre-parse” lots of messages even if no goroutine is available to process them at the moment. This way we won’t have to wait for the parsing to happen whenever a processing goroutine is ready for it because we already banked a bunch of parsed messages in the buffer.&nbsp;</span>

<span style="font-weight: 400;">The overhead is even worse when it comes to the goroutine reading messages from SQS, the network overhead is far larger than the overhead we experience when parsing.</span>

&nbsp;

<span style="font-weight: 400;">The final design schematic:</span>

![](https://similarweb.engineering/wp-content/uploads/2020/08/ColdDb-final-1.jpg)

<span style="font-weight: 400;">For the following explanation let’s define X=10.</span>

<span style="font-weight: 400;">As you can see, this schematic doesn’t look that different from the initial one.</span>

 	

- <span style="font-weight: 400;">X goroutines are reading messages from SQS (each 10 at a time) and putting them into channel </span><b>A</b><span style="font-weight: 400;">, a buffered channel of size X*10.</span>

 	

- <span style="font-weight: 400;">X goroutines are parsing the messages from channel </span><b>A</b><span style="font-weight: 400;"> before passing them on to channel </span><b>B</b><span style="font-weight: 400;">, a buffered channel of size X*10.</span>

 	

- <span style="font-weight: 400;">75 goroutines read the parsed messages and process them (75 is a number I got to by benchmarking performance for different values). The value is then put into channel </span><b>C</b><span style="font-weight: 400;">, a buffered channel of size 50 to match the batch size we use when putting messages into Firehose.</span>

 	

- <span style="font-weight: 400;">Only 2 goroutines are reading those results and putting them into Firehose in batches of 50. 2 goroutines are enough for the job because batch puts happen less frequently, once every 50 results.</span>


<span style="font-weight: 400;">These simple optimizations resulted in even more improvements over our simpler Go service. We ended up with&nbsp; </span><b>~840</b><span style="font-weight: 400;"> Requests/sec (up from ~700) with </span><b>17</b><span style="font-weight: 400;"> instances (down from 25). Or around </span><b>50 Requests/sec per instance.</b>

![](https://media.giphy.com/media/xT0xeJpnrWC4XWblEk/giphy.gif)
### 
### <b>Conclusion</b>
<span style="font-weight: 400;">Comparing our final version </span><span style="font-weight: 400;">to the first version of the Go service we can see nearly x2 improvement in processing speed, or </span><b>x28 compared to our original NodeJs service</b><span style="font-weight: 400;">, and total allocation of 17Ghz (down from 25Ghz) CPU and 8.5Gb (Down from 12.5Gb) memory. That is more than a </span><b>x10 decrease compared to our original NodeJs service</b><span style="font-weight: 400;">.</span>

<span style="font-weight: 400;">The number of calls we need to make can vary from month to month, it might end up being 150, 160 or even 170 Million calls. But with our newfound speed we are able to go through the whole list in just over 2 days!</span>

<span style="font-weight: 400;">What this all demonstrates is that you should always strive to find and use the right tool for the job. Going with the easy solution would have ended up costing us at minimum 8 times more than what we ended up paying for the used resources in our final version.</span>

<span style="font-weight: 400;">Some key takeaways :</span>

 	

- <i><span style="font-weight: 400;">NodeJs</span></i><span style="font-weight: 400;"> is great for asynchronous IO operations but falls short when it comes to CPU intensive work.</span>

 	

- <i><span style="font-weight: 400;">Go</span></i><span style="font-weight: 400;"> can handle both CPU intensive </span><b>and </b><span style="font-weight: 400;">concurrent workloads well.</span>

 	

- <span style="font-weight: 400;">Picking the right tool for the job is one of the most important decisions you will make in any project.</span>

 	

- <span style="font-weight: 400;">The pipeline pattern is a simple way you can dramatically speed up your services.</span>

&nbsp;
### <b>What’s next?</b>
<span style="font-weight: 400;">There are further optimizations possible. Using the GOGC and GOMAXPROCS environment variables we can speed up each container even more,Optimizing the service using these environment variables will speed up each container but will require more allocated CPU/Memory. but it will require more allocated CPU/Memory.</span>

<span style="font-weight: 400;">For more information about these and other useful environment variables, check out Dave Cheney’s blog </span>[<span style="font-weight: 400;">https://dave.cheney.net/tag/gogc</span>](https://dave.cheney.net/tag/gogc)