Optimizing Celery for Workloads
Tags: celery, python, distributedcomputing, redis
Published: 2018-05-21 02:37:45

---
title: Optimizing Celery for Workloads
published: true
description: This article explains my journey to make celery (a distributed computing framework) handle over 10 million tasks a day
tags: Celery, Python, Distributed Computing, Redis
canonical_url: http://techscouter.blogspot.in/2018/05/a-day-in-life-of-software-engineer.html
cover_image: https://thepracticaldev.s3.amazonaws.com/i/tpjg0gw95c75bk8dxlhq.jpeg 
---

<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
Introduction</h2>
<div>
Firstly, a brief background about myself. I am working as a Software Engineer in one of the Alternate Asset Management Organization (<i>Handling 1.4 Trillion&nbsp;with our product suite</i>) responsible for maintaining and developing a product <b>ALT Data Analyzer. </b>My work is focused on making the engine run and feed the machines with their food.</div>
<div>

</div>
<div>
This article explains the problems we faced with scaling up our architecture and solution we followed.
&nbsp;I am dividing the blog in the following different sections:</div>
<div>
<ul style="text-align: left;">
<li><span style="font-size: large;">Product Brief</span></li>
<li><span style="font-size: large;">Current Architecture</span></li>
<li><span style="font-size: large;">Problem With Data loads&nbsp;and Sleepless Nights</span></li>
<li><span style="font-size: large;">Solutions Tried</span></li>
<li><span style="font-size: large;">The Final Destination</span></li>
</ul>
<div>
<span style="font-size: large;">
</span></div>
</div>
<h2 style="text-align: left;">
<span style="font-size: large;">Product Brief</span></h2>
<div>
The idea of building this product was to give users an aggregated view of the WEB around a company. By WEB I mean the data that is flowing freely over all the decentralized nodes of the internet. We try to capture all the financial, technical and fundamental data for the companies, pass that data through our massaging and analyzing pipes and provide an aggregated view on the top of this data. Our dashboards can be one stop for all the information around a company and can aid an investor in his analysis.&nbsp;</div>
<div>
Some of the Data Sources that we support as of now are:</div>
<div>
<ul style="text-align: left;">
<li>Stock News from multiple sources</li>
<li>RSS feed&nbsp;</li>
<li>Twitter</li>
<li>Reddit</li>
<li>Yahoo News</li>
<li>Yahoo Finance</li>
<li>Earning Calenders</li>
<li>Report Filings</li>
<li>Company Financials</li>
<li>Stock Prices</li>
</ul>
</div>
<h2 style="text-align: left;">
Current Architecture</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-zrXUbB9-EVA/WvaSLHGo0fI/AAAAAAAAbmU/jwHR-jSLH7MALNt8Q7ghWotC5H9dhQvcgCLcBGAs/s1600/internet.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="435" data-original-width="1008" height="275" src="https://4.bp.blogspot.com/-zrXUbB9-EVA/WvaSLHGo0fI/AAAAAAAAbmU/jwHR-jSLH7MALNt8Q7ghWotC5H9dhQvcgCLcBGAs/s640/internet.jpeg" width="640" /></a></div>
<div>
<h3 style="text-align: left;">
Broad View</h3>
<div>
We knew that problem that we are solving has to deal with the cruel decentralized Internet. And we need to divide the large task of getting the data from the web and analyzing it into small tasks.</div>
</div>
<div>

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-tPW3nCkXYps/WvaVW4XlHoI/AAAAAAAAbmg/_jpZxvZMOA8M-CpZ0fuffHzeU9IzaYbFgCLcBGAs/s1600/croped1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="115" data-original-width="383" height="120" src="https://1.bp.blogspot.com/-tPW3nCkXYps/WvaVW4XlHoI/AAAAAAAAbmg/_jpZxvZMOA8M-CpZ0fuffHzeU9IzaYbFgCLcBGAs/s400/croped1.jpeg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
Fig 1</div>
<div>

</div>
<div>
On exploring different projects and technologies and analyzing the community support we came to a decision to use <b>Python</b> as our language of choice and <b>Celery</b> as our commander.&nbsp;</div>
<div>

</div>
<div>
Python is a pretty vast language backed by a large set of libraries. Since inception, It has gained a lot&nbsp;of popularity among the developer and Data Science communities. One of the major reason to use python as a backend is the project Celery.&nbsp;</div>
<div>

</div>
<div>
Its website defines celery as<i>&nbsp;An asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. </i>To more about celery you can visit its website <a href="http://www.celeryproject.org/" target="_blank">here</a>.</div>
<div>

</div>
<div>
By now we were clear of how we want to proceed. We wanted to divide the process in Fig 1 into granular&nbsp;units (in celery terminology task). Keeping this as the baseline we identified all the individual units which can work independently in the system.</div>
<div>
This gave rise to a new look to Fig 1</div>
<div>

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-ShzlN3uYpA4/Wvacz_r0wtI/AAAAAAAAbmw/2q0sPcEN4FM3qTkEPOMTqq_zQyL5CID8wCLcBGAs/s1600/cropped1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="274" data-original-width="634" height="276" src="https://2.bp.blogspot.com/-ShzlN3uYpA4/Wvacz_r0wtI/AAAAAAAAbmw/2q0sPcEN4FM3qTkEPOMTqq_zQyL5CID8wCLcBGAs/s640/cropped1.jpeg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
Fig 2</div>
<div>

</div>
<div>
Were are using MongoDB in a replication cluster as a Database engine and Redis DB for queuing and communication between independent celery workers.</div>
<div>
The following figure describes the communication of celery works with the help of broker(Redis)</div>
<div>

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-S1r1aSUiOcc/Wvae5cP0BqI/AAAAAAAAbnE/QNNKir9dgUEGEpDk00KWfOOqLfVxKlAoQCLcBGAs/s1600/workers.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="251" data-original-width="621" height="258" src="https://3.bp.blogspot.com/-S1r1aSUiOcc/Wvae5cP0BqI/AAAAAAAAbnE/QNNKir9dgUEGEpDk00KWfOOqLfVxKlAoQCLcBGAs/s640/workers.jpeg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>

</div>
<div>
<h2 style="text-align: left;">
&nbsp;<span style="font-size: large;">Problem With Data loads&nbsp;and Sleepless Nights</span></h2>
</div>
<div>
Everything was working fine with the development workloads, we had deployed around a cluster of 5 machines in the celery cluster (i7 processors with 16 gigs of memory each) with each machine running some celery workers.&nbsp;</div>
<div>
But this was the calm before the storm. As we planned to increase our workloads and configure more tasks the system started to fade and run out of memory very frequently. We were now generating around 10 million tasks a day. Celery workers were not able to process the tasks at this speed and this was building up tasks in the queues which were slowing down the Redis broker as it was now storing a large number of tasks, this was, in turn, making the whole system run out of memory and gradually to a halt.</div>
<div>

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-yFpJdkfSkB0/WvgHuFRr77I/AAAAAAAAbno/p4siRfrXTVIDEypgbo5qd9LfiKRjoW-qACLcBGAs/s1600/calm.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="712" data-original-width="950" height="478" src="https://3.bp.blogspot.com/-yFpJdkfSkB0/WvgHuFRr77I/AAAAAAAAbno/p4siRfrXTVIDEypgbo5qd9LfiKRjoW-qACLcBGAs/s640/calm.jpg" width="640" /></a></div>
<div>

</div>
<div>
Our beautifully designed ship was sinking and we had to&nbsp;do something to save it.</div>
<div>
<span style="font-size: large;">
</span></div>
<h2 style="text-align: left;">
<span style="font-size: large;">Solutions Tried</span></h2>
<div>
As we faced this issue the first approach we thought, as any other developer would have applied is to add more nodes to the cluster. While discussing this solution there was an argument that if the loads increases&nbsp;again with time, will we be adding more nodes again and again.</div>
<div>

</div>
<div>
So, we skipped this solution and agreed to dig deep into the problem.&nbsp;</div>
<div>
On deep analysis we realized that a large number of tasks were populated by the publishers and these tasks were getting stored in the Redis queues, with the data loads the Redis eventually slows down the rate at which the tasks were being sent to the consumers making the consumers idle.</div>
<div>

</div>
<div>
So our system was stuck in an infinite loop.&nbsp;</div>
<div>

</div>
<div>
<ul style="text-align: left;">
<li>Redis sends tasks at high speed but consumers cannot consume at that rate.</li>
<li>Redis slows down and lowers the rate at which it sends tasks and now consumers stay idle most of the time.</li>
</ul>
<div>
In both the situations, the common name that was in doubt was the Redis DB. On researching and exploring the RedisDB we found that it is a single-threaded system and performs all the tasks in a round robin fashion. Also, it was saving the tasks received by the systems in memory which was increasing the memory consumption and the under heavy loads that single thread was busy performing the persistence and other tasks that it slows down the polling process initiated by the consumers. So, we found the reasons for both the problems that we were facing.</div>
</div>
<div>

</div>
<div>
To handle the heavy workload one of the choices was to shard the Redis server into a cluster. So we created a Redis partition set (tutorials can be seen <a href="https://redis.io/topics/partitioning" target="_blank">here</a>). We created a cluster of three nodes with each node handling an equal number of keys through a <i>consistent hashing technique</i>.&nbsp;</div>
<div>

</div>
<div>
We plugged this cluster with the celery app and the workers were throwing an exception "<i>MOVED to server 192.168.12.12</i>"</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-zw8o3gX19lI/WvibxsT5RII/AAAAAAAAboE/nEeZLU_ia0kqszxTpylxt2FoYDDyfrKfgCEwYBhgL/s1600/fr_201691716455.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="240" data-original-width="350" height="273" src="https://2.bp.blogspot.com/-zw8o3gX19lI/WvibxsT5RII/AAAAAAAAboE/nEeZLU_ia0kqszxTpylxt2FoYDDyfrKfgCEwYBhgL/s400/fr_201691716455.jpg" width="400" /></a></div>
<div>

</div>
<div>
On googling we found that Redis cluster is not yet supported by celery. On one&nbsp;hand we thought we had a solution but on the other, that was not yet supported by the underlying framework :(</div>
<div>

</div>
<div>
Now the exploration began again to get a solution to our problem and we thought of using a proxy server in front of Redis cluster&nbsp;<a href="https://github.com/twitter/twemproxy" style="color: #0066aa; font-family: &quot;Open Sans&quot;, Helvetica, sans-serif; text-decoration-line: none;">Twemproxy</a>. But this time we first choose to check the compatibility with our framework and boom...... we cannot be more wiser in taking this path.</div>
<div style="text-align: center;">
<i>Proxy was not yet supported by Celery.</i></div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
Now frustrated with this incompatibility issue we tried to figure out what all things are compatible with the framework. Some of the compatible brokers were</div>
<div style="text-align: left;">
<ul style="text-align: left;">
<li>SQS</li>
<li>Redis</li>
<li>RabbitMQ</li>
<li>Zookeeper</li>
</ul>
<div>
A straightaway thought was to try a different broker so, we began to explore these options. Following table proved useful in narrowing down our research</div>
<div>

</div>
<div>
<table border="1" class="docutils" style="background-color: white; border-collapse: collapse; border: 1px solid rgb(136, 136, 136); box-shadow: rgb(238, 238, 238) 2px 2px 4px; color: #3e4349; font-family: Optima, Segoe, &quot;Segoe UI&quot;, Candara, Calibri, Arial, sans-serif; font-size: 17px; text-align: left;"><colgroup><col width="23%"></col><col width="22%"></col><col width="25%"></col><col width="31%"></col></colgroup><tbody valign="top">
<tr class="row-odd"><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><strong>Name</strong></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><strong>Status</strong></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><strong>Monitoring</strong></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><strong>Remote Control</strong></td></tr>
<tr class="row-even"><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><em>RabbitMQ</em></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Stable</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Yes</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Yes</td></tr>
<tr class="row-odd"><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><em>Redis</em></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Stable</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Yes</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Yes</td></tr>
<tr class="row-even"><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><em>Amazon SQS</em></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Stable</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">No</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">No</td></tr>
<tr class="row-odd"><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;"><em>Zookeeper</em></td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">Experimental</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">No</td><td style="border: 1px solid rgb(136, 136, 136); padding: 0.25em 0.7em;">No</td></tr>
</tbody></table>
</div>
<div style="text-align: center;">

</div>
<div style="text-align: left;">
Redis is what we have already tried so we went for the second option that is stable and provides more features i.e RabbitMQ.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
<b>Spoiler:</b></div>
<div style="text-align: left;">
By now we knew that RabbitMQ is one the best choice for the brokers and is used by wide variety of clients in production and Redis is the best choice in terms of result backend (intermediate results that are stored by a task in Celery chains and chords).</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
We did necessary changes to the system and tried to run the system.</div>
<div style="text-align: left;">

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-dTzKGioKH8c/Wvig1fw85LI/AAAAAAAAboU/3bugIbOyg6IrPyDrVIVrGqMfr6DC-tJ9QCLcBGAs/s1600/fc6ba9d5149c057a2582a7f07523abd2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="651" data-original-width="650" height="200" src="https://3.bp.blogspot.com/-dTzKGioKH8c/Wvig1fw85LI/AAAAAAAAboU/3bugIbOyg6IrPyDrVIVrGqMfr6DC-tJ9QCLcBGAs/s200/fc6ba9d5149c057a2582a7f07523abd2.jpg" width="198" /></a></div>
<div style="text-align: left;">
This time system nearly went into an irrecoverable state consuming all the memory and we had to restart the servers.</div>
<div style="text-align: left;">
While all this was happening we were monitoring the RabbitMQ through its admin tool and found a fishy thing. It was creating a large number of queues nearly as many as the number of tasks.</div>
<div style="text-align: left;">
On googling we found that the queues were created to store the result of the intermediate tasks and this was consuming too much space in the memory as all this storage was in memory.</div>
<div style="text-align: left;">
<span style="font-size: large;">
</span></div>
<h2 style="text-align: left;">
<span style="font-size: large;">The Final Destination</span></h2>
<div style="text-align: left;">
Collecting all the clues from our exploration we decided to use both the systems i.e Redis and RabbitMQ for the work that they are best at.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
We deployed Redis as a result backend to store the intermediate results and RabbitMQ as a broker for maintaining communication and passing tasks (remember the spoiler above).</div>
<div style="text-align: left;">

</div>
</div>
<div style="text-align: left;">

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-zFlxvWKDZh8/WvijzlNdrfI/AAAAAAAAbog/ZOD6D9Hq9_8i5wL5zq3BK60BCfqCR-bSQCLcBGAs/s1600/fg.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="277" data-original-width="624" height="284" src="https://1.bp.blogspot.com/-zFlxvWKDZh8/WvijzlNdrfI/AAAAAAAAbog/ZOD6D9Hq9_8i5wL5zq3BK60BCfqCR-bSQCLcBGAs/s640/fg.jpeg" width="640" /></a></div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
With this architecture, we were able to run the system under the workload on over 10 million tasks a day which can be scaled easily.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
Hope this helps someone going through the same problem.</div>
<h2 style="text-align: left;">
Acknowledgments</h2>
<div style="text-align: left;">
We took hints from the Celery documentation, Redis documentation, Stackoverflow threads, Github Issue page of Celery, Case studies by Instagram and Trivago.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">

</div>
</div>