How to parallelize for loops in Python and Work with Shared Dictionaries
Tags: python, programming, computerscience, performance
Published: 2021-03-08 21:21:33

<!-- wp:heading {"level":4} -->
<h4>This article will cover the implementation of a for loop with multiprocessing and a for loop with multithreading. We will also make multiple requests and compare the speed.</h4>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3>Table of&nbsp;Contents</h3>
<!-- /wp:heading -->

<!-- wp:list {"ordered":true} -->
<ol><li>Sequential</li><li>MultiProcessing</li><li>MultiThreading</li><li>Sharing Dictionary using Manager</li><li>‘Sharing’ Dictionary by combining Dictionaries at the end</li><li>Comparing Performance of MultiProcessing, MultiThreading(making API requests)</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>I have written all the code in google colab, you can access <a href="https://colab.research.google.com/drive/1Q5tszDxKLyRINp0reXU196h1vJXhz2TQ?usp=sharing" rel="noreferrer noopener" target="_blank">the notebook here</a></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Sequential</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We will be making Post requests to <a href="https://jsonplaceholder.typicode.com/" rel="noreferrer noopener" target="_blank">this API</a>. The function post_req has a parameter called data. This parameter will contain the data which will be sent to the API</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":534,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-1024x303.png" alt="" class="wp-image-534"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Below is the Sequential way of making 500 Post requests to the API.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":535,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-1-1024x340.png" alt="" class="wp-image-535"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>On google colab, this piece of code took around 100 seconds to complete.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Multiprocessing</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We will be making 500 requests to the above-mentioned API. We will be using the <a href="https://docs.python.org/3/library/concurrent.html" rel="noreferrer noopener" target="_blank">concurrent package</a>. Below is the general format to use multiprocessing for a for loop</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":538,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-4-1024x433.png" alt="" class="wp-image-538"/></figure>
<!-- /wp:image -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>concurrent.futures.ProcessPoolExecutor allows you to set the maximum number of proccesses to be used by setting a value for the max_workers parameter</p></blockquote>
<!-- /wp:quote -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>The <strong>function</strong> parameter of executor.submit() should not have any brackets since we do not want to invoke the function.</p></blockquote>
<!-- /wp:quote -->

<!-- wp:paragraph -->
<p><strong>submit()</strong> returns a future object. To get the actual returned value of the function, we will use the <strong>result</strong> method of the future object. The returned values can be in any order.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below is how we would use multiprocessing to make the 500 requests</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":539,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-5-1024x414.png" alt="" class="wp-image-539"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In google colab, this took around 50 seconds. If your piece of code executes in less than a second, there is probably an error somewhere. To check the error during multiprocessing, simply print the result of one of the future values.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Multithreading</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The format for multithreading is pretty similar to multiprocessing. The only difference is instead of <strong>concurrent.furtures.ProcessPollExecutor()</strong>, we will use <strong><a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor" rel="noreferrer noopener" target="_blank">concurrent.futures.ThreadPoolExecutor()</a></strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":540,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-6-1024x396.png" alt="" class="wp-image-540"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Above is the piece of code which makes 500 requests using Multithreading. In google colab it took around 15 seconds. To check the error during multithreading, simply print the result of one of the future values.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>concurrent.futures.ThreadPoolExecutor allows you to set the maximum number of threads to be used by setting a value for the max_workers parameter</p></blockquote>
<!-- /wp:quote -->

<!-- wp:heading {"level":3} -->
<h3>Sharing Dictionary using&nbsp;Manager</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We are going to use a dictionary to store the return values of the function. The key will be the request number and the value will be the response status. Since we are making 500 requests, there will be 500 key-value pairs in our dictionary.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":541,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-7-1024x247.png" alt="" class="wp-image-541"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The parameter <strong>d</strong> is the dictionary that will have to be shared.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To share a dictionary we will use <strong>Manager().&nbsp;</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":542,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-8-1024x247.png" alt="" class="wp-image-542"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Since the function is not returning anything, we do not need to store the future objects returned by the submit function.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":543,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-9-1024x284.png" alt="" class="wp-image-543"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The dictionary created using Manager() is thread-safe and has a lock. This might make your program slow if there are a bunch of inserts.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>‘Sharing’ Dictionary by combining Dictionaries at the&nbsp;end</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We will need to update our function to return a key-value pair instead of directly storing it inside our shared dictionary</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":544,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-10-1024x247.png" alt="" class="wp-image-544"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We will use a dictionary’s update function to store all the returned values. Below is the piece of code which invokes the function multiple times</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":545,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-11-1024x321.png" alt="" class="wp-image-545"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As you can see, this time we are storing the future objects since we will need them to store the returned values. We will essentially iterate over the list of future objects and update our dictionary</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":546,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://www.realpythonproject.com/wp-content/uploads/2021/03/image-12-1024x247.png" alt="" class="wp-image-546"/></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>Comparing Performance of MultiProcessing, MultiThreading</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We won’t be comparing with sequential code since sequential code is pretty slow and will take a long time to make a large number of requests.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We will be making 5&nbsp;, 10&nbsp;, 50, 100, 500, 1000, 2500, 5000 and 10000 requests.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To make it easier, we will create two functions. One of the functions would use multi-processing while the other would use multi-threading.</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://cdn-images-1.medium.com/max/1200/1*Fnl84I0lkSKB7xk9XO42MQ.png" alt=""/><figcaption>Code to&nbsp;compare</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We will use the above times to plot a line graph to compare the performance</p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://cdn-images-1.medium.com/max/1200/1*SA2-3CLvlMFvCBoioe0Y4A.png" alt=""/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As you can see from the above graph, Multithreading performs better but this is because we are currently executing I/O functions(making requests to an API). When executing CPU-heavy functions (a lot of calculations), multiprocessing will be faster.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can refer to the below article to see comparisons b/w Multiprocessing and Multithreading when dealing with CPU-heavy functions. The article also compares the performance with different values for <strong>max_workers</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://towardsdatascience.com/understanding-python-multithreading-and-multiprocessing-via-simulation-3f600dbbfe31">https://towardsdatascience.com/understanding-python-multithreading-and-multiprocessing-via-simulation-3f600dbbfe31</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->