Multiprocessing in Python
Published: 2024-03-27 02:51:51

<p>By default, instructions in a program are executed sequentially, this means that one task has to be completed for another to start. In more familiar terms it means that, for example if you have two functions defined in a program, the first function to be called will be executed until it is complete before the other one starts.&nbsp;</p>

<p>The sequential mode of execution,&nbsp;may not be admirable in some cases,&nbsp;for example,&nbsp;some tasks may take too long to complete thus blocking the execution of other important tasks. We may also want some tasks to run in the background without interfering with the execution of other tasks. This is where concurrency programming comes in handy.</p>

<p><a href="https://www.pynerds.com/introduction-to-concurrent-programming-in-python/">Concurrent</a>&nbsp;programming aims at managing multiple tasks in a program simultaneously. This means that some instructions can be run in parallel rather than sequentially. There are various ways concurrency is achieved in Python, one of this ways is through <code>multiprocessing</code>.</p>

<p>Multiprocessing is normally used with an aim of improving the performance of a program by leveraging multiple processors or cores available in a computer. The primary goal is to parallelize the execution of tasks, allowing them to be executed simultaneously, this generally leads to faster overall execution time.</p>

<p>Before we look into more details on how multiprocessing works, we need to first understand what a&nbsp;process is.</p>

<h3>Processes in Python</h3>

<p>Literally speaking a process refers to an instance of a program that is running. Thus when you run a program you are effectively starting a process.&nbsp;</p>

<p>The operating system manages how processes are created and how they run in the background. Multiple processes can be be running simultaneously but each process is allocated its own chunk of memory where it stores and manipulates its data. A process cannot access or write into the space allocated for another process.&nbsp;</p>

<p>In Python, processes are started by invoking&nbsp;instances of the <code>Process </code>class which is defined in the <code>multiprocessing</code> module of the standard library. This means that when you run a Python program you are basically running an instance of the<code>&nbsp;multiprocessing.Process()&nbsp;</code>class, or to put it better,&nbsp; this is what happens under the hood.</p>

<p>Every program in python starts with a single process which is called the <strong>main process</strong>. We can then create other processes inside that single process, the new processes are referred to as <strong>child processes</strong>.</p>

<p>Multiprocessing happens when we start other instances of<code> multiprosessing.Process()&nbsp;</code>class inside the main process/program, this leads to multiple processes running in parallel alongside the main process.&nbsp;</p>

<h2>Creating&nbsp;and running processes</h2>

<p>To create and run a process, follow the steps shown below:</p>

<div>
<ol>
	<li>Begin by importing the <code>Process</code> class from the <code>multiprocessing </code>module.</li>
	<li>Create an instance of the&nbsp;&nbsp;<code>Process() </code>class, with the necessary arguments.</li>
	<li>Start the created process by calling its&nbsp;<code>start()</code> method.</li>
	<li>Make the main process&nbsp;pause and wait&nbsp;until the started process&nbsp;is done executing by calling the <code>join()</code> method of the created process.</li>
</ol>
</div>

<pre>
<code class="language-python">#import the Process class
from multiprocessing import Process

def squares(num):
art the process
    p.start()
    for i in range(num):
        print(i ** 2)

if __name__ == "__main__":
    #create a process
    p = Process(target = squares, args = (5, ) )
    
    #st
    #the main process waits for this process
    p.join() </code></pre>

<div class="output"><p>Output: </p>
<p>0<br />
1<br />
4<br />
9<br />
16</p>
</div>

<p>In the above example we have created&nbsp;a process, <code>p</code> which we have used to&nbsp;run the<code>&nbsp;squares()</code> function. Let us now look deeper into the <code>Process </code>objects and how they work.</p>

<p>The <code>Process </code>class has the following syntax:</p>

<div class="syntax">
<pre>
<code class="language-python">Process(target = None, name = None, args = None, kwargs = None, daemon = None )</code></pre>
</div>

<div class="parameters">
<table border="1" cellpadding="1" cellspacing="1" class="table table-bordered table-light" style="width:100%">
	<tbody>
		<tr>
			<td><code>target</code></td>
			<td>The function to be run when the process&nbsp;is started.</td>
		</tr>
		<tr>
			<td><code>name</code></td>
			<td>The name to identify the process. Multiple processes may share a common name.&nbsp;</td>
		</tr>
		<tr>
			<td><code>args</code></td>
			<td>A tuple with positional arguments that will be passed to the target function when it is called.</td>
		</tr>
		<tr>
			<td><code>kwargs</code></td>
			<td>A dictionary with keyword arguments that will be passed to the target function when it is called.</td>
		</tr>
		<tr>
			<td><code>daemon</code></td>
			<td>If set to <code>True</code>, the process will run in the background.</td>
		</tr>
	</tbody>
</table>
</div>

<h4>Multiprocessing example</h4>

<p>Our previous example was not really an example of multiprocessing this is because only one process was running at a given time. In the following example, we create two processes to demonstrates how multiprocessing works.</p>

<div class="example">
<pre>
<code class="language-python">#import the Process class
from multiprocessing import Process
import time

#The tasks to be executed
def squares(n):
    for i in range(n):
        print(f"square({i}) = {i ** 2}")
        time.sleep(0.1)

def cubes(n):
    for i in range(n):
         print(f"cube({i}) = {i ** 3}")
         time.sleep(0.1)

if __name__ == "__main__":
     #Create the processes
     psquare = Process(target = squares, name = "psquare", args = (5, ) )
     pcube = Process(target = cubes, name = "pcubes", args = (5, ) )
     
     #start the processes
     psquare.start()
     pcube.start()
     
     # the main process waits for this processes
     psquare.join()
     pcube.join()</code></pre>
</div>

<div class="output"><p>Output: </p>
<p>square(0) = 0<br />
cube(0) = 0<br />
square(1) = 1<br />
cube(1) = 1<br />
square(2) = 4<br />
cube(2) = 8<br />
square(3) = 9<br />
cube(3) = 27<br />
square(4) = 16<br />
cube(4) = 64</p>
</div>

<p>If you observe the above output, you can see that output from the two functions <code>squares() </code>and <code>cubes() </code>are been emitted alternately, this is because the two functions are actually been run in parallel rather than one after the other.</p>

<h4>Daemon Processes</h4>

<p>A daemon process is one that runs in the background. These&nbsp;processes can be used to perform background tasks.</p>

<p>We create a daemon process by setting the <code>daemon </code>argument as <code>True </code>when creating the process.</p>

<div class="example">
<pre>
<code class="language-python">#import the Process class
from multiprocessing import Process

import time

def task():
    for i in range(20):
         print(i)
         time.sleep(0.1)

#a background task that runs at regular intervals
def background_task():
     for i in range(0, 21, 5):
        print("daemon task")
        time.sleep(0.5)

if __name__ == "__main__":
     #regular Process
     P1 = Process(target = task)
     #daemon Process
     P2 = Process(target = background_task, daemon = True)

     #start the Processs
     P1.start()
     P2.start()
   
     P1.join()
     P2.join()</code></pre>
</div>

<div class="output"><p>Output: </p>
<p>daemon task<br />
0<br />
1<br />
2<br />
3<br />
4<br />
daemon task<br />
5<br />
6<br />
7<br />
8<br />
9<br />
daemon task<br />
10<br />
11<br />
12<br />
13<br />
14<br />
daemon task<br />
15<br />
16<br />
17<br />
18<br />
19<br />
daemon task</p>
</div>

<h4>Terminating a&nbsp;process</h4>

<p>Sometimes we may want to terminate a process prematurely. We can achieve this using either <code>kill() </code>or <code>terminate()</code> methods. This function should be used with caution and when absolutely necessary.&nbsp;This is because they makes the process to terminate abruptly without performing the necessary clean-up, thus open resources such as files and sockets will not be closed.</p>

<div class="syntax">
<pre>
<code class="language-python">#terminate
p.terminate()

#kill
p.kill()</code></pre>
</div>

<h4>States of a process.</h4>

<p>A process can be in four states:</p>

<div>
<ol>
	<li><strong>New </strong>- The process has just been instantiated but not yet started.</li>
	<li><strong>Running </strong>- The process is been executed</li>
	<li><strong>Paused </strong>- The process has been paused for some reason, for example it may be waiting for another process to complete its execution.</li>
	<li><strong>Dead </strong>- The process&nbsp;is not runnable. It may have reached this state naturally after completing its task or it may have been terminated using&nbsp;<code>terminate()</code> or killed using <code>kill()</code>.&nbsp;&nbsp;</li>
</ol>
</div>

<p>Processes&nbsp;contains the <code>is_alive()</code> method which can be used to tell whether a process is currently running.&nbsp; The function returns <code>True </code>if the process is in the <strong>running </strong>state and <code>False </code>otherwise.</p>

<div class="syntax">
<pre>
<code class="language-python">p.is_alive()</code></pre>
</div>

<h2>Multprocessing vs multithreading</h2>

<p>Multithreading is another way that we can achieve concurrency in Python.</p>

<p>At first glance, <a href="https://www.pynerds.com/multithreading-in-python/">multithreading</a>&nbsp;looks very similar to multiprocessing,&nbsp; their outer interface and usage looks almost identical, however, the internal workings of the two approaches is very&nbsp;different. You can check on <a href="https://www.pynerds.com/multithreading-in-python/">this</a>&nbsp;link to see more on threads and how multithreading works.</p>

<p>In a nutshell, threads can be thought of as smaller units of&nbsp;a process, this is why they are also referred to as <strong>sub-processes</strong>. A single process can contain multiple threads running simultaneously.&nbsp;All threads in a single process share same resources and data,&nbsp;this generally makes communication between&nbsp;multiple threads quicker.</p>

<p>Multithreading offers a&nbsp;more lightweight alternative approach&nbsp;to&nbsp;multiprocessing. This is partly because threads, unlike processes, share resources and thus communication between two threads is easier. Processes, on the other hand, do not share resources and data, this means that for multiprocessing to work, inter-process communication has to be established.</p>

<p>However, multiple threads sharing&nbsp;resources also comes with its own limitations&nbsp;because&nbsp; a mechanism has to be established to ensure data safety, or otherwise, conflicts may arise like&nbsp;one thread deleting a resource that is been used by another thread .</p>

<p>What Python does to prevent conflicts between threads is ensuring that only a single thread&nbsp;at a time can access or modify shared resources, this is achieved through a mechanism known as the Global Interpreter Lock (GIL). This literally means that true parallelism cannot be achieved with threading like with multiprocessing. Concurrency in multithreading is achieved by switching very quickly between the threads concurrently.&nbsp;The switching is fast enough to&nbsp;prevent most noticeable delays for the end user.</p>

<div class="related"><h3>Related articles</h3><hr><ul><li><p><a class="text-success" href="https://www.pynerds.com/what-is-python/">What is Python?</a>

<li><p><a class="text-success" href="https://www.pynerds.com/the-python-interpreter/">The Python Interpreter</a>

<li><p><a class="text-success" href="https://www.pynerds.com/introduction-to-object-oriented-programming/">Introduction to Object Oriented Programming</a><li><p><a class="text-success" href="https://www.pynerds.com/introduction-to-concurrent-programming-in-python/">Introduction to concurrent programming in Python</a>

<li><p><a class="text-success" href="https://www.pynerds.com/multithreading-in-python/">Multithreading in Python</a><li><p><a class="text-success" href="https://www.pynerds.com/understand-gil-in-python-the-global-interpreter-lock/">Understand GIL in Python - The Global Interpreter Lock</a><li><p><a class="text-success" href="https://www.pynerds.com/subprocesses-in-python/">Subprocesses in Python</a><li><p><a class="text-success" href="https://www.pynerds.com/asyncawait-in-python/">async/await in Python</a><li><p><a class="text-success" href="https://www.pynerds.com/how-to-use-asyncio-in-python/">How to use asyncio in Python</a></ul></div>