How to Make Web Scraping Faster – Python Tutorial
Tags: python, tutorial, webdev, programming
Published: 2023-03-01 14:50:05


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v48257eywlj3krbic0jd.png)Efficient business management in today&rsquo;s competitive environment calls for high-speed public data gathering. You have to access thousands or even millions of pages and do it fast since you want to save as much time as possible to act on this data. But what might prevent you from fetching public information at speed? And how can you make scraping fast?</span></p><p class="c1"><span class="c4">Read this article where we discuss a few useful ways of making public data collection faster as well as provide sample codes that you can implement in your scraping activities. </span><span class="c8 c41 c4">


## What slows down web scraping
</span></p><p class="c1"><span class="c4">The </span><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://www.educative.io/answers/what-are-the-different-kinds-of-computing-network-delays&amp;sa=D&amp;source=editors&amp;ust=1677683290417662&amp;usg=AOvVaw3NydrNhpbp4LiYELPnlD-5">network delay</a></span><span class="c7 c4">&nbsp;is the first obvious bottleneck for any web scraping project. Transmitting a request to the web server takes time. Once the request is received, the web server will send the response, which again causes a delay. </span></p><p class="c1"><span class="c7 c4">When browsing a website, this difference is negligible as we deal with one page at a time. For instance, if sending a request and receiving the response takes a second, it will seem very fast while browsing a small number of pages. But if you are running a web scraping code that has to send requests to ten thousand pages, this will add up to almost three hours, which doesn&#39;t seem that quick anymore. </span></p><p class="c1"><span class="c7 c4">The network delay is only one of the factors that can slow down the process. Your web scraping code will not just send and receive requests but also interact with the data. At this point, the scraping can run into I/O or CPU-bound bottlenecks.</span></p><h3 class="c23 c14" id="h.35jng1rrbf4u"><span class="c36 c8 c4">I/O bound </span></h3><p class="c1"><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://www.baeldung.com/cs/cpu-io-bound&amp;sa=D&amp;source=editors&amp;ust=1677683290418894&amp;usg=AOvVaw2TbKKlTy5tqBadESWJNAPi">I/O bottleneck</a></span><span class="c7 c4">&nbsp;is an issue that relates to a system&rsquo;s input-output performance and its peripherals such as disk drives, internet interface, etc. Any program dependent on the input-output system (e.g., reading and writing data. copying files, downloading files) is an I/O bound program, and the delays are thus called I/O bound delays. </span></p><h3 class="c23 c14" id="h.e4mtc02hd8s5"><span class="c8 c4 c36">CPU bound </span></h3><p class="c1"><span class="c4">The other scenario is when a program is </span><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://www.easytechjunkie.com/what-does-it-mean-to-be-cpu-bound.htm&amp;sa=D&amp;source=editors&amp;ust=1677683290419807&amp;usg=AOvVaw0RptnRVzKXJZKSCHn1K6n3">CPU-bound</a></span><span class="c7 c4">. As the name suggests, in this case, the code execution speed depends on the CPU, which refers to the central processing unit of a computing device. A faster CPU would mean faster code execution.</span></p><p class="c1"><span class="c7 c4">A classic example of CPU-bound application is a task that requires a large number of calculations. For instance, High-Performance Computing (HPC) systems that combine the processing power of multiple processors in the CPU to deliver higher computing performance.</span></p><p class="c1"><span class="c7 c4">The distinction between I/O and CPU is essential to understand since the strategy to make the program run faster largely depends on the bottleneck type. </span></p><h2 class="c14 c39" id="h.faskdo3xomei"><span class="c8 c41 c4">How do you speed up web scraping in Python?</span></h2><p class="c1"><span class="c7 c4">There are a few possible approaches that can help increase the scraping speed:</span></p><ul class="c40 lst-kix_fanidnyh8lds-0 start"><li class="c14 c32 li-bullet-0"><span class="c2">Multiprocessing</span></li><li class="c32 c14 li-bullet-0"><span class="c2">Multithreading</span></li><li class="c32 c14 li-bullet-0"><span class="c2">Asyncio</span></li></ul><p class="c1"><span class="c7 c4">However, let&rsquo;s first take a look at an unoptimized code to make sure the difference between all is clear.</span></p><h3 class="c14 c23" id="h.u09r64d7iy8u"><span class="c36 c8 c4">Web scraping without optimization</span></h3><p class="c1"><span class="c7 c4">We will be scraping 1000 books from books.toscrape.com. This website is a dummy book store that is perfect for learning. </span></p><h3 class="c14 c18" id="h.53cmz0k4tm2s"><span class="c8 c37">Preparation</span></h3><p class="c1"><span class="c7 c4">The first step is to extract all 1000 links to the books and store them in a CSV file. Run this code file to create the links.csv file. You will need to install requests and Beautiful Soup packages for this code to work.</span></p><a id="t.cadfb325b265bbd0b242615f929aa7d57f5a3dc8"></a><a id="t.0"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c0"># prepare</span><span class="c3">.</span><span class="c0 c8">py</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;requests</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">from bs4 </span><span class="c17">import</span><span class="c0 c8">&nbsp;BeautifulSoup</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">from urllib</span><span class="c3">.</span><span class="c0">parse </span><span class="c17">import</span><span class="c0 c8">&nbsp;urljoin</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">fetch_links</span><span class="c3">(</span><span class="c0">url</span><span class="c5">=</span><span class="c10">&quot;https://books.toscrape.com/&quot;</span><span class="c3">,</span><span class="c0">&nbsp;links</span><span class="c5">=</span><span class="c3">[])</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; r </span><span class="c5">=</span><span class="c0">&nbsp;requests</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c0">url</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">r</span><span class="c3">.</span><span class="c0">url</span><span class="c3">,</span><span class="c0">&nbsp;flush</span><span class="c5">=</span><span class="c0">True</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; soup </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c9">BeautifulSoup</span><span class="c3">(</span><span class="c0">r</span><span class="c3">.</span><span class="c0">text</span><span class="c3">,</span><span class="c0">&nbsp;</span><span class="c10">&quot;html.parser&quot;</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;link </span><span class="c17">in</span><span class="c0">&nbsp;soup</span><span class="c3">.</span><span class="c9">select</span><span class="c3">(</span><span class="c10">&quot;h3 a&quot;</span><span class="c3">)</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; links</span><span class="c3">.</span><span class="c9">append</span><span class="c3">(</span><span class="c9">urljoin</span><span class="c3">(</span><span class="c0">url</span><span class="c3">,</span><span class="c0">&nbsp;link</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c10">&quot;href&quot;</span><span class="c3">)))</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; next_page </span><span class="c5">=</span><span class="c0">&nbsp;soup</span><span class="c3">.</span><span class="c9">select_one</span><span class="c3">(</span><span class="c10">&quot;li.next a&quot;</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">if</span><span class="c0">&nbsp;</span><span class="c12">next_page</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">return</span><span class="c0">&nbsp;</span><span class="c9">fetch_links</span><span class="c3">(</span><span class="c9">urljoin</span><span class="c3">(</span><span class="c0">url</span><span class="c3">,</span><span class="c0">&nbsp;next_page</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c10">&quot;href&quot;</span><span class="c3">),</span><span class="c0">&nbsp;links</span><span class="c3">))</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">else</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">return</span><span class="c0 c8">&nbsp;links</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">refresh_links</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; links </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c9">fetch_links</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">with</span><span class="c0">&nbsp;</span><span class="c9">open</span><span class="c3">(</span><span class="c10">&#39;links.csv&#39;</span><span class="c3">,</span><span class="c0">&nbsp;</span><span class="c10">&#39;w&#39;</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">f</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;link </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c12">links</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; f</span><span class="c3">.</span><span class="c9">write</span><span class="c3">(</span><span class="c0">link </span><span class="c5">+</span><span class="c0">&nbsp;</span><span class="c10">&#39;\n&#39;</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0 c8">&nbsp;</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c9">refresh_links</span><span class="c3">()</span></p></td></tr></table><p class="c1 c19"><span class="c7 c4"></span></p><p class="c1"><span class="c7 c4">The fetch_links function will retrieve all the links, and refresh_links() will store the output in a file. We skipped sending the user agent as this is a test site. However, you can do so easily using the requests library.</span></p><h3 class="c18 c14" id="h.fhyzqc3wfj8c"><span class="c37 c8">Writing unoptimized web scraper</span></h3><p class="c1"><span class="c7 c4">We will focus on optimizing 1,000 pages of web scraping in Python.</span></p><p class="c1"><span class="c7 c4">First, install the requests library using pip:</span></p><a id="t.daa1ac23ef06aba8aa819af1a8eb73cbfe4d2536"></a><a id="t.1"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c20"><span class="c8 c21 c4 c15">pip install requests</span></p></td></tr></table><p class="c1"><span class="c4">To keep things simple, we will use regular expressions to extract the title element of the page. Note the get_links functions that loads the urls we saved in the previous step.</span></p><a id="t.8329b8fc3eff786cf541e94fe7d1a6cec110772d"></a><a id="t.2"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;csv</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;re</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;time</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;requests</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">get_links</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; links </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c3">[]</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">with</span><span class="c0">&nbsp;</span><span class="c9">open</span><span class="c3">(</span><span class="c10">&quot;links.csv&quot;</span><span class="c3">,</span><span class="c0">&nbsp;</span><span class="c10">&quot;r&quot;</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">f</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; reader </span><span class="c5">=</span><span class="c0">&nbsp;csv</span><span class="c3">.</span><span class="c9">reader</span><span class="c3">(</span><span class="c0">f</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;i</span><span class="c3">,</span><span class="c0">&nbsp;row </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c9">enumerate</span><span class="c3">(</span><span class="c0">reader</span><span class="c3">)</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; links</span><span class="c3">.</span><span class="c9">append</span><span class="c3">(</span><span class="c0">row</span><span class="c3">[</span><span class="c12">0</span><span class="c3">])</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">return</span><span class="c0 c8">&nbsp;links</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">get_response</span><span class="c3">(</span><span class="c0">session</span><span class="c3">,</span><span class="c0">&nbsp;url</span><span class="c3">)</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">with</span><span class="c0">&nbsp;session</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c0">url</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">resp</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c10">&#39;.&#39;</span><span class="c3">,</span><span class="c0">&nbsp;end</span><span class="c5">=</span><span class="c10">&#39;&#39;</span><span class="c3">,</span><span class="c0">&nbsp;flush</span><span class="c5">=</span><span class="c0">True</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; text </span><span class="c5">=</span><span class="c0">&nbsp;resp</span><span class="c3">.</span><span class="c0 c8">text</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; exp </span><span class="c5">=</span><span class="c0">&nbsp;r</span><span class="c10">&#39;(&lt;title&gt;).*(&lt;\/title&gt;)&#39;</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">return</span><span class="c0">&nbsp;re</span><span class="c3">.</span><span class="c9">search</span><span class="c3">(</span><span class="c0">exp</span><span class="c3">,</span><span class="c0">&nbsp;text</span><span class="c3">,</span><span class="c0">flags</span><span class="c5">=</span><span class="c0">re</span><span class="c3">.</span><span class="c12">DOTALL</span><span class="c3">).</span><span class="c9">group</span><span class="c3">(</span><span class="c12">0</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">main</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; start_time </span><span class="c5">=</span><span class="c0">&nbsp;time</span><span class="c3">.</span><span class="c9">time</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">with</span><span class="c0">&nbsp;requests</span><span class="c3">.</span><span class="c9">Session</span><span class="c3">()</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">session</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; results </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c3">[]</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;url </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c9">get_links</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; result </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c9">get_response</span><span class="c3">(</span><span class="c0">session</span><span class="c3">,</span><span class="c0">&nbsp;url</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">result</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">f</span><span class="c10">&quot;{(time.time() - start_time):.2f} seconds&quot;</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0 c8">&nbsp; &nbsp;</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0 c8">&nbsp; &nbsp; </span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c9">main</span><span class="c3">()</span></p><p class="c11 c19"><span class="c2 c14"></span></p></td></tr></table><p class="c6"><span class="c2 c14"></span></p><p class="c1"><span class="c7 c4">The code without optimization code took 288.62 seconds.</span></p><h3 class="c23 c14" id="h.2fkce8mnxo3k"><span class="c36 c8 c4">Web scraping using multiprocessing</span></h3><p class="c1"><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://www.techtarget.com/searchdatacenter/definition/multiprocessing&amp;sa=D&amp;source=editors&amp;ust=1677683290452880&amp;usg=AOvVaw10gNxCKZtSjvbNDypw_7jp">Multiprocessing</a></span><span class="c7 c4">, as the name suggests, is utilizing more than one processor. Most modern computers have more than one CPU core, if not multiple CPUs. Using the multiprocessing module, included with the Python standard library, we can write code that uses all these cores. </span></p><p class="c1"><span class="c7 c4">For example, if we have an 8-core CPU, we can essentially write code that can split the task into eight different processes where each process runs in a separate CPU core.</span></p><p class="c1"><span class="c7 c4">Note that this approach is more suitable when the bottleneck is CPU or when the code is CPU-Bound. We will still see some improvements in our case, though. </span></p><p class="c1"><span class="c4">The first step is to import </span><span class="c29">Pool</span><span class="c4">&nbsp;and </span><span class="c29">cpu_count</span><span class="c7 c4">&nbsp;from the multiprocessing module</span></p><a id="t.c54ad8b09cb257d24d44daf0b0c7d1dbc1ec5579"></a><a id="t.3"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c20"><span class="c4 c15 c38">from multiprocessing </span><span class="c15 c30">import</span><span class="c8 c4 c15 c21">&nbsp;Pool</span></p></td></tr></table><p class="c1"><span class="c7 c4">The other change is required in both get_response and &nbsp;main functions. </span></p><a id="t.4d02b254c92fa1f5dc966cbb5720473265c73bbb"></a><a id="t.4"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c0">def </span><span class="c9">get_response</span><span class="c3">(</span><span class="c0">url</span><span class="c3">)</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; resp </span><span class="c5">=</span><span class="c0">&nbsp;requests</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c0">url</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c10">&#39;.&#39;</span><span class="c3">,</span><span class="c0">&nbsp;end</span><span class="c5">=</span><span class="c10">&#39;&#39;</span><span class="c3">,</span><span class="c0">&nbsp;flush</span><span class="c5">=</span><span class="c0">True</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; text </span><span class="c5">=</span><span class="c0">&nbsp;resp</span><span class="c3">.</span><span class="c0 c8">text</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; exp </span><span class="c5">=</span><span class="c0">&nbsp;r</span><span class="c10">&#39;(&lt;title&gt;).*(&lt;\/title&gt;)&#39;</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">return</span><span class="c0">&nbsp;re</span><span class="c3">.</span><span class="c9">search</span><span class="c3">(</span><span class="c0">exp</span><span class="c3">,</span><span class="c0">&nbsp;text</span><span class="c3">,</span><span class="c0">&nbsp;flags</span><span class="c5">=</span><span class="c0">re</span><span class="c3">.</span><span class="c12">DOTALL</span><span class="c3">).</span><span class="c9">group</span><span class="c3">(</span><span class="c12">0</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">def </span><span class="c9">main</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; start_time </span><span class="c5">=</span><span class="c0">&nbsp;time</span><span class="c3">.</span><span class="c9">time</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; links </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c9">get_links</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">with</span><span class="c0">&nbsp;</span><span class="c9">Pool</span><span class="c3">(</span><span class="c12">100</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">p</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; results </span><span class="c5">=</span><span class="c0">&nbsp;p</span><span class="c3">.</span><span class="c9">map</span><span class="c3">(</span><span class="c0">get_response</span><span class="c3">,</span><span class="c0">&nbsp;links</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;result </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c12">results</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">result</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">f</span><span class="c10">&quot;{(time.time() - start_time):.2f} seconds&quot;</span><span class="c3">)</span></p></td></tr></table><p class="c1"><span class="c7 c4">The most critical line of the code is where we create a Pool. Note that we are using cpu_count() function to get the count of CPU cores dynamically. This ensures that this code runs on every machine without any change. </span></p><p class="c1"><span class="c7 c4">In our example, the execution time came down to about 142 seconds from 288 seconds on a machine with eight cores. This, as expected, is not a vast improvement. Remember that multiprocessing is suitable when the code is CPU-Bound. Our code is I/O bound; thus we don&rsquo;t see much improvement.</span></p><h3 class="c23 c14" id="h.9y5k7cnltx41"><span class="c36 c8 c4">Web scraping using multithreading</span></h3><p class="c1"><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://www.educative.io/edpresso/what-is-multithreading-in-python&amp;sa=D&amp;source=editors&amp;ust=1677683290464495&amp;usg=AOvVaw0xWhILot_ErnqW9mtNyHcY">Multithreading</a></span><span class="c7 c4">&nbsp;is a great option to optimize web scraping code. A thread is essentially a separate flow of execution. Operating systems typically spawn hundreds of threads and switch the CPU time among these. The switching is so fast that we get the illusion of multitasking. The CPU controls this switching, and it cannot be customized. </span></p><p class="c1"><span class="c4">Using the </span><span class="c29">concurrent.futures</span><span class="c7 c4">&nbsp;module of Python, we can customize how many threads we create to optimize our code. There is only one huge caveat: managing threads can become messy and error-prone as the code becomes more complex. </span></p><p class="c1"><span class="c7 c4">To change our code to utilize multithreading, minimal changes are needed.</span></p><p class="c1"><span class="c4">First, import </span><span class="c29">ThreadPoolExecutor</span><span class="c7 c4">.</span></p><a id="t.88f87462f0a92f1bdbcf4a938d27109e04018fd4"></a><a id="t.5"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c20"><span class="c8 c21 c25 c28">from concurrent.futures import ThreadPoolExecutor</span></p></td></tr></table><p class="c1"><span class="c4">Next, instead of creating a Pool , create a </span><span class="c29">ThreadPoolExecutor</span><span class="c7 c4">:</span></p><a id="t.8ed9e1ce5ce5f4850966b587f3595c17689e7ec5"></a><a id="t.6"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c17">with</span><span class="c0">&nbsp;</span><span class="c9">ThreadPoolExecutor</span><span class="c3">(</span><span class="c0">max_workers</span><span class="c5">=</span><span class="c12">100</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">p</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; results </span><span class="c5">=</span><span class="c0">&nbsp;p</span><span class="c3">.</span><span class="c9">map</span><span class="c3">(</span><span class="c0">get_response</span><span class="c3">,</span><span class="c0">&nbsp;links</span><span class="c3">)</span></p></td></tr></table><p class="c1 c19"><span class="c7 c4"></span></p><p class="c1"><span class="c7 c4">Note that you have to specify max workers. This number will depend on the complexity of the code. A too high number may harm your code as the overload of creating the threads may be too much.</span></p><p class="c1"><span class="c7 c4">For this code, the code execution was complete in 12.10 seconds.</span></p><p class="c1"><span class="c7 c4">For reference, the unoptimized code took 288 seconds. This is a massive improvement. </span></p><h3 class="c23 c14" id="h.bo2esswcfcgv"><span class="c36 c8 c4">Asyncio for asynchronous programming</span></h3><p class="c1"><span class="c27"><a class="c22" href="https://www.google.com/url?q=https://hackernoon.com/a-simple-introduction-to-pythons-asyncio-595d9c9ecf8c&amp;sa=D&amp;source=editors&amp;ust=1677683290468934&amp;usg=AOvVaw2XihCrfrYzlYe7IuRWeYsF">Asynchronous coding</a></span><span class="c7 c4">&nbsp;using the asyncio module is essentially threading where the code controls the context switching. It also makes coding more effortless and less error-prone. Specifically, for web scraping projects, this is the most suitable approach.</span></p><p class="c1"><span class="c7 c4">This approach requires quite a lot of changes. First, the requests library will not work. Instead, we will use the aiohttp library for web scraping in Python. This requires a separate installation:</span></p><a id="t.7bb61dfd17507b066ea0e3e28c49a998eda56dc0"></a><a id="t.7"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c20"><span class="c4 c15">python3 </span><span class="c15 c24">-</span><span class="c7 c4 c15">m pip install aiohttp</span></p></td></tr></table><p class="c1"><span class="c4">Next, import </span><span class="c29">asyncio</span><span class="c4">&nbsp;and </span><span class="c29">aiohttp</span><span class="c7 c4">&nbsp;modules.</span></p><a id="t.b62d8567bf0ff6abeb38b2e106c0c7b3984740d9"></a><a id="t.8"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c17">import</span><span class="c0 c8">&nbsp;aiohttp </span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c17">import</span><span class="c0">&nbsp;asyncio</span></p></td></tr></table><p class="c1"><span class="c7 c4">The get_response() function now needs to change to a coroutine. Also, we will be using the same session for every execution. Optionally, you can send the user agent if needed.</span></p><p class="c1"><span class="c7 c4">Note the use of async and await keywords.</span></p><a id="t.77a7a7e5eddd23971365a6196b24acb6e8383110"></a><a id="t.9"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c17">async</span><span class="c0">&nbsp;def </span><span class="c9">get_response</span><span class="c3">(</span><span class="c0">session</span><span class="c3">,</span><span class="c0">&nbsp;url</span><span class="c3">)</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">async</span><span class="c0">&nbsp;</span><span class="c17">with</span><span class="c0">&nbsp;session</span><span class="c3">.</span><span class="c9">get</span><span class="c3">(</span><span class="c0">url</span><span class="c3">)</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">resp</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; text </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c17">await</span><span class="c0">&nbsp;resp</span><span class="c3">.</span><span class="c9">text</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; exp </span><span class="c5">=</span><span class="c0">&nbsp;r</span><span class="c10">&#39;(&lt;title&gt;).*(&lt;\/title&gt;)&#39;</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">return</span><span class="c0">&nbsp;re</span><span class="c3">.</span><span class="c9">search</span><span class="c3">(</span><span class="c0">exp</span><span class="c3">,</span><span class="c0">&nbsp;text</span><span class="c3">,</span><span class="c0">flags</span><span class="c5">=</span><span class="c0">re</span><span class="c3">.</span><span class="c12">DOTALL</span><span class="c3">).</span><span class="c9">group</span><span class="c3">(</span><span class="c12">0</span><span class="c3">)</span></p></td></tr></table><p class="c1"><span class="c7 c4">The most significant changes are in the main() function.</span></p><p class="c1"><span class="c4">First, it needs to change to a coroutine. Next, we will use </span><span class="c29">aiohttp.ClientSession</span><span class="c4">&nbsp;to create the session object. Most importantly, we will need to create tasks for all the links. Finally, all the tasks will be sent to an event loop using the </span><span class="c29">asyncio.gather</span><span class="c7 c4">&nbsp;method.</span></p><a id="t.4f6c56e25375561e2597db041fdc992b378cb95f"></a><a id="t.10"></a><table class="c13"><tr class="c26"><td class="c34" colspan="1" rowspan="1"><p class="c11"><span class="c17">async</span><span class="c0">&nbsp;def </span><span class="c9">main</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; start_time </span><span class="c5">=</span><span class="c0">&nbsp;time</span><span class="c3">.</span><span class="c9">time</span><span class="c3">()</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c17">async</span><span class="c0">&nbsp;</span><span class="c17">with</span><span class="c0">&nbsp;aiohttp</span><span class="c3">.</span><span class="c9">ClientSession</span><span class="c3">()</span><span class="c0">&nbsp;</span><span class="c17">as</span><span class="c0">&nbsp;</span><span class="c12">session</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; tasks </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c3">[]</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;url </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c9">get_links</span><span class="c3">()</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tasks</span><span class="c3">.</span><span class="c9">append</span><span class="c3">(</span><span class="c0">asyncio</span><span class="c3">.</span><span class="c9">create_task</span><span class="c3">(</span><span class="c9">get_response</span><span class="c3">(</span><span class="c0">session</span><span class="c3">,</span><span class="c0">&nbsp;url</span><span class="c3">)))</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; results </span><span class="c5">=</span><span class="c0">&nbsp;</span><span class="c17">await</span><span class="c0">&nbsp;asyncio</span><span class="c3">.</span><span class="c9">gather</span><span class="c3">(</span><span class="c5">*</span><span class="c0">tasks</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c17">for</span><span class="c0">&nbsp;result </span><span class="c17">in</span><span class="c0">&nbsp;</span><span class="c12">results</span><span class="c5">:</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">result</span><span class="c3">)</span></p><p class="c11 c19"><span class="c0 c8"></span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c9">print</span><span class="c3">(</span><span class="c0">f</span><span class="c10">&quot;{(time.time() - start_time):.2f} seconds&quot;</span><span class="c3">)</span></p></td></tr></table><p class="c1"><span class="c4">Lastly, to run the main() coroutine, we would need to use </span><span class="c7 c29">asyncio.run(main())</span></p><p class="c1"><span class="c7 c4">This execution took 9.43 seconds. </span></p><p class="c1"><span class="c7 c4">As you can see, the asyncio approach was the fastest. This, however, requires an entirely new way of thinking. If you have experience with async-await in any programming language, you will find it familiar.</span></p><h2 class="c39 c14" id="h.410dsi6yjrcp"><span class="c8 c4 c41">Conclusion</span></h2><p class="c1"><span class="c7 c4">In an attempt to gather large amounts of public data, it is not rare for businesses to encounter the problem of slow web scraping. They spend long hours collecting the public information they need, thus losing an opportunity to analyze it and make informed decisions ahead of competitors in the market.</span></p><p class="c1"><span class="c7 c4">This article aimed to explain what contributes to the decreased speed of scraping activities and provide several useful ways to deal with this issue. We looked at such web scraping approaches as multiprocessing, multithreading, and asyncio and compared their execution time so that you could choose the most suitable approach for your specific use case.</span></p><p class="c6"><span class="c7 c4"></span></p><p class="c1 c19"><span class="c7 c4"></span></p><p class="c6"><span class="c2 c14"></span></p><p class="c1 c19"><span class="c4 c7"></span></p><p class="c6"><span class="c2 c14"></span></p><p class="c1 c19"><span class="c7 c4"></span></p><p class="c1 c19"><span class="c7 c4"></span></p><p class="c33 c19"><span class="c7 c25"></span></p></body></html>