Web Scraping With RegEx
Tags: webdev, tutorial, devops, regex
Published: 2023-03-10 11:57:58


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sgys1dw307lqknismhen.png)The demand for digital content has increased exponentially. Since the resulting competition increases, the existing websites are rapidly changing. Quick updates are beneficial to general consumers. However, it&rsquo;s a considerable hassle for businesses that perform public data collection since web scraping uses routines tailored for specific conditions of websites, and frequent updates tend to disrupt them. </span></p><p class="c5"><span class="c8">Let&rsquo;s see how RegEx comes into play here by alleviating some of the more complex elements of certain acquisition and parsing processes.</span></p><h2 class="c34" id="h.twtk5l38bfur"><span class="c37">What is RegEx?</span></h2><p class="c5"><span class="c8">RegEx stands for Regular Expressions, a method to match specific patterns depending on the provided combinations, which can be used as filters to get the desired output.</span></p><h2 class="c34" id="h.d49lbuogx6t6"><span class="c37">How to use RegEx for web scraping?</span></h2><p class="c5"><span class="c8">RegEx can be used to validate all types of character combinations, including special characters like line breaks. One of the biggest pros Regular Expressions have is that no matter the type of data/input (irrespective of its size), it&rsquo;s always compared to the same single regular expression, making the code more efficient.</span></p><p class="c5"><span class="c20">Regular Expressions are universal and can be implemented in any programming language.</span></p><h2 class="c34" id="h.45sbb67kvzf9"><span class="c39">Overview of RegEx tokens</span></h2><a id="t.9e333ffdc5e9d3829c43b968f39216a610f2ec0b"></a><a id="t.0"></a><table class="c45"><tr class="c12"><td class="c46" colspan="1" rowspan="1"><p class="c0"><span class="c3">Token</span></p></td><td class="c49" colspan="1" rowspan="1"><p class="c0"><span class="c3">Matches</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">^</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Start of a string</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">$</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">End of a string</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">.</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Any character (except \n)</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">|</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Characters on either side of the symbol</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Escapes special characters</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">Char</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">The character given</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">*</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Any number of previous characters</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">?</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">1 previous character</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">+</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">1 or more previous characters</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">{Digit}</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Exact number</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">{Digit-Digit)</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Between range</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\d</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Any digit</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\s</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Any whitespace character</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\w</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Any word character</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\b</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Word boundary character</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\D</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Inverse of \d</span></p></td></tr><tr class="c12"><td class="c23" colspan="1" rowspan="1"><p class="c0"><span class="c8">\S</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Inverse of \s</span></p></td></tr><tr class="c12"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c8">\W</span></p></td><td class="c27" colspan="1" rowspan="1"><p class="c0"><span class="c8">Inverse of \w</span></p></td></tr></table><p class="c0 c10"><span class="c26"></span></p><h2 class="c34" id="h.ymvpt78a69n3"><span class="c37">Collecting data using RegEx</span></h2><p class="c5"><span class="c20">In our tutorial, the RegEx scraping target is book titles and prices from </span><span class="c24"><a class="c30" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1678446608977498&amp;usg=AOvVaw29NPvsw8gSFuOramE_xD6F">a dummy website</a></span><span class="c8">&nbsp;intended for training purposes.</span></p><h3 class="c25" id="h.wbocgg19fsv6"><span class="c33">Project requirements:</span></h3><ul class="c38 lst-kix_jt1w3g21hc1n-0 start"><li class="c14 li-bullet-0"><span class="c20">The latest version of </span><span class="c24"><a class="c30" href="https://www.google.com/url?q=https://www.python.org/downloads/&amp;sa=D&amp;source=editors&amp;ust=1678446608978153&amp;usg=AOvVaw0iASF7SxP7Sl7odPg2bEOF">Python</a></span><span class="c8">.</span></li><li class="c14 li-bullet-0"><span class="c24"><a class="c30" href="https://www.google.com/url?q=https://pypi.org/project/beautifulsoup4/&amp;sa=D&amp;source=editors&amp;ust=1678446608978460&amp;usg=AOvVaw2gCpXAcVtaYw84fhmBf1kU">Beautiful Soup 4</a></span><span class="c8">&nbsp;library to parse HTML.</span></li><li class="c14 li-bullet-0"><span class="c24"><a class="c30" href="https://www.google.com/url?q=https://pypi.org/project/requests/&amp;sa=D&amp;source=editors&amp;ust=1678446608978789&amp;usg=AOvVaw34Sl93_Lovx6SvANtqeGiW">Requests</a></span><span class="c8">&nbsp;library to make HTML requests.</span></li></ul><h3 class="c25" id="h.x99b6s70vcsu"><span class="c33">Importing libraries</span></h3><p class="c5"><span class="c8">Let&rsquo;s begin with creating a virtual environment for the project:</span></p><a id="t.60b5693d9fb21a3c657bc194607d816e8450fb96"></a><a id="t.1"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4">python3 </span><span class="c15">-</span><span class="c4">m venv scrapingdemo</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><p class="c5"><span class="c20">Activate the newly created virtual environment (the example for Linux):</span></p><a id="t.ae49d05e13cf4742b8858f36c0ddd508997d5dd5"></a><a id="t.2"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4">source </span><span class="c18">.</span><span class="c15">/</span><span class="c4">scrapingdemo</span><span class="c15">/</span><span class="c4">bin</span><span class="c15">/</span><span class="c22 c20 c29">activate</span></p></td></tr></table><p class="c5 c10"><span class="c20 c29 c48"></span></p><p class="c5"><span class="c8">Now, install the required Python modules.</span></p><p class="c5"><span class="c20 c39">Requests</span><span class="c8">&nbsp;is a library responsible for sending requests to the websites on the internet and returning their response. To install Requests, enter the following:</span></p><a id="t.daa1ac23ef06aba8aa819af1a8eb73cbfe4d2536"></a><a id="t.3"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4">pip install requests</span></p></td></tr></table><p class="c5 c10"><span class="c48 c20 c29"></span></p><p class="c5"><span class="c39 c20">Beautiful Soup</span><span class="c8">&nbsp;is a module used to parse and extract data from the HTML response. To install Beautiful Soup, enter the following:</span></p><a id="t.49366485c230cd2d9f2062890cbbc66aa86d2634"></a><a id="t.4"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4 c16">pip install beautifulsoup4</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><p class="c5"><span class="c39 c20">re</span><span class="c8">&nbsp;is a built-in Python module responsible for working with Regular Expressions.</span></p><p class="c5"><span class="c20">Next, create an empty Python file, for example, </span><span class="c2">demo.py</span><span class="c8">.</span></p><p class="c5"><span class="c8">To import the required libraries, enter the following:</span></p><a id="t.4f4fe01487ccc8259de06dedad3e9d728246a281"></a><a id="t.5"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c11 c17">import</span><span class="c1">&nbsp;requests</span></p><p class="c0"><span class="c4 c16">from bs4 </span><span class="c11 c17">import</span><span class="c1">&nbsp;BeautifulSoup </span></p><p class="c0"><span class="c11 c17">import</span><span class="c4 c16">&nbsp;re</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h3 class="c25" id="h.d20uw3cehb97"><span class="c33">Sending the GET request</span></h3><p class="c5"><span class="c20">Use the Requests library to send a request to a web page from which you want to scrape the data. In this case, </span><span class="c24"><a class="c30" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1678446608984124&amp;usg=AOvVaw0zkGGiFJhz8psLciPXAuYY">https://books.toscrape.com/</a></span><span class="c8">. To commence, enter the following:</span></p><a id="t.e174e758f0309ecf2c117099b4b2b226c225ac0f"></a><a id="t.6"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4 c16">page </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;requests</span><span class="c7">.</span><span class="c11 c2">get</span><span class="c7">(</span><span class="c9">&#39;https://books.toscrape.com/&#39;</span><span class="c7">)</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h3 class="c25" id="h.ombyub1t1vi4"><span class="c33">Selecting data</span></h3><p class="c0"><span class="c20">First, create a Beautiful Soup object and pass the page content received from your request during the initialization, including the parser type. As you&rsquo;re working with an HTML code, select </span><span class="c2">HTML.parser</span><span class="c8">&nbsp;as the parser type.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fi47gihu2j3mcigrf3pu.png)</span></p><p class="c0 c10"><span class="c8"></span></p><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 304.00px;"><img alt="" src="images/image2.png" style="width: 624.00px; height: 304.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c43"><span class="c26">Inspecting the HTML code element</span></p><p class="c0 c10"><span class="c26"></span></p><p class="c5"><span class="c20">By inspecting the elements (right-click and select </span><span class="c39 c20">inspect element</span><span class="c20">) in a browser, you can see that each book title and price are presented inside an </span><span class="c2">article</span><span class="c20">&nbsp;element with the class called </span><span class="c2">product_pod</span><span class="c8">. Use Beautiful Soup to get all the data inside these elements, and then convert it to a string:</span></p><a id="t.6710b54ef22e8dff9d9ef7e0eb436780c0901871"></a><a id="t.7"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4">soup </span><span class="c15">=</span><span class="c4">&nbsp;</span><span class="c31 c2">BeautifulSoup</span><span class="c18">(</span><span class="c4">page</span><span class="c18">.</span><span class="c4">content</span><span class="c18">,</span><span class="c4">&nbsp;</span><span class="c31 c36 c29">&#39;html.parser&#39;</span><span class="c18">)</span></p><p class="c0"><span class="c4">content </span><span class="c15">=</span><span class="c4">&nbsp;soup</span><span class="c18">.</span><span class="c2 c31">find_all</span><span class="c18">(</span><span class="c4">class_</span><span class="c15">=</span><span class="c31 c36 c29">&#39;product_pod&#39;</span><span class="c18">)</span></p><p class="c0"><span class="c4">content </span><span class="c15">=</span><span class="c4">&nbsp;</span><span class="c31 c2">str</span><span class="c18">(</span><span class="c4">content</span><span class="c18">)</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h3 class="c25" id="h.ryy8z3ehwya9"><span class="c33">Processing the data using RegEx</span></h3><p class="c0"><span class="c8">Since the acquired content has a lot of unnecessary data, create two regular expressions to get only the desired data.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6kq8cpt2wocuqz7kpmba.png)</span></p><p class="c0 c10"><span class="c8"></span></p><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 218.67px;"><img alt="" src="images/image3.png" style="width: 624.00px; height: 218.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c43"><span class="c26">Content of the acquired data</span></p><p class="c43 c10"><span class="c26"></span></p><h4 class="c19" id="h.3yumjetebxnx"><span class="c32 c16">Expression # 1</span></h4><p class="c5"><span class="c3">Finding the pattern</span></p><p class="c5"><span class="c20">First, inspect the title of the book to find the pattern. You can see above that every title is present after the text </span><span class="c2">title=</span><span class="c20">&nbsp;in the format </span><span class="c2">title=&ldquo;</span><span class="c2">Titlename</span><span class="c2">&rdquo;</span><span class="c8">.</span></p><p class="c5"><span class="c3">Generating the expression</span></p><p class="c5"><span class="c20">Then, create an expression that returns the data inside quotations after the </span><span class="c2">title=</span><span class="c20">&nbsp;by specifying </span><span class="c2">&quot;(.*?)&quot;</span><span class="c8">.</span></p><p class="c5"><span class="c8">The first expression is as follows:</span></p><a id="t.f73d7fecf9eb9d4a60187912c5869221890227ed"></a><a id="t.8"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4">re_titles </span><span class="c15">=</span><span class="c4">&nbsp;r</span><span class="c31 c29 c36">&#39;title=&quot;(.*?)&quot;&gt;&#39;</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h4 class="c19" id="h.nc0jdq4iw7qv"><span class="c16 c32">Expression # 2</span></h4><p class="c5"><span class="c3">Finding the pattern</span></p><p class="c5"><span class="c20">First, inspect the price of the book. Every price is present after the text </span><span class="c2">&pound;</span><span class="c20">&nbsp;in the format </span><span class="c2">&pound;=price</span><span class="c20">&nbsp;before the paragraph tag </span><span class="c2">&lt;/p&gt;</span><span class="c8">.</span></p><p class="c5"><span class="c3">Generating the expression</span></p><p class="c5"><span class="c20">Then, create an expression that returns the data inside quotations after the </span><span class="c2">&pound;=</span><span class="c20">&nbsp;and before the </span><span class="c2">&lt;/p&gt;</span><span class="c20">&nbsp;by specifying </span><span class="c2">&pound;(.*?)&lt;/p&gt;</span><span class="c8">.</span></p><p class="c5"><span class="c8">The second expression is as follows:</span></p><a id="t.1ceac9f0799a59cfb877d44c8c08efe3f7e3af58"></a><a id="t.9"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4 c16">re_prices </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;</span><span class="c9">&#39;&pound;(.*?)&lt;/p&gt;&#39;</span></p></td></tr></table><p class="c0 c10"><span class="c41"></span></p><p class="c5"><span class="c20">To conclude, use the expressions with </span><span class="c2">re.findall</span><span class="c20">&nbsp;to find the substrings matching the patterns. Lastly, save them in the variables </span><span class="c2">title_list</span><span class="c20">&nbsp;and </span><span class="c2">price_list</span><span class="c8">.</span></p><a id="t.d063baf836922170caa7ba5e799c76ed8eda4477"></a><a id="t.10"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4 c16">titles_list </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;re</span><span class="c7">.</span><span class="c11 c2">findall</span><span class="c7">(</span><span class="c4 c16">re_titles</span><span class="c7">,</span><span class="c4 c16">&nbsp;content</span><span class="c7">)</span></p><p class="c0"><span class="c4 c16">price_list </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;re</span><span class="c7">.</span><span class="c11 c2">findall</span><span class="c7">(</span><span class="c4 c16">re_prices</span><span class="c7">,</span><span class="c4 c16">&nbsp;content</span><span class="c7">)</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h3 class="c25" id="h.l1mcymsy6z2r"><span class="c33">Saving the output</span></h3><p class="c5"><span class="c20">To save the output, loop over the pairs for the titles and prices and write them to the </span><span class="c2">output.txt</span><span class="c8">&nbsp;file.</span></p><a id="t.25045fdb753a73d8cbdb789626309a2249c0e32b"></a><a id="t.11"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c11 c17">with</span><span class="c4 c16">&nbsp;</span><span class="c2 c11">open</span><span class="c7">(</span><span class="c9">&quot;output.txt&quot;</span><span class="c7">,</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;w&quot;</span><span class="c7">)</span><span class="c4 c16">&nbsp;</span><span class="c11 c17">as</span><span class="c4 c16">&nbsp;</span><span class="c11 c40 c29">f</span><span class="c15 c16">:</span></p><p class="c0"><span class="c4 c16">&nbsp; &nbsp;</span><span class="c11 c17">for</span><span class="c4 c16">&nbsp;title</span><span class="c7">,</span><span class="c4 c16">&nbsp;price </span><span class="c11 c17">in</span><span class="c4 c16">&nbsp;</span><span class="c11 c2">zip</span><span class="c7">(</span><span class="c4 c16">titles_list</span><span class="c7">,</span><span class="c4 c16">&nbsp;price_list</span><span class="c7">)</span><span class="c15 c16">:</span></p><p class="c0"><span class="c4 c16">&nbsp; &nbsp; &nbsp; &nbsp;f</span><span class="c7">.</span><span class="c11 c2">write</span><span class="c7">(</span><span class="c4 c16">title </span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;\t&quot;</span><span class="c4 c16">&nbsp;</span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;price </span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;\n&quot;</span><span class="c7">)</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 504.00px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 504.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title="">![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u95umwglodn72sesnfbn.png)</span></p><p class="c43"><span class="c26">The output file</span></p><p class="c0 c10"><span class="c26"></span></p><p class="c5"><span class="c20">Putting everything together, this is the complete code that can be run by calling </span><span class="c2">python demo.py</span><span class="c8">:</span></p><a id="t.df5b2218102d46e6a06248b3f14a1fe13c0bdb13"></a><a id="t.12"></a><table class="c13"><tr class="c12"><td class="c6" colspan="1" rowspan="1"><p class="c0"><span class="c4 c16"># Importing the required libraries</span><span class="c7">.</span></p><p class="c0"><span class="c11 c17">import</span><span class="c1">&nbsp;requests</span></p><p class="c0"><span class="c4 c16">from bs4 </span><span class="c11 c17">import</span><span class="c1">&nbsp;BeautifulSoup</span></p><p class="c0"><span class="c11 c17">import</span><span class="c1">&nbsp;re</span></p><p class="c0 c10"><span class="c1"></span></p><p class="c0"><span class="c4 c16"># Requesting the </span><span class="c11 c29 c40">HTML</span><span class="c4 c16">&nbsp;from the web page</span><span class="c7">.</span></p><p class="c0"><span class="c4 c16">page </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;requests</span><span class="c7">.</span><span class="c11 c2">get</span><span class="c7">(</span><span class="c9">&quot;https://books.toscrape.com/&quot;</span><span class="c7">)</span></p><p class="c0 c10"><span class="c1"></span></p><p class="c0"><span class="c4 c16"># Selecting the data</span><span class="c7">.</span></p><p class="c0"><span class="c4 c16">soup </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;</span><span class="c11 c2">BeautifulSoup</span><span class="c7">(</span><span class="c4 c16">page</span><span class="c7">.</span><span class="c4 c16">content</span><span class="c7">,</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;html.parser&quot;</span><span class="c7">)</span></p><p class="c0"><span class="c4 c16">content </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;soup</span><span class="c7">.</span><span class="c11 c2">find_all</span><span class="c7">(</span><span class="c4 c16">class_</span><span class="c15 c16">=</span><span class="c9">&quot;product_pod&quot;</span><span class="c7">)</span></p><p class="c0"><span class="c4 c16">content </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;</span><span class="c11 c2">str</span><span class="c7">(</span><span class="c4 c16">content</span><span class="c7">)</span></p><p class="c0 c10"><span class="c1"></span></p><p class="c0"><span class="c4 c16"># Processing the data using Regular Expressions</span><span class="c7">.</span></p><p class="c0"><span class="c4 c16">re_titles </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;r</span><span class="c9">&#39;title=&quot;(.*?)&quot;&gt;&#39;</span></p><p class="c0"><span class="c4 c16">titles_list </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;re</span><span class="c7">.</span><span class="c11 c2">findall</span><span class="c7">(</span><span class="c4 c16">re_titles</span><span class="c7">,</span><span class="c4 c16">&nbsp;content</span><span class="c7">)</span></p><p class="c0"><span class="c4 c16">re_prices </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;&pound;(.*?)&lt;/p&gt;&quot;</span></p><p class="c0"><span class="c4 c16">price_list </span><span class="c15 c16">=</span><span class="c4 c16">&nbsp;re</span><span class="c7">.</span><span class="c11 c2">findall</span><span class="c7">(</span><span class="c4 c16">re_prices</span><span class="c7">,</span><span class="c4 c16">&nbsp;content</span><span class="c7">)</span></p><p class="c0 c10"><span class="c1"></span></p><p class="c0"><span class="c4 c16"># &nbsp;Saving the output</span><span class="c7">.</span></p><p class="c0"><span class="c11 c17">with</span><span class="c4 c16">&nbsp;</span><span class="c11 c2">open</span><span class="c7">(</span><span class="c9">&quot;output.txt&quot;</span><span class="c7">,</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;w&quot;</span><span class="c7">)</span><span class="c4 c16">&nbsp;</span><span class="c11 c17">as</span><span class="c4 c16">&nbsp;</span><span class="c11 c40 c29">f</span><span class="c15 c16">:</span></p><p class="c0"><span class="c4 c16">&nbsp; &nbsp;</span><span class="c11 c17">for</span><span class="c4 c16">&nbsp;title</span><span class="c7">,</span><span class="c4 c16">&nbsp;price </span><span class="c11 c17">in</span><span class="c4 c16">&nbsp;</span><span class="c11 c2">zip</span><span class="c7">(</span><span class="c4 c16">titles_list</span><span class="c7">,</span><span class="c4 c16">&nbsp;price_list</span><span class="c7">)</span><span class="c15 c16">:</span></p><p class="c0"><span class="c4 c16">&nbsp; &nbsp; &nbsp; &nbsp;f</span><span class="c7">.</span><span class="c11 c2">write</span><span class="c7">(</span><span class="c4 c16">title </span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;\t&quot;</span><span class="c4 c16">&nbsp;</span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;price </span><span class="c15 c16">+</span><span class="c4 c16">&nbsp;</span><span class="c9">&quot;\n&quot;</span><span class="c7">)</span></p></td></tr></table><p class="c5 c10"><span class="c8"></span></p><h2 class="c34" id="h.8igrhb5t45nj"><span class="c37">Conclusion</span></h2><p class="c5"><span class="c20">In this post, we</span><span class="c8">&nbsp;explained what Regular Expressions are, how to use them, and what most commonly used tokens do. Additionally, we provided an example of scraping the titles and prices from a web page utilizing Python and Regular Expressions.</span></p><p class="c5"><span class="c8">Find this tutorial helpful? Don&rsquo;t hesitate to like and leave a comment below with any questions or impressions regarding the post!</span></p><p class="c0 c10"><span class="c26"></span></p></body></html>