Web Scraping for Machine Learning
Tags: webdev, programming, tutorial, api
Published: 2023-03-03 12:57:03


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hph60skxbvfjahrxuld6.png)<span class="c8">Did you know that Machine Learning (ML) and web scraping often go hand-in-hand? In this article, we&rsquo;ll discuss why ML is so important for web scraping and share a comprehensive guide on how to utilize it for your web scraping endeavors.</span></p><p class="c6"><span class="c8">Let&rsquo;s dive in!</span></p><h2 class="c20" id="h.5h3s2dgd12bc"><span>What is machine learning?</span></h2><p class="c6"><span class="c8">As an essential component of data science and a branch/byproduct of AI, ML aims to imitate the way humans learn. It does so by gathering data and using algorithms, which are then used for gradual self-improvement in terms of predictions and their accuracy.</span></p><p class="c6"><span class="c8">Such features are rather convenient since they allow for a more hands-off approach, i.e., instead of hand-coding various software routines or instructions, this will enable you to achieve a specific given task primarily through a machine with little interference from a developer.</span></p><p class="c6"><span class="c8">The next question is how the benefits of ML translate to real-life scenarios. To answer this, let&rsquo;s look at some use cases.</span></p><p class="c6"><span class="c13 c31">Customer service:</span><span class="c8">&nbsp;chatbots are starting to replace human agents, with FAQs often being answered without a human reply. The Virtual Agents of Slack and Facebook are prime examples of this.</span></p><p class="c6"><span class="c13 c31">Web Unblocker: </span><span class="c8">an AI-powered proxy solution that allows for block-free data gathering. This proxy solution contains ML-driven proxy management as well as ML-powered response recognition, both of which ensure an effortless data collection process.</span></p><p class="c6"><span class="c13 c31">Computer Vision:</span><span class="c8">&nbsp;AI, and as such, ML technology, allows for the extraction of meaningful information taken solely from visual data, upon which recognition tasks can be achieved. A prime example of this use is ML integration within self-driving cars.</span></p><p class="c6"><span class="c13 c31">Stock trading: </span><span class="c8">enables automated trading that optimizes stock portfolios by potentially making millions of automated trades per day.</span></p><h2 class="c20" id="h.c6rvaajmesj1"><span class="c14 c22">The importance of web scraping in machine learning</span></h2><p class="c6"><span class="c8">Having discussed the features of Machine Learning, let&rsquo;s take a look at how they all translate within web scraping. </span></p><p class="c6"><span class="c8">Primarily, web scraping in ML is centered around the core problem of gathering quality data.</span></p><p class="c6"><span class="c8">While the internal information gathered on day-to-day business can provide valuable insights, such data is insufficient. Therefore, gathering from external sources is essential, although a more complex task. Inaccuracy/poor data quality becomes a severe concern when scraping, and a final clean-up step must always be included within any scraping project, though this will be discussed in greater detail later on in this guide.</span></p><h2 class="c20" id="h.87urf7z107nh"><span class="c14 c22">Using machine learning for web scraping</span></h2><p class="c6"><span class="c8">The following example collects historical stock prices using web scraping. Data points, such as daily opening, daily highest, daily lowest, and daily closing, will be collected as well.</span></p><p class="c0"><span class="c8">Thankfully, numerous websites provide such data, and it&rsquo;s usually conveniently presented in a table. Typically, you&rsquo;ll see the HTML code that renders these tables, such as the following image. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/af5u9o01vb5zui6bteaj.png)</span></p><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 256.00px;"><img alt="" src="images/image2.png" style="width: 624.00px; height: 256.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c16 c15"><span class="c8"></span></p><p class="c6"><span class="c8">With that in mind, let&rsquo;s get started with the first step of web scraping.</span></p><h3 class="c29" id="h.ofu7g0310y96"><span class="c37 c24">Project requirements</span></h3><p class="c6"><span class="c8">In this tutorial, we&rsquo;ll be working with Python 3.9. However, this code will work with Python 3.7 and 3.8 as well.</span></p><p class="c6"><span class="c8">There are two sets of requirements for this project:</span></p><ul class="c38 lst-kix_i7dkf9r8qpjt-0 start"><li class="c6 c18 li-bullet-0"><span class="c8">Libraries for web scraping</span></li><li class="c6 c18 li-bullet-0"><span class="c8">Libraries for machine learning</span></li></ul><p class="c6"><span class="c8">For web scraping, we&rsquo;ll need Requests-HTML and BeautifulSoup4. Install these from the terminal as follows:</span></p><p class="c6 c15"><span class="c8"></span></p><p class="c6 c15"><span class="c8"></span></p><a id="t.7fa43ccce4a52cf14b35976291805b6f8612f0de"></a><a id="t.0"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c14 c12">$ python3 -m pip install requests_html beautifulsoup4</span></p></td></tr></table><p class="c6 c15"><span class="c8"></span></p><p class="c6"><span class="c8">In regards to machine learning, we&rsquo;ll be using multiple libraries instead. Primarily, Pandas and Numpy are going to be our choice on how to handle our data. For visualization, Matplotlib will be our choice. For preprocessing data, we&rsquo;ll need help from the SciKit Learn library. Finally, we&rsquo;ll use Tensorflow to create a neural network machine learning model. </span></p><p class="c6"><span class="c8">Install all these libraries from the terminal as follows:</span></p><a id="t.c465684c3157e52a99b2046c1390bdd94d4c7271"></a><a id="t.1"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c14 c12">$ python3 -m pip install pandas numpy matplotlib seaborn tensorflow sklearn</span></p></td></tr></table><h3 class="c29" id="h.utp5mxbockmk"><span class="c37 c24">Extracting the data</span></h3><p class="c6"><span class="c8">If we&rsquo;re looking at machine learning projects, Jupyter Notebook is a great choice as it&rsquo;s easier to run and rerun a few lines of code. Moreover, the plots are in the same Notebook.</span></p><p class="c6"><span class="c8">Begin with importing the required libraries as follows:</span></p><a id="t.132c3c047267cf9bcec44bf1872c4c236a1d229c"></a><a id="t.2"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">from requests_html </span><span class="c4 c9">import</span><span class="c14 c4">&nbsp;HTMLSession</span></p><p class="c0"><span class="c4 c9">import</span><span class="c4">&nbsp;pandas </span><span class="c4 c9">as</span><span class="c14 c4">&nbsp;pd</span></p></td></tr></table><p class="c0 c15"><span class="c14 c34 c25"></span></p><p class="c6"><span class="c8">For web scraping, we only need Requests-HTML. The primary reason is that Requests-HTML is a powerful library that can handle all our web scraping tasks, such as extracting the HTML code from websites and parsing this code into Python objects. Further benefits come from the library&rsquo;s ability to function as an HTML parser, meaning collecting data and labeling can be performed using the same library. </span></p><p class="c6"><span class="c8">Next, we use Pandas for loading the data in a DataFrame for further processing.</span></p><p class="c6"><span class="c8">In the next cell, create a session and get the response from your target URL.</span></p><a id="t.54cbdc2c2c7ff7ee44900012aa92ae79fac47ece"></a><a id="t.3"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">url </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c3">&#39;http://your-target-url&#39;</span></p><p class="c10"><span class="c4">session </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c4 c5">HTMLSession</span><span class="c1">()</span></p><p class="c0"><span class="c4">r </span><span class="c2">=</span><span class="c4">&nbsp;session</span><span class="c1">.</span><span class="c4 c5">get</span><span class="c1">(</span><span class="c4">url</span><span class="c1">)</span></p></td></tr></table><p class="c0 c15"><span class="c24 c25 c40 c41"></span></p><p class="c6"><span class="c8">After this, use XPath to select the desired data. It&rsquo;ll be easier if each row is represented as a dictionary where the key is the column name. All these dictionaries can then be added to a list.</span></p><a id="t.15c83f073afb68f26f3f46e77a4bcd3adfef8c40"></a><a id="t.4"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">rows </span><span class="c2">=</span><span class="c4">&nbsp;r</span><span class="c1">.</span><span class="c4">html</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;//table/tbody/tr&#39;</span><span class="c1">)</span></p><p class="c10"><span class="c4">symbol </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c3">&#39;AAPL&#39;</span></p><p class="c10"><span class="c4">data </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c1">[]</span></p><p class="c10"><span class="c4 c9">for</span><span class="c4">&nbsp;row </span><span class="c4 c9">in</span><span class="c4">&nbsp;</span><span class="c4 c7">rows</span><span class="c2">:</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; </span><span class="c4 c9">if</span><span class="c4">&nbsp;</span><span class="c4 c5">len</span><span class="c1">(</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td&#39;</span><span class="c1">))</span><span class="c4">&nbsp;</span><span class="c2">&lt;</span><span class="c4">&nbsp;</span><span class="c4 c7">7</span><span class="c2">:</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c9">continue</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; data</span><span class="c1">.</span><span class="c4 c5">append</span><span class="c1">({</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Symbol&#39;</span><span class="c2">:</span><span class="c4">symbol</span><span class="c1">,</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Date&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[1]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Open&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[2]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;High&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[3]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Low&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[4]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Close&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[5]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Adj Close&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[6]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">],</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c7">&#39;Volume&#39;</span><span class="c2">:</span><span class="c4">row</span><span class="c1">.</span><span class="c4 c5">xpath</span><span class="c1">(</span><span class="c3">&#39;.//td[7]/span/text()&#39;</span><span class="c1">)[</span><span class="c4 c7">0</span><span class="c1">]</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c1">})</span></p></td></tr></table><p class="c0 c15"><span class="c8"></span></p><p class="c0"><span class="c8">The results of web scraping are stored in the variable data. To understand why such actions are taken, we must consider that these variables are a list of dictionaries that can be easily converted to a data frame. Furthermore, completing the steps mentioned above will also help to complete the vital step of data labeling.</span> 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/30zoq0y4y36woesadv30.png)</p><p class="c35 c25"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 138.67px;"><img alt="" src="images/image4.png" style="width: 624.00px; height: 138.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c6"><span class="c8">The provided example&rsquo;s data frame is not yet ready for the machine learning step. It still needs additional cleaning.</span></p><h3 class="c29" id="h.upnr4q4fvcui"><span class="c37 c24">Cleaning the data</span></h3><p class="c0"><span class="c13">Now that the data has been collected using web scraping, we need to clean it up. The primary reason for this action is the uncertainty of whether the data frame is acceptable; therefore, it&rsquo;s recommended to verify everything by running </span><span class="c12 c5">df.info()</span> 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4xxl7wpy7gupvkwxymcc.png)<span class="c14 c34"></span></p><p class="c25 c35"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 181.33px;"><img alt="" src="images/image5.png" style="width: 624.00px; height: 181.33px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c6"><span class="c13">As evident from the above screen-print, all the columns have data type as</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">object</span><span class="c8">. For machine learning algorithms, these should be numbers.</span></p><p class="c6"><span class="c13">Dates can be handled using</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">Pandas.to_datetime</span><span class="c13">. It&rsquo;ll take a series and convert the values to</span><span class="c12 c5">&nbsp;</span><span class="c5 c12">datetime</span><span class="c8">. This can then be used as follows:</span></p><a id="t.9e7306997dd34d58b6a20f0c84e4196e58ea3e63"></a><a id="t.5"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c4">df</span><span class="c1">[</span><span class="c3">&#39;Date&#39;</span><span class="c1">]</span><span class="c4">&nbsp;</span><span class="c2">=</span><span class="c4">&nbsp;pd</span><span class="c1">.</span><span class="c4 c5">to_datetime</span><span class="c1">(</span><span class="c4">df</span><span class="c1">[</span><span class="c3">&#39;Date&#39;</span><span class="c1">])</span></p></td></tr></table><p class="c0 c15"><span class="c23 c25"></span></p><p class="c6"><span class="c8">The issue we ran into now is that the other columns were not automatically converted to numbers because of comma separators. </span></p><p class="c6"><span class="c13">Thankfully, there are multiple ways to handle this. The easiest one is to remove the comma by calling</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">str.replace()</span><span class="c13">&nbsp;function. The</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">astype</span><span class="c8">&nbsp;function can also be called in the same line which will then return a float.</span></p><a id="t.3179ac47e805f17683272f2bad6f08bdb1f879c5"></a><a id="t.6"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">str_cols </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c1">[</span><span class="c3">&#39;High&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Low&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Close&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Adj Close&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Volume&#39;</span><span class="c1">]</span></p><p class="c0"><span class="c4">df</span><span class="c1">[</span><span class="c4">str_cols</span><span class="c1">]</span><span class="c2">=</span><span class="c4">df</span><span class="c1">[</span><span class="c4">str_cols</span><span class="c1">].</span><span class="c4 c5">replace</span><span class="c1">(</span><span class="c3">&#39;,&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;&#39;</span><span class="c1">,</span><span class="c4">&nbsp;regex</span><span class="c2">=</span><span class="c4">True</span><span class="c1">).</span><span class="c4 c5">astype</span><span class="c1">(</span><span class="c4">float</span><span class="c1">)</span></p></td></tr></table><p class="c6"><span class="c13">Finally, if there are any </span><span class="c12 c5">None</span><span class="c13">&nbsp;or</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">NaN</span><span class="c13">&nbsp;values, these can be deleted by calling the </span><span class="c12 c5">dropna()</span><span class="c13">.</span></p><a id="t.37e295e4aab732de2c88f8ca61b45f6caabc03c3"></a><a id="t.7"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c4">df</span><span class="c1">.</span><span class="c4 c5">dropna</span><span class="c1">(</span><span class="c4">inplace</span><span class="c2">=</span><span class="c4">True</span><span class="c1">)</span></p></td></tr></table><p class="c0 c15"><span class="c14 c34 c25"></span></p><p class="c0 c15"><span class="c23 c25"></span></p><p class="c6"><span class="c13">As the last step, set the</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">Date</span><span class="c8">&nbsp;column as the index and preview the data frame.</span></p><a id="t.add8dad9932feed3e64798ab6fd93dc363170724"></a><a id="t.8"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">df </span><span class="c2">=</span><span class="c4">&nbsp;df</span><span class="c1">.</span><span class="c4 c5">set_index</span><span class="c1">(</span><span class="c3">&#39;Date&#39;</span><span class="c1">)</span></p><p class="c0"><span class="c4">df</span><span class="c1">.</span><span class="c4 c5">head</span><span class="c1">()</span></p></td></tr></table><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 150.67px;"><img alt="" src="images/image3.png" style="width: 624.00px; height: 150.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c6"><span class="c8"> 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vusfx29wshmvh5sc862h.png)The data frame is now clean and ready to be sent to the machine learning model.</span></p><h3 class="c29" id="h.u0s8j41ww6b8"><span class="c37 c24">Visualizing the data</span></h3><p class="c6"><span class="c8">Before we begin the section on machine learning, let&rsquo;s have a quick look at the closing price trend.</span></p><p class="c6"><span class="c8">First, import the packages and set the plot styles:</span></p><p class="c10 c15"><span class="c24 c25 c9 c34"></span></p><a id="t.8ad27e0172c744fdc16a46267f4f3f94a32c4f48"></a><a id="t.9"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4 c9">import</span><span class="c4">&nbsp;matplotlib</span><span class="c1">.</span><span class="c4">pyplot </span><span class="c4 c9">as</span><span class="c14 c4">&nbsp;plt</span></p><p class="c10"><span class="c4 c9">import</span><span class="c4">&nbsp;seaborn </span><span class="c4 c9">as</span><span class="c14 c4">&nbsp;sns</span></p><p class="c10"><span class="c4">sns</span><span class="c1">.</span><span class="c4 c5">set_style</span><span class="c1">(</span><span class="c3">&#39;darkgrid&#39;</span><span class="c1">)</span></p><p class="c0"><span class="c4">plt</span><span class="c1">.</span><span class="c4">style</span><span class="c1">.</span><span class="c4 c5">use</span><span class="c1">(</span><span class="c3">&quot;ggplot&quot;</span><span class="c1">)</span></p></td></tr></table><p class="c10 c15"><span class="c24 c34 c25 c9"></span></p><p class="c6"><span class="c13">Next, enter the following lines to plot the </span><span class="c12 c5">Adj Close</span><span class="c8">, which is the adjusted closing price:</span></p><a id="t.f9ddf6832f2e6cfd18f8734917ed7533b21a4471"></a><a id="t.10"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">figure</span><span class="c1">(</span><span class="c4">figsize</span><span class="c2">=</span><span class="c1">(</span><span class="c4 c7">15</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c4 c7">6</span><span class="c1">))</span></p><p class="c10"><span class="c4">df</span><span class="c1">[</span><span class="c3">&#39;Adj Close&#39;</span><span class="c1">].</span><span class="c4 c5">plot</span><span class="c1">()</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">ylabel</span><span class="c1">(</span><span class="c3">&#39;Adj Close&#39;</span><span class="c1">)</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">xlabel</span><span class="c1">(</span><span class="c4">None</span><span class="c1">)</span></p><p class="c0"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">title</span><span class="c1">(</span><span class="c3">&#39;Closing Price of AAPL&#39;</span><span class="c1">)</span></p></td></tr></table><p class="c6 c15"><span class="c8"></span></p><p class="c0 c15"><span class="c1 c24"></span></p><p class="c16"> ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjvca1v47f6hyqvoy5ch.png)<span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 246.67px;"><img alt="" src="images/image7.png" style="width: 624.00px; height: 246.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c35 c15 c25"><span class="c23"></span></p><h3 class="c29" id="h.x3ib09sr66lj"><span class="c37 c24">Preparing data for machine learning</span></h3><p class="c6"><span class="c8">The first step to machine learning is the selection of features and values we want to predict. </span></p><p class="c6"><span class="c13">In this example, the &ldquo;Adjusted Close&rdquo; is dependent on the &ldquo;Close&rdquo; part. Therefore, we&rsquo;ll ignore the </span><span class="c12 c5">Close</span><span class="c13">&nbsp;column and focus on </span><span class="c12 c5">Adj Close</span><span class="c8">.</span></p><p class="c6"><span class="c13">The features are usually stored in a variable named</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">X</span><span class="c40">,</span><span class="c13">&nbsp;and the values that we want to predict are stored in a variable</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">y</span><span class="c8">.</span></p><a id="t.ea65618745219dc291c7263b2eaaa6b324b9da45"></a><a id="t.11"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">features </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c1">[</span><span class="c3">&#39;Open&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;High&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Low&#39;</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c3">&#39;Volume&#39;</span><span class="c1">]</span></p><p class="c0"><span class="c4">y </span><span class="c2">=</span><span class="c4">&nbsp;df</span><span class="c1">.</span><span class="c4 c5">filter</span><span class="c1">([</span><span class="c3">&#39;Adj Close&#39;</span><span class="c1">])</span></p></td></tr></table><p class="c0 c15"><span class="c1 c24"></span></p><p class="c6"><span class="c13">The next step we have to consider is feature scaling. It&rsquo;s used to normalize the features, i.e., the independent variables. Within our example, we can use</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">MinMaxScaler</span><span class="c8">. This class is part of the preprocessing module of the Sci Kit Learn library.</span></p><p class="c6"><span class="c13">First, we&rsquo;ll create an object of this class. Then, we&rsquo;ll train and transform the values using the</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">fit_transform</span><span class="c12 c5">&nbsp;</span><span class="c8">method as follows:</span></p><a id="t.55c233f8b8e288c948faf2e61f7ada11b8648e4d"></a><a id="t.12"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">from sklearn</span><span class="c1">.</span><span class="c4">preprocessing </span><span class="c4 c9">import</span><span class="c14 c4">&nbsp;MinMaxScaler</span></p><p class="c10"><span class="c4">scaler </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c4 c5">MinMaxScaler</span><span class="c1">()</span></p><p class="c0"><span class="c4 c7">X</span><span class="c4">&nbsp;</span><span class="c2">=</span><span class="c4">&nbsp;scaler</span><span class="c1">.</span><span class="c4 c5">fit_transform</span><span class="c1">(</span><span class="c4">df</span><span class="c1">[</span><span class="c4">features</span><span class="c1 c24">])</span></p><p class="c28 c15"><span class="c14 c4"></span></p></td></tr></table><p class="c6"><span class="c8">The next step is splitting the data we have received into two datasets, test, and training.</span></p><p class="c6"><span class="c13">The example we&rsquo;re working with today is time-series data, meaning data that changes over a time period require specialized handling. The</span><span class="c12 c5">&nbsp;</span><span class="c12 c5">TimeSeriesSplit</span><span class="c12 c5">&nbsp;</span><span class="c13">function from SKLearn&rsquo;s </span><span class="c12 c5">model_selection</span><span class="c8">&nbsp;module will be what we need here.</span></p><p class="c6 c15"><span class="c8"></span></p><a id="t.2dfd18c52596c2c3b48132efa7e6428dbada5139"></a><a id="t.13"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">from sklearn</span><span class="c1">.</span><span class="c4">model_selection </span><span class="c4 c9">import</span><span class="c14 c4">&nbsp;TimeSeriesSplit</span></p><p class="c10"><span class="c4">tscv </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c4 c5">TimeSeriesSplit</span><span class="c1">(</span><span class="c4">n_splits</span><span class="c2">=</span><span class="c4 c7">10</span><span class="c1">)</span><span class="c14 c4">&nbsp;</span></p><p class="c10"><span class="c4 c9">for</span><span class="c4">&nbsp;train_index</span><span class="c1">,</span><span class="c4">&nbsp;test_index </span><span class="c4 c9">in</span><span class="c4">&nbsp;tscv</span><span class="c1">.</span><span class="c4 c5">split</span><span class="c1">(</span><span class="c4 c7">X</span><span class="c1">)</span><span class="c2">:</span></p><p class="c10"><span class="c4">&nbsp; &nbsp; X_train</span><span class="c1">,</span><span class="c4">&nbsp;X_test </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c4 c7">X</span><span class="c1">[</span><span class="c4">train_index</span><span class="c1">],</span><span class="c4">&nbsp;</span><span class="c4 c7">X</span><span class="c1">[</span><span class="c4">test_index</span><span class="c1">]</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; y_train</span><span class="c1">,</span><span class="c4">&nbsp;y_test </span><span class="c2">=</span><span class="c4">&nbsp;y</span><span class="c1">[</span><span class="c4">train_index</span><span class="c1">],</span><span class="c4">&nbsp;y</span><span class="c1">[</span><span class="c4">test_index</span><span class="c1">]</span></p></td></tr></table><p class="c0 c15"><span class="c1 c24"></span></p><p class="c6"><span class="c8">Our approach for today will be creating a neural network that uses an LSTM or a Long Short-Term Memory layer. LSTM expects a 3-dimensional input with information about the batch size, timesteps, and input dimensions. We need to reshape the features as follows:</span></p><a id="t.3186da584f2bf75d4c3a9ee951aa57f63514e9c1"></a><a id="t.14"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">X_train </span><span class="c2">=</span><span class="c4">&nbsp;X_train</span><span class="c1">.</span><span class="c4 c5">reshape</span><span class="c1">(</span><span class="c4">X_train</span><span class="c1">.</span><span class="c4">shape</span><span class="c1">[</span><span class="c4 c7">0</span><span class="c1">],</span><span class="c4">&nbsp;</span><span class="c4 c7">1</span><span class="c1">,</span><span class="c4">&nbsp;X_train</span><span class="c1">.</span><span class="c4">shape</span><span class="c1">[</span><span class="c4 c7">1</span><span class="c1">])</span></p><p class="c0"><span class="c4">X_test </span><span class="c2">=</span><span class="c4">&nbsp;X_test</span><span class="c1">.</span><span class="c4 c5">reshape</span><span class="c1">(</span><span class="c4">X_test</span><span class="c1">.</span><span class="c4">shape</span><span class="c1">[</span><span class="c4 c7">0</span><span class="c1">],</span><span class="c4">&nbsp;</span><span class="c4 c7">1</span><span class="c1">,</span><span class="c4">&nbsp;X_test</span><span class="c1">.</span><span class="c4">shape</span><span class="c1">[</span><span class="c4 c7">1</span><span class="c1 c24">])</span></p><p class="c15 c28"><span class="c8"></span></p></td></tr></table><h3 class="c29" id="h.8br0z3o0djly"><span class="c24 c37">Training the model and predictions</span></h3><p class="c6"><span class="c13">We&rsquo;re now ready to create a model. Import the </span><span class="c12 c5">Sequential</span><span class="c13">&nbsp;model, </span><span class="c12 c5">LSTM</span><span class="c13">&nbsp;layer, and </span><span class="c12 c5">Dense</span><span class="c5 c36">&nbsp;</span><span class="c8">layer from Keras as follows:</span></p><a id="t.3b6cc5c1cfa17e466c7e599e28fd7960697f2864"></a><a id="t.15"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">from keras</span><span class="c1">.</span><span class="c4">models </span><span class="c4 c9">import</span><span class="c14 c4">&nbsp;Sequential</span></p><p class="c0"><span class="c4">from keras</span><span class="c1">.</span><span class="c4">layers </span><span class="c4 c9">import</span><span class="c4">&nbsp;</span><span class="c4 c7">LSTM</span><span class="c1">,</span><span class="c14 c4">&nbsp;Dense</span></p></td></tr></table><p class="c6"><span class="c8">Continue by creating an instance of the Sequential model and adding two layers. The first layer will be an LSTM with 32 units, while the second will be a Dense layer.</span></p><a id="t.d9596b1d6049a6bd653166b711f3cbf38bd10619"></a><a id="t.16"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">model </span><span class="c2">=</span><span class="c4">&nbsp;</span><span class="c4 c5">Sequential</span><span class="c1">()</span></p><p class="c10"><span class="c4">model</span><span class="c1">.</span><span class="c4 c5">add</span><span class="c1">(</span><span class="c4 c7">LSTM</span><span class="c1">(</span><span class="c4 c7">32</span><span class="c1">,</span><span class="c4">&nbsp;activation</span><span class="c2">=</span><span class="c3">&#39;relu&#39;</span><span class="c1">,</span><span class="c4">&nbsp;return_sequences</span><span class="c2">=</span><span class="c4">False</span><span class="c1">))</span></p><p class="c10"><span class="c4">model</span><span class="c1">.</span><span class="c4 c5">add</span><span class="c1">(</span><span class="c4 c5">Dense</span><span class="c1">(</span><span class="c4 c7">1</span><span class="c1">))</span></p><p class="c0"><span class="c4">model</span><span class="c1">.</span><span class="c4 c5">compile</span><span class="c1">(</span><span class="c4">loss</span><span class="c2">=</span><span class="c3">&#39;mean_squared_error&#39;</span><span class="c1">,</span><span class="c4">&nbsp;optimizer</span><span class="c2">=</span><span class="c3">&#39;adam&#39;</span><span class="c1">)</span></p></td></tr></table><p class="c6"><span class="c8">The model can be trained with the following line of code:</span></p><p class="c6 c15"><span class="c8"></span></p><a id="t.c39a8ebb7c73585a1dc59fbc91fe14aeda28521f"></a><a id="t.17"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c4">model</span><span class="c1">.</span><span class="c4 c5">fit</span><span class="c1">(</span><span class="c4">X_train</span><span class="c1">,</span><span class="c4">&nbsp;y_train</span><span class="c1">,</span><span class="c4">&nbsp;epochs</span><span class="c2">=</span><span class="c4 c7">100</span><span class="c1">,</span><span class="c4">&nbsp;batch_size</span><span class="c2">=</span><span class="c4 c7">8</span><span class="c1 c24">)</span></p><p class="c28 c15"><span class="c14 c4"></span></p></td></tr></table><p class="c6"><span class="c8">While the predictions can be made using this line of code:</span></p><p class="c6 c15"><span class="c8"></span></p><a id="t.8b17920278e88da12febfb9abb5c8bf48ce227dc"></a><a id="t.18"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c0"><span class="c4">y_pred</span><span class="c2">=</span><span class="c4">&nbsp;model</span><span class="c1">.</span><span class="c4 c5">predict</span><span class="c1">(</span><span class="c4">X_test</span><span class="c1">)</span></p></td></tr></table><p class="c6"><span class="c8">Finally, let&rsquo;s plot the actual values and predicted values with the following:</span></p><a id="t.7ff853376857782a5c82a862faee4611d61a5549"></a><a id="t.19"></a><table class="c19"><tr class="c11"><td class="c21" colspan="1" rowspan="1"><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">figure</span><span class="c1">(</span><span class="c4">figsize</span><span class="c2">=</span><span class="c1">(</span><span class="c4 c7">15</span><span class="c1">,</span><span class="c4">&nbsp;</span><span class="c4 c7">6</span><span class="c1">))</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">plot</span><span class="c1">(</span><span class="c4">y_test</span><span class="c1">,</span><span class="c4">&nbsp;label</span><span class="c2">=</span><span class="c3">&#39;Actual Value&#39;</span><span class="c1">)</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">plot</span><span class="c1">(</span><span class="c4">y_pred</span><span class="c1">,</span><span class="c4">&nbsp;label</span><span class="c2">=</span><span class="c3">&#39;Predicted Value&#39;</span><span class="c1">)</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">ylabel</span><span class="c1">(</span><span class="c3">&#39;Adjusted Close (Scaled)&#39;</span><span class="c1">)</span></p><p class="c10"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">xlabel</span><span class="c1">(</span><span class="c3">&#39;Time Scale&#39;</span><span class="c1">)</span></p><p class="c0"><span class="c4">plt</span><span class="c1">.</span><span class="c4 c5">legend</span><span class="c1">()</span></p></td></tr></table><p class="c6 c15"><span class="c8"></span></p><p class="c0 c15"><span class="c1 c24"></span></p><p class="c16"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 260.00px;"><img alt="" src="images/image6.png" style="width: 624.00px; height: 260.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c6"><span class="c8"> 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2t3cj82j8v3nvuerx1k.png)The plot shows that the predictions are close to the actual values. Yet, more importantly, the trends are similarly close as well. </span></p><h2 class="c26" id="h.xf69ujnaevv"><span class="c17">Conclusion</span></h2><p class="c6"><span class="c13">This tutorial explored a real-life scenario where web scraping and machine learning work in tandem. We went through writing a web scraping program that can extract data in a format suitable for machine learning. Then, we cleaned the data, updated its type, and applied other preprocessing techniques to make the combined effort ideal for the machine learning program. Finally, we finished by creating a machine learning model that used a neural network to predict stock pricing, which altogether, hopefully, gave you a valuable introduction to how web scraping can be used for machine learning.</span></p></body></html>