Web Scraping With C#
Tags: webscrapin, csharp, beginners, tutorial
Published: 2022-11-22 14:20:33

<span class="c1 c21">After reading this post, you&#39;ll learn how to create a real-life C# web scraper. Keep in mind that even if you&#39;re using C#, you&#39;ll be able to adapt this information to all languages supported by the .NET platform, including VB.NET and F#. </span></p><p class="c11"><span class="c1 c21">Let&#39;s get started!</span></p><h2 class="c35" id="h.280qvyec43l8"><span class="c21 c34">Building a web scraper with C#</span></h2><p class="c31"><span class="c1 c15">As mentioned, you&#39;ll learn how to write a C# public web scraping code using HTML Agility Pack. In this tutorial, we&#39;ll be employing the .NET 5 SDK with Visual Studio Code. This code has been tested with .NET Core 3 and .NET 5, and it should work with other versions of .NET.</span></p><p class="c31"><span class="c15">We&#39;ll follow the hypothetical scenario: scraping a </span><span class="c16"><a class="c25" href="https://www.google.com/url?q=http://books.toscrape.com/catalogue/category/books/mystery_3/index.html&amp;sa=D&amp;source=editors&amp;ust=1669018987579358&amp;usg=AOvVaw30lUIq2lnc-K3ItLW-R5C5">bookstore</a></span><span class="c1 c15">&nbsp;and collecting book names and prices. Let&#39;s set up the development environment before writing a C# web crawler. </span></p><h3 class="c33" id="h.q5sjp259mb8w"><span class="c14">Setup development environment</span></h3><p class="c31"><span class="c15">For the C# development environment, install </span><span class="c16"><a class="c25" href="https://www.google.com/url?q=https://code.visualstudio.com/download&amp;sa=D&amp;source=editors&amp;ust=1669018987579806&amp;usg=AOvVaw3ww6SC2FsHwuSufR-fySl-">Visual Studio Code</a></span><span class="c1 c15">. Note that Visual Studio and Visual Studio Code are two completely different applications if you use them for writing C# code. </span></p><p class="c31"><span class="c15">Once Visual Studio Code is installed, </span><span class="c16"><a class="c25" href="https://www.google.com/url?q=https://dotnet.microsoft.com/download/dotnet/5.0&amp;sa=D&amp;source=editors&amp;ust=1669018987580067&amp;usg=AOvVaw2thoCwTsx-I_DlzTdyhskk">install .NET 5.0</a></span><span class="c1 c15">&nbsp;or newer. You can also use .NET Core 3.1. After installation is complete, open the terminal and run the following command to verify that .NET CLI or Command Line Interface is working properly:</span></p><a id="t.31a080f47024bbde6a8307a47b831c56b7cfe9cd"></a><a id="t.0"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;</span><span class="c3">--</span><span class="c0">version</span></p></td></tr></table><p class="c17"><span class="c1 c19">This should output the version number of the .NET installed.</span></p><h3 class="c33" id="h.r9d2hqriywmy"><span class="c14">Project structure and dependencies</span></h3><p class="c8"><span class="c1 c19">The code will be a part of a .NET project. To keep it simple, create a console application. Then, make a folder where you&#39;ll want to write the C# code. Open the terminal and navigate to that folder. Now, type in this command:</span></p><a id="t.d1f32049bdc88c64f92d4d9de1c917a080e93b8b"></a><a id="t.1"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">console</span></p></td></tr></table><p class="c17"><span class="c1 c19">This command&#39;s output should confirm that the console application has been successfully created. </span></p><p class="c8"><span class="c1 c19">Now, it&#39;s time to install the required packages. To use C# for scraping public web pages, HTML Agility Pack will be a good choice. You can install it for this project using this command:</span></p><a id="t.e17808d106dab35d2ca4666f2986ea8c85987fdc"></a><a id="t.2"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;add </span><span class="c2">package</span><span class="c0">&nbsp;HtmlAgilityPack</span></p></td></tr></table><p class="c17"><span class="c9">You should i</span><span class="c1 c9">nstall one more package so that you can easily export the scraped data to a CSV file:</span></p><a id="t.96cf375eff2e2baac81565914658ce4bd097fd86"></a><a id="t.3"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;add </span><span class="c20">package</span><span class="c0">&nbsp;CsvHelper</span></p></td></tr></table><p class="c26"><span class="c1 c15">If you&#39;re using Visual Studio instead of Visual Studio Code, click &quot;File,&quot; select &quot;New Solution,&quot; and press on &quot;Console Application.&quot; To install the dependencies, follow these steps:</span></p><ul class="c39 lst-kix_pyk27zkywk0p-0 start"><li class="c7 c30 li-bullet-0"><span class="c1 c15">Choose &quot;Project;&quot;</span></li><li class="c7 c30 li-bullet-0"><span class="c1 c15">Click on &quot;Manage Project Dependencies.&quot; This will open the NuGet Packages window;</span></li><li class="c7 c30 li-bullet-0"><span class="c1 c15">Search for HtmlAgilityPack and select it; </span></li><li class="c29 li-bullet-0"><span class="c15">Finally, search for CsvHelper, choose it, and click &quot;Add Packages.&quot;</span></li></ul><p class="c40"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 388.00px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 388.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p>
![project structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m35izi196fxtdrndnpon.png)<p class="c8"><span class="c1 c19">Now that the packages have been installed, you can move on to writing code for web scraping the bookstore. </span></p><h3 class="c33" id="h.qqsjy3zgiyll"><span class="c14">Download and parse web pages</span></h3><p class="c8"><span class="c19">The first step of any web scraping program is to download the HTML of a web page. This HTML will be a string you&#39;ll need to convert into an object that can be processed further. The latter part is called parsing. Html Agility Pack can read and parse files from local files, HTML strings, any URL, or even a browser. </span></p><p class="c8"><span class="c19">In this case, you only need to get HTML from a URL. Instead of using .NET native functions, Html Agility Pack provides a convenient class &ndash; </span><span class="c10">HtmlWeb</span><span class="c19">. This class offers a </span><span class="c10">Load</span><span class="c19">&nbsp;function that can take a URL and return an instance of the </span><span class="c10">HtmlDocument</span><span class="c19">&nbsp;class, which is also part of the package we use. With this information, you can write a function that takes a URL and returns an instance of </span><span class="c10">HtmlDocument</span><span class="c1 c19">.</span></p><p class="c28"><span class="c19">Open </span><span class="c10">Program.cs</span><span class="c19">&nbsp;file and enter this function in the class </span><span class="c10">Program</span><span class="c1 c19">:</span></p><a id="t.0a6d97f92b79db434001a901a2b72f654161b23c"></a><a id="t.4"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c43 c47">// Parses the URL and returns HtmlDocument object</span></p><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;HtmlDocument </span><span class="c10">GetDocument</span><span class="c6">(</span><span class="c0">string url</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; HtmlWeb web </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">HtmlWeb</span><span class="c6">();</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; HtmlDocument doc </span><span class="c3">=</span><span class="c0">&nbsp;web</span><span class="c6">.</span><span class="c10">Load</span><span class="c6">(</span><span class="c0">url</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">return</span><span class="c0">&nbsp;doc</span><span class="c6">;</span></p><p class="c7"><span class="c6">}</span></p></td></tr></table><p class="c17"><span class="c1 c19">With this, the first step of the code is complete. The next step is to parse the document. </span></p><h3 class="c33" id="h.b9atdmcadid5"><span class="c14">Parsing the HTML: getting book links</span></h3><p class="c17"><span class="c19">You&#39;ll extract the required information from the web page in this part of the code. </span><span class="c19">At this stage, a document is now an object of type </span><span class="c10">HtmlDocument</span><span class="c19">. This class exposes two functions to select the elements. Both functions accept XPath as input and return </span><span class="c10">HtmlNode</span><span class="c19">&nbsp;or </span><span class="c10">HtmlNodeCollection</span><span class="c1 c19">. Here is the signature of these two functions:</span></p><a id="t.cdaa194205a47ea61b02acf720c1b08eb36324d6"></a><a id="t.5"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c36"><span class="c2">public</span><span class="c0">&nbsp;HtmlNodeCollection </span><span class="c10">SelectNodes</span><span class="c6">(</span><span class="c0">string </span><span class="c0">xpath</span><span class="c1 c6">);</span></p><p class="c36"><span class="c2">public</span><span class="c0">&nbsp;HtmlNode </span><span class="c10">SelectSingleNode</span><span class="c6">(</span><span class="c0">string </span><span class="c0">xpath</span><span class="c6">);</span></p></td></tr></table><p class="c17"><span class="c19">Let&#39;s discuss </span><span class="c44">SelectNodes</span><span class="c1 c19">&nbsp;first.</span></p><p class="c7"><span class="c15">For this example &ndash; C# web scraper &ndash; you&#39;ll scrape all the book details from </span><span class="c16"><a class="c25" href="https://www.google.com/url?q=http://books.toscrape.com/catalogue/category/books/mystery_3/index.html&amp;sa=D&amp;source=editors&amp;ust=1669018987587572&amp;usg=AOvVaw0Ixji6XEfSCj-cOqWcof1-">this page</a></span><span class="c1 c15">. First, it needs to be parsed so that all the links to the books can be extracted. To do that, open this page in the browser, right-click any book links and click &quot;Inspect.&quot; This will open the Developer Tools. </span></p><p class="c31"><span class="c1 c15">After understanding some time with the markup, your XPath to select should be something like this:</span></p><p class="c31 c13"><span class="c1 c15"></span></p><a id="t.68a9447c928ba125dd04215acfc6abe4dcf731f5"></a><a id="t.6"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c38">//h3/a</span></p></td></tr></table><p class="c13 c31"><span class="c1 c15"></span></p><p class="c8"><span class="c9">This XPath can now be passed to the </span><span class="c10">SelectNodes</span><span class="c1 c9">&nbsp;function.</span></p><a id="t.c0cd4945afbb28ec7e8a2b9462877bc0405d209d"></a><a id="t.7"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c0">HtmlDocument doc </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetDocument</span><span class="c6">(</span><span class="c0">url</span><span class="c6">);</span></p><p class="c7"><span class="c0">HtmlNodeCollection linkNodes </span><span class="c3">=</span><span class="c0">&nbsp;doc</span><span class="c6">.</span><span class="c0">DocumentNode</span><span class="c6">.</span><span class="c10">SelectNodes</span><span class="c6">(</span><span class="c18">&quot;//h3/a&quot;</span><span class="c6">);</span></p></td></tr></table><p class="c17"><span class="c9">Note that the </span><span class="c10">DocumentNode</span><span class="c9">&nbsp;attribute of the </span><span class="c10">HtmlDocument</span><span class="c9">&nbsp;is calling the </span><span class="c10">SelectNodes</span><span class="c1 c9">&nbsp;function.</span></p><p class="c8"><span class="c9">The variable </span><span class="c10">linkNodes</span><span class="c9">&nbsp;is a collection. You can write a </span><span class="c10">foreach</span><span class="c9">&nbsp;loop over it and get the </span><span class="c10">href</span><span class="c1 c9">&nbsp;from each link one by one. There is one tiny problem that you need to take care of &ndash; the links on the page are relative. Hence, they need to be converted into an absolute URL before you can scrape these extracted links. </span></p><p class="c8"><span class="c9">For converting the relative URLs, you can make use of the </span><span class="c10">Uri</span><span class="c9">&nbsp;class. You can use this constructor to get a </span><span class="c10">Uri</span><span class="c1 c9">&nbsp;object with an absolute URL.</span></p><a id="t.a6ddec99728196ea13574f276351840e5d07b91c"></a><a id="t.8"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c10">Uri</span><span class="c6">(</span><span class="c0">Uri baseUri</span><span class="c6">,</span><span class="c0">&nbsp;string</span><span class="c3">?</span><span class="c0">&nbsp;relativeUri</span><span class="c6">);</span></p></td></tr></table><p class="c17"><span class="c9">Once you have the </span><span class="c49">Uri</span><span class="c9">&nbsp;object, you can simply check the </span><span class="c10">AbsoluteUri</span><span class="c4">&nbsp;</span><span class="c1 c9">property to get the complete URL.</span></p><p class="c8"><span class="c1 c9">Write all this in a function to keep the code organized.</span></p><a id="t.7fcffda2a6ab77387b8b1c66460427b63add628c"></a><a id="t.9"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;List</span><span class="c3">&lt;</span><span class="c0">string</span><span class="c3">&gt;</span><span class="c0">&nbsp;</span><span class="c10">GetBookLinks</span><span class="c6">(</span><span class="c0">string url</span><span class="c6">)</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;bookLinks </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">List</span><span class="c3">&lt;</span><span class="c0">string</span><span class="c3">&gt;</span><span class="c6">();</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; HtmlDocument doc </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetDocument</span><span class="c6">(</span><span class="c0">url</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; HtmlNodeCollection linkNodes </span><span class="c3">=</span><span class="c0">&nbsp;doc</span><span class="c6">.</span><span class="c0">DocumentNode</span><span class="c6">.</span><span class="c10">SelectNodes</span><span class="c6">(</span><span class="c18">&quot;//h3/a&quot;</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;baseUri </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">Uri</span><span class="c6">(</span><span class="c0">url</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c10">foreach</span><span class="c0">&nbsp;</span><span class="c6">(</span><span class="c2">var</span><span class="c0">&nbsp;link </span><span class="c2">in</span><span class="c0">&nbsp;linkNodes</span><span class="c6">)</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; string href </span><span class="c3">=</span><span class="c0">&nbsp;link</span><span class="c6">.</span><span class="c0">Attributes</span><span class="c6">[</span><span class="c18">&quot;href&quot;</span><span class="c6">].</span><span class="c0">Value</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; bookLinks</span><span class="c6">.</span><span class="c10">Add</span><span class="c6">(</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">Uri</span><span class="c6">(</span><span class="c0">baseUri</span><span class="c6">,</span><span class="c0">&nbsp;href</span><span class="c6">).</span><span class="c0">AbsoluteUri</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c6">}</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">return</span><span class="c0">&nbsp;bookLinks</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">}</span></p></td></tr></table><p class="c17"><span class="c9">You&#39;re starting with an empty </span><span class="c10">List&lt;string&gt;</span><span class="c9">&nbsp;object in this function. In the </span><span class="c10">foreach</span><span class="c1 c9">&nbsp;loop, you need to add all the links to this object and return it.</span></p><p class="c8"><span class="c9">Now, it&#39;s time to modify the </span><span class="c10">Main()</span><span class="c1 c9">&nbsp;function so that you can test the C# code that you&#39;ve written so far. Modify the function so that it looks like this:</span></p><a id="t.2912ca930c8e5ddbb5279d5c35e39aa8e13d57cd"></a><a id="t.10"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;</span><span class="c2">void</span><span class="c0">&nbsp;</span><span class="c10">Main</span><span class="c6">(</span><span class="c0">string</span><span class="c6">[]</span><span class="c0">&nbsp;args</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;bookLinks </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetBookLinks</span><span class="c6">(</span><span class="c18">&quot;http://books.toscrape.com/catalogue/category/books/mystery_3/index.html&quot;</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; Console</span><span class="c6">.</span><span class="c10">WriteLine</span><span class="c6">(</span><span class="c18">&quot;Found {0} links&quot;</span><span class="c6">,</span><span class="c0">&nbsp;bookLinks</span><span class="c6">.</span><span class="c0">Count</span><span class="c6">);</span></p><p class="c7"><span class="c6">}</span></p></td></tr></table><p class="c26"><span class="c1 c9">To run this code, open the terminal and navigate to the directory which contains this file, and type in the following:</span></p><a id="t.dda33cacdea6b7c9678df23fc8116b0402982a0a"></a><a id="t.11"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;run</span></p></td></tr></table><p class="c26"><span class="c1 c9">The output should be as follows:</span></p><a id="t.26bfbd68416797203b789f954eeb82e6d722bea5"></a><a id="t.12"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">Found </span><span class="c32">20</span><span class="c0">&nbsp;links</span></p></td></tr></table><p class="c11"><span class="c1 c21 c41">Let&#39;s move to the next part, where you&#39;ll learn to process all the links to get the book data. If you have any questions before jumping to the next section, write a comment, and we&#39;ll answer them as soon as possible. </span></p><h3 class="c33" id="h.i6aohf169l3e"><span class="c14">Parsing the HTML: getting book details</span></h3><p class="c8"><span class="c9">At this point, you should have a list of strings that contain the URLs of the books. You can simply write a loop that will get the document using the </span><span class="c10">GetDocument</span><span class="c9">&nbsp;function you&#39;ve already written. After that, you&#39;ll use the </span><span class="c10">SelectSingleNode</span><span class="c9">&nbsp;function to extract the book&#39;s title and price.</span></p><p class="c8"><span class="c9">To keep the data organized, let&#39;s start with a class. This class will represent a book. This class will have two properties &ndash; </span><span class="c10">Title</span><span class="c9">&nbsp;and </span><span class="c10">Price</span><span class="c1 c9">. It looks like this:</span></p><a id="t.52ce9bb1851061123d0cd182767dd5294837a68d"></a><a id="t.13"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c2">public</span><span class="c0">&nbsp;</span><span class="c2">class</span><span class="c0">&nbsp;</span><span class="c10">Book</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">public</span><span class="c0">&nbsp;string Title </span><span class="c6">{</span><span class="c0">&nbsp;get</span><span class="c6">;</span><span class="c0">&nbsp;set</span><span class="c6">;</span><span class="c0">&nbsp;</span><span class="c6">}</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">public</span><span class="c0">&nbsp;string Price </span><span class="c6">{</span><span class="c0">&nbsp;get</span><span class="c6">;</span><span class="c0">&nbsp;set</span><span class="c6">;</span><span class="c0">&nbsp;</span><span class="c6">}</span></p><p class="c7"><span class="c6">}</span></p></td></tr></table><p class="c17"><span class="c9">Now, open a book page in the browser and create the XPath for the </span><span class="c10">Title &ndash; //h1</span><span class="c1 c9">. Creating an XPath for the price is a little trickier because the additional books at the bottom have the same class applied.</span></p><p class="c8"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 230.67px;"><img alt="" src="images/image3.png" style="width: 624.00px; height: 230.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c8 c13"><span class="c1 c9"></span></p>
![parsing html](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tcf5ay496s9jvum2oaf4.png)<p class="c8"><span class="c1 c9">The XPath of the price will be this:</span></p><a id="t.8ac684f3b09722b0ae43a50d2ddc29e0c503be45"></a><a id="t.14"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c43">//div[contains(@class,&quot;product_main&quot;)]/p[@class=&quot;price_color&quot;]</span></p></td></tr></table><p class="c17"><span class="c1 c19">Note that XPath contains double quotes. You&#39;ll have to escape these characters by prefixing them with a backslash. </span></p><p class="c8"><span class="c19">Now, you can use the </span><span class="c10">SelectSingleNode</span><span class="c19">&nbsp;function to get the Node and then employ the </span><span class="c10">InnerText</span><span class="c1 c19">&nbsp;property to get the text contained in the element. You can organize everything in a function as follows:</span></p><a id="t.e7d8b484cc442f26074dc8e03fbc3b27b9ec5b37"></a><a id="t.15"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;List</span><span class="c3">&lt;</span><span class="c0">Book</span><span class="c3">&gt;</span><span class="c0">&nbsp;</span><span class="c10">GetBookDetails</span><span class="c6">(</span><span class="c0">List</span><span class="c3">&lt;</span><span class="c0">string</span><span class="c3">&gt;</span><span class="c0">&nbsp;urls</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;books </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">List</span><span class="c3">&lt;</span><span class="c0">Book</span><span class="c3">&gt;</span><span class="c6">();</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c10">foreach</span><span class="c0">&nbsp;</span><span class="c6">(</span><span class="c2">var</span><span class="c0">&nbsp;url </span><span class="c2">in</span><span class="c0">&nbsp;urls</span><span class="c6">)</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; HtmlDocument document </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetDocument</span><span class="c6">(</span><span class="c0">url</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;titleXPath </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c18">&quot;//h1&quot;</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;priceXPath </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c18">&quot;//div[contains(@class,\&quot;product_main\&quot;)]/p[@class=\&quot;price_color\&quot;]&quot;</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;book </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">Book</span><span class="c6">();</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c6">.</span><span class="c0">Title </span><span class="c3">=</span><span class="c0">&nbsp;document</span><span class="c6">.</span><span class="c0">DocumentNode</span><span class="c6">.</span><span class="c10">SelectSingleNode</span><span class="c6">(</span><span class="c0">titleXPath</span><span class="c6">).</span><span class="c0">InnerText</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c6">.</span><span class="c0">Price </span><span class="c3">=</span><span class="c0">&nbsp;document</span><span class="c6">.</span><span class="c0">DocumentNode</span><span class="c6">.</span><span class="c10">SelectSingleNode</span><span class="c6">(</span><span class="c0">priceXPath</span><span class="c6">).</span><span class="c0">InnerText</span><span class="c6">;</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; books</span><span class="c6">.</span><span class="c10">Add</span><span class="c6">(</span><span class="c0">book</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">}</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">return</span><span class="c0">&nbsp;books</span><span class="c6">;</span></p><p class="c8"><span class="c1 c6">}</span></p><p class="c8"><span class="c0">This function will return a list of </span><span class="c10">Book</span><span class="c0">&nbsp;objects. It&#39;s time to update the </span><span class="c10">Main() </span><span class="c1 c0">function as well:</span></p><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;</span><span class="c2">void</span><span class="c0">&nbsp;</span><span class="c10">Main</span><span class="c6">(</span><span class="c0">string</span><span class="c6">[]</span><span class="c0">&nbsp;args</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;bookLinks </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetBookLinks</span><span class="c6">(</span><span class="c18">&quot;http://books.toscrape.com/catalogue/category/books/mystery_3/index.html&quot;</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; Console</span><span class="c6">.</span><span class="c10">WriteLine</span><span class="c6">(</span><span class="c18">&quot;Found {0} links&quot;</span><span class="c6">,</span><span class="c0">&nbsp;bookLinks</span><span class="c6">.</span><span class="c0">Count</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;books </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetBookDetails</span><span class="c6">(</span><span class="c0">bookLinks</span><span class="c6">);</span></p><p class="c7"><span class="c6">}</span></p></td></tr></table><p class="c17"><span class="c1 c9">The final part of this web scraping project is to export the data in a CSV.</span></p><h3 class="c33" id="h.fgo5d3dl9lkj"><span class="c14">Exporting data</span></h3><p class="c8"><span class="c9">If you haven&#39;t yet installed the </span><span class="c10">CsvHelper</span><span class="c9">, you can do this by running the command </span><span class="c10">dotnet</span><span class="c10">&nbsp;add package CsvHelper</span><span class="c1 c9">&nbsp;from within the terminal.</span></p><p class="c8"><span class="c9">The export function is pretty straightforward. First, you need to create a </span><span class="c10">StreamWriter</span><span class="c9">&nbsp;and send the CSV file name as the parameter. Next, you should use this object to create a </span><span class="c10">CsvWriter</span><span class="c9">. Finally, you can use the </span><span class="c10">WriteRecords</span><span class="c4">&nbsp;</span><span class="c1 c9">function to write all the books in just one line of code. </span></p><p class="c8"><span class="c9">To ensure that all the resources are closed properly, you can use the </span><span class="c10">using</span><span class="c4">&nbsp;</span><span class="c1 c9">block. You can also wrap everything in a function as follows:</span></p><a id="t.f3cb5c92c0f7a7e818bc4c99031ff9a746a9f3a2"></a><a id="t.16"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;</span><span class="c2">void</span><span class="c0">&nbsp;</span><span class="c10">exportToCSV</span><span class="c6">(</span><span class="c0">List</span><span class="c3">&lt;</span><span class="c0">Book</span><span class="c3">&gt;</span><span class="c0">&nbsp;books</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c10">using</span><span class="c0">&nbsp;</span><span class="c6">(</span><span class="c2">var</span><span class="c0">&nbsp;writer </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">StreamWriter</span><span class="c6">(</span><span class="c18">&quot;./books.csv&quot;</span><span class="c6">))</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c10">using</span><span class="c0">&nbsp;</span><span class="c6">(</span><span class="c2">var</span><span class="c0">&nbsp;csv </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c2">new</span><span class="c0">&nbsp;</span><span class="c10">CsvWriter</span><span class="c6">(</span><span class="c0">writer</span><span class="c6">,</span><span class="c0">&nbsp;CultureInfo</span><span class="c6">.</span><span class="c0">InvariantCulture</span><span class="c6">))</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; csv</span><span class="c6">.</span><span class="c10">WriteRecords</span><span class="c6">(</span><span class="c0">books</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c6">}</span></p><p class="c8"><span class="c1 c6">}</span></p><p class="c8"><span class="c9">Finally, you can call this function from the </span><span class="c10">Main()</span><span class="c1 c9">&nbsp;function:</span></p><p class="c8"><span class="c2">static</span><span class="c0">&nbsp;</span><span class="c2">void</span><span class="c0">&nbsp;</span><span class="c10">Main</span><span class="c6">(</span><span class="c0">string</span><span class="c6">[]</span><span class="c0">&nbsp;args</span><span class="c6">)</span></p><p class="c8"><span class="c6">{</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;bookLinks </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetBookLinks</span><span class="c6">(</span><span class="c18">&quot;http://books.toscrape.com/catalogue/category/books/mystery_3/index.html&quot;</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c2">var</span><span class="c0">&nbsp;books </span><span class="c3">=</span><span class="c0">&nbsp;</span><span class="c10">GetBookDetails</span><span class="c6">(</span><span class="c0">bookLinks</span><span class="c6">);</span></p><p class="c8"><span class="c0">&nbsp; &nbsp; </span><span class="c10">exportToCSV</span><span class="c6">(</span><span class="c0">books</span><span class="c6">);</span></p><p class="c8"><span class="c6">}</span></p></td></tr></table><p class="c17"><span class="c1 c9">That&#39;s it! To run this code, open the terminal and run the following command:</span></p><a id="t.dda33cacdea6b7c9678df23fc8116b0402982a0a"></a><a id="t.17"></a><table class="c22"><tr class="c12"><td class="c5" colspan="1" rowspan="1"><p class="c7"><span class="c0">dotnet</span><span class="c0">&nbsp;run</span></p></td></tr></table><p class="c17"><span class="c9">Within seconds, you&#39;ll have a </span><span class="c10">books.csv</span><span class="c1 c9">&nbsp;file created. </span></p><h2 class="c35" id="h.mvsteffmm008"><span class="c46">Summary</span></h2><p class="c8"><span class="c9">You can use multiple packages to write a web scraper with C#.</span><span class="c9">&nbsp;In this post, we&#39;ve </span><span class="c9">shown how to employ HTML Agility Pack, a powerful and easy-to-use package. This was a simple example that can be enhanced further; for instance, you can try adding the above logic to handle multiple pages to this code.</span></p><p class="c8"><span class="c9">If you have any questions about this tutorial or any other web scraping topics, don&#39;t hesitate to comment on this post, we&#39;ll answer you right away. </span></p></body></html>