Web Scraping With PHP | Ultimate Tutorial
Tags: webscraping, php, tutorial, beginners
Published: 2022-10-31 07:37:41

<p class="c11"><span class="c3">You can use various scripting languages to do web scraping, and PHP is certainly one to try! It&rsquo;s a general-purpose language and one of the most popular options for web development. For example, WordPress, the most common content management system for creating websites, is built using PHP.</span></p><p class="c11 c7"><span class="c3"></span></p><p class="c8"><span class="c3">PHP offers various building blocks required to build a web scraper, although it can quickly become an increasingly complicated task. Conveniently, many open-source libraries can make web scraping with PHP more accessible.</span></p><p class="c8"><span class="c3">This post will guide you through the step-by-step process of writing various PHP web scraping routines you can employ to extract public data from static and dynamic web pages.</span></p><p class="c8"><span class="c3">Let&rsquo;s get started!</span></p><h2 class="c39" id="h.jvvtogybb3zg"><span class="c13 c30">Can PHP be used for web scraping?</span></h2><p class="c8"><span class="c3">In short, yes, it certainly can, and the rest of the article will detail precisely how the web page scraping processes should look. However, asking whether it&#39;s a good choice as a language for web scraping is an entirely different question, as numerous programming language alternatives exist. </span></p><p class="c8"><span class="c3">Note that PHP is old. It has existed since the 90s and reached significant version 8. Yet, this is advantageous as it makes PHP a rather easy language to use and has decades of solved problems/errors under its belt. However, simplicity comes at a cost as well. When it comes to complex, dynamic websites, PHP is outperformed by Python and Javascript, although if your requirements are data scraped from simple pages, then PHP is a good choice.</span></p><h2 class="c41" id="h.oicelgt0glfn"><span class="c17 c33 c35">Installing prerequisites</span></h2><p class="c8"><span class="c3">To begin, make sure that you have both PHP and Composer installed.</span></p><p class="c8"><span class="c15">If you&rsquo;re using Windows, visit </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://windows.php.net/download/&amp;sa=D&amp;source=editors&amp;ust=1667206974862205&amp;usg=AOvVaw3JgZnlwaT57g4MhDo6sGbT">this link</a></span><span class="c15">&nbsp;to download PHP. You can also use the </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://chocolatey.org/&amp;sa=D&amp;source=editors&amp;ust=1667206974862666&amp;usg=AOvVaw0g_0j_HO9BUML3NY8nJmDd">Chocolatey</a></span><span class="c3">&nbsp;package manager. </span></p><p class="c8"><span class="c3">Using Chocolatey, run the following command from the command line or PowerShell:</span></p><p class="c2"><span class="c0 c23 c24">choco install php</span></p><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c15">If you&rsquo;re using macOS, the chances are that you already have PHP bundled with the operating system. Otherwise, you can use a package manager such as </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://brew.sh/&amp;sa=D&amp;source=editors&amp;ust=1667206974863654&amp;usg=AOvVaw1e7DBrRX6OTnilSWG3UId8">Homebrew</a></span><span class="c3">&nbsp;to install PHP. Open the terminal and enter the following:</span></p><p class="c2"><span class="c19 c23 c24">brew install php</span></p><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">Once PHP is installed, verify that the version is 7.1 or newer. Open the terminal and enter the following to verify the version:</span></p><p class="c2"><span class="c19 c24">php --version</span></p><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">Next, install Composer. Composer is a dependency manager for PHP. It&rsquo;ll help to install and manage the required packages.</span></p><p class="c8"><span class="c15">To install Composer, </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://getcomposer.org/download/&amp;sa=D&amp;source=editors&amp;ust=1667206974865115&amp;usg=AOvVaw0ws1ZCdJRF_o1Uj1gNuSWv">visit this link</a></span><span class="c3">. Here you&rsquo;ll find the downloads and instructions.</span></p><p class="c8"><span class="c3">If you&rsquo;re using a package manager, the installation is easier. On macOS, run the following command to install Composer:</span></p><p class="c2"><span class="c19 c24">brew install composer</span></p><p class="c8"><span class="c3">On Windows, you can use Chocolatey:</span></p><p class="c2"><span class="c19 c23 c24">choco install composer</span></p><p class="c2 c7"><span class="c19 c23 c24"></span></p><p class="c8"><span class="c3">To verify the installation, run the following command:</span></p><p class="c2"><span class="c19 c23 c24">composer --version</span></p><p class="c2 c7"><span class="c19 c23 c24"></span></p><p class="c8"><span class="c3">The next step is to install the required libraries.</span></p><h2 class="c39" id="h.stnf5u1rl34a"><span class="c13 c30">Making an HTTP GET request</span></h2><p class="c8"><span class="c3">The first step of PHP web scraping is to load the page.</span></p><p class="c8"><span class="c15">In this tutorial, we&rsquo;ll be using </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1667206974866742&amp;usg=AOvVaw186hCJQCnH_tKlgRSBRVh1">books.toscrape.com</a></span><span class="c3">. The website is a dummy book store for practicing web scraping.</span></p><p class="c8"><span class="c15">When viewing a website in a browser, the browser sends an HTTP GET request to the web server as the first step. To send the HTTP GET request using PHP, the built-in function</span><span class="c19 c24">&nbsp;</span><span class="c19 c24">file_get_contents</span><span class="c3">&nbsp;can be used.</span></p><p class="c8"><span class="c3">This function can take a file path or a URL and return the contents as a string.</span></p><p class="c8"><span class="c15">Create a new file and save it as </span><span class="c19 c24">native.php</span><span class="c15">. Open this file in a code editor such as </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://code.visualstudio.com/&amp;sa=D&amp;source=editors&amp;ust=1667206974867463&amp;usg=AOvVaw3m2MX147bPi4EYjwxnIPeD">Visual Studio Code</a></span><span class="c3">. Enter the following lines of code to load the HTML page and print the HTML in the terminal:</span></p><p class="c8 c7"><span class="c3"></span></p><a id="t.352d5260c990ee61af1c26ebfaf33664a9359275"></a><a id="t.0"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c5">&lt;?</span><span class="c13 c0">php</span></p><p class="c11"><span class="c0">$html </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">file_get_contents</span><span class="c0 c4">(</span><span class="c0 c1">&#39;https://books.toscrape.com/&#39;</span><span class="c0 c4">);</span></p><p class="c2"><span class="c0">echo $html</span><span class="c0 c4">;</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">Execute this code from the terminal as follows:</span></p><a id="t.0d0a65929766575c9e0d66ddeb1e007b65eec60e"></a><a id="t.1"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0 c23 c24">php native.php</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">Upon executing this command, the entire HTML of the page will be printed.</span></p><p class="c8"><span class="c3">As of now, it&rsquo;s difficult to locate and extract specific information within the HTML.</span></p><p class="c8"><span class="c3">This is where various open-source, third-party libraries come into play.</span></p><h2 class="c39" id="h.y3kbh6k7nk37"><span class="c13 c30">Web scraping in PHP with Goutte</span></h2><p class="c8"><span class="c3">A wide selection of libraries is available for web scraping with PHP. In this tutorial, Goutte will be used as it&rsquo;s accessible, well-documented, and continuously updated. It&rsquo;s always a good idea to try the most popular solution. Usually, supporting content and preexisting advice are plentiful.</span></p><p class="c8"><span class="c3">Goutte can handle most static websites. For dynamic sites, let&rsquo;s use Symfony Panther.</span></p><p class="c8"><span class="c15">Goutte, pronounced </span><span class="c15 c38">goot</span><span class="c15">, is a wrapper around Symfony&rsquo;s components, such as </span><span class="c19 c6">BrowserKit</span><span class="c36 c37">,</span><span class="c19 c6">&nbsp;CssSelector</span><span class="c36 c37">,</span><span class="c19 c6">&nbsp;DomCrawler</span><span class="c36 c37">, and </span><span class="c19 c6">HTTPClient</span><span class="c17 c40 c37">.</span></p><p class="c8"><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://symfony.com/&amp;sa=D&amp;source=editors&amp;ust=1667206974871666&amp;usg=AOvVaw1UtAp8WPP21mF7trtZ8I54">Symfony</a></span><span class="c3">&nbsp;is a set of reusable PHP components. The components used by Goutte can be used directly. However, Goutte makes it easier to write the code.</span></p><p class="c8"><span class="c3">To install Goutte, create a directory where you intend to keep the source code. Navigate to the directory and enter these commands:</span></p><a id="t.032f33009e0af9aa291aabaa23f8f9995e563b2a"></a><a id="t.2"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">composer init </span><span class="c0 c5">--</span><span class="c0">no</span><span class="c0 c5">-</span><span class="c0">interaction </span><span class="c0 c5">--</span><span class="c0">require</span><span class="c0 c5">=</span><span class="c0 c1">&quot;php &gt;=7.1&quot;</span></p><p class="c11"><span class="c0">composer require fabpot</span><span class="c0 c5">/</span><span class="c13 c0">goutte</span></p><p class="c2"><span class="c13 c0">composer update</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c15">The first command will create the </span><span class="c19 c6">composer.json</span><span class="c19 c36">&nbsp;</span><span class="c15">file. The second command will add the entry for Goutte as well as download and install the required files. It&rsquo;ll also create the </span><span class="c19 c6">composer.lock</span><span class="c3">&nbsp;file.</span></p><p class="c8"><span class="c15">The </span><span class="c19 c6">composer update</span><span class="c19 c36">&nbsp;</span><span class="c3">command will ensure that all the files of the dependencies are up to date.</span></p><h3 class="c26" id="h.p4euilqha7ba"><span class="c16">Sending HTTP requests with Goutte</span></h3><p class="c8"><span class="c15">The most important class for PHP web scraping using Goutte is the</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">Client</span><span class="c19 c6">&nbsp;</span><span class="c3">that acts like a browser. The first step is to create an object of this class:</span></p><a id="t.6469e2edfde763dfccdb352bafcbdd75ed2414f1"></a><a id="t.3"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Client</span><span class="c0 c4">();</span></p></td></tr></table><p class="c8 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">This object can then be used to send a request. The method to send the request is conveniently called</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">request</span><span class="c3">. It takes two parameters &mdash; the HTTP method and the target URL, and returns an instance of the DOM crawler object:</span></p><a id="t.0c29bfdd934d74ac1dcfbb3caad357003ba5cc38"></a><a id="t.4"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">request</span><span class="c0 c4">(</span><span class="c0 c1">&#39;GET&#39;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&#39;https://books.toscrape.com&#39;</span><span class="c0 c4">);</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">This will send the GET request to the HTML page. To print the entire HTML of the page, we can call the </span><span class="c6 c19">html()</span><span class="c3">&nbsp;method.</span></p><p class="c8"><span class="c3">Putting together everything we&rsquo;ve built so far, this is how the code file looks like:</span></p><a id="t.98ab0ccc5ccc4e265d8f775902e9ad243233807d"></a><a id="t.5"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c5">&lt;?</span><span class="c13 c0">php</span></p><p class="c11"><span class="c0">require </span><span class="c0 c1">&#39;vendor/autoload.php&#39;</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">use Goutte\Client</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Client</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">$crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">request</span><span class="c0 c4">(</span><span class="c0 c1">&#39;GET&#39;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&#39;https://books.toscrape.com&#39;</span><span class="c0 c4">);</span></p><p class="c2"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">html</span><span class="c0 c4">();</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">Save this new PHP file as</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">books.php</span><span class="c3">&nbsp;and run it from the terminal. This will print the entire HTML:</span></p><a id="t.04cff84a8abb0bfb32f72e290a875169cd72300e"></a><a id="t.6"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">php books</span><span class="c0 c4">.</span><span class="c0">php</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">Next, we need a way to locate specific elements from the page.</span></p><h3 class="c26" id="h.y92z81c8xdla"><span class="c16">Locating HTML elements via CSS Selectors</span></h3><p class="c8"><span class="c15">Goutte uses the Symfony component</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">CssSelector</span><span class="c3">. It facilitates the use of CSS Selectors in locating HTML elements.</span></p><p class="c8"><span class="c15">The CSS Selector can be supplied to the </span><span class="c19 c6">filter</span><span class="c19 c6">&nbsp;</span><span class="c15">method. For example, to print the title of the page, enter the following line to the</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">books.php</span><span class="c19 c6">&nbsp;</span><span class="c3">file that we&rsquo;re working with:</span></p><a id="t.29b89988e0fd97452875acf04692c689f9a118f8"></a><a id="t.7"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;title&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">Note that</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">title</span><span class="c19 c6">&nbsp;</span><span class="c15">is the CSS Selector that selects the </span><span class="c19 c6">&lt;title&gt;</span><span class="c3">&nbsp;node from the HTML.</span></p><p class="c8"><span class="c15">Keep in mind that in this particular case,</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">text()</span><span class="c15">&nbsp;returns a text contained in the HTML element. In the earlier example, we&rsquo;ve used</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">html()</span><span class="c3">&nbsp;to return the entire HTML of the selected element.</span></p><p class="c8"><span class="c15">If you prefer to work with XPath, use the </span><span class="c19 c6">filterXPath()</span><span class="c3">&nbsp;method instead. The following line of code produces the same output:</span></p><a id="t.5c14a02816138fc746ebebb0c9b9724cc88f2e32"></a><a id="t.8"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//title&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">Now, let&rsquo;s move on to extracting the book titles and prices.</span></p><h3 class="c26" id="h.129vut1szvjm"><span class="c16">Extracting the elements</span></h3><p class="c2"><span class="c15">Open </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1667206974886184&amp;usg=AOvVaw1oRFir9AcgEgw6CtsG2qgI">https://books.toscrape.com</a></span><span class="c3">&nbsp;in Chrome, right-click on a book and select Inspect. Before we write the web scraping code, we need to analyze the HTML of our page first.</span></p><p class="c42"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 348.00px;"><img alt="" src="images/image2.png" style="width: 624.00px; height: 348.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c28"><span class="c23 c40 c4 c37">The books are located in the &lt;article&gt; tags</span></p><p class="c8"><span class="c15">Upon examining the HTML of the target web page, we can see that each book is contained in an </span><span class="c19 c6">article</span><span class="c15">&nbsp;tag, which has a</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">product_pod</span><span class="c15">&nbsp;class. Here, the CSS Selector would be</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">.product_pod</span><span class="c3">.</span></p><p class="c8"><span class="c15">In each</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">article</span><span class="c19 c6">&nbsp;</span><span class="c15">tag, the complete book title is located in the thumbnail image as an </span><span class="c19 c6">alt</span><span class="c15">&nbsp;attribute value. The CSS Selector for the book title would be</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">.image_container img</span><span class="c3">.</span></p><p class="c8"><span class="c15">Finally, the CSS Selector for the book price would be</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">.price_color</span><span class="c3">.</span></p><p class="c8"><span class="c15">To get all the titles and prices from this page, first, we need to locate the container and then run the </span><span class="c19 c6">each</span><span class="c3">&nbsp;loop.</span></p><p class="c8"><span class="c3">In this loop, an anonymous function will extract and print the title along with the price as follows:</span></p><a id="t.30b126a5030eff8d0dc809f8dd2840b6a8ffa3bf"></a><a id="t.9"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c6">scrapePage</span><span class="c0 c4">(</span><span class="c0">$url</span><span class="c0 c4">,</span><span class="c0">&nbsp;$client</span><span class="c0 c4">){</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">request</span><span class="c0 c4">(</span><span class="c0 c1">&#39;GET&#39;</span><span class="c0 c4">,</span><span class="c0">&nbsp;$url</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.product_pod&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $title </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.image_container img&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">attr</span><span class="c0 c4">(</span><span class="c0 c1">&#39;alt&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $price </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.price_color&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; echo $title </span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;-&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;$price </span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c25">PHP_EOL</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c4">});</span></p><p class="c2"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">The functionality of web data extraction was isolated in a function. The same function can be used for extracting data from different websites.</span></p><h3 class="c26" id="h.uy6sxkcfj8n4"><span class="c16">Handling pagination</span></h3><p class="c8"><span class="c3">At this point, your PHP web scraper is performing data extraction from only a single URL. In real-life web scraping scenarios, multiple pages would be involved.</span></p><p class="c8"><span class="c15">In this particular site, the pagination is controlled by a </span><span class="c15 c33">Next</span><span class="c15">&nbsp;link (button). The CSS Selector for the </span><span class="c15 c33">Next</span><span class="c15">&nbsp;link is </span><span class="c19 c6">.next &gt; a</span><span class="c3">.</span></p><p class="c8"><span class="c15">In the function</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">scrapePage</span><span class="c3">&nbsp;that we&rsquo;ve created earlier, add the following lines:</span></p><a id="t.3b310f549edf77e95726774c465c83fe75d94efb"></a><a id="t.10"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c32">try</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $next_page </span><span class="c0 c5">=</span><span class="c0">&nbsp;$crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.next &gt; a&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">attr</span><span class="c0 c4">(</span><span class="c0 c1">&#39;href&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0 c4">}</span><span class="c0">&nbsp;</span><span class="c0 c32">catch</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">InvalidArgumentException</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span><span class="c0">&nbsp;</span><span class="c0 c34">//Next page not found</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c32">return</span><span class="c0">&nbsp;</span><span class="c0 c32">null</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0 c4">}</span></p><p class="c2"><span class="c0 c32">return</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;https://books.toscrape.com/catalogue/&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;$next_page</span><span class="c0 c4">;</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">This code uses the CSS Selector to locate the </span><span class="c15 c33">Next</span><span class="c15">&nbsp;button and to extract the value of the </span><span class="c19 c6">href</span><span class="c19 c6">&nbsp;</span><span class="c15">attribute, returning the relative URL of the subsequent page. On the last page, this line of code will raise the </span><span class="c19 c6">InvalidArgumentException</span><span class="c3">.</span></p><p class="c8"><span class="c15">If the next page is found, this function will return its URL. Otherwise, it will return </span><span class="c19 c6">null</span><span class="c3">.</span></p><p class="c8"><span class="c3">From now on, you&rsquo;ll be initiating each scraping cycle with a different URL. This will make the conversion from a relative URL to an absolute one easier.</span></p><p class="c8"><span class="c15">Lastly, you can use a </span><span class="c19 c6">while</span><span class="c3">&nbsp;loop to call this function:</span></p><a id="t.3ef39ac03c7c0026fb3d868f9dfcc7d4e16e329c"></a><a id="t.11"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Client</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">$nextUrl </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;https://books.toscrape.com/catalogue/page-1.html&quot;</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0 c32">while</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$nextUrl</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $nextUrl </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">scrapePage</span><span class="c0 c4">(</span><span class="c0">$nextUrl</span><span class="c0 c4">,</span><span class="c0">&nbsp;$client</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0 c4">}</span></p><p class="c2"><span class="c0 c6">scrapePage</span><span class="c0 c4">(</span><span class="c0">$url</span><span class="c0 c4">,</span><span class="c0">&nbsp;$client</span><span class="c0 c4">);</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">The web scraping code is almost complete.</span></p><h3 class="c26" id="h.pagvmkuao5xv"><span class="c16">Writing data to a CSV file</span></h3><p class="c8"><span class="c15">The final step of the PHP web scraping process is to export the data to a storage. PHP&rsquo;s built-in </span><span class="c6 c27"><a class="c12" href="https://www.google.com/url?q=https://www.php.net/manual/en/function.fputcsv.php&amp;sa=D&amp;source=editors&amp;ust=1667206974956986&amp;usg=AOvVaw0GUrqznioLngWQo__GEoUO">fputcsv</a></span><span class="c3">&nbsp;function can be used to export the data to a CSV file.</span></p><p class="c8"><span class="c3">First, open the CSV file in write or append mode and store the file handle in a variable.</span></p><p class="c8"><span class="c15">Next, send the variable to the</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">scrapePage</span><span class="c15">&nbsp;function. Then, call the</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">fputcsv</span><span class="c3">&nbsp;function for each book to write the title and price in one row.</span></p><p class="c8"><span class="c15">Lastly, after the </span><span class="c19 c6">while</span><span class="c15">&nbsp;loop, close the file by calling</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">fclose</span><span class="c3">.</span></p><p class="c8"><span class="c3">The final code file will be as follows:</span></p><a id="t.4b0969ba6a0d86ceba3726ac8d7a6f1e3c022e07"></a><a id="t.12"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c6">scrapePage</span><span class="c0 c4">(</span><span class="c0">$url</span><span class="c0 c4">,</span><span class="c0">&nbsp;$client</span><span class="c0 c4">,</span><span class="c0">&nbsp;$file</span><span class="c0 c4">)</span></p><p class="c11"><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">request</span><span class="c0 c4">(</span><span class="c0 c1">&#39;GET&#39;</span><span class="c0 c4">,</span><span class="c0">&nbsp;$url</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.product_pod&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c6">use</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $title </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.image_container img&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">attr</span><span class="c0 c4">(</span><span class="c0 c1">&#39;alt&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $price </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.price_color&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c6">fputcsv</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c4">[</span><span class="c0">$title</span><span class="c0 c4">,</span><span class="c0">&nbsp;$price</span><span class="c0 c4">]);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">});</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c32">try</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $next_page </span><span class="c0 c5">=</span><span class="c0">&nbsp;$crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.next &gt; a&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">attr</span><span class="c0 c4">(</span><span class="c0 c1">&#39;href&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span><span class="c0">&nbsp;</span><span class="c0 c32">catch</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">InvalidArgumentException</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span><span class="c0">&nbsp;</span><span class="c0 c34">//Next page not found</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c32">return</span><span class="c0">&nbsp;</span><span class="c0 c32">null</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c32">return</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;https://books.toscrape.com/catalogue/&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;$next_page</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0 c4">}</span></p><p class="c11 c7"><span class="c13 c0"></span></p><p class="c11"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Client</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">$file </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">fopen</span><span class="c0 c4">(</span><span class="c0 c1">&quot;books.csv&quot;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;a&quot;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">$nextUrl </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;https://books.toscrape.com/catalogue/page-1.html&quot;</span><span class="c0 c4">;</span></p><p class="c11 c7"><span class="c13 c0"></span></p><p class="c11"><span class="c0 c32">while</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$nextUrl</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; echo </span><span class="c0 c1">&quot;&lt;h2&gt;&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;$nextUrl </span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;&lt;/h2&gt;&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c25">PHP_EOL</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $nextUrl </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">scrapePage</span><span class="c0 c4">(</span><span class="c0">$nextUrl</span><span class="c0 c4">,</span><span class="c0">&nbsp;$client</span><span class="c0 c4">,</span><span class="c0">&nbsp;$file</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0 c4">}</span></p><p class="c2"><span class="c0 c6">fclose</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">);</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">Run this file from the terminal:</span></p><a id="t.04cff84a8abb0bfb32f72e290a875169cd72300e"></a><a id="t.13"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">php books</span><span class="c0 c4">.</span><span class="c0">php</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">This will create a</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">books.csv</span><span class="c3">&nbsp;file with 1,000 rows of data.</span></p><h2 class="c39" id="h.pjjsvs7dfgjg"><span class="c13 c30">Web scraping with Guzzle, XML, and XPath</span></h2><p class="c8"><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://docs.guzzlephp.org/en/stable/&amp;sa=D&amp;source=editors&amp;ust=1667206974978458&amp;usg=AOvVaw3Si5D1c-EgjO_cvwX9nO_A">Guzzle</a></span><span class="c3">&nbsp;is a PHP library that sends HTTP requests to web pages in order to get a response. In other words, Guzzle is a PHP HTTP client that you can use to scrape data. Note that before working with a web page, you&rsquo;d need to understand two more concepts--XML and XPath.</span></p><p class="c8"><span class="c3">XML stands for eXtensible Markup Language. It&rsquo;ll be used to create files for storing structured data. These files can then be transmitted and the data constructed.</span></p><p class="c8"><span class="c3">There is the issue of reading &nbsp;XML files and this is where XPath comes into the picture.</span></p><p class="c8"><span class="c3">XPath stands for XML Path and is used for navigation and selecting XML nodes.</span></p><p class="c8"><span class="c3">HTML files are very similar to XML files. In some cases, you might need a parser to make adjustments to the minor differences and make HTML at least somewhat compliant with XML file standards. There are some parsers that can read even poorly formatted XML. </span></p><p class="c8"><span class="c3">In any case, the parsers will then make necessary HTML modifications so that you can work with XPath to query and navigate the HTML.</span></p><h3 class="c26" id="h.sf07e5eraenz"><span class="c16">Setting up a Guzzle Project</span></h3><p class="c8"><span class="c3">To install Guzzle, create a directory where you intend to keep the source code. Navigate to the directory and enter these commands:</span></p><a id="t.06dde1f20a8852a6640a3c1cf44b25cba665327b"></a><a id="t.14"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">composer init </span><span class="c0 c5">--</span><span class="c0">no</span><span class="c0 c5">-</span><span class="c0">interaction </span><span class="c0 c5">--</span><span class="c0">require</span><span class="c0 c5">=</span><span class="c0 c1">&quot;php &gt;=7.1&quot;</span></p><p class="c2"><span class="c0">composer require guzzlehttp</span><span class="c0 c5">/</span><span class="c13 c0">guzzle</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">In addition to Guzzle, let&rsquo;s also use a library for parsing HTML code. There are many PHP libraries available such as simple HTML dom parser and Symphony DOMCrawler. In this tutorial, Symphony DOMCrawler is chosen. Its syntax is very similar to Goutte, and you&rsquo;ll be able to apply what you already know in this section. Another point in favor of DomCrawler over the simple HTML dom parser is that it supports working with invalid HTML code very well. So, let&rsquo;s get going.</span></p><p class="c8"><span class="c3">Install DOMCrawler using the following command:</span></p><a id="t.42146308d31088759fc2bdcd46c8f6b0e818d835"></a><a id="t.15"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">composer require symfony</span><span class="c0 c5">/</span><span class="c0">dom</span><span class="c0 c5">-</span><span class="c0">crawler</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c15">These commands will download all the necessary files. The next step is to create a new file and save it as </span><span class="c19 c6">scraper.php</span><span class="c3">.</span></p><h3 class="c26" id="h.miii3sbq665s"><span class="c16">Sending HTTP requests with Guzzle</span></h3><p class="c8"><span class="c3">Similar to Goutte, the most important class of Guzzle is Client. Begin by creating a new file scraper.php and enter the following lines of PHP code:</span></p><a id="t.dece5ddf6cfcf1b9f2a4e7a902ff45ad62eafef5"></a><a id="t.16"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c5">&lt;?</span><span class="c13 c0">php</span></p><p class="c11"><span class="c0">require </span><span class="c0 c1">&#39;vendor/autoload.php&#39;</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">use GuzzleHttp\Client</span><span class="c0 c4">;</span></p><p class="c2"><span class="c0">use Symfony\Component\DomCrawler\Crawler</span><span class="c0 c4">;</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">Now we&rsquo;re ready to create an object of the Client class:</span></p><a id="t.6469e2edfde763dfccdb352bafcbdd75ed2414f1"></a><a id="t.17"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Client</span><span class="c0 c4">();</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">You can then use the client object to send a request. The method to send the request is conveniently called request. It takes two parameters &mdash; the HTTP method and the target URL, and returns a response:</span></p><a id="t.35df9a6bece4bbabbed77287ab4865fc9f366fae"></a><a id="t.18"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$response </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">request</span><span class="c0 c4">(</span><span class="c0 c1">&#39;GET&#39;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&#39;https://books.toscrape.com&#39;</span><span class="c0 c4">);</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">From this response, we can extract the web page&#39;s HTML as follows:</span></p><a id="t.0e595f53bbfbf6a2c2301fb4490ce163d54fc5d5"></a><a id="t.19"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$html </span><span class="c0 c5">=</span><span class="c0">&nbsp;$response</span><span class="c0 c5">-&gt;</span><span class="c0 c6">getBody</span><span class="c0 c4">()</span><span class="c0 c5">-&gt;</span><span class="c0 c6">getContents</span><span class="c0 c4">();</span></p><p class="c2"><span class="c13 c0">echo $html</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">Note that in this example, the response contains HTML code. If you&rsquo;re working with a web page that returns JSON, you can save the JSON to a file and stop the script. The next section will be applicable only if the response contains HTML or XML data.</span></p><p class="c8"><span class="c3">Continuing, the DomCrawler will be used to extract specific elements from this web page.</span></p><h3 class="c26" id="h.9vb731fd51le"><span class="c16">Locating HTML elements via XPath</span></h3><p class="c8"><span class="c3">Import the Crawler class and create an instance of the Crawler class as shown in the following PHP code snippet:</span></p><a id="t.5937501b993670eefca71788687006f756f266ea"></a><a id="t.20"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">use Symfony\Component\DomCrawler\Crawler</span><span class="c0 c4">;</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c3">We can create an instance of the crawler class as follows:</span></p><a id="t.f547719afbc4b8c03b228884d23ec7362c82a340"></a><a id="t.21"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c32">new</span><span class="c0">&nbsp;</span><span class="c0 c6">Crawler</span><span class="c0 c4">(</span><span class="c0">$html</span><span class="c0 c4">);</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">Now we can use the</span><span class="c0 c6">&nbsp;</span><span class="c0 c6">filterXPath</span><span class="c3">&nbsp;method to extract any XML node. For example, the following line prints only the title of the page:</span></p><a id="t.5c14a02816138fc746ebebb0c9b9724cc88f2e32"></a><a id="t.22"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//title&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">A quick note about XML Nodes: In XML, everything is a node-- an element is a node, an attribute is a node, and text is also a node. The </span><span class="c0 c6">filterXPath</span><span class="c15">&nbsp;method returns a node. So, to extract the text from an element, even if you use the </span><span class="c0 c6">text()</span><span class="c3">&nbsp;function in XPath, you still have to call the text() method to extract text as a string.</span></p><p class="c8"><span class="c3">In other words, both the following lines of code will return the same value:</span></p><a id="t.9b5b819d02045492217367fe5b17b929e54d3417"></a><a id="t.23"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//title&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c2"><span class="c0">echo $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//title/text()&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p></td></tr></table><p class="c2 c7"><span class="c4 c21"></span></p><p class="c8"><span class="c3">Now, let&#39;s move on to extracting the book titles and prices.</span></p><h3 class="c26" id="h.csl6xlw17krp"><span class="c16">Extracting the elements</span></h3><p class="c8"><span class="c3">Before writing web scraping code, let&rsquo;s start with analyzing the HTML of our page.</span></p><p class="c8"><span class="c15">Open the web page </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1667206974998023&amp;usg=AOvVaw27NJRv5B4zN5gGniX2h3Hb">https://books.toscrape.com</a></span><span class="c15">&nbsp;in Chrome, right-click on a book and select </span><span class="c15 c33">Inspect</span><span class="c3">. </span></p><p class="c8"><span class="c3">The books are located in &lt;article&gt; elements with the class attribute set to product_pod. The XPath to select these nodes will be as follows:</span></p><a id="t.f2c5396a61e5a2c8bc40cadb97ecd285dc852def"></a><a id="t.24"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0 c34">//*[@class=&quot;product_pod&quot;]</span></p></td></tr></table><p class="c2 c7"><span class="c21 c34"></span></p><p class="c8"><span class="c3">In each article tag, the complete book title is located in the thumbnail image as an alt attribute value. The XPath for book title and book price would be as follows:</span></p><a id="t.2fc2ae22f27fd72d6f4b2a4cad22a7f7b894d208"></a><a id="t.25"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c34">//*[@class=&quot;image_container&quot;]/a/img/@alt</span></p><p class="c2"><span class="c0 c23 c34">//*[@class=&quot;price_color&quot;]/text()</span></p></td></tr></table><p class="c2 c7"><span class="c21 c34"></span></p><p class="c8"><span class="c3">To get all of the titles and prices from this page, you first need to locate the container and then use a loop to get to each of the elements containing the data you need.</span></p><p class="c8"><span class="c3">In this loop, an anonymous function will extract and print the title along with the price, as shown in the following PHP code snippet:</span></p><a id="t.9d20c3a9b809279be80c9b24b6c4c3b68e0a5a44"></a><a id="t.26"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXpath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//*[@class=&quot;product_pod&quot;]&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">$title </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXpath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.//*[@class=&quot;image_container&quot;]/a/img/@alt&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">$price </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.//*[@class=&quot;price_color&quot;]/text()&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">echo $title </span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;-&quot;</span><span class="c0">&nbsp;</span><span class="c0 c4">.</span><span class="c0">&nbsp;$price </span><span class="c0 c4">.</span><span class="c0">&nbsp;</span><span class="c0 c25">PHP_EOL</span><span class="c0 c4">;</span></p><p class="c2"><span class="c0 c4">});</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">This was a simple demonstration of how you can scrape data from any page using Guzzle or DOMCrawler parsers. Note that this method won&rsquo;t work with a dynamic website. These websites use JavaScript code that cannot be handled by DOMCrawler. In cases like this, you&rsquo;ll need to use Symphony Panther.</span></p><p class="c8"><span class="c3">The next step after extracting data is to save it.</span></p><h3 class="c26" id="h.wr5t2yks4ey4"><span class="c16">Saving extracted data to a file</span></h3><p class="c8"><span class="c3">To store the extracted data, you can change the script to use the built-in PHP and create a CSV file. </span></p><p class="c8"><span class="c3">Write the following PHP code snippet as this:</span></p><a id="t.68331851184013eaf470765be84740ff0483247e"></a><a id="t.27"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$file </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">fopen</span><span class="c0 c4">(</span><span class="c0 c1">&quot;books.csv&quot;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;a&quot;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">$crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXpath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;//*[@class=&quot;product_pod&quot;]&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c6">use</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">$title </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXpath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.//*[@class=&quot;image_container&quot;]/a/img/@alt&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">$price </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filterXPath</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.//*[@class=&quot;price_color&quot;]/text()&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0 c6">fputcsv</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c4">[</span><span class="c0">$title</span><span class="c0 c4">,</span><span class="c0">&nbsp;$price</span><span class="c0 c4">]);</span></p><p class="c11"><span class="c0 c4">});</span></p><p class="c2"><span class="c0 c6">fclose</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">);</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c8"><span class="c15">This code snippet, when run, will save all the data to the</span><span class="c6 c29">&nbsp;</span><span class="c29 c6">books.csv</span><span class="c29 c6">&nbsp;</span><span class="c3">file.</span></p><h2 class="c39" id="h.5u6a2n37ofbx"><span class="c13 c30">Web scraping with Symfony Panther</span></h2><p class="c8"><span class="c3">Dynamic websites use JavaScript to render the contents. For such websites, Goutte wouldn&rsquo;t be a suitable option.</span></p><p class="c8"><span class="c15">For these websites, the solution is to employ a browser to render the page. It can be done using another component from Symfony &ndash; </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://github.com/symfony/panther&amp;sa=D&amp;source=editors&amp;ust=1667206975010183&amp;usg=AOvVaw2RX5x0USr8D4_sPkqbr4eU">Panther</a></span><span class="c3">. Panther is a standalone PHP library for web scraping using real browsers.</span></p><p class="c8"><span class="c15">In this section, let&rsquo;s scrape quotes and authors from </span><span class="c27 c6"><a class="c12" href="https://www.google.com/url?q=https://quotes.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1667206975010830&amp;usg=AOvVaw0q0fny1_MeneU6gCVBzG13">quotes.toscrape.com</a></span><span class="c3">. It&rsquo;s a dummy website for learning the basics of scraping dynamic web pages.</span></p><h3 class="c26" id="h.h76i5m3hrwoz"><span class="c16">Installing Panther and its dependencies</span></h3><p class="c8"><span class="c3">To install Panther, open the terminal, navigate to the directory where you&rsquo;ll be storing your source code, and run the following commands:</span></p><a id="t.1c3956283ceb82eb5e1f1f8da70e87a156dd0675"></a><a id="t.28"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">composer init </span><span class="c0 c5">--</span><span class="c0">no</span><span class="c0 c5">-</span><span class="c0">interaction </span><span class="c0 c5">--</span><span class="c0">require</span><span class="c0 c5">=</span><span class="c0 c1">&quot;php &gt;=7.1&quot;</span><span class="c13 c0">&nbsp;</span></p><p class="c11"><span class="c0">composer require symfony</span><span class="c0 c5">/</span><span class="c13 c0">panther</span></p><p class="c2"><span class="c13 c0">composer update</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c15">These commands will create a new </span><span class="c19 c6">composer.json</span><span class="c19 c36">&nbsp;</span><span class="c3">file and install Symfony/Panther.</span></p><p class="c8"><span class="c3">The other two dependencies are a browser and a driver. The common browser choices are Chrome and Firefox. The chances are that you already have one of these browsers installed.</span></p><p class="c8"><span class="c3">The driver for your browser can be downloaded using any of the package managers.</span></p><p class="c8"><span class="c3">On Windows, run:</span></p><a id="t.9e0700c0b20c28768f8a975b143822d7ff3d4a1c"></a><a id="t.29"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">choco install chromedriver</span></p></td></tr></table><p class="c2 c7"><span class="c21 c22"></span></p><p class="c8"><span class="c3">On macOS, run:</span></p><a id="t.87c48f029f6116cb06d0c7e61e4488f062c233e1"></a><a id="t.30"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">brew install chromedriver</span></p></td></tr></table><h3 class="c26" id="h.blo6xpq8wfk0"><span class="c16">Sending HTTP requests with Panther</span></h3><p class="c8"><span class="c15">Panther uses the </span><span class="c19 c6">Client</span><span class="c15">&nbsp;class to expose the </span><span class="c19 c6">get()</span><span class="c3">&nbsp;method. This method can be used to load URLs, or in other words, to send HTTP requests.</span></p><p class="c8"><span class="c15">The first step is to create the Chrome</span><span class="c19 c6">&nbsp;</span><span class="c19 c6">Client</span><span class="c3">. Create a new PHP file and enter the following lines of code:</span></p><a id="t.c853227808ead85345307691ef65cfd647c34f32"></a><a id="t.31"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c5">&lt;?</span><span class="c13 c0">php</span></p><p class="c11"><span class="c0">require </span><span class="c0 c1">&#39;vendor/autoload.php&#39;</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">use \Symfony\Component\Panther\Client</span><span class="c0 c4">;</span></p><p class="c2"><span class="c0">$client </span><span class="c0 c5">=</span><span class="c0">&nbsp;Client</span><span class="c0 c5">::</span><span class="c0 c6">createChromeClient</span><span class="c0 c4">();</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">The </span><span class="c0 c6">$client</span><span class="c3">&nbsp;object can then be used to load the web page:</span></p><a id="t.8f32660d0eb9c0f95b25e2e4cf559d706fc76b40"></a><a id="t.32"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">get</span><span class="c0 c4">(</span><span class="c0 c1">&#39;https://quotes.toscrape.com/js/&#39;</span><span class="c0 c4">);</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c3">This line will load the page in a headless Chrome browser.</span></p><h3 class="c26" id="h.cunfku60kulu"><span class="c16">Locating HTML elements via CSS Selectors</span></h3><p class="c8"><span class="c15">To locate the elements, first, you need to get a reference for the</span><span class="c0 c6">&nbsp;crawler</span><span class="c15">&nbsp;object. The best way to get an object is to wait for a specific element on a page using the </span><span class="c0 c6">waitFor()</span><span class="c3">&nbsp;method. It takes the CSS Selector as a parameter:</span></p><a id="t.bc23879d8c2605d8aeda523bee7f7467136024e6"></a><a id="t.33"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c2"><span class="c0">$crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">waitFor</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.quote&#39;</span><span class="c0 c4">);</span></p></td></tr></table><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">The code line waits for the element with this selector to become available and then returns an instance of the </span><span class="c0 c6">crawler</span><span class="c3">.</span></p><p class="c2"><span class="c15">The rest of the code is similar to Goutte&rsquo;s as both use the same </span><span class="c0 c6">CssSelector</span><span class="c3">&nbsp;component of Symfony.</span></p><p class="c42"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 366.67px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 366.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c28"><span class="c23 c4 c37 c40">The container HTML element of a quote</span></p><p class="c8"><span class="c15">First, the</span><span class="c0 c6">&nbsp;filter</span><span class="c3">&nbsp;method is supplied by the CSS Selector to get all of the quote elements. Then, the anonymous function is supplied to each quote to extract the author and the text:</span></p><a id="t.073fd035b7089125b8fbf2e6ae0323cad59742c6"></a><a id="t.34"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.quote&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $author </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.author&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $quote </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.text&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp;echo $autor</span><span class="c0 c4">.</span><span class="c0 c1">&quot; - &quot;</span><span class="c0 c4">.</span><span class="c13 c0">$quote</span></p><p class="c2"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">});</span></p></td></tr></table><h3 class="c26" id="h.6ob5wsr0d8hm"><span class="c16">Handling pagination</span></h3><p class="c8"><span class="c15">To scrape data from all of the subsequent pages of this website, you can simply click the </span><span class="c15 c33">Next</span><span class="c15">&nbsp;button. For clicking the links, the </span><span class="c19 c6">clickLink()</span><span class="c3">&nbsp;method can be used. This method works directly with the link text.</span></p><p class="c8"><span class="c3">On the last page, the link won&rsquo;t be present, and calling this method will throw an exception. This can be handled by using a try-catch block:</span></p><a id="t.127cbd45fb699c659bdb4aac858471fa303449f7"></a><a id="t.35"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0 c32">while</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0 c25">true</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">waitFor</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.quote&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c13 c0">&hellip;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c32">try</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">clickLink</span><span class="c0 c4">(</span><span class="c0 c1">&#39;Next&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span><span class="c0">&nbsp;</span><span class="c0 c32">catch</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">Exception</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c32">break</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span></p><p class="c2"><span class="c0 c4">}</span></p></td></tr></table><h3 class="c26" id="h.dr6pbxrnx4d6"><span class="c16">Writing data to a CSV file</span></h3><p class="c8"><span class="c15">Writing the data to CSV is straightforward when using PHP&rsquo;s </span><span class="c0 c6">fputcsv()</span><span class="c15">&nbsp;function. Open the CSV file before the </span><span class="c0 c6">while</span><span class="c15">&nbsp;loop, write every row using the </span><span class="c0 c6">fputcsv()</span><span class="c3">&nbsp;function, and close the file after the loop.</span></p><p class="c8"><span class="c3">Here&rsquo;s the final code:</span></p><a id="t.a8c07de118d6277e596d383b3fbfdf58cde43754"></a><a id="t.36"></a><table class="c14"><tr class="c20"><td class="c10" colspan="1" rowspan="1"><p class="c11"><span class="c0">$file </span><span class="c0 c5">=</span><span class="c0">&nbsp;</span><span class="c0 c6">fopen</span><span class="c0 c4">(</span><span class="c0 c1">&quot;quotes.csv&quot;</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c1">&quot;a&quot;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0 c32">while</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0 c25">true</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler </span><span class="c0 c5">=</span><span class="c0">&nbsp;$client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">waitFor</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.quote&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; $crawler</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.quote&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">each</span><span class="c0 c4">(</span><span class="c0 c32">function</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$node</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c6">use</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $author </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.author&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $quote </span><span class="c0 c5">=</span><span class="c0">&nbsp;$node</span><span class="c0 c5">-&gt;</span><span class="c0 c6">filter</span><span class="c0 c4">(</span><span class="c0 c1">&#39;.text&#39;</span><span class="c0 c4">)</span><span class="c0 c5">-&gt;</span><span class="c0 c6">text</span><span class="c0 c4">();</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c6">fputcsv</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">,</span><span class="c0">&nbsp;</span><span class="c0 c4">[</span><span class="c0">$author</span><span class="c0 c4">,</span><span class="c0">&nbsp;$quote</span><span class="c0 c4">]);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">});</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c32">try</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; $client</span><span class="c0 c5">-&gt;</span><span class="c0 c6">clickLink</span><span class="c0 c4">(</span><span class="c0 c1">&#39;Next&#39;</span><span class="c0 c4">);</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span><span class="c0">&nbsp;</span><span class="c0 c32">catch</span><span class="c0">&nbsp;</span><span class="c0 c4">(</span><span class="c0">Exception</span><span class="c0 c4">)</span><span class="c0">&nbsp;</span><span class="c0 c4">{</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0 c32">break</span><span class="c0 c4">;</span></p><p class="c11"><span class="c0">&nbsp; &nbsp; </span><span class="c0 c4">}</span></p><p class="c11"><span class="c0 c4">}</span></p><p class="c2"><span class="c0 c6">fclose</span><span class="c0 c4">(</span><span class="c0">$file</span><span class="c0 c4">);</span></p></td></tr></table><p class="c8 c7"><span class="c3"></span></p><p class="c2 c7"><span class="c21 c4"></span></p><p class="c8"><span class="c15">Once you execute the web scraper contained in this PHP script, you&rsquo;ll have a </span><span class="c0 c6">quotes.csv</span><span class="c3">&nbsp;file with all the quotes and authors ready for further analysis.</span></p><p class="c8"><span class="c27 c6"><a class="c12" href="https://github.com/oxylabs/web-scraping-php">Click here</a></span><span class="c3">&nbsp;and check out a repository on GitHub to find the complete code used in this article.</span></p><h2 class="c39" id="h.hpd0jbe2ldi6"><span class="c13 c30">Conclusion</span></h2><p class="c8"><span class="c3">You shouldn&rsquo;t run into major hiccups when using Goutte for most static web pages, as this popular library offers sufficient functionality and extensive documentation. However, if the typical HTML extraction methods aren&rsquo;t up to the task when dynamic elements come into play, then Symfony Panther is the right way to deal with more complicated loads.</span></p><p class="c8"><span class="c15">If you&rsquo;re working with a site developed using Laravel, Code Igniter, or just plain PHP, writing the web scraping part directly in PHP can be very useful, for example, when creating your own WordPress plugin. As PHP is also a scripting language, you can write web scraping code even when it&rsquo;s not meant to be deployed to a website.</span></p></body></html>