Web Scraping With Playwright: Tutorial for 2022
Tags: webscraping, playwright, tutorial, beginners
Published: 2022-11-03 07:48:55

<p class="c16"><span class="c13 c11">You most probably won&rsquo;t get surprised if we tell you that in recent years, the internet and its impact have grown tremendously. This can be attributed to the growth of the technologies that help create more user-friendly applications. Moreover, there is more and more automation at every step &ndash; from the development to the testing of web applications. </span></p><p class="c16"><span class="c13 c11">Having good tools to test web applications is crucial. Libraries such as Playwright help speed up processes by opening the web application in a browser and other user interactions such as clicking elements, typing text, and, of course, extracting public data from the web.</span></p><p class="c16"><span class="c13 c11">In this post, we&rsquo;ll explain everything you need to know about Playwright and how it can be used for automation and even web scraping.</span></p><h2 class="c23" id="h.b0a9ugv1v1te"><span class="c10 c26">What is Playwright?</span></h2><p class="c16"><span class="c13 c11">Playwright is a testing and automation framework that can automate web browser interactions. Simply put, you can write code that can open a browser. This means that all the web browser capabilities are available for use. The automation scripts can navigate to URLs, enter text, click buttons, extract text, etc. The most exciting feature of Playwright is that it can work with multiple pages at the same time, without getting blocked or having to wait for operations to complete in any of them.</span></p><p class="c16"><span class="c13 c11">It supports most browsers such as Google Chrome, Microsoft Edge using Chromium, Firefox. Safari is supported when using WebKit. In fact, cross-browser web automation is Playwright&rsquo;s strength. The same code can be efficiently executed for all the browsers. Moreover, Playwright supports various programming languages such as Node.js, Python, Java, and .NET. You can write the code that opens websites and interacts with them using any of these languages.</span></p><p class="c16"><span class="c14">Playwright&rsquo;s </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://playwright.dev/python/docs/intro&amp;sa=D&amp;source=editors&amp;ust=1667205600027209&amp;usg=AOvVaw3jpWcQwHSlXmLA1-AWx3h_">documentation</a></span><span class="c13 c11">&nbsp;is extensive. It covers everything from getting started to a detailed explanation about all the classes and methods.</span></p><h2 class="c23" id="h.6xxmwsykko8y"><span class="c10 c26">Support for proxies in Playwright</span></h2><p class="c16"><span class="c13 c11">Playwright supports the use of proxies. Before we explore this subject further, here is a quick code snippet showing how to start using a proxy with Chromium:</span></p><p class="c16"><span class="c13 c11">Node.js:</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;</span><span class="c0">{</span><span class="c2">&nbsp;chromium </span><span class="c0">}</span><span class="c2">&nbsp;</span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c5">require</span><span class="c0">(</span><span class="c2 c18">&#39;playwright&#39;</span><span class="c0">);</span><span class="c10 c2">&nbsp;&quot;</span></p><p class="c4"><span class="c2 c8">const</span><span class="c2">&nbsp;browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0 c11">();</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c13 c11">Python:</span></p><p class="c9"><span class="c2">from playwright</span><span class="c0">.</span><span class="c2">async_api </span><span class="c2 c8">import</span><span class="c10 c2">&nbsp;async_playwright</span></p><p class="c9"><span class="c2 c8">import</span><span class="c10 c2">&nbsp;asyncio</span></p><p class="c9"><span class="c2 c8">with</span><span class="c2">&nbsp;</span><span class="c2 c5">async_playwright</span><span class="c0">()</span><span class="c2">&nbsp;</span><span class="c2 c8">as</span><span class="c2">&nbsp;</span><span class="c2 c19">p</span><span class="c1">:</span></p><p class="c4"><span class="c2">browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;p</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0 c11">()</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c13 c11">This code needs only slight modifications to fully utilize proxies.</span></p><p class="c16"><span class="c14">In the case of Node.js, the launch function can accept an optional parameter of </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://playwright.dev/docs/api/class-browsertype/%23browser-type-launch&amp;sa=D&amp;source=editors&amp;ust=1667205600029717&amp;usg=AOvVaw00y23c7-GGLykOJOzqbsbl">LauchOptions</a></span><span class="c14">&nbsp;type. This </span><span class="c7 c5">LaunchOption</span><span class="c14">&nbsp;object can, in turn, send several other parameters, e.g., &nbsp;</span><span class="c7 c5">headless</span><span class="c14">. The other parameter needed is </span><span class="c7 c5">proxy</span><span class="c14">. This proxy is another object with properties such as </span><span class="c7 c5">server</span><span class="c14">, </span><span class="c7 c5">username</span><span class="c14">, </span><span class="c7 c5">password</span><span class="c13 c11">, etc. The first step is to create an object where these parameters can be specified.</span></p><p class="c9"><span class="c2 c34">// Node.js</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;launchOptions </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c19">proxy</span><span class="c1">:</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c19">server</span><span class="c1">:</span><span class="c2">&nbsp;</span><span class="c2 c19">123.123.123.123</span><span class="c1">:</span><span class="c2 c19">80</span><span class="c10 c2">&#39;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c0">},</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c19">headless</span><span class="c1">:</span><span class="c2">&nbsp;</span><span class="c2 c19">false</span></p><p class="c4"><span class="c0 c11">}</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c13 c11">The next step is to pass this object to the launch function:</span></p><p class="c4"><span class="c2 c8">const</span><span class="c2">&nbsp;browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">(</span><span class="c2">launchOptions</span><span class="c0 c11">);</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">In the case of Python, it&rsquo;s slightly different. There&rsquo;s no need to create an object of </span><span class="c7 c5">LaunchOptions</span><span class="c14">. Instead, all the values can be sent as separate parameters. Here&rsquo;s how the proxy dictionary will be sent:</span></p><p class="c9"><span class="c10 c2"># Python</span></p><p class="c9"><span class="c2">proxy_to_use </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c19">&#39;server&#39;</span><span class="c1">:</span><span class="c2">&nbsp;</span><span class="c2 c18">&#39;123.123.123.123:80&#39;</span></p><p class="c9"><span class="c0 c11">}</span></p><p class="c4"><span class="c2">browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;pw</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">(</span><span class="c2">proxy</span><span class="c1">=</span><span class="c2">proxy_to_use</span><span class="c0">,</span><span class="c2">&nbsp;headless</span><span class="c1">=</span><span class="c2">False</span><span class="c0 c11">)</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">When deciding on which proxy to use, it&rsquo;s best to use residential proxies as they don&rsquo;t leave a footprint and won&rsquo;t trigger any security alarms. For example, our own Oxylabs&rsquo; </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://oxylabs.io/products/residential-proxy-pool&amp;sa=D&amp;source=editors&amp;ust=1667205600034032&amp;usg=AOvVaw2jEjGHJ1GWGI63Uv9rXger">Residential Proxies</a></span><span class="c13 c11">&nbsp;can help you with an extensive and stable proxy network. You can access proxies in a specific country, state, or even a city. What&rsquo;s essential, you can integrate them easily with Playwright as well.</span></p><h2 class="c23" id="h.silc7ax8o5d7"><span class="c10 c26">Basic scraping with Playwright</span></h2><p class="c16"><span class="c13 c11">Let&rsquo;s move to another topic where we&rsquo;ll cover how to get started with Playwright using Node.js and Python. </span></p><p class="c16"><span class="c13 c11">If you&rsquo;re using Node.js, create a new project and install the Playwright library. This can be done using these two simple commands:</span></p><p class="c9"><span class="c2">npm init </span><span class="c1">-</span><span class="c10 c2">y</span></p><p class="c4"><span class="c10 c2">npm install playwright</span></p><p class="c4 c12"><span class="c10 c2"></span></p><p class="c16"><span class="c13 c11">A basic script that opens a dynamic page is as follows:</span></p><p class="c9 c12"><span class="c11 c2 c8"></span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;playwright </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c5">require</span><span class="c0">(</span><span class="c2 c18">&#39;playwright&#39;</span><span class="c0">);</span></p><p class="c9"><span class="c0">(</span><span class="c2 c8">async</span><span class="c2">&nbsp;</span><span class="c0">()</span><span class="c2">&nbsp;</span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;playwright</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">({</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c19">headless</span><span class="c1">:</span><span class="c2">&nbsp;</span><span class="c2 c19">false</span><span class="c2">&nbsp;</span><span class="c2 c34">// Show the browser. </span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c0">});</span></p><p class="c9"><span class="c10 c2">&nbsp; &nbsp; </span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;page </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">newPage</span><span class="c0">();</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">goto</span><span class="c0">(</span><span class="c2 c18">&#39;https://books.toscrape.com/&#39;</span><span class="c0">);</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">waitForTimeout</span><span class="c0">(</span><span class="c2 c19">1000</span><span class="c0">);</span><span class="c2">&nbsp;</span><span class="c2 c34">// wait for 1 seconds</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">close</span><span class="c0">();</span></p><p class="c4"><span class="c0 c11">})();</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">Let&rsquo;s take a look at the provided code &ndash; the first line of the code imports Playwright. Then, an instance of Chromium is launched. It allows the script to automate Chromium. Also, note that this script is running with a visible UI. We did it by passing </span><span class="c7 c5">headless:false</span><span class="c14">. Then, a new browser page is opened. After that, the </span><span class="c7 c5">page.goto</span><span class="c14">&nbsp;function navigates to the </span><span class="c14 c29">Books to Scrape </span><span class="c13 c11">web page. After that, there&rsquo;s a wait of 1 second to show the page to the end-user. Finally, the browser is closed.</span></p><p class="c16"><span class="c14">The same code can be written in Python easily. First, install Playwright using </span><span class="c7 c5">pip</span><span class="c13 c11">&nbsp;command:</span></p><p class="c4"><span class="c10 c2">pip install playwright</span></p><p class="c4 c12"><span class="c10 c2"></span></p><p class="c16"><span class="c13 c11">Note that Playwright supports two variations &ndash; synchronous and asynchronous. The following example uses the asynchronous API:</span></p><p class="c9"><span class="c2">from playwright</span><span class="c0">.</span><span class="c2">async_api </span><span class="c2 c8">import</span><span class="c10 c2">&nbsp;async_playwright</span></p><p class="c9"><span class="c2 c8">import</span><span class="c10 c2">&nbsp;asyncio</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c2 c8">async</span><span class="c2">&nbsp;def </span><span class="c2 c5">main</span><span class="c0">()</span><span class="c1">:</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">async</span><span class="c2">&nbsp;</span><span class="c2 c8">with</span><span class="c2">&nbsp;</span><span class="c2 c5">async_playwright</span><span class="c0">()</span><span class="c2">&nbsp;</span><span class="c2 c8">as</span><span class="c2">&nbsp;</span><span class="c2 c19">pw</span><span class="c1">:</span><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;pw</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">(</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; headless</span><span class="c1">=</span><span class="c10 c2">False &nbsp;# Show the browser</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; page </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">new_page</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">goto</span><span class="c0">(</span><span class="c2 c18">&#39;https://books.toscrape.com/&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c10 c2">&nbsp; &nbsp; &nbsp; &nbsp; # Data Extraction Code Here</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">wait_for_timeout</span><span class="c0">(</span><span class="c2 c19">1000</span><span class="c0">)</span><span class="c2">&nbsp; # Wait </span><span class="c2 c8">for</span><span class="c2">&nbsp;</span><span class="c2 c19">1</span><span class="c10 c2">&nbsp;second</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">close</span><span class="c0">()</span></p><p class="c9"><span class="c10 c2">&nbsp; &nbsp; &nbsp; &nbsp; </span></p><p class="c9"><span class="c2 c8">if</span><span class="c2">&nbsp;__name__ </span><span class="c1">==</span><span class="c2">&nbsp;</span><span class="c2 c18">&#39;__main__&#39;</span><span class="c1">:</span></p><p class="c4"><span class="c2">&nbsp; &nbsp; asyncio</span><span class="c0">.</span><span class="c2 c5">run</span><span class="c0">(</span><span class="c2 c5">main</span><span class="c0 c11">())</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">This code is similar to the Node.js code. The biggest difference is the use of </span><span class="c7 c5">asyncio</span><span class="c14">&nbsp;library. Another difference is that the function names change from </span><span class="c7 c5">camelCase</span><span class="c14">&nbsp;to </span><span class="c7 c5">snake_case</span><span class="c13 c11">. </span></p><p class="c16"><span class="c13 c11">If you want to create more than one browser context or want to have finer control, you can create a context object and create multiple pages in that context. This would open pages in new tabs:</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;context </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">newContext</span><span class="c0">();</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;page1 </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;context</span><span class="c0">.</span><span class="c2 c5">newPage</span><span class="c0">();</span></p><p class="c4"><span class="c2 c8">const</span><span class="c2">&nbsp;page2 </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;context</span><span class="c0">.</span><span class="c2 c5">newPage</span><span class="c0 c11">();</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">You may also want to handle page context in your code. It&rsquo;s possible to get the browser context that the page belongs to using the </span><span class="c7 c5">page.context()</span><span class="c13 c11">function.</span></p><h2 class="c23" id="h.6f9rf73fjbp2"><span class="c10 c26">Locating elements</span></h2><p class="c16"><span class="c14">To extract information from any element or to click any element, the first step is to locate the element. Playwright supports both </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://oxylabs.io/blog/xpath-vs-css&amp;sa=D&amp;source=editors&amp;ust=1667205600043412&amp;usg=AOvVaw3O8v0f98T4199vwOe7e1yJ">CSS and XPath</a></span><span class="c13 c11">&nbsp;selectors.</span></p><p class="c4"><span class="c14">This can be understood better with a practical example. Open </span><span class="c14 c29">https://books.toscrape.com/ </span><span class="c14">in Chrome. Right-click the first book and select inspect</span>
![web scraping with playwright](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jmf2pxo0bx01x6iodcm6.png)<span class="c13 c11">.</span></p><p class="c4 c12"><span class="c13 c11"></span></p><p class="c4"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 364.00px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 364.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c12 c33"><span class="c11 c2 c30"></span></p><p class="c16"><span class="c14">You can see that all the books are under the </span><span class="c7 c5">article</span><span class="c14">&nbsp;element, which has a class </span><span class="c7 c5">product_prod</span><span class="c13 c11">.</span></p><p class="c16"><span class="c14">To select all the books, you need to run a loop over all these </span><span class="c7 c5">article </span><span class="c14">elements. These </span><span class="c7 c5">article</span><span class="c13 c11">&nbsp;elements can be selected using the CSS selector:</span></p><p class="c4"><span class="c0">.</span><span class="c10 c2">product_pod</span></p><p class="c4 c12"><span class="c10 c2"></span></p><p class="c16"><span class="c13 c11">Similarly, the XPath selector would be as following:</span></p><p class="c4"><span class="c11 c2 c34">//*[@class=&quot;product_pod&quot;]</span></p><p class="c4 c12"><span class="c11 c2 c34"></span></p><p class="c16"><span class="c13 c11">To use these selectors, the most common functions are as following:</span></p><ul class="c37 lst-kix_ch9hd93qbj8i-0 start"><li class="c22 li-bullet-0"><span class="c14">$eval</span><span class="c7 c5">(selector, function)</span><span class="c13 c11">&nbsp;&ndash; selects the first element, sends the element to the function, and the result of the function is returned;</span></li><li class="c22 li-bullet-0"><span class="c14">$$eval</span><span class="c7 c5">(selector, function)</span><span class="c13 c11">&nbsp;&ndash; same as above, except that it selects all elements; </span></li><li class="c22 li-bullet-0"><span class="c7 c5">querySelector(selector)</span><span class="c13 c11">&nbsp;&ndash; returns the first element;</span></li><li class="c22 li-bullet-0"><span class="c7 c5">querySelectorAll(selector)</span><span class="c13 c11">&ndash; return all the elements.</span></li></ul><p class="c16"><span class="c13 c11">These methods will work correctly with both CSS and XPath Selectors. </span></p><h2 class="c23" id="h.7oto0n84i4ay"><span class="c10 c26">Scraping text</span></h2><p class="c16"><span class="c14">Continuing with the example of </span><span class="c14 c29">Books to Scrape</span><span class="c14">, after the page has been loaded, you can use a selector to extract all book containers using the </span><span class="c7 c5">$$eval</span><span class="c13 c11">&nbsp;function.</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;books </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">$$eval</span><span class="c0">(</span><span class="c2 c18">&#39;.product_pod&#39;</span><span class="c0">,</span><span class="c2">&nbsp;all_items </span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2 c34">// run a loop here</span></p><p class="c4"><span class="c0 c11">})</span></p><p class="c4 c12"><span class="c11 c20"></span></p><p class="c16"><span class="c13 c11">Now all the elements that contain book data can be extracted in a loop:</span></p><p class="c9"><span class="c2">all_items</span><span class="c0">.</span><span class="c2 c5">forEach</span><span class="c0">(</span><span class="c2">book </span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;name </span><span class="c1">=</span><span class="c2">&nbsp;book</span><span class="c0">.</span><span class="c2 c5">querySelector</span><span class="c0">(</span><span class="c2 c18">&#39;h3&#39;</span><span class="c0">).</span><span class="c2">innerText</span><span class="c0">;</span></p><p class="c4"><span class="c0 c11">})</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">Finally, the </span><span class="c7 c5">innerText</span><span class="c13 c11">&nbsp;attribute can be used to extract the data from each data point. Here&rsquo;s the complete code in Node.js:</span></p><p class="c9"><span class="c2 c8">const</span><span class="c2">&nbsp;playwright </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c5">require</span><span class="c0">(</span><span class="c2 c18">&#39;playwright&#39;</span><span class="c0">);</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c0">(</span><span class="c2 c8">async</span><span class="c2">&nbsp;</span><span class="c0">()</span><span class="c2">&nbsp;</span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;playwright</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">();</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;page </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">newPage</span><span class="c0">();</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">goto</span><span class="c0">(</span><span class="c2 c18">&#39;https://books.toscrape.com/&#39;</span><span class="c0">);</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;books </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">$$eval</span><span class="c0">(</span><span class="c2 c18">&#39;.product_pod&#39;</span><span class="c0">,</span><span class="c2">&nbsp;all_items </span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;data </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c0">[];</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; all_items</span><span class="c0">.</span><span class="c2 c5">forEach</span><span class="c0">(</span><span class="c2">book </span><span class="c1">=&gt;</span><span class="c2">&nbsp;</span><span class="c0">{</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;name </span><span class="c1">=</span><span class="c2">&nbsp;book</span><span class="c0">.</span><span class="c2 c5">querySelector</span><span class="c0">(</span><span class="c2 c18">&#39;h3&#39;</span><span class="c0">).</span><span class="c2">innerText</span><span class="c0">;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;price </span><span class="c1">=</span><span class="c2">&nbsp;book</span><span class="c0">.</span><span class="c2 c5">querySelector</span><span class="c0">(</span><span class="c2 c18">&#39;.price_color&#39;</span><span class="c0">).</span><span class="c2">innerText</span><span class="c0">;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">const</span><span class="c2">&nbsp;stock </span><span class="c1">=</span><span class="c2">&nbsp;book</span><span class="c0">.</span><span class="c2 c5">querySelector</span><span class="c0">(</span><span class="c2 c18">&#39;.availability&#39;</span><span class="c0">).</span><span class="c2">innerText</span><span class="c0">;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; data</span><span class="c0">.</span><span class="c2 c5">push</span><span class="c0">({</span><span class="c2">&nbsp;name</span><span class="c0">,</span><span class="c2">&nbsp;price</span><span class="c0">,</span><span class="c2">&nbsp;stock</span><span class="c0">});</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c0">});</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">return</span><span class="c2">&nbsp;data</span><span class="c0">;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c0">});</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; console</span><span class="c0">.</span><span class="c2 c5">log</span><span class="c0">(</span><span class="c2">books</span><span class="c0">);</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">close</span><span class="c0">();</span></p><p class="c4"><span class="c0 c11">})();</span></p><p class="c4 c12"><span class="c0 c11"></span></p><p class="c16"><span class="c14">The code in Python will be a bit different. Python has a function </span><span class="c7 c5">eval_on_selector</span><span class="c14">, which is similar to </span><span class="c7 c5">$eval</span><span class="c13 c11">&nbsp;of Node.js, but it&rsquo;s not suitable for this scenario. The reason is that the second parameter still needs to be JavaScript. This can be good in a certain scenario, but in this case, it will be much better to write the entire code in Python.</span></p><p class="c16"><span class="c14">It would be better to use </span><span class="c5 c7">query_selector</span><span class="c14">&nbsp;and </span><span class="c7 c5">query_selector_all</span><span class="c13 c11">&nbsp;which will return an element and a list of elements respectively.</span></p><p class="c9"><span class="c2">from playwright</span><span class="c0">.</span><span class="c2">async_api </span><span class="c2 c8">import</span><span class="c10 c2">&nbsp;async_playwright</span></p><p class="c9"><span class="c2 c8">import</span><span class="c10 c2">&nbsp;asyncio</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c2 c8">async</span><span class="c2">&nbsp;def </span><span class="c2 c5">main</span><span class="c0">()</span><span class="c1">:</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; </span><span class="c2 c8">async</span><span class="c2">&nbsp;</span><span class="c2 c8">with</span><span class="c2">&nbsp;</span><span class="c2 c5">async_playwright</span><span class="c0">()</span><span class="c2">&nbsp;</span><span class="c2 c8">as</span><span class="c2">&nbsp;</span><span class="c2 c19">pw</span><span class="c1">:</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; browser </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;pw</span><span class="c0">.</span><span class="c2">chromium</span><span class="c0">.</span><span class="c2 c5">launch</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; page </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">new_page</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">goto</span><span class="c0">(</span><span class="c2 c18">&#39;https://books.toscrape.com&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; all_items </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;page</span><span class="c0">.</span><span class="c2 c5">query_selector_all</span><span class="c0">(</span><span class="c2 c18">&#39;.product_pod&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; books </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c0">[]</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">for</span><span class="c2">&nbsp;item </span><span class="c2 c8">in</span><span class="c2">&nbsp;</span><span class="c2 c19">all_items</span><span class="c1">:</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; book </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c0">{}</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; name_el </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;item</span><span class="c0">.</span><span class="c2 c5">query_selector</span><span class="c0">(</span><span class="c2 c18">&#39;h3&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c0">[</span><span class="c2 c18">&#39;name&#39;</span><span class="c0">]</span><span class="c2">&nbsp;</span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;name_el</span><span class="c0">.</span><span class="c2 c5">inner_text</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; price_el </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;item</span><span class="c0">.</span><span class="c2 c5">query_selector</span><span class="c0">(</span><span class="c2 c18">&#39;.price_color&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c0">[</span><span class="c2 c18">&#39;price&#39;</span><span class="c0">]</span><span class="c2">&nbsp;</span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;price_el</span><span class="c0">.</span><span class="c2 c5">inner_text</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; stock_el </span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;item</span><span class="c0">.</span><span class="c2 c5">query_selector</span><span class="c0">(</span><span class="c2 c18">&#39;.availability&#39;</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c0">[</span><span class="c2 c18">&#39;stock&#39;</span><span class="c0">]</span><span class="c2">&nbsp;</span><span class="c1">=</span><span class="c2">&nbsp;</span><span class="c2 c8">await</span><span class="c2">&nbsp;stock_el</span><span class="c0">.</span><span class="c2 c5">inner_text</span><span class="c0">()</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; books</span><span class="c0">.</span><span class="c2 c5">append</span><span class="c0">(</span><span class="c2">book</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c5">print</span><span class="c0">(</span><span class="c2">books</span><span class="c0">)</span></p><p class="c9"><span class="c2">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c2 c8">await</span><span class="c2">&nbsp;browser</span><span class="c0">.</span><span class="c2 c5">close</span><span class="c0">()</span></p><p class="c9"><span class="c10 c2">&nbsp;</span></p><p class="c9"><span class="c2 c8">if</span><span class="c2">&nbsp;__name__ </span><span class="c1">==</span><span class="c2">&nbsp;</span><span class="c2 c18">&#39;__main__&#39;</span><span class="c1">:</span></p><p class="c4"><span class="c2">&nbsp; &nbsp; asyncio</span><span class="c0">.</span><span class="c2 c5">run</span><span class="c0">(</span><span class="c2 c5">main</span><span class="c0 c11">())</span></p><p class="c4 c12"><span class="c20 c11"></span></p><p class="c16"><span class="c14">The output of both the Node.js and the Python code will be the same. You can </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://github.com/oxylabs/playwright-web-scraping&amp;sa=D&amp;source=editors&amp;ust=1667205600061019&amp;usg=AOvVaw3_gV49Bpx6lo0yUuYZ2wDk">click here</a></span><span class="c13 c11">&nbsp;to find the complete code used in this post for your convenience.</span></p><h2 class="c23" id="h.2hwql2g3xiia"><span class="c10 c26">Playwright vs Puppeteer and Selenium</span></h2><p class="c16"><span class="c13 c11">There are other tools like Selenium and Puppeteer that can also do the same thing as Playwright. </span></p><p class="c16"><span class="c13 c11">However, Puppeteer is limited when it comes to browsers and programming languages. The only language that can be used is JavaScript, and the only browser that works with it is Chromium. </span></p><p class="c16"><span class="c13 c11">Selenium, on the other hand, supports all major browsers and a lot of programming languages. It is, however, slow and less developer-friendly. </span></p><p class="c16"><span class="c14">Also note that Playwright can intercept network requests. For more details about network requests, see </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://playwright.dev/docs/network/&amp;sa=D&amp;source=editors&amp;ust=1667205600062100&amp;usg=AOvVaw2cvQsULjBcfU2-T0jQ-Pcs">this page</a></span><span class="c11 c13">.</span></p><p class="c4"><span class="c13 c11">The following table is a quick summary of the differences and similarities:</span></p><a id="t.0"></a><table class="c41"><tr class="c42"><td class="c15" colspan="1" rowspan="1"><p class="c4 c12"><span class="c13 c11"></span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">PLAYWRIGHT</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">PUPPETEER</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">SELENIUM</span></p></td></tr><tr class="c27"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">SPEED</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Fast</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Fast</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Slower</span></p></td></tr><tr class="c27"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">DOCUMENTATION</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Excellent</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Excellent</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Fair</span></p></td></tr><tr class="c6"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">DEVELOPER EXPERIENCE</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Best</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Good</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Fair</span></p></td></tr><tr class="c6"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">PROGRAMMING LANGUAGES</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">JavaScript, Python, C#, Java</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">JavaScript</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Java, Python, C#, Ruby</span></p><p class="c4"><span class="c13 c11">JavaScript, Kotlin</span></p></td></tr><tr class="c6"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">BACKED BY</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Microsoft</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Google</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Community and Sponsors</span></p></td></tr><tr class="c6"><td class="c15" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">COMMUNITY</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Small but active</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Large and active</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Large and active</span></p></td></tr><tr class="c40"><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">BROWSER SUPPORT</span></p></td><td class="c21" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Chromium, Firefox, and WebKit</span></p></td><td class="c31" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Chromium</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c4"><span class="c13 c11">Chrome, Firefox, IE, Edge, Opera, Safari, and more</span></p></td></tr></table><h2 class="c23" id="h.uqplxw6w4cz4"><span class="c10 c26">Comparison of performance</span></h2><p class="c16"><span class="c13 c11">As we mentioned in the previous section, because of the vast difference in the programming languages and supported browsers, it isn&rsquo;t easy to compare every scenario.</span></p><p class="c16"><span class="c13 c11">The only combination that can be compared is when scripts are written in JavaScript to automate Chromium. This is the only combination that all three tools support.</span></p><p class="c16"><span class="c14">A detailed comparison would be out of the scope of this post. You can read more about the performance of Puppeteer, Selenium, and Playwright in this </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/%23conclusion&amp;sa=D&amp;source=editors&amp;ust=1667205600071101&amp;usg=AOvVaw3LD9mKamJme04aL_6Zdy5P">article</a></span><span class="c13 c11">. The key takeaway is that Puppeteer is the fastest, followed by Playwright. Note that in some scenarios, Playwright was faster. Selenium is the slowest of the three.</span></p><p class="c16"><span class="c13 c11">Again, remember that Playwright has other advantages, such as multi-browser support, supporting multiple programming languages. </span></p><p class="c16"><span class="c13 c11">If you&rsquo;re looking for a fast cross-browser web automation or don&rsquo;t know JavaScript, Playwright will be your only choice.</span></p><h2 class="c23" id="h.rv82gxjx4u5i"><span class="c10 c26">Conclusion</span></h2><p class="c16"><span class="c14">In today&rsquo;s post, we</span><span class="c13 c11">&nbsp;explored the capabilities of Playwright as a web testing tool that can be used for web scraping dynamic sites. Due to its asynchronous nature and cross-browser support, it&rsquo;s a popular alternative to other tools. We also covered code examples in both Node.js and Python. </span></p><p class="c16"><span class="c13 c11">Playwright can help navigate to URLs, enter text, click buttons, extract text, etc. Most importantly, it can extract text that is rendered dynamically. These things can also be done by other tools such as Puppeteer and Selenium, but if you need to work with multiple browsers, or have to work with language other than JavaScript/Node.js, then Playwright would be a great choice.</span></p><p class="c4"><span class="c14">If you&rsquo;re interested to read more about other similar topics, check out our blog posts on </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://oxylabs.io/blog/selenium-web-scraping&amp;sa=D&amp;source=editors&amp;ust=1667205600072002&amp;usg=AOvVaw1rAaC9hRXkp1v1MMY5Bgl0">web scraping with Selenium</a></span><span class="c14">&nbsp;or </span><span class="c3"><a class="c24" href="https://www.google.com/url?q=https://oxylabs.io/blog/puppeteer-tutorial&amp;sa=D&amp;source=editors&amp;ust=1667205600072163&amp;usg=AOvVaw2ZkhyzQ5eWj-Y6ow24Hmf4">Puppeteer tutorial</a></span><span class="c13 c11">. </span></p><p class="c4 c12"><span class="c13 c11"></span></p><p class="c4"><span class="c14">And, of course, in case you have any questions or impressions about today&rsquo;s tutorial, don&rsquo;t hesitate to leave a comment below!</span></p></body></html>