Puppeteer on AWS Lambda
Tags: beginners, tutorial, aws, webscraping
Published: 2022-10-06 07:21:08

<p class="c13"><span class="c6">Wondering how you can get Puppeteer to work properly on AWS Lambda? </span></p><p class="c13 c18"><span class="c6"></span></p><p class="c13"><span>You&rsquo;re in the right place! In this post, we&rsquo;ll cover the main challenges you can encounter while trying to do that. </span><span class="c6 c17">But first, let&rsquo;s start with introducing both Puppeteer and AWS Lambda.</span></p><h2 class="c16" id="h.v5rs5wi10t1b"><span class="c24 c22">What is Puppeteer?</span></h2><p class="c0"><span class="c1">Simply put, </span><span class="c31"><a class="c10" href="https://www.google.com/url?q=https://pptr.dev/&amp;sa=D&amp;source=editors&amp;ust=1665042820449299&amp;usg=AOvVaw3ikzBNURuEiBJkg9Y4fF9F">Puppeteer</a></span><span class="c8 c1">&nbsp;is a software for controlling a (headless) browser. It&rsquo;s a piece of open-source software developed and supported by Google&rsquo;s developer tools team. It allows you to simulate user interaction with a browser through a simple API. This is very helpful for doing things like automated tests or web scraping.</span></p><p class="c13"><span class="c1">A picture&rsquo;s worth a thousand words. How much is a </span><span class="c1 c20">gif</span><span class="c1">&nbsp;worth? With a little bit of code shown in the </span><span class="c1 c20">gif </span><span class="c1">below, I can log in to a Google account. You simply need to click, enter text, paginate, and scrape all the publicly available data you need.</span><sup> ![Puppeteer on AWS Lambda tutorial](https://images.prismic.io/oxylabs-sm/MWU3MDdmNjAtNmZmOS00M2E5LWIxNDUtOTdlZGM0MTQ2NjNl_puppeeteer.gif?auto=compress,format&rect=0,0,1417,1328&w=1417&h=1328&fm=webp&q=75)</sup></p><h2 class="c16" id="h.hmepr5577djc"><span class="c24 c22">What is AWS Lambda?</span></h2><p class="c0"><span class="c31"><a class="c10" href="https://www.google.com/url?q=https://aws.amazon.com/lambda/&amp;sa=D&amp;source=editors&amp;ust=1665042820450070&amp;usg=AOvVaw1m5OWi96eDa2fUqXBIVLaU">AWS Lambda</a></span><span class="c8 c1">&nbsp;is what Amazon calls &ldquo;Run code without thinking about servers or clusters.&rdquo; You can simply create a function on Lambda and then execute it. It&rsquo;s that easy.</span></p><p class="c0"><span class="c8 c1">Simply put, you can do everything on AWS Lambda. Okay, everything is a strong word, but almost. For example, it is possible to scrape thousands of public web pages every night with AWS Lambda functions. Also, it manages to insert data into databases. </span></p><p class="c0"><span class="c8 c1">Getting started with AWS Lambda is simple and inexpensive. You only need to pay for what you use, and they also have a generous free trial.</span></p><h2 class="c11" id="h.hmtwuhy0r2d3"><span class="c24 c22">Problem #1 &ndash; Puppeteer is too big to push to Lambda</span></h2><p class="c0"><span class="c1">AWS Lambda has a 50 MB limit on the zip file you push directly to it. Due to the fact that it installs Chromium, the </span><span class="c31"><a class="c10" href="https://www.google.com/url?q=https://github.com/puppeteer/puppeteer&amp;sa=D&amp;source=editors&amp;ust=1665042820450812&amp;usg=AOvVaw3kjW6mHWn65Yc7vZtNH2DG">Puppeteer package</a></span><span class="c1">&nbsp;is significantly larger than that. However, this 50 MB limit doesn&rsquo;t apply when you load the function from S3! See the documentation </span><span class="c31"><a class="c10" href="https://www.google.com/url?q=https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html&amp;sa=D&amp;source=editors&amp;ust=1665042820451125&amp;usg=AOvVaw32_bH8lGHA_QQsSe7cP8u6">here</a></span><span class="c8 c1">.</span></p><p class="c13"><span class="c1 c20">AWS Lambda quotas can be tight for Puppeteer</span>
![Puppeteer lambda](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnsc2z8cttp84im0lgl0.png)<span class="c8 c1"></span></p><p class="c13 c18"><span class="c8 c1"></span></p><p class="c13"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 133.33px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 133.33px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c13"><span class="c1 c17">The 250 MB unzipped can be bypassed by uploading directly from an S3 bucket. So we create a bucket in S3, use a node script to upload to S3, and then update our Lambda code from that bucket. The script looks something like this:</span></p><p class="c13 c18"><span class="c6"></span></p><p class="c13"><span class="c9">&quot;zip&quot;</span><span class="c12">:</span><span class="c7 c1">&nbsp;</span><span class="c23">&quot;npm run build &amp;&amp; 7z a -r function.zip ./dist/* &nbsp;node_modules/&quot;</span><span class="c4">,</span></p><p class="c13"><span class="c9">&quot;sendToLambda&quot;</span><span class="c12">:</span><span class="c7 c1">&nbsp;</span><span class="c23 c22">&quot;npm run zip &amp;&amp; aws s3 cp function.zip s3://chrome-aws &amp;&amp; rm function.zip &amp;&amp; aws lambda update-function-code --function-name puppeteer-examples --s3-bucket chrome-aws --s3-key function.zip&quot;</span></p><h2 class="c16" id="h.o8l4acnki3ey"><span class="c22 c24">Puppeteer on AWS Lambda doesn&rsquo;t work</span></h2><p class="c0"><span class="c8 c1">By default, Linux (including AWS Lambda) doesn&rsquo;t include the necessary libraries required to allow Puppeteer to function.</span></p><p class="c0"><span class="c1">Fortunately, there already exists a package of Chromium built for AWS Lambda. You can find it </span><span class="c31"><a class="c10" href="https://www.google.com/url?q=https://www.npmjs.com/package/chrome-aws-lambda&amp;sa=D&amp;source=editors&amp;ust=1665042820452644&amp;usg=AOvVaw36wXCj0o79esfsCTQ1qQWd">here</a></span><span class="c1">. You will need to install it and </span><span class="c1 c20">puppeteer-core</span><span class="c8 c1">&nbsp;in your function that you are sending to Lambda. </span></p><p class="c0"><span class="c8 c1">The regular Puppeteer package will not be needed and, in fact, counts against your 250 MB limit.</span></p><p class="c0"><span class="c23 c22">`npm i --save chrome-aws-lambda puppeteer-core`</span></p><p class="c0"><span class="c8 c1">And then, when you are setting it up to launch a browser from Puppeteer, it will look like this:</span></p><p class="c0"><span class="c19">const</span><span class="c7 c1">&nbsp;browser </span><span class="c12">=</span><span class="c7 c1">&nbsp;</span><span class="c19">await</span><span class="c7 c1">&nbsp;chromium</span><span class="c4">.</span><span class="c7 c22 c1">puppeteer</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4">.</span><span class="c2">launch</span><span class="c4">({</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">args</span><span class="c12">:</span><span class="c7 c1">&nbsp;chromium</span><span class="c4">.</span><span class="c7 c1">args</span><span class="c4">,</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">defaultViewport</span><span class="c12">:</span><span class="c7 c1">&nbsp;chromium</span><span class="c4">.</span><span class="c7 c1">defaultViewport</span><span class="c4">,</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">executablePath</span><span class="c12">:</span><span class="c7 c1">&nbsp;</span><span class="c19">await</span><span class="c7 c1">&nbsp;chromium</span><span class="c4">.</span><span class="c7 c1">executablePath</span><span class="c4">,</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c9">headless</span><span class="c12">:</span><span class="c1 c7">&nbsp;chromium</span><span class="c4">.</span><span class="c7 c22 c1">headless</span></p><p class="c0"><span class="c7 c1">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4 c22">});</span></p><h2 class="c11" id="h.cuonvbfwxtfq"><span class="c24 c22">Final note</span></h2><p class="c13"><span class="c1">Puppeteer requires more memory than a regular script, so keep an eye on your max memory usage. When using Puppeteer, we recommend at least 512 MB on your AWS Lambda function. Also, don&rsquo;t forget to run </span><span class="c21">`await browser.close()`</span><span class="c8 c1">&nbsp;at the end of your script. Otherwise, you may end up with your function running until timeout for no reason because the browser is still alive and waiting for commands.</span>