Avoid being blocked with puppeteer
Tags: learningtowebscra, puppeteer, javascript
Published: 2019-12-31 10:33:14

---
title: Avoid being blocked with puppeteer
published: true
date: 2019-12-30 07:25:18 UTC
tags: Learning to Web Scra,Puppeteer,Javascript,Typescript
canonical_url: https://javascriptwebscrapingguy.com/avoid-being-blocked-with-puppeteer/
---

[Demo code here](https://github.com/aarmora/avoid-being-blocked-with-puppeteer)

![](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-6.png)

One of the main questions I see on forums and reddit with regards to web scraping is…”how do I avoid being blocked?”. This is a problem that I certainly have had to address and the best solution to avoid being blocked is puppeteer and some of the great tools in [puppeteer-extra](https://github.com/berstend/puppeteer-extra). I also feel that it is important to mention how any web scraping should be done with care. While I feel that anything public is fine to web scrape, you shouldn’t be doing anything that puts undue burden on the target site. Feel free to take a look at the post I wrote on [ethical web scraping](https://javascriptwebscrapingguy.com/ethics-of-web-scraping/).

Officially this is going to be part of the [Learn to Web Scrape](https://javascriptwebscrapingguy.com/learn-to-web-scrape/) series but this isn’t targeted towards beginners. While I don’t feel it is very difficult to start using the puppeteer-extra plugins, I’m not going to go into the depth that a complete beginner to programming would need.

## To the trials!

![doctor who fun gif](https://media.giphy.com/media/gDCv3jkNYSgSY/giphy.gif)

We are going to use Zillow as a test target today. I have a simple bit of puppeteer code visiting a random address in Ohio on Zillow. I perform the action five times, waiting 1.5 seconds between each new attempt. Check the code:

```
	const browser = await puppeteer.launch({ headless: false });

	const url = 'https://www.zillow.com/homes/%0913905--ROYAL-BOULEVARD-cleveland-ohio_rb/33601155_zpid/';

	for (let i = 0; i < 5; i++) {
		const page = await browser.newPage();

		await page.goto(url);

		await page.waitFor(1500);

		await page.close();
	}

	await browser.close();
```

I was blocked on my third attempt. Zillow let me visit the page twice and then:

![puppeteer being blocked by zillow](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-7.png)

Ouch. That is some pretty impressive and swift blocking. I tried to add a humanish user agent.

`page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'); `

Two visits and then blocked again. Good for Zillow. I honestly applaud websites taking measures to slow down behavior they don’t want. The more friction there is, the less likely people are to want to deal with web scraping it.

## Stealth mode

![psych fun gif jackal switch](https://media.giphy.com/media/OmiRKhBnS8y1W/giphy.gif)

It’s time for the great stuff. [Berstend](https://github.com/berstend) has made some really powerful tools that come with something called puppeteer-extra. There is a large list of the tools [here](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra), with some cool ones like adblocker, flash, and….stealth.

It’s extremely easy to setup. We import the packages with require since there aren’t typescript definition files yet.

```
const puppeteerExtra = require('puppeteer-extra');
const pluginStealth = require('puppeteer-extra-plugin-stealth');
```

Then, we just setup puppeteer from puppeteer extra.

```
	puppeteerExtra.use(pluginStealth());
	const browser = await puppeteerExtra.launch({ headless: false });

	// Normal browser from normal puppeteer
	// const browser = await puppeteer.launch({ headless: false });

	const url = 'https://www.zillow.com/homes/%0913905--ROYAL-BOULEVARD-cleveland-ohio_rb/33601155_zpid/';

	for (let i = 0; i < 5; i++) {
		console.log('starting attempt:', i);
		const page = await browser.newPage();

		await page.goto(url);

		await page.waitFor(1500);

		await page.close();
	}

	await browser.close();
```

Now, back to Zillow. Out of my five attempts…none were blocked. Let’s try 20.

![avoiding getting blocked with puppeteer on zillow](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-8-1024x769.png)

20 atttempts. No recaptchas. That easy. It’s THE best package and tool I’ve seen to avoid getting blocked while web scraping with puppeteer or any package for that matter.

Now, let’s try with 100 attempts. Eventually Zillow catches the stealth plugin and throws a recaptcha.

![](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-9.png)

So, avoiding recaptchas entirely isn’t quite possible. Let’s talk about recaptchas.

## reCaptcha land

![fun recaptcha robot gif](https://media.giphy.com/media/Xv4UHRJLyVZZe/giphy.gif)

reCaptchas are tough to deal with but not impossible. Berstend comes to our rescue once again with [puppeteer-extra-plugin-recaptcha](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-recaptcha). The thing about reCaptchas, though, is that they can’t really be beat with pure automation. At least, I haven’t found a way.

How this plugin works is it leverages services that beat reCaptchas. One of these services is [2Captcha](http://2captcha.com/?from=7390140) (this is an affiliate link. But, I use this product myself and really like it. Easy to use, very inexpensive, and works great.). You have to pay to use it and the plugin uses this integration to beat reCaptchas. But it’s not a program doing it. It’s actual humans. As I did a little more investigation, it turns out 2Captcha hires people to break the reCaptchas.

![2Captcha hiring](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-10.png)

So what it does (or at least, what I assume it does) is send the reCaptcha to 2Captcha and then someone solves it immediately and sends back the completed token. Here’s the code to handle the reCaptcha:

```
	// Use the reCaptcha plugin
	puppeteerExtra.use(
		RecaptchaPlugin({
			provider: { id: '2captcha', token: process.env.captchaToken },
			visualFeedback: true // colorize reCAPTCHAs (violet = detected, green = solved)
		})
	);	

```

You’ll get your captchaToken from 2Captcha and place it there. In this package I’m using a `.env` file. I’ve included a `.sample.env` file to which you can add a token and just rename to `.env`.

```
                // Handle the reCaptcha
		await page.goto(url);

		try {
			await page.waitForSelector('.error-content-block', { timeout: 750 });

			await page.waitFor(5000);
			await (<any>page).solveRecaptchas();
			await Promise.all([
				page.waitForNavigation(),
				page.click('[type="submit"]')
			]);
			console.log('we found a recaptcha on attempt:', i);
		}
		catch (e) {
			console.log('no recaptcha found');
		}
```

Bam, this is all. Now when it pops up, it finds that the reCaptcha is there and then solves it. Easy. I was going to record a gif of it being solved but once I did it once it must have flagged my IP as good because it now hardly ever prompts me to solve reCaptchas. I started another 100 attempt check WITHOUT the stealth plugin and it didn’t prompt to solve a recaptcha until attempt number 75 and then it solved it and continued on.

![avoid being blocked with puppeteer and handling recaptchas](https://javascriptwebscrapingguy.com/wp-content/uploads/2019/12/image-11.png)

Pretty awesome, right?

## Conclusion

Star of the show is the [puppeteer-extra](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra). Combo that with its stealth plugins and its recaptcha plugin and [2Captcha](https://2captcha.com?from=7390140) and you can avoid, or handle, almost any blocking. Happy scraping!

[Demo code here](https://github.com/aarmora/avoid-being-blocked-with-puppeteer)

## Looking for business leads?

Using the techniques talked about here at [javascriptwebscrapingguy.com](https://javascriptwebscrapingguy.com), we’ve been able to launch a way to access awesome business leads. Learn more at [Cobalt Intelligence](https://cobaltintelligence.com)!

The post [Avoid being blocked with puppeteer](https://javascriptwebscrapingguy.com/avoid-being-blocked-with-puppeteer/) appeared first on [JavaScript Web Scraping Guy](https://javascriptwebscrapingguy.com).