Creating a High-Speed Blog Crawler with Scrapy
Published: 2020-12-15 11:52:44

Scrapy is one of the most accessible tools that you can use to scrape and also spider a website with effortless ease.

One of the most common applications of web scraping according to the patterns we see with many of our customers at Proxies API is scraping blog posts. Today lets look at how we can build a simple scraper to pull out and save blog posts from a blog like CopyBlogger.
First, we need to install scrapy if you haven't already.
Once installed, we will add a simple file with some barebones code.
Let's examine the code before we proceed.

The allowed_domains array restricts all further crawling to the domain paths specified here.

start_urls is the list of URLs to crawl. For us, in this example, we only need one URL.

The LOG_LEVEL settings make the scrapy output less verbose, so it is not confusing.

The def parse(self, response): function is called by scrapy after every successful URL crawl. Here is where we can write our code to extract the data we want.

Now let's see what we can write in the parse function...

For this, let's find the CSS patterns that we can use as selectors for finding the blog posts on this page.
When we inspect this in the Google Chrome inspect tool (right-click on the page in Chrome and click Inspect to bring it up), we can see that the article headlines are always inside an H2 tag with the CSS class entry-title. This is good enough for us. We can just select this using the CSS selector function.
This will give us the Headline. We also need the href in the 'a' which has the class entry-title-link, so we need to extract this as well.
So lets put this all together.
Let's save it as BlogCrawler.py and then run it with these parameters, which tells scrapy to disobey Robots.txt and also to simulate a web browser.
When you run, it should return.
Those are all the blog posts. Let's save them into files.
When you run it now, it will save all the blog posts into a file folder.

But if you look at it, there are more than 320 odd pages like this on CopyBlogger. We need a way to paginate through to them and fetch them all.
When we inspect this in the Google Chrome inspect tool, we can see that the link is inside an LI element with the CSS class pagination-next. This is good enough for us. We can just select this using the CSS selector function.
This will give us the text 'Next Page,' though. What we need is the href in the 'a' tag inside the LI tag. So we modify it.
The moment we have the URL, we can ask Scrapy to fetch the URL contents.
When you run it, it should download all the blog posts that were ever written on CopyBlogger onto your system.
Making the Crawler Faster

The above example does the job, but it does it only linearly. If you want to be able to spend less time waiting for the crawlers to finish, you can increase the speed of the crawler by optimizing the concurrency and other settings and also the order in which Scrapy downloads stuff. Here is what we are going to use.
The concurrency settings are limited to your system's capacity and also your network speeds. Here setting it to 10 means the spider will crawl ten times faster. We also change the depth-first setting that Scrapy defaults to a breadth-first setting.

They also make sure we dont get into some unexpected large file downloads by limiting the file size. The retry setting can be changed for slow websites.

When you run this, you will notice the same end results, but it will be much much faster.

Scaling Scrapy

The example above is ok for small scale web crawling projects. But if you try to scrape large quantities of data at high speeds, you will find that sooner or later, your access will be restricted. Web servers can tell you are a bot, so one of the things you can do is run the crawler impersonating a web browser. This is done by passing the user agent string to the Wikipedia web server, so it doesn't block you.
In more advanced implementations, you will need even to rotate this string, so Wikipedia can't tell it the same browser! Welcome to web scraping.

If we get a little bit more advanced, you will realize that Wikipedia can simply block your IP, ignoring all your other tricks. This is a bummer, and this is where most web crawling projects fail.

Overcoming IP Blocks

Investing in a private rotating proxy service like Proxies API can most of the time make the difference between a successful and headache-free web scraping project, which gets the job done consistently and one that never really works.

Plus, with the 1000 free API calls running an offer, you have almost nothing to lose by using our rotating proxy and comparing notes. It only takes one line of integration to its hardly disruptive.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

A simple API can access the whole thing like below in any programming language.