How to Scrape the Web using Python with ScraPy Spiders
Tags: programming, python, scrapy, webscraping
Published: 2019-04-14 07:35:29

---
title: How to Scrape the Web using Python with ScraPy Spiders
published: true
tags: Programming,python,scrapy,web scraping
canonical_url: https://www.datastuff.tech/programming/how-to-scrape-the-web-using-python-with-scrapy-spiders/
---

Sometimes Kaggle is not enough, and you need to generate your own data set.

<meta name="twitter:card" content="summary"><meta name="twitter:site" content="@strikingloo"><meta name="twitter:title" content="How to Scrape the Web using Python with ScraPy Spiders"><meta name="twitter:description" content="Sometimes Kaggle is not enough, and you need to generate your own data set."><meta name="twitter:image" content="http://www.datastuff.tech/wp-content/uploads/2019/03/spider-564685_1920-1038x576.jpg">

Maybe you need pictures of spiders for this crazy Convolutional Neural Network you’re training, or maybe you want to scrape the NSFW subreddits for, um, scientific purposes.   
Whatever your reasons, scraping the web can give you very interesting data, and help you compile awesome data sets.

In this article we’ll use ScraPy to scrape a Reddit subreddit and get pictures.  
Some will tell me [using Reddit’s API](https://github.com/praw-dev/praw) is a much more practical method to get their data, and that’s strictly true. So true, I’ll probably write an article about it soon.   
But as long as we do it in a very small dose, and don’t overwork Reddit’s busy servers, it should be alright. So keep in mind, this tutorial is for educational purposes only, and if you ever need Reddit’s data you should use the official channels, like their [awesome API](https://github.com/praw-dev/praw).

So how do we go about Scraping a website? Let’s start from the beginning.

### Checking the robots.txt

First we’ll go into [reddit.com/robots.txt](http://reddit.com/robots.txt). It’s customary for a site to make their _robots.txt_ file accessible from their main domain. It respects the following format:

```
User-agent: \<pattern\>
Disallow: \<patterns\>
```

Where U_ser-agent_ describes a type of device (we fall in \*, the wildcard pattern), and _Disallow_ points to a list of url-patterns we can’t crawl.   
I don’t see /r/\* in there, so I _think_ it’s ok to scrape a subreddit’s main page.  
I’d still advise you to use the API for any serious project, as a matter of etiquette.

Not respecting a site’s _robots.txt_ file may have legal ramifications, but it mainly just makes you look like a mean person, and we don’t want that.

### Setting up our Project.

In order to scrape a website in Python, we’ll use ScraPy, its main scraping framework. Some people prefer BeautifulSoup, but I find ScraPy to be more dynamic.

ScraPy’s basic units for scraping are called _spiders, _and we’ll start off this program by creating an empty one.

So, first of all, we’ll install ScraPy:

```
pip install --user scrapy
```

And then we’ll start a ScraPy project:

```
scrapy startproject project\_name
```

Here you can enter anything instead of project\_name. What this command will do is create a directory with a lot of files and python scripts in it.

Now for our last initialization command, we’ll create our first spider. To do that we’ll run scrapy’s _genspider_ command, which takes a spider’s _name_ and a_domain url_ as its arguments.  
 I’ll name mine _kitten-getter _(beware: spoilers) and crawl _reddit.com/r/cats_.

```
scrapy genspider kitten\_getter reddit.com/r/cats
```

Now we’ll just go into the /_spiders_ directory and not focus in the rest. As always, I’ve made my code available in [this GitHub project](https://github.com/StrikingLoo/kitten-getter).

### Setting up our first spider

In the _spiders_ directory, we’ll open the file called _kitten\_getter.py _and paste this code:

{% gist https://gist.github.com/StrikingLoo/0e70d7018df5913ce11e21893493c0eb.js %}

What’s happening here? Well, each spider needs 3 things: a _parse_ method, a _start\_requests_ method, and a _name_.

- The spider’s _name_ will be used whenever we start the spider from the console.
- Running the spider from the console will make it start from the _start\_requests_ routine.
- We make the routine do _http_ requests on a list of urls, and call our _parse_method on their _http_ responses.

In order to run this, all we have to do is open our terminal in the project’s directory and run:

```
scrapy crawl kitten\_getter
```

To set your spiders free! Let them roam the web, snatching its precious data.

If you run that command, it will run the spider we just wrote, so it’ll make a request, get the HTML for the first _url_ in the _url\_list_ we supplied, and parse it the way we asked it to. In this case, all we’re doing is writing the whole response straight into a (~140Kb in size) file called ‘kitten\_response0’.

If you open it, you’ll see it’s just the HTML code for the website we scraped. This’ll come in handy for our next goal.

### Identifying patterns

If you go to the link [reddit.com/r/cats](http://reddit.com/r/cats) with the intention of scraping the subreddit for kitten pictures, you’ll notice there are two kinds of posts.

- Posts that link to their comments section when clicked.
- Posts that lead straight to a pic

We noticed also that we can’t scrape anything that matches _reddit.com/r/\*/comments/\* _without violating_ robots.txt, _so extracting a picture from a post would be wrong. We can however get the picture URLs if they’re directly linked from the subreddit’s main page. We see those links are always the _href_ property in an _\<a\>_ tag, so what we’ll do to get them is call the the response object’s _xpath_ method.

_xPath _is a way to move in a website’s HTML tree and get some of its elements. Scrapy also provides us with the _css _method, which allows for a different way of indexing and tagging elements. I personally find right clicking an element in the browser, hitting inspect and then _copy xpath_ is a quick way to get started, and then I just play around with the output a bit.

In this particular case, since all we need is the _href_ value for each _\<a\> _element, we’ll call

```
response.xpath(‘//a/@href’)
```

on the response, which will return an iterator for every _href_ value (an object from the ScraPy library). We then extract the string form of that value by calling the extract method, and check whether it’s actually a link to an image by seeing if it ends with ‘.png’ or ‘.jpg’.   
Here’s the whole improved parse method, which now also creates an html file to display all the images without downloading them:

{% gist https://gist.github.com/StrikingLoo/70c8752a50780227617cd46950401af3.js %}

So we make our spider crawl again, and the output should look something like this:

```
Crawled (200) \<GET [https://www.reddit.com/r/cats/](https://www.reddit.com/r/cats/)\> (referer: None)
[https://i.imgur.com/Au0aqkj.jpg](https://i.imgur.com/Au0aqkj.jpg)
[https://i.imgur.com/Xw90WFo.jpg](https://i.imgur.com/Xw90WFo.jpg)
[https://i.imgur.com/fOINLvP.jpg](https://i.imgur.com/fOINLvP.jpg)
```

Where each link is a cute kitten’s picture. As a bonus, the file _kittens.html_should be overflowing with cuteness.

That’s it! You’ve successfully crawled your first site!

#### Saving the images

Suppose instead of making an HTML file, we wanted to download the images. What we’d do then is import Python’s _requests_ library, and the _unicodedata_one. _Requests_ is gonna do the grunt work, but we’ll need _unicodedata_ since extracted strings are in _unicode_ by default, and _requests_ expects an ASCII one.

Now instead of the parse method, we’ll pass our _scrapy.Request_ function the following function as callback argument:

{% gist https://gist.github.com/StrikingLoo/5268541dbd29ce25b239cf80df90cae2.js %}

All it does is download an image and save it as a JPG. It also auto increases an index attribute stored in the spider, which gives each image its name.

#### Playing around: interactive shell

ScraPy provides us with an interactive shell where we can try out different commands, expressions and _xpaths_. This is a much more productive way of iterating and debugging a spider than running the whole thing over and over with a _crawl_ command. All we need to do to start the shell is running this:

```
scrapy shell ‘http://reddit.com/r/cats’
```

Of course the URL can be replaced with any other.

#### Extending our Spider

If we wanted to get more images, we could make the _download\_pictures _method call _scrapy.Request_ on the URL of the next page, which can be obtained from the _href_ attribute of the ‘next page’ button. We could also make the spider take a subreddit as argument, or change the downloaded file extensions.

All in all though, the best solution is usually the simplest one, and so using Reddit’s API will save us a lot of headaches.

I hope you now feel empowered to make your own spider and, obtain your own data. Please tell me if you found this useful, and what’s a good data set you think you could generate using this tool — the more creative the better.

Finally, There is an O’Reilly book I love. I found it very useful when I started my Data Science journey, and it exposed me to a different, easier to use (though less flexible) Web Scraping framework. It’s called [Data Science from Scratch with Python](https://www.bookdepository.com/book/9781491901427/?a_aid=strikingloo&chan=ws), and it’s probably half the reason I got my job. If you read this far, you may enjoy it!

_Follow me on [Medium](http://www.medium.com/@strikingloo) or [Twitter](http://twitter.com/strikingloo) for more Python tutorials, tips and tricks! If you liked my article, share it with a friend or tweet it!_

The post [How to Scrape the Web using Python with ScraPy Spiders](http://www.datastuff.tech/programming/how-to-scrape-the-web-using-python-with-scrapy-spiders/) appeared first on [Data Stuff](http://www.datastuff.tech).