How to go to the next page - 03 - Python scrapy tutorial for beginners
Tags: python, scrapy, tutorial
Published: 2019-09-12 19:45:54

Original post [Python Scrapy tutorial for beginners – 03 – How to go to the next page](https://letslearnabout.net/tutorial/python-scrapy-tutorial-for-beginners-03-how-to-go-to-the-next-page/ "Permalink to Python Scrapy tutorial for beginners - 03")

# Python Scrapy tutorial for beginners - 03

![][1]

On our last lesson, [extracting all the data with Scrapy,][2] we managed to get all the books URL and then extracted the data from each one. We were limited to the books on the main page, as we didn't know how to go to the next page using Scrapy.

Until now.

In this post you will learn how to: 

* Navigate to the 'next page'
* Solve routing problems
* Extract all the data of every book available

{% youtube mDUCyn6pxwQ%}


* * *

### Our game-plan

![next page using Scrapy - gameplan][3]

Initially we just listed all the book URLs and then, one by one, we extracted the data. 

As we had 20 books, we just listed 20 book URLs, and then parsed those 20 URLs, yielding the result.

We just need to add another step.

Now, we'll list 20 book URLs, parse them, and then, if there is a 'Next' page, we'll navigate to it to repeat the process, listing and yielding the new 20 book URLs, until there are no more pages.

In our [Beautiful Soup tutorial][4] we used the same strategy:

![next page using Scrapy - recursive flow][5]

And that's what we are going to start using right now.

* * *

### Checking if there is a 'next page' available 

Let's start from the code we used in our second lesson, [extract all the data][2]:
    
    
    # -*- coding: utf-8 -*-
    import scrapy
    
    
    class SpiderSpider(scrapy.Spider):
        name = 'spider'
        allowed_domains = ['books.toscrape.com']
        start_urls = ['http://books.toscrape.com/']
        base_url = 'http://books.toscrape.com/'
    
        def parse(self, response):
            all_books = response.xpath('//article[@class="product_pod"]')
    
            for book in all_books:
                book_url = self.start_urls[0] + 
                    book.xpath('.//h3/a/@href').extract_first()
    
                yield scrapy.Request(book_url, callback=self.parse_book)
    
        def parse_book(self, response):
            title = response.xpath('//div/h1/text()').extract_first()
    
            relative_image = response.xpath('//div[@class="item active"]/img/@src').extract_first()
            final_image = self.base_url + relative_image.replace('../..', '')
    
            price = response.xpath(
                '//div[contains(@class, "product_main")]/p[@class="price_color"]/text()').extract_first()
            stock = response.xpath(
                '//div[contains(@class, "product_main")]/p[contains(@class, "instock")]/text()').extract()[1].strip()
            stars = response.xpath(
                '//div/p[contains(@class, "star-rating")]/@class').extract_first().replace('star-rating ', '')
            description = response.xpath(
                '//div[@id="product_description"]/following-sibling::p/text()').extract_first()
            upc = response.xpath(
                '//table[@class="table table-striped"]/tr[1]/td/text()').extract_first()
            price_excl_tax = response.xpath(
                '//table[@class="table table-striped"]/tr[3]/td/text()').extract_first()
            price_inc_tax = response.xpath(
                '//table[@class="table table-striped"]/tr[4]/td/text()').extract_first()
            tax = response.xpath(
                '//table[@class="table table-striped"]/tr[5]/td/text()').extract_first()
    
            yield {
                'Title': title,
                'Image': final_image,
                'Price': price,
                'Stock': stock,
                'Stars': stars,
                'Description': description,
                'Upc': upc,
                'Price after tax': price_excl_tax,
                'Price incl tax': price_inc_tax,
                'Tax': tax,
            }
    

Since this is currently working, we just need to check if there is a 'Next' button after the for loop is finished. Right-click on the next button:

![][6]

The next page URL is inside an _a_ tag, within a _li_ tag. You know how to extract it, so create a _next_page_url _ we can navigate to. Beware, it is a partial URL, so you need to add the base URL. As we did it before, you can do it yourself. Give it a try.

This is how I did it:
    
    
            for book in all_books:
                book_url = self.start_urls[0] + 
                    book.xpath('.//h3/a/@href').extract_first()
    
                yield scrapy.Request(book_url, callback=self.parse_book)
    
            # New code:
            next_page_partial_url = response.xpath(
                '//li[@class="next"]/a/@href').extract_first()
    
            next_page_url = self.base_url + next_page_partial_url
            yield scrapy.Request(next_page_url, callback=self.parse)

Run the code with _scrapy crawl spider -o next_page.json_ and check the result.

What's going on? There is only 20 elements in the file! Let's check the logging to see what's going on.

![][7]

We managed to get the first 20 books, but then, suddenly, we can't get more books…

The _books.toscrape.com_ is a website made by [Scraping Hub][8] to train people on web scraping, and they have little traps you need to notice. Compare the successful URLs (blue underline) with the failed ones (red underline). There is a _/catalogue _missing on each routing. They didn't add it to make you fail.

Let's solve that problem.

* * *

### Solving the 'book' routing problem

As _/catalogue_ is missing from some URLs, let's have a check: If the routing doesn't have it, let's prefix it to the partial URL. As simple as that.

Try it on your own before continuing. You can check my code here:
    
    
            for book in all_books:
                book_url = book.xpath('.//h3/a/@href').extract_first()
    
                if 'catalogue/' not in book_url:
                    book_url = 'catalogue/' + book_url
    
                book_url = self.base_url + book_url

Let's run the code again! It should work, right? _scrapy crawl spider -o next_page.json_

Now we have more books! But only 40. We managed to get the first 20, then the next 20. Then, something happened. We didn't get the third page from the second one. Let's go to the second page and see what's going on with the next button and compare it with the first one (and its link to the second one)

![][9]

We have the same problem we had with the books: Some links have _/catalogue_, some others don't.

* * *

### Solving the 'next' routing problem

As we have the same problem, we have the same solution. One you can solve easily. Why don't you try? Again, you just need to check the link and prefix _/catalogue_ in case that sub-string isn't there.

If you couldn't solve it, this is my solution:
    
    
    next_page_partial_url = response.xpath(
                '//li[@class="next"]/a/@href').extract_first()
    
            if next_page_partial_url:
                if 'catalogue/' not in next_page_partial_url:
                    next_page_partial_url = "catalogue/" + next_page_partial_url
    
                next_page_url = self.base_url + next_page_partial_url
                yield scrapy.Request(next_page_url, callback=self.parse)
    

You can see the pattern: We get the partial URL, we check if _/catalogue_ is missing and if it does, we add it. Then, we add the base_url and we have our absolute URL.

Run the spider again: _scrapy crawl spider -o next_page.json_.

Now we have our 1000 books. Every single one. 🙂

This is the final code:
    
    
    # -*- coding: utf-8 -*-
    import scrapy
    
    
    class SpiderSpider(scrapy.Spider):
        name = 'spider'
        allowed_domains = ['books.toscrape.com']
        start_urls = ['http://books.toscrape.com/']
        base_url = 'http://books.toscrape.com/'
    
        def parse(self, response):
            all_books = response.xpath('//article[@class="product_pod"]')
    
            for book in all_books:
                book_url = book.xpath('.//h3/a/@href').extract_first()
    
                if 'catalogue/' not in book_url:
                    book_url = 'catalogue/' + book_url
    
                book_url = self.base_url + book_url
    
                yield scrapy.Request(book_url, callback=self.parse_book)
    
            next_page_partial_url = response.xpath(
                '//li[@class="next"]/a/@href').extract_first()
    
            if next_page_partial_url:
                if 'catalogue/' not in next_page_partial_url:
                    next_page_partial_url = "catalogue/" + next_page_partial_url
    
                next_page_url = self.base_url + next_page_partial_url
                yield scrapy.Request(next_page_url, callback=self.parse)
    
    
        def parse_book(self, response):
            title = response.xpath('//div/h1/text()').extract_first()
    
            relative_image = response.xpath(
                '//div[@class="item active"]/img/@src').extract_first()
            final_image = self.base_url + relative_image.replace('../..', '')
    
            price = response.xpath(
                '//div[contains(@class, "product_main")]/p[@class="price_color"]/text()').extract_first()
            stock = response.xpath(
                '//div[contains(@class, "product_main")]/p[contains(@class, "instock")]/text()').extract()[1].strip()
            stars = response.xpath(
                '//div/p[contains(@class, "star-rating")]/@class').extract_first().replace('star-rating ', '')
            description = response.xpath(
                '//div[@id="product_description"]/following-sibling::p/text()').extract_first()
            upc = response.xpath(
                '//table[@class="table table-striped"]/tr[1]/td/text()').extract_first()
            price_excl_tax = response.xpath(
                '//table[@class="table table-striped"]/tr[3]/td/text()').extract_first()
            price_inc_tax = response.xpath(
                '//table[@class="table table-striped"]/tr[4]/td/text()').extract_first()
            tax = response.xpath(
                '//table[@class="table table-striped"]/tr[5]/td/text()').extract_first()
    
            yield {
                'Title': title,
                'Image': final_image,
                'Price': price,
                'Stock': stock,
                'Stars': stars,
                'Description': description,
                'Upc': upc,
                'Price after tax': price_excl_tax,
                'Price incl tax': price_inc_tax,
                'Tax': tax,
            }
    

* * *

### Conclusion

You hit a milestone today. Now you are able to extract every single element from a website.

You have learnt that you need to get all the elements on the first page, scrap them individually, and how to go to the next page to repeat this process. Let me show the diagram once again:

![next page using Scrapy - Recursive flow][5]

And not only that. This example was a tricky one as we had to check if the partial URL had _/catalogue_ to add it.

Normally, paginating websites with Scrapy is easier as the 'next' button contains the full URL, so this example was even harder than normal and yet you managed to get it!

But… what if I tell you that this can be even easier than what we did?

Instead of grabbing your pitchfork and heading to my home, go to the [fourth lesson][13] where you will learn how to scrape every single item in an even easier way using crawlers.


* * *

[My Youtube tutorial videos][10]

[Final code on Github][11]

[Reach to me on Twitter][12]

[Previous lesson: 02 – Creating your first spider ][2]

[1]: https://i2.wp.com/letslearnabout.net/wp-content/uploads/2019/09/image-8.png?w=688&ssl=1
[2]: https://letslearnabout.net/tutorial/python-scrapy-tutorial-for-beginners-02-extract-all-the-data/
[3]: https://i0.wp.com/www.hondurassoccer.com/wp-content/uploads/2014/06/pizarra-estrategia.png?w=688
[4]: https://letslearnabout.net/python/beautiful-soup/your-first-web-scraping-script-with-python-beautiful-soup/
[5]: https://i0.wp.com/letslearnabout.net/wp-content/uploads/2019/08/Untitled-Project.jpg?w=688&ssl=1
[6]: https://i0.wp.com/letslearnabout.net/wp-content/uploads/2019/09/image-9.png?fit=688%2C398&ssl=1
[7]: https://i2.wp.com/letslearnabout.net/wp-content/uploads/2019/09/image-10.png?fit=688%2C269&ssl=1
[8]: https://scrapinghub.com/
[9]: https://i1.wp.com/letslearnabout.net/wp-content/uploads/2019/09/image-11.png?fit=688%2C851&ssl=1
[10]: https://www.youtube.com/channel/UC9OLm6YFRzr4yjlw4xNWYvg?sub_confirmation=1
[11]: https://github.com/david1707/scrapy_tutorial/tree/03_lesson
[12]: https://twitter.com/DavidMM1707
[13]: https://letslearnabout.net/tutorial/python-scrapy-tutorial-for-beginners-04-crawler-rules-and-linkextractor/