How to use Scrapy Items - 05 - Python Scrapy tutorial for beginners
Tags: python, scrapy, tutorial
Published: 2019-10-09 18:31:59


Original post [Python Scrapy tutorial for beginners – 05 – How to use Scrapy Items](https://letslearnabout.net/python/python-scrapy-tutorial-for-beginners-05-how-to-use-scrapy-items/ "Permalink to Python Scrapy tutorial for beginners - 05")


The goal of scraping is to extract data from websites. Without Scrapy Items, we return unstructured data in the form of Python dictionaries: An easy way to introduce typos and return faulty data.

Luckily, Scrapy provides us with the Item class: A class we can inherit to make our data more structured and stronger, yielding a Python object.

In this post you will learn how to:

* Create Scrapy Items
* Use them to return a structured object

{% youtube gzpNbrnWQHM %}


* * *

### Starting point

![man preparing for race at oval track][1]

While you can use your own Scrapy projects for this tutorial, I'll recommend you to follow along by using the last version of this tutorial series, where we added [Rules and a LinkExtractor][2] to our spider.

Clone the [Github Repo][3], and you are set to go!

* * *

### Creating our Scrapy Item

To use our Item, first we need to create it and… it is already done! 

On the root project, we have an items.py file with the skeleton of an item:

![Scrapy Items][4]

Besides the scrapy import and a valuable link, we have nothing there. Let's solve that!

Our BooksItem it is going to be the class we are going to use for every scraped element. Think about it like a blueprint that tells you what we are going to need.

And what do we need on each item? A title, an image, a price… Exactly what our spider.py file yields. 

Copy the elements and paste them inside the class, and assign them 'scrapy.Field()':
    
    
    class BooksItem(scrapy.Item):
        title = scrapy.Field()
        final_image = scrapy.Field()
        price = scrapy.Field()
        stock = scrapy.Field()
        stars = scrapy.Field()
        description = scrapy.Field()
        upc = scrapy.Field()
        price_excl_tax = scrapy.Field()
        price_inc_tax = scrapy.Field()
        tax = scrapy.Field()
    

That's it! We are done! 

Our BooksItem class is created. The only fields we can add are the ones we explicitly wrote inside the class. Let's test that theory.

* * *

### Testing our Scrapy Item 

Let's see that our theory is solid. Load the scrapy shell (with _scrapy shell_ on your terminal), import the item and create an object with some fields. Nothing wrong happens, as every field is optional.

But then, try to create another object with non-existing fields. You'll get an error:

![Scrapy Items][5]

Our theory is right: We can only add the existing fields that we declared on our Item.

We talked enough about the Item, let's use it.

* * *

### Using our Scrapy Item in our Spider 

Open your items.py (finally!) and add the import on top of the file:
    
    
    # -*- coding: utf-8 -*-
    from scrapy.spiders import CrawlSpider, Rule
    from scrapy.linkextractors import LinkExtractor
    
    from ..items import BooksItem # New line
    
    import scrapy

Then, inside the parser method, create an object somewhere. For example, I created it after every data is scraped:
    
    
                   ....
                   '//table[@class="table table-striped"]/tr[1]/td/text()').extract_first()
                price_excl_tax = response.xpath(
                    '//table[@class="table table-striped"]/tr[3]/td/text()').extract_first()
                price_inc_tax = response.xpath(
                    '//table[@class="table table-striped"]/tr[4]/td/text()').extract_first()
                tax = response.xpath(
                    '//table[@class="table table-striped"]/tr[5]/td/text()').extract_first()
    
                book = BooksItem() # New line
    

Now we have a nice yield returning a dictionary with all the data.

Delete it.

And then, assign each field to the book object. And then, yield the object instance:
    
    
                book = BooksItem()
    
                book['title'] = title
                book['final_image'] = final_image
                book['price'] = price
                book['stock'] = stock
                book['stars'] = stars
                book['description'] = description
                book['tax'] = tax
    
                yield book

That's enough. Let's run the code. Run _scrapy crawl spider -o scrapy_item_version.json_ and wait until the spider is done.

As always, we have our 1000 books, this time, with a stronger and more solid code, by using Items:

![Scrapy Items - Json][6]
* * *

### Conclusion 

It is easy to make your spiders less buggy, and one of the easier improvements are using Scrapy Items. The Item class let us inherit a class that enables us to use Scrapy classes that by declaring its fields. To use them, we just need to:

* Create an Item by specifying the fields it is going to have
* Import the class created
* Create an instance of that class
* For every field extracted, add it to the Item instance
* Finally, return the object instance.

This opens the door to the Item Pipeline, which processes the item scraped. We can tell how Scrapy should process the scraped item, for example, cleaning it, validating the fields and more.

And of course, we'll learn that and more on our next lesson*.

*_The sixth lesson is being built right now. Thanks for your patience._

* * *

[My Youtube tutorial videos][7]

[Final code on Github][8]

[Reach to me on Twitter][9]

[Previous lesson: 04 – Crawler, Rules and LinkExtractor][10]

[1]: https://images.unsplash.com/photo-1523497804259-88c4c134ca90?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1000&q=80
[2]: https://letslearnabout.net/tutorial/python-scrapy-tutorial-for-beginners-04-crawler-rules-and-linkextractor/
[3]: https://github.com/david1707/scrapy_tutorial/tree/04_lesson
[4]: https://i2.wp.com/letslearnabout.net/wp-content/uploads/2019/10/image-1.png?w=810&ssl=1
[5]: https://i2.wp.com/letslearnabout.net/wp-content/uploads/2019/10/image-2.png?w=810&ssl=1
[6]: https://i2.wp.com/letslearnabout.net/wp-content/uploads/2019/10/image-3.png?w=810&ssl=1
[7]: https://www.youtube.com/channel/UC9OLm6YFRzr4yjlw4xNWYvg?sub_confirmation=1
[8]: https://github.com/david1707/scrapy_tutorial/tree/05_lesson
[9]: https://twitter.com/DavidMM1707
[10]: https://letslearnabout.net/tutorial/python-scrapy-tutorial-for-beginners-03-how-to-go-to-the-next-page/