OpenSea NFT Scraper
Tags: nft, opensea, python, tutorial
Published: 2022-09-23 11:00:48

#Overview
OpenSea has a lot of NFT pictures. It's a good example of learning how to scrape. Compared to other typical static websites, pictures in OpenSea are dynamic loading along with the endless scroll. Pictures are loading if they are on the screen and will be gone 
when they are out of the screen. 
## Clicknium 
[Clicknium](https://www.clicknium.com/documents/) is a Python automation library and very easy to do this kind of scraping work. It's more than a Python library but also provides a smooth and interactive coding experience with VS Code plugins and browser plugins. 

## Requirement
- Windows7 SP1+ 
- Python 3.7+
- VS Code

## How to 
Let's scrape [Okay Bear](https://opensea.io/collection/okay-bears) as an example. 
### Picture info   
1. Open Okay Bear's page with Chrome.  
2. Press F12 to open the DevTools, click the mouse icon, and then click the NFT picture.  
In the following pic, we can see:
- Image alt can be used as picture name
- Picture src and srcset with different sizes. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/010605txekudq6lq6s2z.png)

### Capture Pictures
Capturing similar elements is quite easy for this scenario:
1. Click the capture button in VS code extension to start the recorder.   
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/12hlhcvbvtlsio98mexj.png)  
2. Click `Similar elements` .
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8sx4k8fhy8qha7b8qw2m.png)
3. Ctrl + Click on the first pic and then capture the second.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o4jgltyzkojkvo01k912.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2qod69mddae93sxvv1nn.png)

You can also use this way to capture the price. 
### Handle scroll to load pictures
Clicknium provides API to control the mouse and keyboard. In this Scenario, Page-down is the easiest way. More about [HotKey](https://learn.microsoft.com/en-au/dotnet/api/system.windows.forms.sendkeys?view=windowsdesktop-6.0#remarks) 
```
clicknium.send_hotkey("{PGDN}")
```

### Download picture
use requests(`pip install requests`)


## Source code:
Github: [OpenSeaScraper](https://github.com/automation9417/OpenSeaScraper)


```
import requests
from time import sleep
from clicknium import clicknium as cc, locator
def main():
    cc.chrome.open("https://opensea.io/collection/okay-bears")
    nftDic = {}

    while True:
        newDic = {}
        finish = True
        sleep(4)
        nfts = cc.find_elements(locator.sample.opensea.img_okay_bear)
        for nft in nfts:
            src = nft.get_property("src")
            name = nft.get_property("alt")
            if name not in nftDic:
                newDic[name]=src
                finish = False
        
        for key,value in newDic.items():
            r = requests.get(value)
            print("Downloading:"+ key)
            imagePath = ".\\image\\" + key + ".jpg"
            with open(imagePath,'wb') as f:
                f.write(r.content)

        nftDic = nftDic | newDic
        if finish:
            break
        else:
            print("next page")
            cc.send_hotkey("{PGDN}")

if __name__ == "__main__":
    main()

```


More sample:
[10 lines of Python code to upload a video to TikTok, Instagram, Twitter](https://dev.to/kayyolo/10-lines-of-python-code-to-upload-a-video-to-tiktok-instagram-twitter-c9n)

[A Spotify robot](https://dev.to/kayyolo/a-spotify-robot-824)