Web Scraping CoinsMarketCap with Python: Selenium
Tags: python
Published: 2022-12-14 17:08:54

When it comes to web scraping in Python, people usually have two choices: 
1. bs4 + requests
2. Selenium (the so called webdriver!)

Often, it suffices with approach one (beautifulsoup), and one can scrape the majority of websites by adding a header. However, for some websites that are equipped with strong anti-scraping, selenium is a must in your toolkit. 

Today, we are going to look at an example of scraping historical price of bitcoin at coinmarketcap. 


![historical data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t5ptzxhayp6c2yokijfa.png)

We need the historical data of bitcoin, but instead of manually copy paste, can we automate this process? Wouldn't it be nice to have a scraper, so that each time we run it, it just scrapes everything we want? 

Sure, it would be very nice! But how do we do this? 

First, import necessary packages 

```python
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
import calendar
from pprint import pprint
```

Next, our target page is https://coinmarketcap.com/currencies/bitcoin/historical-data/, or to make it more general, it is of the format `https://coinmarketcap.com/currencies/{exchange_name}/historical-data/`, where exchange_name could be bitcoin, ethereum etc. 

We then open the url by webdriver
```
url: str = f"https://coinmarketcap.com/currencies/{exchange_name}/historical-data/"
driver = webdriver.Firefox()
driver.get(url)
```

Next, we need to manually inspect the page (using development tool), and see how can we select specific element. 

![inspect](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ucqt1y078hr4fnj2jcv3.png)

So, what we are interested lies in a table. The table has a parent class with class "history". That should be pretty much enough for specifying the elements we want. 

```python
elem = driver.find_element(By.CSS_SELECTOR, ".history tbody tr")
```

This selects the top most row of the table. To get the inside text of the elem, we just use `elem.text`.

What left is just getting target information with playing strings. Very straightforward. 

Below is a fully working code:

```python

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
import calendar
from pprint import pprint


def get_latest_price(exchange_name="bitcoin"):
    exchange_name = exchange_name.lower()
    url: str = f"https://coinmarketcap.com/currencies/{exchange_name}/historical-data/"
    driver = webdriver.Firefox()
    driver.get(url)
    # wait the page to load
    time.sleep(2)
    # get the latest date
    elem = driver.find_element(By.CSS_SELECTOR, ".history tbody tr")
    res = elem.text
    res = res.split(" $")
    date = res[0]
    open_price = res[1]
    high_price = res[2]
    low_price = res[3]
    close_price = res[4]
    driver.close()
    return {
        "exchange_name": exchange_name,
        "url": url,
        "date": date,
        "open_price": open_price,
        "high_price": high_price,
        "low_price": low_price,
        "close_price": close_price,
    }
```

Let's try it out:

```bash
➜  python3 scraper_exchange.py 
{'close_price': '17,781.32',
 'date': 'Dec 13, 2022',
 'exchange_name': 'bitcoin',
 'high_price': '17,930.09',
 'low_price': '17,111.76',
 'open_price': '17,206.44',
 'url': 'https://coinmarketcap.com/currencies/bitcoin/historical-data/'}
```


![real price](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/atcw7yp4mtleav6q2kqm.png)


Yeah! We do get the latest real price.