Fetching programmatically token addresses from coingecko - using NodeJs
Tags: coingecko, scraping, tokenaddress, blockchain
Published: 2022-10-02 09:54:07

The **objective** is to extract the addresses of tokens.

We can use this data for several purpouse : 
Ex. Ussually each token is deployed in more than one Chain with different contract address. Automatizating task of having to search for a token address manually. 


**Set-Up**
To do this we will use "node" and "puppeteer" 
library to scrape coingecko website.

```javascript
npm init --y
npm install puppeteer
```

**Step 1_ Fetching web's urls**
We will study the DOM of Coingecko's web in order to automate the data extraction process.
For it, we will follow the following steps:
- We will locate in which DOM elements the data we want to extract is stored. (using web inspector)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/on1k9lv9bj14p90qwxzn.PNG)

> We're extracting each token url to extract smart contract's address.

- Once located the DOM elements, we will look at the attributes (class, id, type...) to be able to extract them using the following commands:

First, wrap all the code inside an IIFE async function:

```javascript
const fs = require("fs");
const puppeteer = require("puppeteer");

(async () => {

      Code...

}();
```
Create a new "page" object using "puppeteer" and go to https://www.coingecko.com/es
Use page objects built-in "evaluate" method to acces to the website DOM. 
Use "querySelectorAll" pointing to the element "a" inside element with "coin-name" class.

```javascript
const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  await page.goto("https://www.coingecko.com/es");

  // Evaluating page's DOM elements and extracting data
  const duplicatedUrls = await page.evaluate(() => {
    // Extracting DOM elemets to an array // Contains href and ticker
    const _rawUrls = document.querySelectorAll(".coin-name a");

    // Looping to extract and filter data
    const _duplicatedUrls = [];
    for (let i = 0; i < _rawUrls.length; i++) {
      let _url = _rawUrls[i].href;
      _duplicatedUrls.push(_url);
    }

    return _duplicatedUrls;
  });

  // Deleting duplicated data from urls
  const urls = [...new Set(duplicatedUrls)];

```

> @Dev: Coingecko website may be updated, the attributes of the extracted elements may change. In this case, you will have to re-write the logic to extract DOM elements.


**Step 2_ Looping in each token url and fething data**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eoik1dnij732s86a5flh.PNG)

For loop, wraps the logic that will take care of getting data, filtering, structuring, and returning to an array.

Use "evaluate" method to fetch DOM element using "querySelectorAll(`[data-address]`)" and use "getAttribute()" method to get token's data: symbol, chainId, address and decimals.


> @notice Loop ends before step 5
```javascript
// Fetching token addresses from urls, "addresses" will be final array
const addresses = [];

  for (let i = 0; i < urls.length; i++) {
    await page.goto(urls[i]);

    const tokenAddress = await page.evaluate(() => {
      /* 
        DATA FETCHING LOGIC
      */
      var _rawElement;
      try {
        _rawElement = document.querySelectorAll(`[data-address]`);
      } catch (error) {
        // Most coins, also have "data-address" atribute in DOM element
        console.log("Is a coin, has not an address"); // Logs are only visible in pupperteer opened browser
        return undefined;
      }
      // If is a coin, we don't search for an address, "_rawElement" will be false
      if (_rawElement) {
        // We will run code inside puppeteer's opened browser
        // Extracting raw data
        let _tokenAddress = [];

        for (let i = 0; i < _rawElement.length; i++) {
          _tokenAddress.push([
            _rawElement[i].getAttribute("data-symbol"),
            _rawElement[i].getAttribute("data-chain-id"),
            _rawElement[i].getAttribute("data-address"),
            _rawElement[i].getAttribute("data-decimals"),
            /* search for more INFO in DOM element */
          ]);
        }
```

**Step 3_ Filtering data** 
When our logic checks Coin's data (not Token's) will return undefined. Usign following logic:
Check "data-address" if is equal to "0x" or not.
Check if "data-chain-id" is null or undefined or empty.

More info about differences between Coin and token
**AGREGAR UN BLOG MIO DE DIFERENCIA ENTRE COIN Y TOKEN**
https://www.outlookindia.com/business/demystified-the-difference-between-crypto-coins-and-crypto-tokens-read-here-for-details-news-197683

Use "[...new Set(_array)]" to delete duplicated data.
Use array.prototype.filter method to delete "null" variables.
```javascript
// As mentioned before, we need to guarantee to return "undefined" if it is a Coin
        // 2 checks
        // Comparing "data-address" if starts with "0x"
        let isToken = false;
        // Checking if there is a "data-chain-id" value // In near future, maybe we'll need more filters
        let isChain = false;
        for (let i = 0; i < _rawElement.length; i++) {
          const addr = _rawElement[i]
            .getAttribute("data-address") // hasta en los tokens, puede veinr uno con un string
            .substring(0, 2);
          if (addr === "0x") {
            isToken = true;
            console.log("is a token"); // Logs are only visible in pupperteer opened browser
          }
          const chainTest = _rawElement[i].getAttribute("data-chain-id");
          if (chainTest) {
            isChain = true;
          }
        }
        if (!isToken || !isChain) return undefined;

        // Cleaning data
        const _elements = [...new Set(_tokenAddress)];

        // Checking duplicated arrays with null values to delete them
        const _tokenData = _elements.filter(
          (item) => item[0] !== null && item[1] !== null && item[2] !== null
        );
```
**Step 4_ Structuring data**
Create a new object and fill it in new properties (using filtered data). Thus, we're not trasfering any useless data.
```javascript

        const tokenObject = {};
        // un objeto con su ticker para cada token
        tokenObject[`${_tokenData[0][0]}`] = {
          symbol: `${_tokenData[0][0]}`,
        };

        // Dividing in an array of chains where the token is deployed
        const chains = [];
        // Dividing in an array of addresses of the token
        const tokenAddressPerChain = [];
        // Dividing in an array token`s decimals
        const tokenDecimals = [];
        for (let i = 0; i < _tokenData.length; i++) {
          chains.push(_tokenData[i][1]);
          tokenAddressPerChain.push(_tokenData[i][2]);
          tokenDecimals.push(_tokenData[i][3]);
        }
        // Adding data to final object, overrides duplicated data
        for (let i = 0; i < chains.length; i++) {
          tokenObject[`${_tokenData[0][0]}`][`${chains[i]}`] = {
            address: [`${tokenAddressPerChain[i]}`],
            decimals: [`${tokenDecimals[i]}`],
            /* ADD more INFO to json*/
          };
        }

        return tokenObject;
      } else return undefined;
    }); 
// THE LOOP ENDS HERE

    await page.goBack();
    if (tokenAddress) {
      addresses.push(tokenAddress);
    }
  }
```
**Step 5_ Create a json file**

```javascript
fs.writeFileSync("json/TokenAddresses.json", JSON.stringify(addresses));

  // Closing the browser
  await browser.close();
```
**Final json pre-view**

```javascript
[
  {
    LEO: {
      1: {
        address: ["0x2af5d2ad76741191d15dfe7bf6ac92d4bd912ca3"],
        decimals: ["18"],
      },
      symbol: "LEO",
    },
  },
  {
    MATIC: {
      1: {
        address: ["0x7d1afa7b718fb893db30a3abc0cfc608aacfebb0"],
        decimals: ["18"],
      },
      56: {
        address: ["0xcc42724c6683b7e57334c4e856f4c9965ed682bd"],
        decimals: ["18"],
      },
      137: {
        address: ["0x0000000000000000000000000000000000001010"],
        decimals: ["18"],
      },
      1666600000: {
        address: ["0x301259f392b551ca8c592c9f676fcd2f9a0a84c5"],
        decimals: ["18"],
      },
      symbol: "MATIC",
    },
  },
...
]
```
I hope it has been helpful.