Part 1: Locating the HTML for the website data you are interested
Tags: html, node, scraping
Published: 2020-05-04 15:37:05

You maybe might be in the same position, you want to gather some data of different objects (like financial data from different entities) from a website, and you copy/paste thousands times from website to your local worksheet. The work of collection makes you exhausting. Believe me, I have been there as you did.

To collect and format data more efficiently and nicely, we will create a little program to solve the problem for us. Afterwards, it will only take a few seconds to retrieve the online data.

The article is split into two parts:

- The **first part** introduces how to locate the data points we want in the HTML of the website. 
- The **second part** shows you how to apply that knowledge in an automated way with a small NodeJS script.

This part is going to start with an example of collecting financial information of Apple Inc. from a stock market website, **martketwatch.com**. 
The URL path is: [https://www.marketwatch.com/investing/stock/aapl/financials]().

The approach I will show is based on the assumption that there is no API available which could be used directly. In the case of **martketwatch.com**, they render the data on the server, which means the rendered page needs to be parsed. So let us do that!

Now, to help us research how to get to the precious data, let us make use of **jQuery**. It has the same interface as **cheerio**, which will be used later.

To use **jQuery** on the current website, execute the following in the developer console. It adds a script tag to the website which will load the jQuery library for our use.

```javascript
var script = document.createElement('script');script.src = "https://code.jquery.com/jquery-latest.min.js";document.getElementsByTagName('head')[0].appendChild(script);
```

*Then the $ should be available. In case it is not available, maybe due to CSR policy of the website, the last option is to copy the content of [https://code.jquery.com/jquery-latest.min.js]() and paste and execute it in the developer console.*

Give it a test and select all input tags on the website:

```javascript
$('input')
```

![Result of $('input') command on **martketwatch.com**](https://dev-to-uploads.s3.amazonaws.com/i/iq297ll3qzg4qmf1nwxp.png)

You should now see that there are some input tags found by the **jQuery** selector. This is great, the groundwork is laid for our adventure!


---

Now, let us see how we can select the data we want, in this case, the financial data of the stocks of brand with a bitten fruit.

![Apple stock data in the years 2015 to 2019](https://dev-to-uploads.s3.amazonaws.com/i/a0vg61b8m7ca163s0jpj.png)

The data we are interested here are the sales from the year 2015 to 2019, the question is, how can we extract them? Well, you might have guessed it, with the jQuery selector. Let us have a look at the HTML of it. With the element selector of your developers tools, you can select the element in the view.

![Element selection to find corresponding tag in HTML](https://dev-to-uploads.s3.amazonaws.com/i/5frdrwdxr8gfl35fjdu3.png)

After selecting a field, the element will be located in the HTML.

![](https://dev-to-uploads.s3.amazonaws.com/i/i93u79jxiv9ml1ze4t7x.png)

Looking at the surrounding HTML, there is a **div** tag with the class **financials**. Inside of that tag is the actual **table** tag, which has a **tbody** tag. That one contains a **tr** tag which contains the **td** tags with our precious data we want to collect! I hope you could follow me here so far :)

Let us try to come up with a selector that will work for jQuery. Clearly the **financials** class looks like a perfect anchor, due to being unique in the HTML. Next let us try to select only the class **valueCell**.

```javascript
$('.financials td.valueCell')
```

![Too many tags with class valueCell are selected](https://dev-to-uploads.s3.amazonaws.com/i/7cq4iqxdtrb19n1r1xlf.png)

Now this is a bit to broad and we have too many **valueCell** class tags selected, we need to narrow it down a bit. After looking at what kind of other tags were selected, it seems that we need to constrain it down to the first row of **valueCell** class tags. Luckily, there is a **tr** tag which has the class **partialSum**.

```javascript
$('.financials tr.partialSum:nth-child(1) td.valueCell')
```

![](https://dev-to-uploads.s3.amazonaws.com/i/gaulbnuau8mxsu1q0rha.png)

This leaves us with exact the cells we want to look up! Next step is too use it in our automation!

References:

https://www.w3schools.com/jquery/jquery_ref_selectors.asp

https://api.jquery.com/category/selectors/