Web scraping Google Books Ngram Viewer with Nodejs
Tags: webscraping, node, serpapi
Published: 2022-11-03 15:43:52

<h2 id='intro'>Intro</h2>

Currently, we don't have an API that supports extracting data from Google Books Ngram Viewer page.

This blog post is to show you the way how you can do it yourself with the provided DIY solution below while we're working on releasing our proper API.

The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA.

You can check our public roadmap to track the progress for this API:

* 🗺️ [[New API] Add Google Ngram Viewer API](https://github.com/serpapi/public-roadmap/issues/51)

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/197813259-b8494120-3d84-484e-82c1-b77124546763.png)

Comparing with the scraped data chart:

![scraped](https://user-images.githubusercontent.com/64033139/197814887-79a85ebe-03ac-49e4-b7cc-a2d7a608e7ba.png)

<h2 id='full_code'>Full code</h2>

```javascript
const axios = require("axios");
const fs = require("fs");
const { ChartJSNodeCanvas } = require("chartjs-node-canvas");

const searchString = "Albert Einstein,Sherlock Holmes,Frankenstein,Steve Jobs,Taras Shevchenko,William Shakespeare"; // what we want to get
const startYear = 1800; // the start year of the search
const endYear = 2019; // the end year of the search

const AXIOS_OPTIONS = {
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
  }, // adding the User-Agent header as one way to prevent the request from being blocked
  params: {
    content: searchString, // what we want to search
    year_start: startYear, // parameter defines the start year of the search
    year_end: endYear, // parameter defines the end year of the search
  },
};

async function saveChart(chartData) {
  const width = 1920; //chart width in pixels
  const height = 1080; //chart height in pixels
  const backgroundColour = "white"; // Uses https://www.w3schools.com/tags/canvas_fillstyle.asp
  const chartJSNodeCanvas = new ChartJSNodeCanvas({ width, height, backgroundColour });

  const labels = new Array(endYear - startYear + 1).fill(startYear).map((el, i) => (el += i));

  const configuration = {
    type: "line", // for line chart
    data: {
      labels,
      datasets: chartData?.map((el) => {
        const data = el.timeseries.map((el) => el * 100);
        return {
          label: el.ngram,
          data,
          borderColor: [`rgb(${parseInt(Math.random() * 255)}, ${parseInt(Math.random() * 255)}, ${parseInt(Math.random() * 255)})`],
        };
      }),
    },
    options: {
      scales: {
        y: {
          title: {
            display: true,
            text: "%",
          },
        },
      },
    },
  };

  const base64Image = await chartJSNodeCanvas.renderToDataURL(configuration);

  const base64Data = base64Image.replace(/^data:image\/png;base64,/, "");

  fs.writeFile("chart.png", base64Data, "base64", function (err) {
    if (err) {
      console.log(err);
    }
  });
}

function getChart() {
  return axios.get(`https://books.google.com/ngrams/json`, AXIOS_OPTIONS).then(({ data }) => data);
}

getChart().then(saveChart);
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`axios`](https://www.npmjs.com/package/axios) to make a request to a website, [`chart.js`](https://github.com/chartjs/Chart.js) to build chart from received data and [`chartjs-node-canvas`](https://www.npmjs.com/package/chartjs-node-canvas) to render chart with Chart.js using [`canvas`](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial).

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y
```

And then:

```bash
$ npm i axios chart.js chartjs-node-canvas
```

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

<h2 id='process'>Process</h2>

We'll receive Books Ngram data in JSON format, so we need only handle the received data, and create our own chart (if needed):

Request:

```javascript
axios.get(`https://books.google.com/ngrams/json`, AXIOS_OPTIONS).then(({ data }) => data);
```

Response JSON:

```json
[
  {
    "ngram": "Albert Einstein",
    "parent": "",
    "type": "NGRAM",
    "timeseries": [
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9.077474010561153e-10, 9.077474010561153e-10, 9.077474010561153e-10,
      ...and other chart data
      ]
  },
  {
    "ngram": "Sherlock Holmes",
    "parent": "",
    "type": "NGRAM",
    "timeseries": [
      4.731798064483428e-9, 3.785438451586742e-9, 3.154532042988952e-9, 2.7038846082762446e-9, 0, 2.47730296593878e-10,
      ...and other chart data
    ]
  },
  ...and other Books Ngram data
]
```

<h3 id='code_explanation'>Code explanation</h3>

Declare constants from [`axios`](https://www.npmjs.com/package/axios), [`fs`](https://nodejs.org/api/fs.html) (`fs` library allows you to work with the file system on your computer) and [`chartjs-node-canvas`](https://www.npmjs.com/package/chartjs-node-canvas) libraries:

```javascript
const axios = require("axios");
const fs = require("fs");
const { ChartJSNodeCanvas } = require("chartjs-node-canvas");
```

Next, we write what we want to get, start year and end year:

```javascript
const searchString = "Albert Einstein,Sherlock Holmes,Frankenstein,Steve Jobs,Taras Shevchenko,William Shakespeare";
const startYear = 1800;
const endYear = 2019;
```

Next, we write a request options: [`HTTP headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) with [`User-Agent`](https://developer.mozilla.org/en-US/docs/Glossary/User_agent) which is used to act as a "real" user visit, and the necessary parameters for making a request.

[Default `axios` request user-agent is `axios/<axios_version>`](https://github.com/axios/axios/blob/892c241773e7dda78a969ac1faa9b365e24f6cc8/lib/adapters/http.js#L224) so websites understand that it's a script that sends a request and might block it. [Check what's your user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent/):

```javascript
const AXIOS_OPTIONS = {
  headers: {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
  }, // adding the User-Agent header as one way to prevent the request from being blocked
  params: {
    content: searchString, // what we want to search
    year_start: startYear, // parameter defines the start year of the search
    year_end: endYear, // parameter defines the end year of the search
  },
};
```

Next, we write a function that handles and saves received data to the ".png" file:

```javascript
async function saveChart(chartData) {
    ...
}
```

In this function we need to declare the [`canvas`](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial) `width`, `height` and `backgroundColor`, then build it using [`chartjs-node-canvas`](https://www.npmjs.com/package/chartjs-node-canvas):

```javascript
const width = 1920; //chart width in pixels
const height = 1080; //chart height in pixels
const backgroundColour = "white"; // Uses https://www.w3schools.com/tags/canvas_fillstyle.asp
const chartJSNodeCanvas = new ChartJSNodeCanvas({ width, height, backgroundColour });
```

Then, we need to define and create the "x" axis labels. To do this we need to create a [new array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Array) with a length that equals the numbers of years from `startYear` to `endYear` (we add '1' because we need to include these years also).

Then we [`fill`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/fill) an array with `startYear` and add element position (`i`) to each value (using [`map()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map) method):

```javascript
const labels = new Array(endYear - startYear + 1)
  .fill(startYear)
  .map((el, i) => (el += i));
```

Next, we need to create `configuration` object for [`chart.js`](https://github.com/chartjs/Chart.js) library. In this object, we define chart `type`, `data`, and `options`.

In the chart `data` we define the main axis `labels` and make `datasets` from received `chartData` in which we set for each line label, data, and random color (using [`Math.random()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/random) and [`parseInt()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt) methods).

In the chart `options` we set the 'y' axis name and allow to show it (`display` property):

```javascript
const configuration = {
  type: "line", // for line chart
  data: {
    labels,
    datasets: chartData?.map((el) => {
      const data = el.timeseries.map((el) => el * 100);
      return {
        label: el.ngram,
        data,
        borderColor: [`rgb(${parseInt(Math.random() * 255)}, ${parseInt(Math.random() * 255)}, ${parseInt(Math.random() * 255)})`],
      };
    }),
  },
  options: {
    scales: {
      y: {
        title: {
          display: true,
          text: "%",
        },
      },
    },
  },
};
```

Next, we wait for building chart in [`base64`](https://en.wikipedia.org/wiki/Base64) encoding, remove data type properties from `base64` string ([`replace()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace) method) and save the "chart.png" file with [`writeFile()`](https://nodejs.org/api/fs.html#fswritefilefile-data-options-callback) method:

```javascript
const base64Image = await chartJSNodeCanvas.renderToDataURL(configuration);

const base64Data = base64Image.replace(/^data:image\/png;base64,/, "");

fs.writeFile("chart.png", base64Data, "base64", function (err) {
  if (err) {
    console.log(err);
  }
});
```

Then, we write a function that makes the request and returns the received data. We received the response from [`axios`](https://www.npmjs.com/package/axios) request that has `data` key that we [destructured](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment) and return it:

```javascript
function getChart() {
  return axios
    .get(`https://books.google.com/ngrams/json`, AXIOS_OPTIONS)
    .then(({ data }) => data);
}
```

And finally, we need to run our functions:

```javascript
getChart().then(saveChart);
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='savedFile'>Saved file</h2>

![scraped](https://user-images.githubusercontent.com/64033139/197814887-79a85ebe-03ac-49e4-b7cc-a2d7a608e7ba.png)

If you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>