Convert a Website into an API: Building a Serverless Web Scraper with the AWS Cloud Development Kit
Tags: aws, javascript, web, lambda
Published: 2020-04-26 10:49:44

---
title: Convert a Website into an API: Building a Serverless Web Scraper with the AWS Cloud Development Kit
published: true
description: Learn to build a web scraper with JS and the AWS CDK
tags: aws, javascript, web, lambda
---

Imagine you are looking at a website and you see some really nice data that you would love to use to build an app or integration for. After furiously googling for an API providing this data, you come up short. This is when you need to make use of a web scraper.

> A web scraper is an application or program that fetches HTML from a web page, parses it, and stores the data in a human readable format for future reference.

Web scrapers are very useful for things like automation, social media scraping, statistics and monitoring for changes on a web page such as prices or products. 

Today, I’m going to show you how to convert a website into a serverless API using NodeJS running in Amazon Web Services. We’re going to scrape the [UFC Stat Leaders Site](http://statleaders.ufc.com/), which contains statistics for the top 10 fighters of all time in a whole host of different categories. 

![UFC Stat Leaders Page](https://martinmck.com/images/screenshot_2020-04-18_at_12.19.48.png)

We want to scrape the results for each fighter in every section and set up a REST API that allows us to query for a fighter by their **name**. This will return the list of statistics for which that fighter is in the top 10 of all time, including:

* The **statistic** (eg. Total fights, wins)
* The **value** for that statistic (eg. 12)
* The **position** in the leaderboard they sit (eg. 2)

Let’s get into it.

# How?

At a high level, here are the main problems we need to solve to complete our web scraper. 	 

* Trigger a scrape the UFC website by performing an HTTP request 	 
* Store the data from the scrape in a database 	 
* Provide a way to query data out of the database via an HTTP request 	 
* Deploy everything above to the cloud in a simple, self-contained way. 	

## AWS Cloud Development Kit

In order to perform the above steps, we are going to make use of the AWS Cloud Development Kit.

> The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to model and provision your cloud application resources using familiar programming languages.

The AWS CDK lets us describe all of our cloud resources using code, and deploy them to AWS easily. This is useful for developers because it allows us to group our resources together.  This enables us to tear down or update everything related to the scraper (database, API) at once, reducing the chance of potential mistakes. Another benefit of the CDK is the ability to version control your infrastructure and roll back to an old version if something goes wrong. This makes it a powerful tool for teams.

## Creating your AWS Account and IAM User

If you haven’t already got an AWS account, you will need to [create one and configure your security credentials](https://docs.aws.amazon.com/cdk/latest/guide/work-with.html). 

## Getting Started with the AWS CDK

Now we are all set up with our AWS credentials, we can get set up with the CDK. The AWS CDK is developed at Amazon with TypeScript, so we can install it globally from NPM.

```bash
npm install -g aws-cdk
```

You can then create a new AWS CDK project by invoking `cdk init` in an empty directory. Since we will be writing our Lambdas in JavaScript, we’ll use the JavaScript CDK language option as well. The AWS CDK also supports several other languages such as Python, Java and C#. Let’s run the following to set up our project.

```bash
mkdir aws-cdk-scraper
cd aws-cdk-scraper
cdk init app —language javascript
cdk bootstrap
```

That’s it! The CDK will create some files and directories for us, as well as set up the relevant config in AWS and we are ready to start building. We’ll come back to the CDK whenever we are ready to start deploying our resources in the cloud, but now it’s time to.. 

### Design our Scraper

When programming any web scraper, it is imperative that you know the structure of the HTML of the webpage that you are trying to scrape. You can see this by using the developer tools in your preferred browser.

If you look at http://statleaders.ufc.com/ you will see it looks like this: 

![UFC Stat Leaders Page](https://martinmck.com/images/screenshot_2020-04-18_at_12.19.48.png)

We  want to scrape **each section** of this site and populate our database with the data inside each section. Let’s break it down: 	 

* Get the title of the section (Total Fights, Wins, etc) 	 

  * Iterate through the top 10 fighters for that section 	
  * Write each fighter into the database with their name, statistic, leaderboard position, and value for that statistic. For example:

```
{ name: “Donald Cerrone”, statistic: “Wins”, value: 23, position: 1 } 
```

We know our steps. Let’s write our scraper!

### Coding the Scraper

In order to build our scraper, we will need to install some modules from NPM.

#### cheerio

[Cheerio](https://github.com/cheeriojs/cheerio) is a library that parses markup and exposes a simple, jQuery-like API. It’s a very popular choice for web scraping with JavaScript. We will be using it to parse the HTML we fetch from the UFC website.

#### node-fetch

We need to make an HTTP GET request to fetch the HTML from the UFC website so we can parse it. Feel free to use your library of preference here, or even the native node `http` module. We will be using `node-fetch` in this tutorial for our code examples.

We will store our scraper code in a `src` directory in the root of our project. Let’s create that and install the modules above. We need to install these modules in the `src` directory so that they will be available in the AWS Lambda environment when we deploy our code to the cloud.

```
mkdir src
cd src
npm init -y
npm i —-save cheerio node-fetch
```

Next up, we can create a scraper by defining the `src/scraper.js` file with the code below. Please take some time to read the comments and understand the control flow.

```javascript
// src/scraper.js
const cheerio = require("cheerio");
const fetch = require("node-fetch");

const TABLE_NAME = process.env.TABLE_NAME;
const SCAPER_URL = "http://statleaders.ufc.com/";

exports.scrape = async function (event, context) {
	// fetch the HTML from the UFC website
  const response = await fetch(SCAPER_URL);
  const html = await response.text();
  const $ = cheerio.load(html);

	// get each section
  const resultsGroup = $(".results-group");

  const fighterStatistics = [];

	 // iterate over the sections
  resultsGroup.each(function (i, result) {
	  // get the name of the statistic (“Total Fights”, etc)
    const statistic = $(result).find("header > h3").text();

	  // find each row in the table for that statistic
    $(result)
      .find(".results-table--tr")
      .each(function (i, row) {
        const result = $(row)
          .find("span")
          .map((i, el) => $(el).text())
          .get();
			
		  // get the data from each table row
        const [position, name, value] = result;
        const isHeaderRow = result.length > 3;

        if (!isHeaderRow) {
          fighterStatistics.push({
            name,
            statistic,
            value,
            position
          });
        }
      });
  });
	
	// let’s print the results for debugging purposes
  fighterStatistics.forEach(console.log);
};
```

Let’s test it out.

```bash
node src/scraper
```

Looking good! The scraper has scraped the UFC stats website and printed out a list of our results.

![List of UFC Scraper Results](https://martinmck.com/images/screenshot_2020-04-18_at_18.33.37.png)

### Deploying Our Scraper to AWS

Our scraper is now working locally, but we want to get it up and running in the cloud. As mentioned before, we are using the AWS CDK to create the cloud resources we need for our scraper. Let’s dive in.

### Creating a Serverless App with the CDK

The scraper will be composed of several components and AWS services:

* AWS Lambda functions for the scraper and our API written in NodeJS
* An API to expose our lambda functions over HTTP using AWS API Gateway
* A DynamoDB database to store and query our data

![AWS Architecture for Scraper](https://martinmck.com/images/aws-2019-horizontal-framework-1-.png)

Let’s start by installing all the necessary dependencies in the root of our project so we can create these resources with the AWS CDK.

```
npm install @aws-cdk/aws-apigateway @aws-cdk/aws-lambda @aws-cdk/aws-s3 @aws-cdk/aws-dynamodb
```

#### Creating Our DynamoDB Table and Serverless REST API

We can now create our dynamoDB table. Update `/lib/aws-cdk-scraper-stack.js` with the following code. 

```javascript
const cdk = require("@aws-cdk/core");
const dynamodb = require("@aws-cdk/aws-dynamodb");
const lambda = require("@aws-cdk/aws-lambda");
const apiGateway = require("@aws-cdk/aws-apigateway");

class AwsCdkScraperStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create our DynamoDB table
    const fighterStatsTable = new dynamodb.Table(this, "FighterStatsTable", {
      partitionKey: {
        name: "name",
        type: dynamodb.AttributeType.STRING
      },
      sortKey: {
        name: "statistic",
        type: dynamodb.AttributeType.STRING
      },
    });

    // Create our lambda
    const scraperLambda = new lambda.Function(this, "scraper", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "scraper.scrape",
      timeout: cdk.Duration.seconds(20),
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    // Create API Gateway API
    const api = new apiGateway.RestApi(this, "ufc-scraper-api", {
      restApiName: "UFC Stats Scraper",
      description: "UFC Stats Scraper API.",
    });

    // call our lambda when someone makes a GET request to /scrape
    const scrape = api.root.addResource("scrape");
    const scraperIntegration = new apiGateway.LambdaIntegration(scraperLambda);
    scrape.addMethod("GET", scraperIntegration);

    // allow the scraper lambda to write to dynamoDB
    fighterStatsTable.grantWriteData(scraperLambda);
  }
}

module.exports = { AwsCdkScraperStack };
```

As with all software development, it’s usually a good idea to test incrementally. Let’s deploy these resources to the cloud and test that our scraper and shiny new REST API are working by sending an HTTP request to the `/scrape` endpoint we’ve just created. Deploying with the CDK couldn’t be simpler - just run

```
cdk deploy
```

AWS will do its thing - creating your Lambda, DynamoDB table, and API Gateway REST API. When done, you should receive some output in your console with the URL for your new API! Copy the URL to your clipboard.

![AWS Scraper Stack URL](https://martinmck.com/images/screenshot_2020-04-18_at_19.20.10.png)

Before we move on - let’s test our new scraper API with `curl`.

```bash
curl -H ‘Content-Type: application/json’ -H ‘Accept: application/json’ <your-api-url>
```

You should now see the results of the scrape in your terminal. 🥳

### Interacting with DynamoDB

Okay, our scraper is working, but it isn’t really that useful yet. We need to store the data from our scrapes in our database, and provide a way to query this data.

We will be using dynamoDB to store each of the fighters statistics keyed by their **name** so we can fetch all of the areas where a fighter ranks in the UFC top ten by passing their name to our API. This is not recommended in production apps because a persons name is not always unique.

#### Writing the Scrape Result into Dynamo

To write the array of results from our scrape into DynamoDB, we must use the AWS SDK for node. We are going to use the DynamoDB `DocumentClient` which is a nice wrapper for interacting with Dynamo. 

We will need to update `src/scraper.js` to write our scrape results to the `FighterStatsTable` in DynamoDB. 

```js
const AWS = require(“aws-sdk”); 
const cheerio = require(“cheerio”); 
const fetch = require(“node-fetch”);

const TABLE_NAME = process.env.TABLE_NAME;
const SCAPER_URL = “http://statleaders.ufc.com”;

const client = new AWS.DynamoDB.DocumentClient();

exports.scrape = async function (event, context) {
  // fetch the HTML from the UFC website
  const response = await fetch(SCAPER_URL);
  const html = await response.text();
  const $ = cheerio.load(html);
	
  // get each section
  const resultsGroup = $(“.results-group”);

  const fighterStatistics = [];

  // iterate over the sections
  resultsGroup.each(function (i, result) {
    const statistic = $(result).find(“header > h3”).text();
	
	  
    // find each row in the table for that statistic
    $(result)
      .find(“.results-table—tr”)
      .each(function (i, row) {
        const result = $(row)
          .find("span")
          .map((I, el) => $(el).text())
          .get();
        
        // get the data from each table row
        const [position, name, value] = result;
        const isHeaderRow = result.length > 3;

        if (!isHeaderRow) {
          fighterStatistics.push({
            name,
            statistic,
            value,
            position
          });
        }
      });
  });

  
  try {
    // Write all the fighter statistics into DynamoDB
    await Promise.all(fighterStatistics.map((result) =>
      client
        .put({
          TableName: TABLE_NAME,
          Item: result,
        })
        .promise()
    ));

    return {
      statusCode: 200,
      headers: {
        “Content-Type”: “application/json”,
      },
      body: JSON.stringify({
        results: fighterStatistics,
      }),
    };
  } catch (error) {
    return {
      statusCode: 500,
      body: JSON.stringify(error),
    };
  }
};
```

Let’s test this out.

```bash
curl -H ‘Content-Type: application/json’ -H ‘Accept: application/json’ <your-api-url>
```

If you check your [DynamoDB console](https://console.aws.amazon.com/dynamodb/home) in AWS , you should be able to see the results from the updated scraper which is now successfully persisting to our database!

![Scraper Results in the DynamoDB console](https://martinmck.com/images/screenshot_2020-04-21_at_22.29.35.png)

#### Querying DynamoDB

Now that we have results in Dynamo, we now want to **query** these results from our Fighter Statistics Table. Since we've used a composite key, this enables us to query using a fighters name, and get back all the statistics for which they are in the top 10 of all time in the UFC. Let’s start by creating a lambda at `src/queryStats.js` that will query the results from Dynamo.

```js
const AWS = require("aws-sdk");

const TABLE_NAME = process.env.TABLE_NAME;

const client = new AWS.DynamoDB.DocumentClient();

exports.query = async function (event) {
  if (event.body) {
    const body = JSON.parse(event.body);
    const result = await client
      .query({
        TableName: TABLE_NAME,
        ExpressionAttributeNames: { "#column": "name" },
        KeyConditionExpression: "#column = :name",
        ExpressionAttributeValues: { ":name": body.name, },
      })
      .promise();

    return {
      statusCode: 200,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(result.Items),
    };
  } else {
    return {
      statusCode: 400,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        message: "No Fighter Name Provided.",
      }),
    };
  }
};
```

#### Creating our new API in AWS
Now that we've written the code to actually query the database, we need to deploy it to AWS and put it behind API Gateway so that we can send HTTP requests to it. Update your `lib/aws-cdk-scraper-stack.js` to create our lambda and attach it to API Gateway at the `/fighter` path, meaning we can POST to that endpoint to query our data in Dynamo.
```javascript
const cdk = require("@aws-cdk/core");
const dynamodb = require("@aws-cdk/aws-dynamodb");
const lambda = require("@aws-cdk/aws-lambda");
const apiGateway = require("@aws-cdk/aws-apigateway");

class AwsCdkScraperStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create our DynamoDB table
    const fighterStatsTable = new dynamodb.Table(this, "FighterStatsTable", {
      partitionKey: {
        name: "name",
        type: dynamodb.AttributeType.STRING,
      },
      sortKey: {
        name: "statistic",
        type: dynamodb.AttributeType.STRING,
      },
    });

    // Create our lambda
    const scraperLambda = new lambda.Function(this, "scraper", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "scraper.scrape",
      timeout: cdk.Duration.seconds(20),
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    const fighterApiLambda = new lambda.Function(this, "fighterApi", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "queryStats.query",
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    // Create API Gateway API
    const api = new apiGateway.RestApi(this, "ufc-scraper-api", {
      restApiName: "UFC Stats Scraper",
      description: "UFC Stats Scraper API.",
    });

    // call our lambda when someone makes a GET request to /scrape
    const scrape = api.root.addResource("scrape");
    const scraperIntegration = new apiGateway.LambdaIntegration(scraperLambda);
    scrape.addMethod("GET", scraperIntegration);

    const fighterApi = api.root.addResource("fighter");
    const apiIntegration = new apiGateway.LambdaIntegration(fighterApiLambda);
    fighterApi.addMethod("POST", apiIntegration);

    // allow the scraper lambda to write to dynamoDB
    fighterStatsTable.grantWriteData(scraperLambda);
    // allow the api lambda to read from dynamoDB
    fighterStatsTable.grantReadData(fighterApiLambda);
  }
}

module.exports = { AwsCdkScraperStack };
```

Now run `cdk deploy` again to deploy this latest stack.

#### Querying our API

We can query our new API using an HTTP POST request, with a request body containing the name of the fighter we want to query for. For example, if you wanted to see all the statistics where Jon Jones ranks in the top 10 of all time, you would perform the following `curl` request.

```bash
curl <your-api-url>/fighter -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "name": "Jon Jones" }'
```

You should get back the following results:

```json
[
    {
        "value": "0.08%",
        "statistic": "Bottom Position Percentage",
        "name": "Jon Jones",
        "position": "3"
    },
    {
        "value": "10",
        "statistic": "Decision Wins",
        "name": "Jon Jones",
        "position": "4"
    },
    {
        "value": "1463",
        "statistic": "Sig Strikes Landed",
        "name": "Jon Jones",
        "position": "6"
    },
    {
        "value": "95.0%",
        "statistic": "Takedown Defense",
        "name": "Jon Jones",
        "position": "2"
    },
    {
        "value": "14",
        "statistic": "Title Fight Wins",
        "name": "Jon Jones",
        "position": "1"
    },
    {
        "value": "5:40:15",
        "statistic": "Total Fight Time",
        "name": "Jon Jones",
        "position": "9"
    },
    {
        "value": "13",
        "statistic": "Win Streak",
        "name": "Jon Jones",
        "position": "2"
    },
    {
        "value": "20",
        "statistic": "Wins",
        "name": "Jon Jones",
        "position": "3"
    }
]
```

Nice one! Looks like everything is working as expected 😃

### Wrapping Up

We scraped the UFC website in this particular case, but these concepts are applicable to any web scraping you may want to do. Feel free to check out the full code for this tutorial here on [GitHub](https://github.com/shogunpurple/aws-cdk-scraper) or reach out to me on [Twitter](https://twitter.com/shogun_purple).

Thanks for reading!