Strapi, another use case: Build your own API from any website with Puppeteer
Tags: node, headless, cms, react
Published: 2020-04-17 17:12:10

The objective of this tutorial is to build a simple job search API with Strapi and Puppeteer. [Strapi](https://strapi.io/) is an open-source [Headless CMS](https://headlesscms.org/) written in NodeJS and [Puppeteer](https://pptr.dev/) is an open-source [Headless Browser](https://github.com/dhamaniasad/HeadlessBrowsers) (Chrome) NodeJS API.

It seems that the time is for headless tools...:laughing: (Anyway, there is no direct link between Strapi & Puppeteer except the "Headless" word.)

### Strapi
Strapi is used to build powerful APIs without efforts. Several features are available in Strapi including CRON tasks configuration (And this is a good thing because we will use them to schedule the Puppeteer script execution).

#### 1. Strapi installation
Well, let's start this tutorial. The first thing we need to do is to install Strapi.
```
yarn create strapi-app job-api --quickstart
```
> If you don't want to use `yarn`, there are other possibilities to install Strapi in the [documentation](https://strapi.io/documentation/3.0.0-beta.x/getting-started/installation.html). 

#### 2. Strapi admin user

This command should install Strapi and open your browser. Then, you will be able to create your admin user.
![Strapi admin user creation](https://dev-to-uploads.s3.amazonaws.com/i/aowh6o2zoc301et8psnl.png)

#### 3. Job Collection type

In the Strapi admin home page, click on the blue button `CREATE YOUR FIRST CONTENT-TYPE`.
![Strapi admin home page](https://dev-to-uploads.s3.amazonaws.com/i/ddjgh6bslluyuruvo1oj.png)
You will be redirected to the collection type creation form.
![Strapi collection type creation](https://dev-to-uploads.s3.amazonaws.com/i/l4xjmpwd4751mzxhpp22.png)

After that, you will be able to add fields to the Job collection type.
![Strapi fields list](https://dev-to-uploads.s3.amazonaws.com/i/7hu3j3hkdu8ca96sn86n.png)
![Strapi field form](https://dev-to-uploads.s3.amazonaws.com/i/idjrea9nvlaibwrow5br.png)

For our basic example, we will need to create five text fields (title, linkedinUrl, companyName, descriptionSnippet, and timeFromNow).
![Strapi Job fields](https://dev-to-uploads.s3.amazonaws.com/i/dth15nt8qfhskjgto0s9.png)

#### Don't forget to click on the Save button to restart the Strapi server
![Strapi server restart](https://dev-to-uploads.s3.amazonaws.com/i/qb8skbw0rjj38roulryo.png)
After that, we can put the Strapi admin aside for the moment and open the Strapi repository in an editor.

### Strapi CRON task
Firstly, we'll need to enable CRON in the Strapi server configuration. 
Open the `config/environments/development/server.json` file
```json
{
  "host": "localhost",
  "port": 1337,
  "proxy": {
    "enabled": false
  },
  "cron": {
    "enabled": true
  },
  "admin": {
    "autoOpen": false
  }
}
```
Then let's create the CRON task. Open the `~/job-api/config/functions/cron.js` file and replace the content by this
```javascript
"use strict";
module.exports = {
  // The cron should display "{date} : My super cron task!" at every minute.
  "*/1 * * * *": (date) => {
    console.log(`${date} : My super cron task!\n`);
  },
};
```
Now, restart the Strapi server and let's see if our cron task is running properly.
```
yarn develop
yarn run v1.21.1
$ strapi develop

 Project information

┌────────────────────┬──────────────────────────────────────────────────┐
│ Time               │ Thu Apr 16 2020 01:40:49 GMT+0200 (GMT+02:00)    │
│ Launched in        │ 1647 ms                                          │
│ Environment        │ development                                      │
│ Process PID        │ 20988                                            │
│ Version            │ 3.0.0-beta.18.7 (node v10.16.0)                  │
└────────────────────┴──────────────────────────────────────────────────┘

 Actions available

Welcome back!
To manage your project 🚀, go to the administration panel at:
http://localhost:1337/admin

To access the server ⚡️, go to:
http://localhost:1337

Thu Apr 16 2020 01:41:00 GMT+0200 (GMT+02:00) : My super cron task !

Thu Apr 16 2020 01:42:00 GMT+0200 (GMT+02:00) : My super cron task !

Thu Apr 16 2020 01:43:00 GMT+0200 (GMT+02:00) : My super cron task !

...
```
We can see that `{date} : My super cron task !` is displayed every minute in the terminal.

### Puppeteer
Puppeteer is used to automating any action you can perform in the browser. You can use it to automate flows, take screenshots and generate PDFs. In this tutorial, we will use Puppeteer to get the list of ReactJS jobs from Linkedin. We will also use Cheerio to select the data in the received markup.

Now that the CRON task is working well, we will install Puppeteer and Cheerio in the Strapi project.
```bash
cd job-api
yarn add puppeteer cheerio 
```
Let's adapt the CRON task to get a [list of ReactJS job published on linkedin the last 24 hours in San Francisco](https://fr.linkedin.com/jobs/search?keywords=React.js&location=R%C3%A9gion%20de%20la%20baie%20de%20San%20Francisco&trk=guest_job_search_jobs-search-bar_search-submit&redirect=false&position=1&pageNum=0&f_TP=1).

In the `~/job-api/config/functions/cron.js`
```javascript
"use strict";
// Require the puppeteer module.
const puppeteer = require("puppeteer");

module.exports = {
  // Execute this script every 24 hours. (If you need to change the cron 
  // expression, you can find an online cron expression editor like 
  // https://crontab.guru
  "0 */24 * * *": async (date) => {
    // 1 - Create a new browser.
    const browser = await puppeteer.launch({
      args: ["--no-sandbox", "--disable-setuid-sandbox", "--lang=fr-FR"],
    });

    // 2 - Open a new page on that browser.
    const page = await browser.newPage();

    // 3 - Navigate to the linkedin url with the right filters.
    await page.goto(
      "https://fr.linkedin.com/jobs/search?keywords=React.js&location=R%C3%A9gion%20de%20la%20baie%20de%20San%20Francisco&trk=guest_job_search_jobs-search-bar_search-submit&redirect=false&position=1&pageNum=0&f_TP=1"
    );

    // 4 - Get the content of the page.
    let content = await page.content();
  },
};
```

Parse the html `content` with Cheerio and store the job with the Strapi global.
```javascript
"use strict";
const puppeteer = require("puppeteer");
const cheerio = require("cheerio");

module.exports = {
  "0 */24 * * *": async (date) => {
    const browser = await puppeteer.launch({
      args: ["--no-sandbox", "--disable-setuid-sandbox", "--lang=fr-FR"],
    });
    const page = await browser.newPage();
    await page.goto(
      "https://fr.linkedin.com/jobs/search?keywords=React.js&location=R%C3%A9gion%20de%20la%20baie%20de%20San%20Francisco&trk=guest_job_search_jobs-search-bar_search-submit&redirect=false&position=1&pageNum=0&f_TP=1"
    );
    let content = await page.content();

    // 1 - Load the HTML
    const $ = cheerio.load(content);

    // 2 - Select the HTML element you need
    // For the tutorial case, we need to select the list of jobs and for each element, we will
    // create a new job object to store it in the database with Strapi.
    $("li.result-card.job-result-card").each((i, el) => {
      if (Array.isArray(el.children)) {
        const job = {
          title: el.children[0].children[0].children[0].data,
          linkedinUrl: el.children[0].attribs.href,
          companyName:
            el.children[2].children[1].children[0].data ||
            el.children[2].children[1].children[0].children[0].data,
          descriptionSnippet:
            el.children[2].children[2].children[1].children[0].data,
          timeFromNow: el.children[2].children[2].children[2].children[0].data,
        };
  
        // 4 - Store the job with the Strapi global.
        strapi.services.job.create(job);
      }
    });

    // 5 - Close the browser
    browser.close();
  },
};
```

Restart the Strapi server and let's go back to the admin 
`http://localhost:1337/admin`.
In the Job content manager, you should see the data from LinkedIn
![Strapi content manager list view](https://dev-to-uploads.s3.amazonaws.com/i/b2jrqg0dbuayh9tvtfyd.png)
![Strapi content manager details view](https://dev-to-uploads.s3.amazonaws.com/i/hsnzfwdi3c3expx2sovg.png)

Good job ! You've just build an API from another website in few minutes :smile: