Build a CLI to crawl a web page with web-crawljs
Tags: node, beginners, webcrawler, tutorial
Published: 2020-04-19 23:11:24

---
title: Build a CLI to crawl a web page with web-crawljs
published: true
description: Building a simple CLI web crawler
tags: node,beginner,web-crawler,tutorial
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/vy5wf35wfwb9asbm8yq2.jpeg
---

**Moving Articles from medium to dev.to**

In this tutorial, we are going to create a web crawler that scraps information from Wikipedia pages. This web crawler would run
from a command-line interface (e.g. terminal, command prompt).


The code for this article is on [github](https://github.com/kayslay/Build-a-CLI-to-crawl-a-web-page-with-web-crawljs).

An example of the command that would crawl the page would look like

```bash
$ node crawl.js -d 3 -x wiki
```

The command will get a config file named `wiki` and saves the crawled data to a MongoDB collection called `wiki`.

## Web Crawling
   
Web crawlers are programs written to get information from a web page.

```
“A Web crawler, sometimes called a spider, is an Internet bot that systematically 
browses the World Wide Web, typically for the purpose of Web indexing”
 — Wikipedia
```


## What we will be needing
   
For this project, we will be needing `commander`, `web-crawljs`, and `mongoose`.

### Commander

Commander is an npm module that makes working with the command line interface easier. It makes it easy to handle command-line arguments. Check out it’s [documentation](https://www.npmjs.com/package/commander).

### web-crawljs
    
web-crawljs is an npm module that crawls web pages and extracts information from the page. It makes crawling web pages with Nodejs easy.

The only thing that web-crawljs needs, is a configuration object for it to start crawling.

### why web-crawljs

One of the reasons I chose web-crawljs is because of how easy it is to crawl web pages with it. It is also a light weight web crawler, that is, it uses far less CPU and RAM compared to using a headless browser (e.g Phanthomjs).

Due to the less CPU and RAM usage, it can not render a SPA (single page application) pages. And also because I built It :).

All that is required to run it is Nodejs, no need for installing Phanthomjs on your machine; so far you have node installed, you are good to go.

### mongoose
    
Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment. It’s an Object Data Modeling library that gives a modeling environment for MongoDB and it enforces a more structured data model.

Mongoose gives us the ability to create MongoDB data Models and Schemas.

We are going to use mongoose to save the information extracted from a page to the MongoDB database.

## Project Structure

The structure of this project would look like this.

        ├── config
        │   ├── db.js
        │   └── wiki.js
        ├── crawl.js
        ├── package.json
        ├── package-lock.json
        └── readme.md
        
### crawler/config
    
The main file in the crawler/config folder is the db.js. This file contains the configuration for our database. The wiki.js is the javascript file that will hold the configuration for web-crawljs.

Apart from `db.js`, all other files are configurations for `web-crawljs`.

### crawler/config
     
 The main file in the crawler/config folder is the db.js. This file contains the configuration for our database. The wiki.js is the javascript file that will hold the configuration for web-crawljs.
 
 Apart from db.js, all other files are configurations for web-crawljs.
 
 
## What we will crawl
    
In this article, we are going to extract some information out of Wikipedia and save it to a MongoDB database. The information we want to extract from the page are:

- title of the wiki content
- content of the wiki page
- all the reference link

## Requirements
    
For this tutorial, Nodejs and MongoDB must be installed on your machine. And I’ll be making use of node 7.8.0 and MongoDB version 2.6.10. I am also making use of ES6 syntax (arrow function, destructuring e.t.c).

- node >=v7.0.0
- mongodb

## Let’s get started
   
   Now let’s go straight to business. we will start by creating a new folder called crawler
   
   
```bash
$ mkdir crawler
$ cd crawler #move into the folder
```

Now that it is done, let’s create the config directory inside the crawler directory

```bash
$ mkdir config
#create the config files
$ touch config/wiki.js config/db.js
#create the crawl.js file
$ touch crawl.js
```

time to create the package.json file. use the npm init -y command to create the package.json (using it because it’s easy).

```bash
$ npm init -y
```

## Installing the dependencies
   
 We are making use of only three dependencies in this project, the mongoose, commander, and web-crawljs module. To install this module we will use our good friend npm. run npm install --save web-crawljs mongoose to install the dependencies.
 
```bash
$ npm install --save web-crawljs mongoose commander
```
   
Now that it is installed let's move to next stuff

### config/db.js
    
This file holds the configuration details of our MongoDB database

```javascript
/**
 * Created by kayslay on 6/3/17.
 */
module.exports = {
    dbName: "crawl",
    dbHost: "localhost",
};
```

### config/wiki.js
    
The `config/wiki.js` file holds the configuration we will use to crawl our Wikipedia page.

```javascript
/**
 * Created by kayslay on 6/3/17.
 */
const mongoose = require('mongoose');
const dbConfig = require('../config/db');
//mongoose configs
const Schema = mongoose.Schema;
//creating a schema for the extracted data
const wikiSchema = new Schema({
    title: String,
    body: String,
    references: [String]
});
//connect to mongo db
mongoose.connect(`mongodb://${dbConfig.dbHost}/${dbConfig.dbName}`);
//create the model
const wikiModel = mongoose.model('Wiki', wikiSchema);

//crawl config
module.exports = {
	//the selectors on page we want to select
	//here we are selecting the title, a div with an id of mw-content-text and links with a
	//class name of external and text
    fetchSelector: {title: "title", body: "div#mw-content-text",references: 'a.external.text'},
	//what we want to select from the selector
	//for the title and body we want the text
	//for the references we want to get the href of the links
    fetchSelectBy: {title: "text", body: "text",references:['attr','href']},
	// the same rules apply to the nextSelector and nextSelectBy
	//but this is used to get the links of the page to crawl next
    nextSelector: {links: 'a[href^="/wiki"]'},
    nextSelectBy: {links: ['attr','href']},
	//this changes the next selector when the links match .svg
    dynamicSchemas:{
        nextSelector:[{url:/\.svg/,schema:{links:""}}]
    },
	//formats the url
    formatUrl: function (url) {
        if((/\.svg?/.test(url) || /[A-Z]\w+:\w+?/.test(url))){
            //returning a visited string so that it does not visit the link
            //when the url ends with `.svg` or something like `Wikipedia:About`
        return 'https://en.wikipedia.org/wiki/Web_crawler/'
        }
        return url;
    },
	//what we want to do with the data extracted from the page
	//we want to save it to a mongodb database
    fetchFn: (err, data, url) => {

        if (err) {
            return console.error(err.message);
        }
        let {title, body, references} = data;
        let wikiData = {title: title[0], body: body[0], references};
        wikiModel.create(wikiData, function (err, wiki) {
            console.log(`page with a title ${wiki.title}, has been saved to the database`);
        });
    },
	//called at the end of the whole crawl
    finalFn: function () {
        console.log('finished crawling wiki');
    },
    depth: 3, //how deep the crawl should go
    limitNextLinks: 10,// limit the amount of links we get from wikipedia to 10. this helps when you dont want to get all the links
    urls: ['https://en.wikipedia.org/wiki/Web_crawler/'] //the default urls to crawl if one is not specified
};
```

### crawl.js

```javascript
#!/usr/bin/env node
/**
 * Created by kayslay on 5/31/17.
 */
const crawler = require('web-crawljs');
const program = require('commander');

//commander configuration
function list(val) {
    "use strict";
    return val.split(',');
}

program
    .option('-x --execute <string>', 'the configurtion to execute')
    .option('-d --depth [number]', 'the depth of the crawl')
    .option('-u --urls [items]', 'change the urls',list)
    .parse(process.argv);

//throw an error if the execute flag is not used
if (!program.execute) {
    throw new Error('the configuration to use must be set use the -x flag to define configuration;' +
        ' use the --help for help')
}
//holds the additional configuration that will be added to crawlConfig
const additionalConfig = {};

//set the object that will override the default crawlConfig
(function (config) {
    //depth
    if (program.depth) config['depth'] = program.depth;
    if(!!program.urls) config['urls'] = program.urls

})(additionalConfig);

//the action is the file name that holds the crawlConfig
let action = program.execute;


try {
    //set the crawlConfig 
    //adds the additional config if need
    let crawlConfig = Object.assign(require(`./config/${action}`), additionalConfig);
    const Crawler = crawler(crawlConfig);
    Crawler.CrawlAllUrl()
} catch (err) {
    console.error(`An Error occurred: ${err.message}`);
}
```

The crawl.js file is the main file of this project. This file is what we will run using the `node` command. It’s our entry point.

It depends on two packages: web-crawljs and commander. Which were imported on lines 5 and 6.

From line 9 to line 18 we set up the flags needed to be used by our CLI.

Thanks to commander this is very easy to achieve. Check its documentation for more.

Line 21 all the way down to line 37, configures the values gotten from the CLI.

The comment in the file should explain what's going on.

The lines that follow just performs the web crawl operation.

## Let’s test our crawler
   
Now that's all the code has been written, it’s time to test the crawler.

Type the following in your terminal

```bash
$ node crawl.js -x wiki
```

When we check our MongoDB collection we will see the title, body and reference added to it.

Instead of using the default Wikipedia URL, we are going to use our own wiki page URL.

```bash
$ node crawl -u https://en.wikipedia.org/wiki/Web_crawler -x wiki
```

This will not start crawling from the default https://en.wikipedia.org/ but would start crawling from `https://en.wikipedia.org/wiki/Web_crawler`. 
To add more URLs, separate the URLs by commas.

## Conclusion
   
We now know how to create a web crawler using `web-crawljs`, `commander` and `mongoose` :).

And to those who don’t know how easy it is to create a Command Line Interface with Nodejs is; Now you know.

This is at least one more thing you know.

Thanks for reading and please recommend this post.