GitDesc: Extract ReadMe from GitHub
Tags: github, webdev, cheerio, tutorial
Published: 2021-03-02 15:49:57

###Hello there!
####Welcome to my first post and my first ever tutorial.
#####I started learning web development last year, just HTML and CSS. Started learning again last month and I thought I'd learn best by writing about it as I learn and I am fairly new to this.
######We will be making a website to extract the description from GitHub repositories, all we need is the link to the repository.

**Note:** I am using [Hyper terminal](hyper.is) for the execution of the commands and I am hoping that you have the latest version of [Node.js](nodejs.org) installed as well as [npm](npmjs.com). You could use any other terminal, or even do things manually if you're such a masochist. XD
Alright now, let's get to work!

* Make a directory or folder anywhere you like and name it anything you like. For more user-friendliness I'd be using the name: `GitDesc`.
To create the folder, use command: `mkdir GitDesc`
After you have done so, change your current directory to the `GitDesc` directory by using command: `cd GitDesc`.


* Now it's time to initialize `npm` and install the required packages for our project. So go to the terminal and run the following commands to initialize and install packages.

1.  `npm init`
    You will be asked some details to fill about the project you
    are creating. If you don't want to fill anything, you can just
    press enter until it stops asking.
    Or there is another shortcut to automatically fill in the
    details using command `npm init -y`.

2.  `npm install express --save`
    This is a framework for creating the web application, which is
    just what we want. If you wanna learn more, here's the link: [Express] (https://expressjs.com).

3.  `npm install body-parser --save`
    We will make a HTML file for the user to put the link of
    certain GitHub repository and this link would be transferred
    to the server-side JavaScript using this package.
    After that, we will make a connection request from that link
    to GitHub and the required ReadMe content will be extracted.

4.  `npm install cheerio --save`
    With its help, we will narrow down our required data into
    text. This is an API used for **Web Scraping** and that's what
    we are doing here too.

5.  `npm install request --save`
    This is required in order to make HTTP request to the website.

####You can omit the use of `--save`


* Now we will create two files, HTML file for the user-interface and a JavaScript file which would be responsible for the server-side scripting.
`touch` command is used to make files and the name we will use are, "index" and "app".
So do the following,
`touch index.html`
`touch app.js`


* Open the project in code editor. I am using [Atom](atom.io) and to open the project in atom via the command-line, we use command: `atom .` in the working directory. This will open up Atom with the project we are currently working on.


* Open `app.js` file.
First, we will have to import all the packages that we have installed so far.
```javascript
const cheerio = require('cheerio');
const request = require('request');
const express = require('express');
const bodyParser = require('body-parser');
```

* Create a new app and set-up the body-parser.
```javascript
const app = express();

app.use(bodyParser.urlencoded({extended: true}));
```
For more information on body-parser, you can check [this] (https://stackoverflow.com/questions/55558402/what-is-the-mean-of-bodyparser-urlencoded-extended-true-and-bodyparser)


* Now as everything has been set-up and initialized, we can finally get down to the meat. But before that, we should think about how the interface is going to be.
####1. There will be a simple HTML page with an input box and a submit button.
####2. After pressing the button, the results will be displayed in the page.


* Now we will send the whole HTML file as the GET request.
```javascript
app.get("/", function(req, res)
{
  res.sendFile(__dirname + "/index.html");
})
```


* Below we are going to post the information we have received from GitHub to the webpage using POST.

```javascript
app.post("/", function(req, res)
{
  // This is the URL we have received from the HTML file.
  const url = req.body.githubLink;

  // This is HTTP request to the website.
  // response is received from the website.
  // html contains the website's html.
  request(url, function(error, response, html)
  {
    // This command will load the HTML of the
    // repository and store it in a constant.
    const $ = cheerio.load(html);
    
    // This is the class under which the GitHub's ReadMe
    // information is put, we are going to extract that
    // information and will store it in a constant.
    const body = $('.markdown-body');
    res.write("<h3>" + "Description" + "</h3>");
    
    // To extract just the text from the body,
    // we are using body.text().
    res.write("<textArea cols=\"90\" rows = \"30\">" + body.text() + "</textArea>");
    res.send();
  })
})
```


* To start the server, we will do the following:
```javascript
// You can use any port number you like.
// Most common is 3000.
app.listen(666, function()
{
    console.log("Server started...");
})
```


* Now for the final step, open up `index.html` and add the following lines within the `<body>` tag:
```html
<form action="/" method="post">
  <input type="link" name="gitHubLink" placeholder="Enter link here...">
  <button type="submit" name="submit">Submit</button>
</form>
```


* Run the web application we have just made after starting the server by typing `node app.js` in the terminal.
You'll see a log as, "Server started..." which means, the server is listening at port you have specified and there are no errors.


* Open your web browser and type in `localhost:666/`, don't forget to change the port number with the one you have specified.


* Copy the repository link from GitHub, paste it into the input box and Voila! You have made a web app which scrapes information about whatever is written in the ReadMe by the developer.

#####The link for the project on GitHub is [here](https://github.com/Syncrofosatron/Git-Desc).

######I really hope it was not very hard to follow.
######Suggestions to improve are always welcome :)