Scraping memes from reddit using node.js
Tags: webdev, javascript, beginners, node
Published: 2020-07-05 13:59:20

We all like memes, don't we? If you thought of making an application which serves memes from the internet but didn't know how to, you've come to the right post! Here I will show you how to scrap memes from [Reddit](https://reddit.com) yourself, and not relying on any other APIs. So let's get started!

We'll be using axios and cheerio for web scraping.

### tl;dr 
All the code demonstrated in this post is up on [GitHub](https://github.com/k4u5h4L/meme-scrape-reddit)

## Prerequisites:
* Node.js installed.
* Npm installed. This should come with node.js.

To check if they are installed, type
```
node --version
```
and
```
npm --version
```

Now that everything is installed, we can start.

## Start

We will start on an empty folder. Run
```
npm init -y
```
to generate a `package.json` file. Now we can install required dependencies. Run
```
npm install axios cheerio
```

Now let's actually start coding some JavaScript!

Make a file names `index.js` in the root directory of the project, and open it in your preferred text editor.

Now import the required libraries into your project.
```js
const axios = require("axios");
const cheerio = require("cheerio");
```

Now we will choose a site to scrap from. For the sake of this guide I will be scraping memes from [r/dankmemes](https://reddit.com/r/dankmemes).

```js
const mainUrl = `https://reddit.com/r/dankmemes`;
```

According to the documentation of [axios](https://github.com/axios/axios), we will set up the intial code.
```js
axios
    .get(mainUrl)
    .then((response) => {
        console.log(response.data);
    })
    .catch((err) => {
        console.log(err);
    });
```

The `.get()` method takes in the URL of the site. Because axios runs asynchronously, you have to add a `.then()` method to do something with the data. We will just take the data and log it in the console. 
In this process, if something goes wrong, the `.catch()` method catches the error and displays it. This is put for better error handling.
So now let's finally run the code and see what we get!
```
node index.js
```

And whew! We get loads of text. But this "text" is actually the HTML code what is hosted in the URL we specified.

But we needed only the image source right? So now we should parse and filter the HTML got using an amazing library called cheerio. Cheerio helps us parse HTML in a jQuery like fashion, which makes is amazingly easy to do out job. And it's fast too!

But we need to see what to filter right? To know that, we have to visit the URL we specified, which is [https://reddit.com/r/dankmemes](https://reddit.com/r/dankmemes). So head on to the site in another tab. 

When everything finished loading, right click on any image post and choose `inspect element`. This should open chrome's developer tools. When the image element is highlighted, you should see some other attributes inside the `<img />` tag.

Below I have taken a random post on the subreddit, and you can see a `src=""` attribute on the right side of the screen. That is the data we need to scrap! But how to we exactly locate that image? Simple, we look into other attributes of the same HTML element.

![reddit screenshot](https://dev-to-uploads.s3.amazonaws.com/i/pl6v33pe1585vl0qtixu.png)

Here in our case we can see that the image as a class of
```html
<img alt="Post image" class="_2_tDEnGMLxpM6uOa2kaDB3 ImageBox-image media-element _1XWObl-3b9tPy64oaG6fax" src="https://preview.redd.it/g64fe51e6z851.jpg?width=640&crop=smart&auto=webp&s=c5917f6…" style="max-height: 512px;">
```

Bingo! We got hold of the different classes in the image tag. So now let's continue with parsing this HTML data.

SO instead of console logging the HTML, we will pass it into another function to parse this data.
So this should be your axios part so far.
```js
axios
    .get(mainUrl)
    .then((response) => {
        dealWithData(response.data);
    })
    .catch((err) => {
        console.log(err);
    });
```

Now create a function called `dealWithData()` or any other name you have given in the `.then()` method.

Now we will add some code inside that new function.
```js
const dealWithData = (html) => {
    const $ = cheerio.load(html);
    const urlMeme = $("._2_tDEnGMLxpM6uOa2kaDB3.ImageBox-image.media-element._1XWObl-3b9tPy64oaG6fax");
    const indexValue = 0;
    console.log(`Source is:\n${urlMeme[indexValue].attribs.src}`);
};
```

Here, we are assigning the $ sign with the parsed html content using cheerio, just to make it more jQuery-like.

Now that we know our image element's attributes, we assign the HTML content of the image to a variable. Note that all images will have the same classname, so you will get an array of image elements returned. You can try to console log to see it. So we will assign an index value 0, which gets us the first image. And then we log the src of the image element with its index value. This will take some time to run but you should eventually get the result.

But there is a problem here. Notice that if you run this multiple times, there is a chance you get the same image source again and again. So instead of hard coding the index value, we will generate a random value.
```js
const randNo = (limit) => {
    const thatNo = Math.floor(Math.random() * limit);
    return thatNo;
};
```

Also don't forget to update the same in your `dealWithData()` function or whatever name you have given it.

```js
const indexValue = randNo(urlMeme.length);
```

We will pass the length of the array to get a random index number. Now putting together all this code, we will run the full program.

And wow! you should get an output of this kind.
```
Source is:
https://preview.redd.it/gnmgdb09q0951.jpg?width=640&crop=smart&auto=webp&s=8175c12e8aaa356af8f7cc78fe4e0b83d37341e2
```

And done. You can visit the link and check. You now have your very own meme scraper!

The same code can be used to scrap from different subbreddits. Just specify the required URL when starting out.

Happy coding! Cheers!