Scraping GeeksforGeeks User Profiles with Node.js
Tags: webdev, javascript, tutorial, programming
Published: 2023-06-10 03:19:41

In this tutorial, we will explore how to scrape user profiles on GeeksforGeeks using Node.js. We'll utilize the Express framework, Axios for making HTTP requests, Cheerio for parsing HTML, and Mongoose for working with MongoDB.


### Project Setup

To get started, make sure you have Node.js and npm (Node Package Manager) installed on your machine. Then, follow these steps:

1. Create a new directory for your project and navigate to it in your terminal.
2. Initialize a new Node.js project by running the following command:

```
npm init -y
```
3. Install the necessary dependencies:

```
npm install express axios cheerio cors dotenv mongoose

```

### Setting up the Express Server

Let's start by setting up an Express server that will handle the scraping and API requests.

Create a new file called `server.js` and add the following code:

```
const express = require('express');
const axios = require('axios');
const cheerio = require('cheerio');
const cors = require('cors');
require("dotenv").config();
const mongoose = require("mongoose");

const app = express();
app.use(express.json());

app.use(cors());

app.get('/profile/:username', async (req, res) => {
  
});

const port = process.env.PORT || 3000;
app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

```

### Scraping GeeksforGeeks User Profiles

To scrape user profiles on GeeksforGeeks, we'll use Axios to make HTTP requests and Cheerio to parse the HTML response.

Within the `/profile/:username` route, add the following code:

```
app.get('/profile/:username', async (req, res) => {
  const { username } = req.params;
  try {
    const url = `https://auth.geeksforgeeks.org/user/${username}`;
    const response = await axios.get(url);
    const $ = cheerio.load(response.data);

    const name = $('.profile_name').text().trim();
    const institution = $('.basic_details_data a').text().trim();
    const longestStrek = $('.streakCnt.tooltipped').text().trim();
    const rank = $('.rankNum').text().trim();
    const coddingScore = $('.score_card_value').text().trim();
    const overallScore = coddingScore.slice(0, 3);
    const solvedProblemsCount = coddingScore.slice(3).replace(/_/g, '');
    const totalSubmissions = $('.heatmap_header .numProblems').text().trim();

    const profile = {
      name,
      username,
      institution,
      rank,
      overallScore,
      solvedProblemsCount,
      longestStrek,
      totalSubmissions
    };

    res.json(profile);
  } catch (error) {
    console.log(error);
    res.status(500).json({ error: 'Failed to fetch profile data' });
  }
});

```
In the above code, we make a GET request to the GeeksforGeeks user profile URL, load the HTML response using Cheerio, and then scrape the required profile information.

## Building the Frontend

Now that we have the backend set up to scrape and store profiles, let's build a simple frontend to display the profile data.

Create a new file called `index.html` and add the following code:

```
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>GeeksforGeeks User Profiles</title>
</head>
<body>
  <h1>GeeksforGeeks User Profiles</h1>
  <input type="text" id="username" placeholder="Enter a username">
  <button onclick="fetchProfile()">Fetch Profile</button>
  <div id="profile"></div>

  <script src="script.js"></script>
</body>
</html>

```
In the above HTML, we have a simple form to enter a username and a button to fetch the profile data.

Create a new file called `script.js` and add the following code:

```
async function fetchProfile() {
  const username = document.getElementById('username').value;
  const response = await fetch(`http://localhost:3000/profile/${username}`);
  const data = await response.json();
  const profileDiv = document.getElementById('profile');
  profileDiv.innerHTML = JSON.stringify(data, null, 2);
}

```
This JavaScript code fetches the profile data from the server and displays it in the `profile` div.

### Running the Project

To run the project, start the Express server by running the following command:

```
node server.js
```
Open `index.html` in a web browser, enter a GeeksforGeeks username, and click the "Fetch Profile" button. The profile data will be displayed on the page.

That's it! You have successfully created a Node.js project that scrapes user profiles on GeeksforGeeks. You can now deploy it to a hosting platform and share it with others.

Happy coding!