Building an HackerNews to Discord notification system using Node.js, node-cron and discord-notification.
Tags: javascript, node, beginners, webdev
Published: 2023-07-31 12:30:16

## **TL;DR**

In this article I am going to show how to build a simple service which monitors our HackerNews account and sends a notification to our Discord channel when there are new comments to our submissions.

We are going to use th following packages/technolgies:

- Node.js / express
- [discord-notification](https://www.npmjs.com/package/@penseapp/discord-notification)
- [node-cron](https://www.npmjs.com/package/node-cron)

---


## Before we start… I have a favour to ask. 🤗

I am building an open source feature flags platform and if you could **star the project on github** it would really help me to grow the project and would allow me keep writing articles like this!

[https://github.com/switchfeat-com/switchfeat](https://github.com/switchfeat-com/switchfeat) 


![thank you image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r78x0pf7qhdrlzf0rk2i.gif)


---

### Setup the Express server

Let’s start creating a `server` folder and install `express` and some other depedencies we are going use:

``` shell
cd server & npm init -y
npm install express cors typescript 
npm install @types/node @types/express @types/cors dotenv
npx tsc --init
```

Let’s create the `index.ts` file, and push the esieast Express configuration you can have in tyescript:

```tsx
import express, { Express } from 'express';
import cors from "cors"; 

const app: Express = express();
const port = 4000;

app.use(express.urlencoded({ extended: false }));
app.use(express.json());
app.use(cors());

app.get("/api", (req, res) => {
    res.json({
        message: "(HackerNews -> Discord) notifier",
    });
});

app.listen(port, () => {
    console.log(`⚡️[server]: Server is running on port 4000`);
});
```

In `package.json` of the sever, we need to add a script to run it:

```json
"scripts": {
    "start": "node index.js"
  }
```

To finish our setup, let’s compile typescript and run the server:

``` shell
tsc & npm run start
``` 

> Cool! Let’s go!

---

### Installing server dependancies

In the server folder let’s install the dependancies we need to interact with both HackerNews API and Discord webhook.

``` shell
npm i discord-notification node-cron @types/node-cron
```

---

### Building the HackerNews API connection

The HackerNews API exposes a bunch of public endpoints, there is no authentication, so I would assume there is some sort of rate limits in place, so be careful not to overload it with requests. More information [here](https://github.com/HackerNews/API). 

Let’s create a simple abstraction layer which would hold the connection to HackerNews and run simple `GET` requests to their API. Having these calls in separate file enforces separation of concerns and allows to use other libraries in the future with minimal effort.

The `hackerNews.ts` file would look like this: 

```tsx
export const getUser = async (userName: string) => {
    const resp = await fetch(getUserURL(userName), {
        method: "GET",
        headers: {
            Accept: "application/json"
        }
    });
    return await resp.json();
};

export const getSubmission = async (submissionId: string) => {
    const resp = await fetch(getItemURL(submissionId), {
        method: "GET",
        headers: {
            Accept: "application/json"
        }
    });
    return await resp.json();
};
```

Let’s start building the scheduling logic, in a way that our app will automatically fetch the latest data from HN on a schedule. For this, we are going to use the `node-cron` library which is a very solid and mature library which allows to run different sorts of scheduling patterns. 

We are going to setup the scheduler to check every 15min if there are new comments to our latest submission.

> `node-cron` uses cron-tab based cron expressions, you can see how to write more complex once [here](https://crontab.guru/).

Let’s create a `scheduler.ts` file as follows:

```tsx
import * as cronJob from "node-cron";

export const initScheduledJobs = () => {
  const scheduledJobFunction = cronJob.schedule("*/15 * * * *", () => {
    // - intaract with HN API
    // - send nofitication to Discord
  });

  scheduledJobFunction.start();
}
```

Finally let’s instruct our express server to start the scheduler at startup in the `index.ts`:

```tsx
import * as scheduler from "./scheduler";
scheduler.initScheduledJobs();
```

Great! Now that we have a scheduler in place, let’s add the HackerNews fetching logic to the `initScheduledJobs` function:

```tsx
const userData = await hackerNews.getUser(userName);
console.log(userData);
```

Running this code, we are going to see the response from HN for my user data `dev-bre` on the console.

```json
{
  created: 1587898826,
  id: 'dev-bre',
  karma: 23,
  submitted: [
    36846077, 36712635, 36712606, 36548356,
    36547441, 36147979, 36131968, 36131217,
    36131215, 36128924, 36022807, 34781854,
    34109805, 34109230, 34109200, 34105053,
    33909549, 33909537, 33887733, 33388763
  ]
}
```

The `submitted` field in the response contains an array of itemIds each one referring to a single submission I made, sorted by date. For the purpose of this article we are going only to monitor the latest submission, which means we are only interested in the first item in that array.

In order to get the list of comments for this post, we need to run an additional API call as follows: 

```tsx
const submissionData = await hackerNews.getSubmission(selectedPost);
```

The HN API returns the data in the form of a **Tree datastructure**, which means this API response would contain a field: `kids` in the form of an array of numbers which are the direct children of the current item, i.e. direct comments to the requested comment. In order to traverse the entire set of comments, on each level we need to use recursion.

---

## A recursive approach to get HN data

In order to get all comments to a post on any level, we need to go with a **recursive approach**!

Without going too much into the details of how recursion works, in a nutshell a recursive function is a function which calls itself progressively on a smaller data sets, until it reaches the base case which allows the function to return.

In our scenario, the base case is when we find a comment with no children (no one replied to that comment).  Here it is how our function would look like: 

```tsx
const callRecursively = 
	async (node: ResultType, results: ResultType[]): Promise<void> => {

  // call HN for the current node
	let curr = await hackerNews.getSubmission(node.id);
	const kids = curr.kids as number[];
	results.push({ id: node.id, text: node.text, time: node.time });

  // check kids and go recursive
	if (kids && kids.length > 0) {
		for (let x = 0; x < kids.length; x++) {
			const childNode = {
				id: kids[x],
				text: curr.text,
				time: curr.time
			};
			await callRecursively(childNode, results);
		}
	}
};
```

At the end of the processing, the `results` array is going to contain all the comments for the requested post. 

Next step is to keep track of the previous execution and compare the current execution with the previous one. If the two arrays are different, then we have updates and we need to send a discord notification.

Here it is how all this is going to be integrated with the rest of the app we are builduing: 

```tsx
    console.log("Starting check HackerNews -> Discord");
    const currentResults = await grabContentFromHN(
       userName, 
       selectedPostId);
    console.log(JSON.stringify(currentResults));

    // Compare the latest results with the previous run, 
    // if there is a change, then there are updates to this post.
    const updateAvailable = compareWithLastRun(currentResults);

    if (updateAvailable) {
        console.log("New updates available!");
	// notify Discord! 
    }

    // update the global state
    globalResults = [...currentResults];
```

The super simple comparing logic simply converts the 2 arrays in JSON strings and use the string comparator to do the work. This not the best solution, simply because it's not the fastest, but does the job:

```tsx
const compareWithLastRun = 
    (currentResults: ResultType[]) : boolean => {
	if (globalResults.length < currentResults.length) {
		return true;
	}

	if (JSON.stringify(globalResults) !== JSON.stringify(currentResults)) {
		return true;
	}

	return false;
}
```

For the purpose of this article we are going to save the state of the previous run locally, which means everytime we restart our app, we are going to receive an intial notification, the first execution sets the baseline. A more elegant approach would save the previous run in a storage instead.

---

## Discord integration

In order to send Discord notifications, we are going to use the `discord-notification` library, this is not a library which covers every single feature that the Discord API offers, but it is good enough for our usecase.

Before using the library we need to create a new Webhook on Discord. I am not going into the details on how to create a webhook, the Discord documentation is really well done. [Here it is the link](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks). 

Now that we have our webhook, let’s save it in the `.env` file of our project. The `.env` should have this structure:

```
HN_USERNAME=
HN_POST_ID=
DISCORD_WEBHOOK=
```

Let’s create a new file `discordNotifier.ts` and let’s push this code in it.

```tsx
const discordWebHook = process.env.DISCORD_WEBHOOK as string;
export const discordNotification = 
     new DiscordNotification('HN-Notifier', discordWebHook);

export const notifyDiscord = (message: string, 
     postId: string, 
     postText: string, 
     comments: number) : Promise<void> => {
    return discordNotification
    .sucessfulMessage()
    .addTitle(message)
    .addDescription(`[Link to the post](https://news.ycombinator.com/item?id=${postId})`)
    .addContent(postText)
    .addField({name: 'postId', value: postId, inline: true })
    .addField({name: 'comments', value: comments.toString() }) 
    .sendMessage();
};

```

The code above receives in input the notification data and uses the Webhook to send the notification. Couldn’t be easier than this!

Finally let’s integrate this logic in our scheduler, simply using this function like this: 

```tsx
if (updateAvailable) {
	console.log("New updates available!");
	// notify Discord!
	await discordNotifier.notifyDiscord(
			"New comments(s) available!", 
			selectedPostId, 
			currentResults[0].text, 
			currentResults.length - 1);
}
```

Here it is how our new shiny Discord notification will look like!


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xfa3tgw5gt7mr4djb0ep.png)

---

## Well done!

We now have fully working Hacker News monitoring system and we won’t lose comments anymore!

We could expand on this putting additional data into the notification, like, what was the text of the latest comment, the time when the latest comment has been published, etc. This system we just built is flexible enough to allow these chanegs with minimal effort. 

[Github link here!](https://github.com/dev-bre/hackernews-to-discord-notifier).

---

## So.. Can you help? 😉
I hope this article was somehow interesting. If you could give a star to my repo would really make my day!

https://github.com/switchfeat-com/switchfeat


![thank you gif](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hh32g4jm2tixqbonlldm.gif)