Dev Diary Week 2 - Reading a Feed
Tags: node, javascript, showdev
Published: 2020-04-23 15:09:50

---
title: Dev Diary Week 2 - Reading a Feed
published: true
description: This post is an entry in a weekly development diary on building a feed-aggregator-based blog on Glitch. This week, I added an API endpoint and a module to parse an RSS feed.
tags: node, javascript, showdev
series: Dev Diary
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/06l1787yedv7wynyna3m.jpg
---

> This post is an entry in a weekly development diary on building a feed-aggregator-based blog on [Glitch]. Only a small part of building an app is code; the bulk of your time is spent planning, experimenting, making mistakes, getting frustrated, and making progress in baby steps.

## The plan

As I discussed in [last week's entry], I'm building a web app that will do three things:

1. pull in [RSS] feeds of my content from various platforms;
2. combine them into a single superfeed; and
3. automatically generate a blog from that firehose feed.

If we look at this project as a black box, here's how it should work:

```
INPUT          +---------------+          OUTPUTS
               |               |
               |               +----> Aggregated RSS feed
OPML file ---->+ ~rss-firehose |
               |               +----> "Content firehose" blog
               |               |
               +---------------+
```

In other words, to get your own "content firehose" blog, you would:

1. [Remix] the `~rss-firehose` Glitch app.
2. Drop in an [OPML file] of your own content feeds.
3. There is no step three!

We're not even close to this yet — right now, `~rss-firehose` is just a simple remix of the `~hello-sqlite` starter app on Glitch. Let's think about the steps to get us from where we are, to where we want to be! As a side benefit, this will give me a rough outline for the work I want to do each week. I only have a few hours per week to dedicate to this project, so I want to be thoughtful about how I approach that time.

Roughly speaking, here's how I expect things to go. This might change! I'm _estimating_ based on a few things, like how comfortable I am working in Node, SQLite, and how deeply I understand the overall scope of the project. I've built [blog templates] before, but I've never built a feed reader!

- **[Week 1]:** Define the scope of the project ✅
- **Week 2:** Pull in a single RSS feed
- **Week 3:** Parse an OPML file into multiple RSS feeds
- **Week 4:** Save feed entries to a SQLite database
- **Week 5:** Generate an aggregated "firehose" RSS feed
- **Week 6:** Create a basic [linklog] front-end
- **Week 7:** Improve the front-end design

With our pre-requisite week 1 done, let's move on to week 2.

## Reading an RSS Feed

Here's what I plan on doing this week; the next section will discuss how it went, and what I learned.

We want to implement a way to read a single RSS feed in our app. Now, we _could_ figure out how to implement a basic RSS feed reader, but there are a few different formats — I can think of at least three different versions of the RSS spec itself, plus [Atom], plus... well, you get the point.

Instead, we can build upon the work of those that came before us. RSS has been around for a _very_ long time, so there are great packages we can use in our project to parse feeds in our app.

The package I see being used most in similar projects is [feedparser], so we're going to add that to our app. I've never used feedparser before, so I'm going use my time budget for the week experimenting with reading a single RSS feed and outputting it to the server console.

Just to make it a bit more fun, I'll work with [my own Dev.to feed]!

## How did it go?

I worked on the app over a few days, for a total of about four hours.

I started by remixing the [actual app] so that it's not affected by this week's work. In [my remix], I added the feedparser package to **package.json** and got down to work.

When I'm working off a remix of a Glitch starter app, I usually start by deleting all the files and code I don't need. I'm skipped that this time, because it's nice to have some boilerplate code that I can examine for the SQLite stuff that will come later.

I then added [feedparser.js], which takes a feed, tries to parse it, and returns a `feedResponse` object. The parsing-related work was mostly taken straight from the [feedparser package README]. The `feedResponse` object has two properties that get initialized as the parser does its thing: a `meta` object containing feed metadata, and an `items` array for feed entries.

Finally, I added [an `/api/parse/:feed` endpoint to server.js] — you can add a feed address to this endpoint in the browser, and it'll return the JSON of the `feedResponse` object. Try it by opening [this link] in a new tab, or add an RSS/Atom feed of your choice at the end of this link (but make sure it's [url-encoded]): https://rss-firehose-readfeed.glitch.me/api/parse/

I usually like to ask myself a few questions at the end of my workweek, and I think they're a good fit here too.

### What went well?

Generally, adding the feedparser library went well. The README and sample code is thorough enough that I could follow along and make it do what I needed it to do. 

### What did I have trouble with?

I'm not very experienced with using the ([deprecated]) [request] library, and spent a while trying to get port the feedparser sample code to newer libraries in vain. Not wanting to blow my time budget for the week porting code that otherwise worked fine, I decided instead to just stick with request. I can always change that later.

### What did I learn?

I learned how to use the feedparser library! I didn't dive into any of the more complex example code with additional dependencies, because the feeds I tested with worked fine as-is.

### What will I work on next week?

Next week, I'm going to work on adding some feeds to an OPML file, parsing that file, and handing them off to the `/api/parse/:feed` endpoint I created this week!

So, that's week 2 wrapped up! Here's where the app is at now:

{% glitch rss-firehose-readfeed %}

## Do you use a feed reader?

Personally, I like [NetNewsWire], and I sync feeds between my devices using [Feedbin].

Which on is your favourite? Let me know in the comments!


<!--Links-->

[Glitch]: https://glitch.com?utm_medium=weblink&utm_source=dev.to&utm_campaign=blog&utm_content=dev
[last week's entry]: https://dev.to/glitch/dev-diary-week-1-the-kickoff-1ed5/
[RSS]: https://cyber.harvard.edu/rss/rss.html
[Remix]: https://glitch.com/help/remix/
[OPML file]: http://dev.opml.org/
[blog templates]: https://themes.gohugo.io/indigo/
[Week 1]: https://dev.to/glitch/dev-diary-week-1-the-kickoff-1ed5/
[linklog]: https://en.wikipedia.org/wiki/Linklog
[Atom]: https://tools.ietf.org/html/rfc4287/
[feedparser]: https://www.npmjs.com/package/feedparser/
[actual app]: https://glitch.com/~rss-firehose/?utm_medium=weblink&utm_source=dev.to&utm_campaign=blog&utm_content=dev
[my remix]: https://glitch.com/~rss-firehose-readfeed/?utm_medium=weblink&utm_source=dev.to&utm_campaign=blog&utm_content=dev
[feedparser package README]: https://github.com/danmactough/node-feedparser#usage
[my own Dev.to feed]: https://dev.to/feed/angelostavrow/
[feedparser.js]: https://glitch.com/edit/#!/rss-firehose-readfeed?path=feedparser.js:1:0
[an `/api/parse/:feed` endpoint to server.js]: https://glitch.com/edit/#!/rss-firehose-readfeed?path=server.js:54:0
[url-encoded]: https://www.urlencoder.org/
[request]: https://www.npmjs.com/package/request/
[deprecated]: https://github.com/request/request/issues/3142
[NetNewsWire]: https://ranchero.com/netnewswire/
[Feedbin]: https://feedbin.com/
[this link]: https://rss-firehose-readfeed.glitch.me/api/parse/https%3A%2F%2Fdev.to%2Ffeed%2Fangelostavrow%2F