Converting a dev.to feed to markdown
Tags: kotlin, markdown, writing
Published: 2021-03-29 13:29:06

dev.to has become my primary writing platform. Although personal articles are nicely grouped on one's landing page I wanted to reference them on my [Jekyll](https://jekyllrb.com/)-powered homepage, too. Something like this:

![Info about articles on dev.to](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/73qfm4ofrit7gjyygo7l.png)
 
Being a programmer enables you to build tools that just do what you want. Nothing more. Nothing less.

Take a look.

As Jekyll is based upon markdown, mine will create *.md' files. Here's the template:

```kotlin
val template = """
---
layout: post
title: !!!TITLE!!!
date: !!!FULL_DATE_AND_TIME!!!
categories: articles
---
!!!DATE!!! on dev.to  
[&#x1F4D6; Read article](!!!URL!!!)
""".trimIndent()
```

Obviously, text starting and ending in `!!!` will be replaced.

```kotlin
val counter = 1
rssFeed.forEach {
  val title = it.title.get()
    .replace("#", "&#x23;")
    .replace(":", "&#058;")
  val url = it.link.get()
  val date: String
  val fullDateAndTime: String
  val year: Int
  it.pubDateZonedDateTime.get().run {
    fullDateAndTime = toString()
    date = toLocalDate().toString()
    year = this.year
  }
  val dir = File("$baseDir${File.separatorChar}$year")
  dir.mkdirs()
  val file = File(dir, "${date}-${counter}.md")
  val fileContent = template.replace("!!!TITLE!!!", title)
    .replace("!!!FULL_DATE_AND_TIME!!!", fullDateAndTime)
    .replace("!!!DATE!!!", date)
    .replace("!!!URL!!!", url)
  file.writeText(fileContent)
}
```

So, once the values have been obtained, I simply invoke `replace()`. `title` gets a special treatment as the title is processed somewhat differently inside Jekyll, so characters like `#` and `:` must be escaped.

The markdown files are saved in folders named like the year of publication. ...just like I said, tailored to my needs. 😎

While the code for the loop is complete you are surely wondering where the feed data comes from... what this `rssFeed` thing is.

```kotlin
fun main(args: Array<String>) {
  val baseDir = if (args.isNotEmpty()) args[0] else "./_posts"
  val client = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).build()
  val request = HttpRequest.newBuilder()
    .uri(URI.create("https://dev.to/feed/tkuenneth/"))
    .header(
      "User-Agent",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
    )
    .build()
  val response = client.send(request, BodyHandlers.ofString())
  if (response.statusCode() == 200) {
    val inputStream = response.body().byteInputStream()
    val reader = RssReader()
    val rssFeed = reader.read(inputStream)
```

I am using Java's `HttpClient`. The code is straight forward, just be sure to change the url accordingly. 

One thing that caused me a tiny headache...: while doing my first experiments I always got a 403. Using some C# code did work, however. And firing up the url in the browser worked, too. It turns out that to be succesful I needed to change `User-Agent`. Now, while one could argue the url should not be called from an app, feeds generally are processed by feed readers and the likes, aren't they?

To parse the fetched data I use [rssreader](https://github.com/w3stling/rssreader) by Peter Westling. The library is easy to include using Maven or Gradle. And it is published under the MIT license.

This applies to my tiny project, too. You can find it on [GitHub](https://github.com/tkuenneth/devto_extractor).