Convert HTML to Markdown with a serverless function 
Tags: serverless, aws, markdown
Published: 2020-07-09 14:22:12

Outlined below is the setup for a AWS lambda function which combines fetching the HTML for a URL, stripping it back to just the essential article content, and then converting it to Markdown. To deploy it you'll need an AWS account, and to have the [serverless framework](https://www.serverless.com/framework/docs/getting-started/) installed.

### Step 1 - Download the full HTML for the URL

First get the full html of the url getting converted. As this is running in a lambda function I decided to try out an ultra-lightweight node http client called [phin](https://github.com/ethanent/phin) (which is 95% smaller than my usual favourite Axios):

```js
const phin = require('phin')
const fetchPageHtml  async fetchUrl => {
  const response = await phin(fetchUrl)
  return response.body;
};
```

### Step 2 - Convert to readable HTML

Converting to readable HTML is a feature originally offered by Instapaper (going back to 2008) as part of the core experience of a "read it later" service, but is now built into most browsers. Before converting to markdown its a good idea to strip out the unnecessary parts of the HTML (adverts, menus, images, etc), and just display the text of the main article in a clean and less distracting way. 

This process won't work for every web page - it is designed for blog posts, news articles etc which have a clear "body content" section which can be the focus of the output.

Mozilla have open sourced their code for doing this in a [Readability](https://github.com/mozilla/readability) library, which can be reused here:

```js
const readability = require("readability");
const JSDOM = require("jsdom").JSDOM;

const extractMainContent = (pageHtml, url) => {
  const doc = new JSDOM(pageHtml, {
    url,
  });
  const reader = new Readability(doc.window.document);
  const article = reader.parse();
  return article.content;
};
```

This returns the HTML for just the article in a more readable form.

### Step 3 - Convert readable HTML to markdown

There is a CLI tool called [pandoc](https://pandoc.org/) which converts HTML to markdown. The elevator pitch for pandoc is:

> If you need to convert files from one markup format into another, pandoc is your swiss-army knife.

To try this out locally before running it from the lambda function, you can follow one of their [installation methods](https://pandoc.org/installing.html), and then test it from the command line by piping a html file as the input:

```bash
cat sample.html | pandoc -f html -t commonmark-raw_html+backtick_code_blocks --wrap none
```

The options used here are:

`-f html` is the input format

`-t commonmark` is the output format (a particular markdown flavour)

You can add extra configuration options to the output by adding them to the output name. 

`commonmark-raw_html+backtick_code_blocks` sets the converter to disable the `raw_html` extension, so no plain html is included in the output. It enables the `backtick_code_blocks` extension so that any code blocks are fenced with backticks rather than being indented.

The pandoc tool needs to be executed from within the node script, which involves spawning it in a child process, writing the html to the child `stdin` and then collect the markdown output via the child `stdout`.

Most of these functions have been taken from [this very helpful blog post](https://2ality.com/2018/05/child-process-streams.html) on working with stdout and stdin in nodejs.

First off this is the generic streamWrite function, which allows you to pipe the html to the pandoc process, by writing to the `stdin` stream of the child process.

```js
const streamWrite = async (stream, chunk, encoding = 'utf8') =>
  new Promise((resolve, reject) => {
    const errListener = (err) => {
      stream.removeListener('error', errListener);
      reject(err);
    };
    stream.addListener('error', errListener);
    const callback = () => {
      stream.removeListener('error', errListener);
      resolve(undefined);
    };
    stream.write(chunk, encoding, callback);
  });
```

This similar function reads from the `stdout` stream of the child process, so you can collect the markdown that is output:

```js
const {chunksToLinesAsync, chomp} = require('@rauschma/stringio');
const collectFromReadable = async (readable) => {
  let lines = [];
 for await (const line of chunksToLinesAsync(readable)) {
   lines.push(chomp(line));
 }
 return lines;
}
```

Finally this helper function converts the callback events for the child process into an "awaitable" async function:

```js
const onExit = async (childProcess) =>
  new Promise((resolve, reject) => {
    childProcess.once('exit', (code) => {
      if (code === 0) {
        resolve(undefined);
      } else {
        reject(new Error('Exit with error code: '+code));
      }
    });
    childProcess.once('error', (err) => {
      reject(err);
    });
  });
```

To make the API a bit cleaner, here is that all wrapped up in a single helper function:

```js
// spawns a child process, supplying stdin to the child STDIN, then reads from the child STDOUT and
// returns this as a string
const spawnHelper = async (command, stdin) => {
  const commandParts = command.split(" ");
  const childProcess = spawn(commandParts[0], commandParts.slice(1))
  await streamWrite(childProcess.stdin, stdin);
  childProcess.stdin.end();
  const outputLines = await collectFromReadable(childProcess.stdout);
  await onExit(childProcess);
  return outputLines.join("\n");
}
```

This makes calling pandoc from the node script much simpler:

```js
const convertToMarkdown = async (html) => {
  const convertedOutput = await spawnHelper('/opt/bin/pandoc -f html -t commonmark-raw_html+backtick_code_blocks --wrap none', html)
  return convertedOutput;
}
```

To run this as an AWS lambda you need to include the pandoc binary. This is achieved by adding a shared lambda layer which includes a precompiled pandoc binary. You can [build this yourself](https://github.com/serverlesspub/pandoc-aws-lambda-binary), or just include the [public published layer](https://github.com/mthenw/awesome-layers#utilities) in your serverless config.

```yaml
# function config
layers:
  - arn:aws:lambda:us-east-1:145266761615:layer:pandoc:1
```

### Step 4 - Wrapping this up in the lambda handler function

Export a function from this module which has been configured as the handler. This is the function AWS will run every time the lambda receives a request.

```js
module.exports.endpoint = async (event) => {
  const url = event.body
  const pageHtml = await fetchPageHtml(url);
  const article = await extractMainContent(pageHtml, url);
  const bodyMarkdown = await convertToMarkdown(article.content);
  // add the title and source url to the top of the markdown
  const markdown = `# ${article.title}\n\nSource: ${url}\n\n${bodyMarkdown}`
  return {
    statusCode: 200,
    body: markdown,
    headers: {
      'Content-type': 'text/markdown'
    }
  }
}
```

This is the full `serverless.yml` configuration that is needed for serverless to deploy everything:

```yaml
service: url-to-markdown

frameworkVersion: ">=1.1.0 <2.0.0"

provider:
  name: aws
  runtime: nodejs12.x
  region: us-east-1

functions:
  downloadAndConvert:
    handler: handler.endpoint
    timeout: 10
    layers:
      - arn:aws:lambda:us-east-1:145266761615:layer:pandoc:1
    events:
      - http:
          path: convert
          method: post
```

### Wrap Up

The full source code is [available on github](https://github.com/michaelvigor/url-to-markdown). Once deployed you can test it from the command line like so:

```bash
curl -X POST -d 'https://www.atlasobscura.com/articles/actual-1950s-proposal-nuke-alaska' https://zm13c3gpzh.execute-api.us-east-1.amazonaws.com/dev/convert
```