DevTip#3: Strip Down and Convert HTML to Markdown for Importing Notes into Joplin
Tags: tutorial, techtip, tooling, productivity
Published: 2021-02-28 21:02:16

<a name="top"></a>

>  * [▶ Background](#background)  
>  * [🛑 Problem Statement](#problem)
>  * [📜 Solution](#solution)
>    * [1️⃣ Step 1](#step1)  
>    * [2️⃣ Step 2](#step2)
>  * [✉ Summary](#summary)

---
<a name="background"></a>
## ▶ Background [↑top](#top)
Even though this article specifically mentioned about importing notes into Joplin, the steps laid out are generic. Most of the details of exporting and importing notes are left out to keep this article as concise as possible. If you are just looking for the steps to clean and convert HTML to Markdown, skip right into the [Solution](#solution).

I had been using [OneNote](https://www.onenote.com), [Evernote](https://evernote.com/) and [Leanote](https://leanote.com/) as my note taking software. OneNote is primarily used for personal notes while Evernote is used for work. Later on, I switched to self-hosted Leanote from Evernote and used that for the past 2-3 years. 

Now, I plan to consolidate both OneNote and Leanote contents into one place. In the end I decided to settle for [Joplin](https://joplinapp.org/). This article focus on part of the whole export and import notes process. More details are laid out in the [Solution](#solution) section below. 

---
<a name="problem"></a>
## 🛑 Problem Statement [↑top](#top)

A quick web search will return results on how to import OneNote and Leanote into Joplin. Despite the methods from the search results work considerably well for simple notes format, the imported note looks messed up once the note contains tables, images, styles, nested structures and combination of all these. 

---
<a name="solution"></a>
## 📜 Solution [↑top](#top)

After some trial and error, the following workflow works the best. 
- exporting all the notes from OneNote and Leanote into HTML,
- remove all HTML attributes except for `src` and `href` (kept for image source and links),
- follow by converting them into Markdown, 
- finally importing all the markdown notes into Joplin using the built-in importer

This article will focus on the 2 intermediary steps - sanitize & convert HTML into Markdown.

<a name="step1"></a>
### 1️⃣ Step 1 [↑top](#top)

A typical HTML will contains `script`, `id`, `class`, `style`, data attributes and much more other HTML tags/attributes. As most of these are not supported in Markdown, removal of all HTML tags/attributes while remaining only a handful of what we need will drastically the markdown output later on. 

[Beautiful Soup 4](https://beautiful-soup-4.readthedocs.io/en/latest/#quick-start) is used for this step. The command below uses pip to install the python library. 
```
pip install beautifulsoup4
```

After installing, running the code example below will remove all HTML attributes except for `src` and `href`, and removing all `script` and `style` tags. I also posted this solution in [this SO Q&A](https://stackoverflow.com/a/66268812/2809828).
```python
# https://beautiful-soup-4.readthedocs.io/en/latest/#searching-the-tree
from bs4 import BeautifulSoup, NavigableString

def unstyle_html(html):
    soup = BeautifulSoup(html, features="html.parser")

    # remove all attributes except for `src` and `href`
    for tag in soup.descendants:
        keys = []
        if not isinstance(tag, NavigableString):
            for k in tag.attrs.keys():
                if k not in ["src", "href"]:
                    keys.append(k)
            for k in keys:
                del tag[k]

    # remove all script and style tags
    for tag in soup.find_all(["script", "style"]):
        tag.decompose()

    # return html text
    return soup.prettify()
``` 

<a name="step2"></a>
### 2️⃣ Step 2 [↑top](#top)

To convert from the sanitize HTML into Markdown, [pandoc](https://pandoc.org/) is used. This is a command line tool, an external library is installed using pip to use pandoc easier in Python.
```
pip install pypandoc
```

After installing, the code snippet below shows how to call pypandoc to convert HTML into Markdown. 
```python
pypandoc.convert_file(html_path,
    'markdown+pipe_tables+backtick_code_blocks-markdown_attribute',
    format='html',
    outputfile=md_path)
```

This pandoc documentation shows [all the supported input and output formats](https://pandoc.org/MANUAL.html#options). If you are curious about the 'plus' and 'minus' strings after the format, those are for adding or removing pandoc extensions respectively. The Markdown files generated using these extensions provide the best imported Joplin notes. Check out [this section](https://pandoc.org/MANUAL.html#extensions) to understand more details about the extensions. 

---
<a name="summary"></a>
## ✉ Summary [↑top](#top)
1. Use [Beautiful Soup 4](https://beautiful-soup-4.readthedocs.io/en/latest/#quick-start) to sanitize unneeded HTML tags and attributes
2. Use [pandoc](https://pandoc.org/) to convert HTML to Markdown.