HTML Parser - Developer Tools
Tags: htmlparser, tools, templates, appseed
Published: 2019-07-23 13:09:09

---
title: HTML Parser - Developer Tools
published: true
description: Interactive tool to convert flat HTML (themes or stand-alone files) to Pug, Jinja2 and Blade templates
cover_image: https://raw.githubusercontent.com/app-generator/static/master/developer-tools/html-parser-select-component.jpg
tags: htmlparser, tools, templates, appseed
---
Hello Coders, 

In this article, I will present a simple **[HTML Parser](https://appseed.us/developer-tools/html-parser)** used by me to integrate much faster HTML themes into legacy apps, coded in different technologies. When a customer requests a new UI for his app, the manual processing can take some time, and I decided to automate a little bit of the whole flow. Using the tool, I'm able to update the design in less than 2h for a simple website with 2/3 pages. 

> *Thanks for reading!* - Content provided by **[App Generator](https://appseed.us/)**.

---

## Main Feature
The tool converts flat HTML to production-ready components for different engines: [PUG](https://pugjs.org/api/getting-started.html), [Jinja2](http://jinja.pocoo.org/docs/2.10/), [Blade](https://laravel.com/docs/5.8/blade), [Mustache](https://laravel.com/docs/5.8/blade), Core Php.  

---

> Note: the tool is not open-source, but I will consider releasing a light version as an open-source project in the future. In the [HTML Parser public repository](https://github.com/app-generator/html-parser), I will publish processed HTML themes converted to PUG, Jinja, and Blade to be used by anyone.  

---

## Technologies

The [HTML parser](https://docs.python.org/3/library/html.parser.html) Tool is developed in [Python3](https://www.python.org/) / [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) library as an interactive console. I was able to use the tool for real projects after 3mo of R&D work. 

---

## [HTML Parser](https://appseed.us/developer-tools/html-parser) features

- Normalize the HTML file to load the assets from a standard directories ( /assets/ [ img, js, css ] ) making the integration in webpack related tools much easier
- Edit / traverse the HTML tree
- Edit attributes like anchor HREF, span texts, remove elements, edit class names
- Extract components for production use for various engines like PUG, Jinja2, Blade
- Migrate legacy Bootstrap layouts to Bulma and Tailwind CSS frameworks

---

## [HTML Parser](https://appseed.us/developer-tools/html-parser) Implementation 

In order to process the HTML and process the HTML tree, we need to load first the whole file. [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) has a simple constructor that accepts the string to parse and load into memory and the desired parser. 

### Load HTML in memory

```python
# read_file retun the file content as string
html_content = read_file('index.html')
soup  = bs(html_content,'html.parser') 
# At this point, we can interact with the HTML 
# elements stored in memory using all helpers offered by BS library
```  

BeautifulSoup library supports more than one parser (e.g. lxml, xml, html5lib), the differences between them become clear on non well-formed HTML documents. For instance, lxml will add missing closing tags for all elements. For more information please access the [dedicated section](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers) in the documentation regarding this topic.

### Parse Head section

To select the whole HEAD node, and interact with all elements we need to write just a few lines of code:

```python
header = soup.find('head')

# If we want to change the title

header.title.string.replace_with('My new title') 
```  

### Parse HTML for JS Scripts

Javascript files are present in the HTML using `script` nodes:
```
...
<script type='text/javascript' src='js/bootstrap.js'></script>
<script type='text/javascript' src='js/custom.js'></script>
...
```

To scan the HTML soup for script tags, we can use the `find_all` helper:

```
for script in soup.body.find_all('script', recursive=False):

   # Print the path 
   print(' JS source = ' + script[src]) 

   # Update (normalize) the path
   js_path = script['src']
   js_file = js_path.split('/')[-1] # select the last segment
   script[src] = '/assets/js/' + js_file
```

### Parse HTML for Images

Using the same technique as for JS files, we can normalize the Images to be loaded from a standard directory. 

```
for img in soup.body.find_all('img'):

   # Print the path 
   print(' IMG src = ' + img[src]) 
   
   img_path = img['src']
   img_file = img_path.split('/')[-1]  
   img[src] = '/assets/img/' + img_file
```

### Save the HTML

All our changes are made in memory. To make these changes permanent we need to extract the string representation of our processed HTML from BS, and dump it into a file for later usage:

```
processed_html = soup.prettify(formatter="html")
f = open( 'index2.html', 'w+')
f.write(processed_html)
f.close
```

---

## Real life sample 

The sample, extracted from [Stellar HTML5Up theme](https://github.com/app-generator/html-parser/tree/master/sample-stellar) is a simple navigation bar, extracted from [this file](https://github.com/app-generator/html-parser/blob/master/sample-stellar/index.html)

- Index file: [original version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/index.html) and [normalized version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/index2.html)
- [JSON descriptor](https://github.com/app-generator/html-parser/blob/master/sample-stellar/index2.json) is generated by the **HTML parser** tool and encapsulate the assets and resources used by the HTML files
- Navigation component
  - [HTML version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/components/nav.html)
  - [PUG version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/components/nav.pug)
  - [Jinja2 Version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/components/nav.j2)
  - [Php version](https://github.com/app-generator/html-parser/blob/master/sample-stellar/components/nav.php)
  - [JSON descriptor](https://github.com/app-generator/html-parser/blob/master/sample-stellar/components/nav.json)
  

### Pug version

```yaml
nav#nav
  ul
    li
      a.active.newclass(href='https://appseed.us/html-parser').
        
        Introduction
        
    li
      a(href='#first').
        
        First Section
        
    li
      a(href='#second').
        
        Second Section
        
    li
      a(href='#cta').
        
        Get Started
```

<br />

### PHP version

```php
<nav id="nav">
 <ul>
  <li>
   <a class="active newclass" href="https://appseed.us/html-parser">
    <?php echo $var_1?>
   </a>
  </li>
  <li>
   <a href="#first">
    <?php echo $var_2?>
   </a>
  </li>
  <li>
   <a href="#second">
    <?php echo $var_3?>
   </a>
  </li>
  <li>
   <a href="#cta">
    <?php echo $var_4?>
   </a>
  </li>
 </ul>
</nav>
```

---

## Projects built with this tool 

All are open-source, with live DEMO. 
  - [JAMstack Fractal](https://github.com/app-generator/jamstack-fractal) - HTML5Up design coded in JAMstack pattern
  - [JAMstack BigPicture](https://github.com/app-generator/jamstack-big-picture) - HTML5Up design coded in JAMstack pattern
  - [JAMstack Landed](https://github.com/app-generator/jamstack-landed) - HTML5Up Landed design coded in JAMstack pattern
  - [Flask Dashboard Material Design](https://github.com/app-generator/flask-material-dashboard) - Admin Dashboard with Material Design
  - [Flask Dashboard NowUI](https://github.com/app-generator/flask-now-ui-dashboard) - Admin Dashboard with NowUI Design
  - [Flask Dashboard Black](https://github.com/app-generator/flask-black-dashboard) - Open-Source Admin Panel
  - [Flask Dashboard Argon](https://github.com/app-generator/flask-argon-dashboard) - Open-Source Admin Panel
  - [Flask Dashboard Light](https://github.com/app-generator/flask-dashboard-light-bootstrap) - Open-Source Admin Panel

---

## Resources

- [HTML Parser - How to use Python BS4 to work less](https://blog.appseed.us/html-parser-python-beautifulsoup-use-cases/)  
- [Developer Tools - Open-Source HTML Parser](https://blog.appseed.us/developer-tools-html-parser/) - related article
- [HTML Parser](https://blog.appseed.us/appseed-app-generator-html-parsing/) - used by the AppSeed App Generator to parse flat HTML 
- [BeautifulSoup Html Parser](https://www.crummy.com/software/BeautifulSoup) documentation
- [HTML Parser sources](https://github.com/app-generator/html-parser) - the official public repository
- [HTML Parser](https://appseed.us/developer-tools/html-parser) provided by AppSeed
- [HTML Parser - Convert HTML to Jinja2 and Php components](https://blog.appseed.us/html-parser-convert-html-to-jinja2-and-php/) - related blog article
- Video presentation [HTML parsing](https://www.youtube.com/watch?v=N8l1tTurKP0) and components extraction

---

> Thank you!