Parse LIVE Website - Extract component and convert to React
Tags: webdev, python
Published: 2021-05-21 17:04:57

---
title: Parse LIVE Website - Extract component and convert to React
published: true
description: A simple introduction to HTML parsing using Python tools
tags: webdev, python
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mv6q4lhc37iexo8yviwc.png
canonical_url: https://docs.appseed.us/content/tutorials/parse-html-components
---
Hello Coders, 

This article explains how to parse and extract components from a LIVE website using open-source libraries and tools. Personally, I'm using HTML parsing to convert automatically components from one technology (Bootstrap) to others like React, Vue, Svelte with less manual work and better quality. 

*Thanks for reading!* - The article is heavily inspired from here: **[Parse HTML Components](https://docs.appseed.us/content/tutorials/parse-html-components)** 

--- 

Parsing LIVE websites or lifeless HTML files might be useful in many scenarios. I will mention only a few:

- code a pricing scanner to detect changes
- check health for a LIVE system 
- extract components and reuse previous work for evolutions
- extract texts from a LIVE website and check text errors

In the end, I will mention an [open-source Django product](https://github.com/app-generator/django-pixel-lite) that uses a UI built with components extracted from a Bootstrap 5 Kit using parsing code quite similar to the one presented in this article. 

--- 

## Tools we need

- [Python](https://www.python.org/) - the interpreter 
- [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - a well-known parsing library 
- [Lxml](https://lxml.de/) - used to compensate BS4 limitations

---

> The process

- Load the HTML content - this can be done from a local file or using a LIVE website
- Analyze the page and extract XPATH expression for a component
- Use Lxml library to extract the HTML using *Xpath* selector
- Format the component and save it on disk
 
---

> Install tools 

```bash
$ pip install requests 
$ pip install lxml
$ pip install beautifulsoup4
```

Once all tools and libraries are installed and accessible in the terminal, we can start coding using `Python console`. 

```python
$ python [ENTER]
>>>
```

The HTML content can be a local file or a remote one, deployed and rendered by a LIVE system. 

**Load the HTML from a local file** (a simple file read)

```python
>>> f = open('./app/templates/index.html','r')
>>> html_page = f.read()
```

**Load content from a live website** - [Pixel Lite](https://demo.themesberg.com/pixel-lite/index.html)

```python
>>> import requests
>>> page = requests.get('https://demo.themesberg.com/pixel-lite/index.html')
>>> html_page = page.content
```

At this point, `html_page` variable contains the entire HTML content (string type) and we can use it in BS4 or Lxml to extract the components. To visualize the page structure we can use browser tools:  

![HTML Parser - Target Component Inspection.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ck60uqemtra5p1g8mseb.png)

The target component will be extracted using an `XPATH expression` provided by the browser:

```javascript
//*[@id="features"]
```

Once we have the selector, let's extract the components using LXML library:

```python
>>> from lxml import html
>>> html_dom = html.fromstring( html_page )
>>> component = html_dom.xpath( '//*[@id="features"]' )
```

If the `XPATH selector` returns a valid component, we should have a valid LXML object that holds the HTML code - Let's use it:

```python
>>> from lxml.etree import tostring
>>> component_html = tostring( component[0] )
```

To have a nice formatted component and gain access to all properties like nodes, `css style`, texts .. etc, the HTML is used to build a `Beautiful Soup` object. 

```python
>>> from bs4 import BeautifulSoup as bs
>>> soup = bs( component_html )
>>> soup.prettify()
```

The component is now fully parsed and we can traverse all information and proceed further with a conversion to React. 

```html
  <section class="section section-lg pb-0" id="features">
   <div class="container">
    <div class="row">
     
     ...
     
     <div class="col-12 col-md-4">
      <div class="icon-box text-center mb-5 mb-md-0">
       <div class="icon icon-shape icon-lg bg-white shadow-lg border-light rounded-circle icon-secondary mb-3">
        <span class="fas fa-box-open">
        </span>
       </div>
       <h2 class="my-3 h5">
        80 components
       </h2>
       <p class="px-lg-4">
        Beatifully crafted and creative components made with great care for each pixel
       </p>
      </div>
     </div>
     
     ...
     
     </div>
    </div>
   </div>
  </section>
```

This tool-chain will check and validate the component to be a valid HTML block with valid tags. 

---

**The extracted component**

![HTML Parser - Extracted Component.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6wo3l8vtvpd6y5ie6x34.png)
 
---

**React component**

```javascript
class Comp extends React.Component {
  render() {
    return COMPONENT_HTML_GOES_HERE;
  }
}
```

**React Component usage**

```javascript
ReactDOM.render(<Comp />, document.getElementById('root'));
```

--- 

This process can be extended for more tasks and automation:

- detect page layouts
- validate links (inner and outer)
- check images size

--- 

To see a final product built using a component extractor please access [Pixel Lite Django](https://github.com/app-generator/django-pixel-lite), an open-source product that uses a Bootstrap 5 design. 

The project can be used by anyone to code faster a nice website  using Django as backend technology and Bootstrap 5 for styling.

- [Django Pixel Lite](https://django-pixel-lite.appseed-srv1.com/) - LIVE Demo
- [Pixel Lite](https://demo.themesberg.com/pixel-lite/index.html) - the original HTML design 

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fx9yfsoylsnpk7a45r6k.png)

--- 

> Thanks for reading! For more resources please access:

--- 

- [Use XPath in Beautiful Soup](https://stackoverflow.com/questions/11465555/can-we-use-xpath-with-beautifulsoup) - related article published on StackOverflow
- [Web Scraping](https://docs.python-guide.org/scenarios/scrape/) - the right way (with sample)