Use puppeteer on the server in non-headless mode
Tags: webdev, productivity, devops, beginners
Published: 2021-07-23 02:49:40

## 📓📓 TL:DR

Using puppeteer on the server is necessary use case when you want to

1. Crawl millions of web pages of a certain URL 
2. Perform Web Automation 
3. Automate the testing of web apps
4. Implement Bot-like activities on web sites 
etc.

But, using it in non - headless mode is such a pain. Especially, when you have any Unix-like OS i.e. ubuntu, centos installed on the remote machine and you are using the machine over SSH or any other remote connection tool like putty.

-----------------


##🛠 How to do it :

So, in spite of all these extreme cases we can still use it and in this article we will take a look at one of the ways to implement it.

-----------------

## 🕶  Some Background :

Puppeteer at it's core uses the chromium browser, which does all the heavy lifting stuff like 


1. Exposing  DOM API's for interactions
2. Perform DOM manipulations and code injection 
3. Cookie management and session handling 
4. Navigation and context maintenance 
etc.

Chromium needs the display adapters to launch the chromium window on the host machine. Regardless of the host OS, it uses the available display adapter API available. 

When we are serving puppeteer using virtual machine or any other remote machine, by default we don't have any displays because all the work is done by Bash (AKA command line). 

If we have windows server, then there is no big deal. Just connect  the server using Remote Desktop Connection and  you are good to go. But, if you have Ubuntu, centos or any other UNIX based OS installed on the remote machine, then you have to do some extra efforts.

---------

## 👉 Steps

---------

### ⚒ 1. Regular stpes :-

Before doing anything else, lets update host by some OLD school stuff 

```bash 

sudo apt-get update

sudo apt-get upgrade 

```     

-----------------------


### 2. 🖥 Install desktop-like enviroments 

For installing the desktop based utility known as XRDP, I have installed core utilities like `ubuntu-mate-core` and `ubuntu-mate-desktop` by using this

```bash
sudo apt-get install --no-install-recommends ubuntu-mate-core ubuntu-mate-desktop -y
```
-----------------------

Ubuntu Mate Core :-

This utility converts the basic Ubuntu machine to complete workstation by adding some extra applications.


Ubuntu Mate Desktop :-

This is on such desktop like environments for Ubuntu. This exposes  the interface for handling the local as well as networked files, perform calculations etc.  

-----------------------

### 3. 💻 Install XRDP :-

After doing above steps, we have to install XRDP server which will give us  the access of Remote Desktop Connection for host machine. We will also install some secondary dependencies for implementing complete desktop like environment.

```bash

sudo apt-get install mate-core mate-desktop-environment mate-notification-daemon xrdp -y

``` 

-----------------------

### 👮 4. Create remote user :-

As suggested by numerous manuals, we will now create non-root user for performing all of our remaining tasks. 

Create a XYZ user and grant the sudo access to newly created user for avoiding any permission conflicts.


```bash 

adduser xseven

usermod -aG admin xseven

usermod -aG sudo xseven

su - xseven

``` 
--------------

### 🔧 5. Configuring Mate Session :

We  now will configure the necessary Mate Session. This will help use to customize the available desktop like enviroment as per our needs 


```bash 

echo mate-session> ~/.xsession

sudo cp /home/xseven/.xsession /etc/skel

```  

-------------------------


### 🚀🚀🚀 6. Restart XRDP :-

Final step is to restart XRDP to apply all the configrations.

``` bash
sudo service xrdp restart

```

Once restarted, we can connect to the remote machine by using RDC  or any other compatible tool



This is how we can run puppteer in non-headless mode on server. There are many other alternatives like 

1. Installing GNome or KDE for desktop machines 
2. Host applications on desktop ubuntu enviroment 

etc.
 

Thanks For Reading....