Python + Flask + Selenium + Chromedriver + Lightsail + SSL
Tags: python, selenium, lightsail, aws
Published: 2020-09-17 21:20:19

### 1. Secure the server

---

```
# edit the configuration file
sudo nano /etc/ssh/sshd_config

# set these lines, if not already set
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes

# restart the service
sudo service ssh restart
```

### 2. Install dependencies

---

```
sudo apt-get -y update
sudo apt-get -y install python3 python3-venv python3-dev
sudo apt-get -y install supervisor nginx git
```

### 3. Install the application

---

Follow repo instructions. Remember to install everything inside the virtual environment.

### 4. Setup Gunicorn and Supervisor

---

```
sudo nano /etc/supervisor/conf.d/transit.conf

# example file
[program:transit-cu]
command=/home/ubuntu/transit-cu/venv/bin/gunicorn -b localhost:8000 -w 3 transit:app
directory=/home/ubuntu/transit-cu
user=ubuntu
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
```

### 5. Enable Lightsail HTTPS rule and create a private address

---

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pgymrwskfdgb3uz6idyp.png)

### 6. Update A record on DNS

---

My domain is registered with Godaddy and also created a sub-domain to setup as my server name. Go to your DNS manager page and update or create an A record with your host pointing to the Static IP address provided by Lightsail.

### 7. Setup NGINX

---

```
sudo rm /etc/nginx/sites-enabled/default
sudo nano /etc/nginx/sites-enabled/transit
```

Replace "transit" with your app name.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5fps3sao7zchg72edro0.png)

### 8. Setup Let's Encrypt

---

I decided to install the certificate manually since `certbot --nginx` failed to reach my domain. However it worked later after I was able to solve the certbot challenge.

`sudo certbot -d mydomain.com --manual --preferred-challenges dns certonly`

Confirm any required step and then create a TXT record on your domain DNS manager. In my case, It went like this:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/l5hz8rxgmx4ko2ssyw7m.png)

The name was *_acme-challenge.rsscraper* which was different from what cerbot displayed. Actually just the first part of the entire string.

### 9. Install Chrome (headless) and Chromedriver on Ubuntu Server 18.04

---

Let's go back into the project dependencies since this is a tricky part, use the following commands to install the latest Chrome version available:

```jsx
curl https://intoli.com/install-google-chrome.sh | bash

sudo mv /usr/bin/google-chrome-stable /usr/bin/google-chrome

google-chrome --version && which google-chrome
```

Then you'll have to install the respective chrome driver, go to this [site](http://chromedriver.chromium.org/downloads) and download it using `wget` then unzip the package and move to the binaries folder.

```jsx
cd/tmp/
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
chromedriver --version
```

*The above code is just an example.*

Make sure the way you use Selenium with the Chrome driver goes as follows:

```jsx
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
browser = webdriver.Chrome("./chromedriver", options=options)
browser.set_window_size(1920, 1080)
```

Not sure if the `set_window_size` is required with this driver but that fixes issues when the browser screen is not able to see other elements because of the responsiveness of the site and your scrapping code might fail when trying to find elements.

The Important part here is that you have to either specify the binary location of the chrome driver or leave it on the root location of the project (like in my case).

You may experience issues due to driver incompatibility with the Chrome version so that's why it was better for me to have it inside the project.

Another possible issue that you could have is related to some `lib` libraries because we are using a server version that is not connected to a monitor. The issues are related with display and can be easily fixed with a google search.

### 10. Initializing services

---

If everything was done correctly with all your project dependencies, you should be able to access to  your endpoints using your custom domain name after running the following commands:

```jsx
// Reload supervisor

sudo supervisorctl reload
sudo supervisorctl status

// Check Nginx syntax and reload

sudo nginx -t
sudo service nginx reload
```

### References

- [https://github.com/dandalpiaz/lightsail-flask](https://github.com/dandalpiaz/lightsail-flask)
- [https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux)
- [http://chromedriver.chromium.org/downloads](http://chromedriver.chromium.org/downloads)
- [https://medium.com/@praneeth.jm/running-chromedriver-and-selenium-in-python-on-an-aws-ec2-instance-2fb4ad633bb5](https://medium.com/@praneeth.jm/running-chromedriver-and-selenium-in-python-on-an-aws-ec2-instance-2fb4ad633bb5)
- [https://stackoverflow.com/questions/43382548/can-i-still-specify-a-path-to-chromedriver-using-chromeoptions-in-python](https://stackoverflow.com/questions/43382548/can-i-still-specify-a-path-to-chromedriver-using-chromeoptions-in-python)
- [https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-nginx#request-a-lets-encrypt-certificate-nginx](https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-nginx#request-a-lets-encrypt-certificate-nginx)
- [https://www.godaddy.com/community/Managing-Domains/Problem-adding-txt-record-for-letsencrypt/td-p/115376#](https://www.godaddy.com/community/Managing-Domains/Problem-adding-txt-record-for-letsencrypt/td-p/115376#)