How to use Beautiful Soup in AWS Lambda for Web Scraping
Tags: aws, python, serverless, linux
Published: 2022-02-20 00:59:19

The purpose of this post is to show how to use the Beautiful Soup module in AWS Lambda with Python Runtimes.Keep in mind, AWS Lambda is not integrated with all the modules available for Python. And the only way to import the modules to Lambda is to bundle the lambda function alongside the modules in an isolated environment.

The idea is similar to containers, we create an isolated environment, we import only the dependencies we need to make our application working, we write our code, we bundle it and we export it to the Cloud.

**1) Install Pipenv**

Pipenv is the module that will enable us to create an isolated environment for our lambda function.

For Debian:

```
sudo apt update -y
sudo apt upgrade -y
sudo apt install python-pip
pip3 install pipenv 
```

For RPM:

```
sudo apt update -y
sudo apt upgrade -y
sudo yum install python-pip
pip3 install pipenv
```

**2) Create a Python 3.8 environment**

At the time I am writing, the documentation for Beautiful Soup has been written for Python 3.8 [https://www.crummy.com/software/BeautifulSoup/bs4/doc/](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) . As result,we need a Python 3.8 isolated environment. Let's create it:

```
pipenv --python 3.8
```

![Creating Python Environment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bkszi6s2u7izj4lhhqxw.png)

```
pipenv shell
```

![Log in the Virtual Environment ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4wxfzfdi5uoiey7f1u42.png)

**3) Install our bs4 dependency**

```
pip install bs4
```

![Installing bs4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bffmmnw8aw83xdwf1ubi.png)

**4) Write the Lambda function**
Create a file name lambda_function.py, **it's very important to name it "lambda_function.py", otherwise the Lambda handler will not work**
Copy the code supplied in the following link, paste it onto the the lambda_function.py file, and save it. [https://github.com/aissa-laribi/bs4-in-lambda/blob/main/lambda_function.py](https://github.com/aissa-laribi/bs4-in-lambda/blob/main/lambda_function.py)
![Lambda function](https://i.imgur.com/pVI5nLc.png)

**5) Bundle up the lambda function and the dependencies**

Now it's time to move the lambda function alongside the dependencies of our environment.

```
cp lambda_function.py ~/.local/share/virtualenvs/<yourenvname>/lib/python3.8/site-packages
cd ~/.local/share/virtualenvs/<yourenvname>/lib/python3.8/site-packages
ls
```
And normally, you should be able to see your lambda function alongside the dependencies.

![Lambda function & dependencies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/183xk4efijimb1rfme8j.png) 

Now, we need to zip the whole directory

```
zip -r9 bs4_in_lambda.zip *
cp bs4_in_lambda.zip ~/Desktop
cd  ~/Desktop
```

![Zipping The Lambda Function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vovgzguq3selyvbbsfk3.png)

**
6) Upload the Zip File to your Lambda Function****

Then, create a Lambda Function, and make sure that Python 3.8 Runtime is selected.


![Creating the Lambda function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nomjydkyoo6jq8utbbmg.png)

Go to Code and in the top right corner, click on Upload from

![Lambda function uploading zip file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7qa6b4j67ynj2cmaiqn8.png)

And select the Zip file we have created.

![Lambda function uploading zip file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/50957hwb56akpfsw3eio.png)

The Lambda function will show up and we can see on the left side all the pip packages stored in folders.

![Imgur Image](https://imgur.com/oNKXjHL.png)

Then go to Configuration and increase the runtime because there are 10 pages to be scraped.


![Lambda Configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aa9x9zte1o0dsg5oit4q.png)

Let's set a 5 minute runtime to make sure it will scrape all the pages.


![Lambda Runtime](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/msdza9mqrql2q275u8y5.png)

Then,return to Code > Test.

Leave the Configuration Test content by default and add any name to the Event Name and Save.



![Configuratio Test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0enndgjps88bjr1nmfiu.png)

Click on Test.

And sometimes we will get this error message.

![Test Failing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ps0jx88a0v69u8ox4fu2.png)

The trick is to switch between http and https in our function.

Press Ctrl + F Scroll Down and replace all "https" with "http", Deploy and test the function.


![Tweak the function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jopejvepwuasd3np9j1r.png)

Et Voila!
We have a list of websites.

![Imgur Image](https://imgur.com/RR0ThyF.png)

Then go to Monitor > Logs > Click in the LogStream of the first invocation.Then, a new window will open, and you will get access to the full list.

![Imgur Image](https://imgur.com/ToBe3lD.png)

![Imgur Image](https://imgur.com/thkbASt.png)