My pain with Serverless and AWS Lambda
Tags: serverless, aws, lambda, python
Published: 2020-03-31 07:13:54

---
title: My pain with Serverless and AWS Lambda
published: true
description: I want to share my story with Serverless Development in AWS Lambda and Python
tags: serverless,aws,lambda,python
---
Just recently, I got to work with Serverless on AWS Lambda. It's a great technology, and I love the idea not to manage and provision underlying servers. I do much programming in Python, and luckily AWS Lambda comes with a Python Runtime. The Serverless Framework is an excellent way to start; that's what I thought...
Here is my story with Serverless Development in AWS Lambda and Python and some of my pain.

# Python and Serverless Framework

Probably, you have heard the term **Serverless** which is a technology where you don't care about managing servers and its underlying infrastructure. Still, they are there, but you care about *server less*. I recently had a task where I needed to deploy a Deep Learning function with [Tensorflow](https://www.tensorflow.org/) and [Keras](https://keras.io/). 
Our Data Scientist passed me his `predict` - function, and I needed to productionize it. So good so far.
I thought of using the [Serverless Framework](https://serverless.com/) to deploy this function on AWS.
During the set up by using `virtualenv`, I encountered my first problem. After googling, I found out that the library is supporting currently Python3.7 and I had Python3.8 installed. Hmm... okay, no problem, I will use [Docker](https://github.com/docker-library/python/blob/695bd3c10cdf1692a2af9abdc51f0eff99731e78/3.7/stretch/slim/Dockerfile) instead.
```Bash
cd /path/to/my/project
docker run -it --name tensorflow -v $(pwd):/root/home imagehash bash
```
The Docker-Image comes with `pip` out of the box.
```Bash
# Installing Tensorflow
pip install tensorflow==1.15
    
# Installing Keras
pip install Keras==2.2.5
```
Notice that I don't run `virtualenv` here because I am now in a new fresh Docker container with zero packages. So, I removed it. I also know that there is a [solution](https://stackoverflow.com/questions/59302043/error-when-installing-tensorflow-python-3-8) for using Tensorflow with Python 3.8 but put that in the `requirements.txt` 😉 We should stick to a [stable version](https://www.tensorflow.org/install/pip).

# Deploying
Now was the time to deploy it on AWS. That means we need to create a handler. Luckily, **Serverless Framework** comes with a Python template
```Bash
# Python
sls create --template aws-python3 --name name-of-service --path path/to/store
# Note: sls is short for serverless, both are possible
```
This command generates a `handler.py` for your function and a `serverless.yml` for the serverless setup. After removing all the comments and changing the Python runtime, it looks as follows
```Yaml
service: aws-python
    
    provider:
      name: aws
      runtime: python3.8
    
    functions:
      hello:                    # 1st hello function name for AWS
        handler: handler.hello  # 1st handler tells sls where to look for a handler			
                                # 2nd handler is the Python file
                                # 2nd hello is the function inside the Python file
```
I still have some trouble understanding the functions part and its keys and how to call the function. I hope I got it right as above described. When I started the Serverless Framework, I had something like
```YAML
functions:
      handler:
        handler: handler.handler # Brainf*ck 🤪
```
Anyhow, now I am ready to deploy by running the command
```Bash
# Requirement: AWS Cli must be installed
sls deploy
```
and execute it with its logging
```Bash
sls invoke -f handler --log
# Output
[ERROR] Runtime.ImportModuleError: Unable to import module 'tensorflow'
```
Hmm, it seems like AWS is not running `pip install -r requirements.txt`. Which is weird because that file is also deployed. That's annoying, but there is a [plugin](https://github.com/UnitedIncome/serverless-python-requirements#readme) for that.
```Bash
sls plugin install -n serverless-python-requirements
```
It creates `package.json` and a `node_modules` because the Serverless Framework is written in Javascript and Node.
Okay, now we should be ready to go 💪
```Bash
sls deploy
# Error
An error occurred: UploadLambdaFunction - Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: c3b94dc7-6a06-11e9-8823-bb373647997a).
```
What??😱
262144000 bytes, yep that's the error you get. In theory, you could upload 0 bytes (I don't know how that is possible 😅), I guess that's why the message is in bytes. In other terms, the function should not be more than 262 MB. Well, my `handler.py` is 4 kB and the project in total with Serverless Framework and its plugin 5 MB.

# What causes this number?
It must be the dependencies causes by the `requirements.txt`. That plugin must be doing something, and it installs the whole Tensorflow and Keras library! And guess what Tensorflow is already ~500 MB large 😱.

Alright, the **Serverless-Python-Requirements** can deal with Lambda's size limitations.
```Yaml
custom:
  pythonRequirements:
    zip: true
```
And before the `handler.py` I had to paste following code snippet in order to unzip the requirements:
```Python
try:
  import unzip_requirements
except ImportError:
  pass
```
It creates a `unzip_requirements.py` also. However, the Lambda gets even bigger...
```Yaml
custom:
  pythonRequirements:
    zip: true
    slim: true # This had been added.
```
And guess what, it still fails🙈
It is still too big...
After googling, I found an excellent comprehensive [blog post](https://medium.com/@mike.p.moritz/running-tensorflow-on-aws-lambda-using-serverless-5acf20e00033), and I applied the change (without `dockerizePip` though)
```Yaml
custom:
  pythonRequirements:
    dockerizePip: non-linux
    zip: true
    slim: true
    noDeploy:
      - boto3
      - botocore
      - docutils
      - jmespath
      - pip
      - python-dateutil
      - s3transfer
      - setuptools
      - six
      - tensorboard
package:
  exclude:
    - node_modules/**
    - model/**
    - .vscode/**
```
Et voilá, we have a zip file suddenly with 150 MB. Whenever we install a package from `pip`, we also install its dependencies. And often, we install unconsciously heavy packages such as [NumPy](https://numpy.org/) or [SciPy](https://www.scipy.org/). With the `noDeploy`, we omit specific packages from the deployment. The list above is a standard list which already built into Lambda, as well as Tensorboard.

# Running the Function

To invoke the function, we need to run
```Yaml
sls invoke --function name-of-function
```
And of course it fails 😭
```Json
{
  "errorMessage": "[Errno 28] No space left on device",
  "errorType": "OSError",
	..
}
```
Locally, I had run it on my Mac with 8-Core Intel Core i9 and 16 GB RAM. On AWS, I had used the maximum Memory of 3008 MB.
Well, Lambda was not able to run my Data Science function. 
In the end, I put that into a Docker Container and shipped that one function to AWS.

# Conclusion

AWS has evolved the computing process and application development with Lambdas. Serverless was born. However, it's not ideal for every use case. You borrow some computational power where you don't have any influence on capacity or resources. That is, in many cases, sufficient, but when it comes to processing an image or using a vast library, then you might overthink your usage of Serverless.

AWS has evolved the computing process and application development with Lambdas. Serverless was born. However, it's not ideal for every use case. You borrow some computational power where you don't have any influence on capacity or resources. That is, in many cases, sufficient, but when it comes to processing an image or using a vast library, then you might overthink your usage of Serverless.

I don't say I won't use it. In fact, I have many successful stories to tell, and I love Serverless 💓 It is just not as easy as everyone praised. You still need to know what's behind the **handler**, the size of the Lambda, and the computational power you need. You need to configure on many places such as `YAML`- file(s), AWS region, because not every region is the same,  and the dependencies of your dependencies. Maybe it really depends on the language. I had less pain with Node than with Python. Regardless, n the beginning, it is a lot of try and error, which is also time-consuming and can be frustrating.

Nevertheless, **Serverless** is my first choice when developing a service, and I think after many ups and downs, you will have a better feeling when to use it and when not.