AWS Glue first experience - part 4 - Deployment & packaging
Tags: aws, python, datascience, serverless
Published: 2020-08-29 18:13:00

In this episode, we are going to explore how can we reuse our code and how to deploy AWS Glue application which consists of more than one file.
I expected the workflow to be very similar to AWS Lambda which is already well known and optimised for python but due to involvement of Spark this is not entirely true for AWS Glue.

## Challenge number 5: Stay DRY

Because the initialisation process for all data sources and jobs was similar I decided it would be a good idea not having to repeat myself much and create a library with a set of functions which take care of parsing arguments, getting configuration data, simplify the PySpark or Pandas interface.

Due to the nature of different types of dependencies each job type requires. PySpark - `.py`, `.zip`, Python Shell - `.egg`, `.whl`. And the fact that all our code is held in monorepo.

I decided to create a simple python package with `setuptools` and follow the [src structure][src_py].

This gives me enough flexibility to produce needed formats and also reference to the library from inside `requirements.txt`.

{% gist https://gist.github.com/1oglop1/f5ee29229f3b9b0a10d354c2f59aa7ad file=simple_setup.py %}

## Challenge number 6: Deployment & packaging

Okay so now that I have all the necessary components covered let's put them together and deploy with [Terraform][Terraform].

For each data source, we have defined two transitions `raw to refined` and `refined to curated`.

AWS Glue requires 1 `.py` file as an entry point and rest of the files must be plain `.py` or contained inside `.zip` or `.whl` and each job should be able to have a different set of requirements.

Another requirement from AWS Glue is that entry point script file and dependencies have to be uploaded to S3.

Anything uploaded to S3 then also has to be listed in [Terraform][Terraform] as a [Special parameter][special_parameters] `--extra-py-files` in form of comma separated list of S3 URLs, eg. `s3://bucket/dep1.zip, s3://bucket/deb2.zip` or `s3://bucket/dep1.whl, s3://bucket/deb2.whl`.

Since this list can be very dynamic it's best to keep it as short as possible. As you can see there are a number of operations require from the developer and developing more than one job requires a significant effort. Therefore I decided to use the following structure

```
/
├── glue/
│   ├── data_sources/
│   ├── ├── ds1/
│   │   └── ├── raw_to_refined/
│   │       │   ├── Makefile
│   │       │   ├── config.py
│   │       │   ├── raw_to_refined.py
│   │       │   └── requirements.txt
│   │       └── refined_to_curated/
│   │           ├── Makefile
│   │           ├── config.py
│   │           ├── another_dependency.py
│   │           ├── refined_to_curated.py
│   │           └── requirements.txt
│   └── shared/
│       └── glue_shared_lib/
│           ├── Makefile
│           ├── setup.py
│           ├── src
│           │   └── glue_shared/__init__.py
│           └── tests/
└── terraform/
```

Let's describe the structure above.

* `/glue/` holds all the python code

* `/glue/data_sources/` holds the code of jobs for each data source

* `/glue/data_sources/ds1/` - is a directory of 1 particular data source containing transformation

* `/glue/data_sources/ds1/raw_to_refined` and `/glue/data_sources/ds1/raw_to_refined`
   are the transformations whose content is then deployed as a particular AWS Glue Job

* `/glue/shared/` - contains shared items among the glue (jobs, files, etc...)

* `/glue/shared/glue_shared_lib` - is the library used by the jobs, contains configuration interface and other useful functions

* `/terraform/` holds all resources required to be deployed our Glue Jobs, IAM roles, lambda functions, etc...

Now that we understand the structure, we can look closer at a particular job.

### Glue Job structure

This is a standard blueprint which fit my purpose of developing and deploying several AWS Glue jobs.

```
ds1/
└── raw_to_refined/
   ├── Makefile
   ├── config.py
   ├── raw_to_refined.py
   └── requirements.txt
```

In this case, we are looking at a transformation job from raw zone to refined zone.

* `Makefile` - contains several make targets which names are common across all jobs, `clean, package, test, upload-job, upload, deploy` then implementation of each target is job-specific.

  * `clean` - Cleans up the local temporary files.

  * `package` - For PySpark job creates a `.zip` file with dependencies. For Python shell job it runs `pip` and downloads all the wheel files.

  * `upload-job` - uploads entry point script to S3 - useful for quick updates during the development in case you are not doing any changes inside dependent files.

  * `upload` - upload all related files `.zip`, `.whl` and entrypoint `.py` file to S3.

  * `deploy` - performs `clean`, `package` and `upload`

* `config.py` - is responsible for creating a configuration object. This is an `extra .py` file which is later packaged and used as a dependency. For the sake of saving some time I used python dictionary but with growing complexity of the job I'd recommend spending time on creating a better approach.

* `raw_to_refined.py` - this is the main entry point file executed by AWS Glue. U can use this file execute the code in dependencies or directly implement transformation logic. The name of this file is purposely the same as it's parent directory which will be explained later.

* `requirements.txt` - Is a standard [requirements file][requirements_file] It's a very simple way of managing your dependencies.

This setup gives me enough flexibility as a developer to run and update jobs in the cloud from within my local environment as well as using CI/CD. Another benefit is that if you have PySpark with Glue running locally, you can use that as well!


### Terraform part

This is an example of deploying PySpark Job via Terraform, Python Shell job follows the same process with a slight difference (mentioned later).

To create or update the job via Terraform we need to supply several parameters Glue API which Terraform resource requires. Plus the parameters our job expects.

{% gist https://gist.github.com/1oglop1/f5ee29229f3b9b0a10d354c2f59aa7ad file=tf1_sample.tf %}

We need to provide:

* `Command.ScriptLocation` - represented as `ds1_raw_to_refined_job_script_location` - this is our entrypoint script

* `DefaultArguments` - represented as map `ds1_raw_to_refined_job_default_arguments` -- this holds the main configuration

Key `--extra-py-files` in the map `ds1_raw_to_refined_job_default_arguments` is a comma separated string of S3 urls pointing to our dependencies eg. `s3://bucket/dep1.zip,s3://bucket/deb2.zip`

All extra dependencies fit in 1 `.zip` file and once you get the shape of these parameters there is no need to change it.

This brings a potential problem of human oversight, especially with Python Shell jobs. Where dependencies are wheels and by default, wheel name contains a version number `numpy-1.19.0-cp36-cp36m-manylinux2010_x86_64.whl`.

Then any change in `requirements.txt` or job arguments also requires a change in Terraform resource which is maintained in a different directory.

I haven't solved this problem during the project but this could be potentially avoided by maintaining a list of dependencies in the S3 bucket in the form of a file which could be generated during the `make`. And then Terraform would just download this information. However this theoretical the solution can lead to chicken-egg problems and I wish AWS Glue had a better option to maintain dependencies and the job config. Just allowing to use S3 prefix instead of the full URL would be a good start.

The code for the examples in this article can be found in my GitHub repository [aws-glue-monorepo-style]


[aws-glue-monorepo-style]: https://github.com/1oglop1/aws-glue-monorepo-style

[src_py]: https://blog.ionelmc.ro/2014/05/25/python-packaging/
[Terraform]: https://www.terraform.io
[special_parameters]: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
[requirements_file]: https://pip.pypa.io/en/latest/user_guide/#requirements-files