Working on AWS Glue locally with Jupyter Notebook
Tags: awsglue, jupyter, aws
Published: 2022-03-03 05:31:32

This post is written with intent to help users get started who wish to develop ETL scripts in AWS Glue interactively whilst saving on the cost. There are two ways for interactive development of ETL scripts

- Develop Interactively with Notebook servers that are provided by AWS Glue.
- Run Notebooks on Local Machine and use glue resources through a dev 
endpoint.

We are going to discuss and go forward with the second option. For the first option you just need an appropriate IAM access role plus the ability to pass that role to AWS Glue and you can simply start the server.

## Key Steps

1. Setting up IAM Role.
2. Setting up a dev endpoint.
3. Setting up Jupyter Notebook and spark dependencies.
4. Connecting dev endpoint with local Jupyter notebook.
5. Testing the notebook connection

## Setting up IAM Role

Follow [this](https://docs.aws.amazon.com/glue/latest/dg/create-notebook-policy.html) aws doc and you should be good to go. In case you face an error detailing Access denied exception with something about iam:PassRole [this](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) aws doc detailing passRole and it’s use case can be a good starting point.

## Setting up a dev endpoint

Before starting do remember that dev endpoints are charged for every moment they are in ready state, irrespective of whether they are being used or not. Provisioning an endpoint takes roughly about ten minutes, It might be economical to go on to next section first and once done return to this, but anyways here we go.
### Getting your SSH Fingerprint
_(skip this if you know how to generate a ssh key)_

A dev endpoint requires an ssh public key to verify that only devices with listed ssh fingerprints get access to the endpoint.

To generate a ssh fingerprint you could simply copy paste the command below in your terminal or watch this video and get a hold of what’s happening before pasting the command given below.

<iframe width="692" height="389" src="https://www.youtube.com/embed/vSvNBmZN548" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

`$ ssh-keygen -t rsa
`
When you are asked to name the file where ssh key is stored name it anything but just ensure that it ends in .pem. Such as if I wish to store my key in a file called test then when prompted for file name I would enter test.pem .This is an AWS mandate.

When prompted for a passphrase, enter something rememberable you will be asked for it every-time you connect to the dev endpoint.

Now two files are created file_name.pem and file_name.pem.pub. The .pub file contains a public key and this will be given to dev endpoint. The .pem file contains private key and this is what will verify you, remember it’s location.

### Creating the dev Endpoint
_(skip this if you know how to create a dev-endpoint)_

Go to AWS console and in the Navigation tab under ETL click on Dev Endpoints.
![Image describing location of dev-endpoint on glue console page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bydkj8lxxhs5v66z062u.png)

Click on Add endpoint and the draft appears.
![Create Endpoint draft](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yt9lxjjotpwezozg68hp.png)
 
Give an apt IAM role to the point with access to your data storage and permission to create EC2 instances since you are planning to run a notebook.

Under the security configuration select worker type (Standard, G1x …) and DPU’s to be assigned : Default is 5 and Min value is 2.

( PS: If you wish to use one dev endpoint for running multiple spark instances in parallel. [This](https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-sharing.html) aws doc might be helpful in getting a rough idea of resource allocation. )

Click on next and choose Skip Networking Information. Go to next page, here you will paste the key generated in file_name.pem.pub file from ssh-keygen command.

Finally review the draft and click on finish. The dev endpoint is created and goes into provisioning mode. It takes 8–12 mins in provisioning. Once done the endpoint is ready to be used.
After the endpoint is in ready state click on the endpoint and inside you should see something like this.
![Endpoint Ready Status stating commands to connect with the endpoint locally](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9kwryn77blhz4fer0fjr.png) 

The SSH to Python/Scala/ tunnel remote … are the commands you will need to use to connect to dev endpoint from your machine locally. The important bit is the **glue@ec2-….compute.amazonaws.com** .

To run the server for notebook (using livy) locally the command would be

`$ ssh -i <PrivateKey.pem> -vnNT -L 8998:169.254.76.1:8998 glue@ec2-....compute.amazonaws.com`

## Setting up Jupyter Notebook and Spark Dependencies

Follow the commands below

`$ pip3 install jupyter
$ pip install sparkmagic`

If you face errors in building sparkmagic wheel following two links might be helpfull

- [Install krb5-config on your machine](https://command-not-found.com/krb5-config) (once this is done re run the pip install sparkmagic command)
- [Error in building krb-5config](https://stackoverflow.com/questions/19955775/error-command-gcc-failed-with-exit-status-1-on-centos) (after this re run the above command for krb5 install )

`python3 -m jupyter nbextension enable --py --sys-prefix widgetsnbextension`

If you get an error stating jupyter not found it’s mostly because of installing commands in two different python versions (python 2.7 and python 3.6 or above). If you face this then run install jupyter and widgetsnbextension command in same python version (“pip and python” or “python3 and pip3”)

### Install spark kernels

`## find sparkmagic's location`
`$ pip show sparkmagic`
`## cd to that directory.`
`## location will be diffrent for you`
`$ cd home/user/.local/lib/python3.6/site-packages`
`## Depending on your use case install the below given kernels`
`# for spark`
`$ jupyter-kernelspec install sparkmagic/kernels/sparkkernel`
`# for python`
`$ jupyter-kernelspec install` `sparkmagic/kernels/pysparkkernel`

If the above kernel installation command fails the try running them as give below

`# for spark`
`$ jupyter-kernelspec install --user` `sparkmagic/kernels/sparkkernel`
`# for python`
`$ jupyter-kernelspec install --user` `sparkmagic/kernels/pysparkkernel`

Now is the time to ensure that your dev endpoint is ready before moving on with next step.

## Connecting Local jupyter notebook with dev endpoint

If you don’t know much about Apache Livy this is the place to get context. Anyways it comes preinstalled for linux machines. It’s default port is 8998

From your provisioned endpoint pick up the command for SSH tunneling to remote interpreter. It looks something like

`sh -i <private-key.pem> -vnNT -L :9007:169.254.76.1:9007 glue@ec2–3–70–179–133.compute-1.amazonaws.com`

Now change it to connect with livy port (8998) and it becomes

`ssh -i <private-key.pem> -vnNT -L 8998:169.254.76.1:8998 glue@ec2–3–70–179–133.compute-1.amazonaws.com`

The <private-key.pem> is substituted by a path to your ssh private key file created. Once done run the command enter the passphrase and you are connected.

Leave this window open and open another terminal tab and run jupyter notebook. The jupyter notebook opens and you are ready to go.

## Testing the notebook connection

Run the below given code in a pyspark kernel. If it runs without error you know you are good to go.

`spark`

Wait for it to create a spark application with an output like
![Spark Instance creation output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/75fxs8uwobcx2z69ziu5.png)
 

`import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
glueContext = GlueContext(SparkContext.getOrCreate())`

If it all runs you are ready to start developing. You could import tables from your glue catalogs and try different transformations now.

If you get an error detailing insufficient yarn resource on running the spark command : you probably have multiple spark sessions going on. Open **https://localhost:8998/ui** . This opens livy server where you can manage all the instances running

**PS: Stop the dev- endpoint from aws as soon as your work gets over. You are being charged heavily for every minute of its existence.**

## Conclusion

Hopefully your jupyter notebook ran successfully by following the tutorial. Please find the benefits and pricing of the dev endpoint below.

### Benefits

- No need to keep endpoint running as all codes are saved to your notebook.
- No need to extract data out of S3 to your notebook to perform feature engineering.
- Can share the spark cluster among the team, resulting in cost saving.
- Can perform ad hoc logic of TB’s of data.
- Endpoint can scale to 50 DPU which is approximately (16 x 50 = 800 GB memory and 4 x 50 = 200 cores) 100 executors.

### Pricing

Glue is priced per DPU which stands for Data Processing Unit
$0.44/Hour/DPU, billed per second, with a 10-minute minimum.
Cheers 🥂

## Bibliography

- https://medium.com/explore-ml/connect-jupyter-notebook-to-aws-glue-endpoint-f5fc78b4e00