Running AWS Glue locally on a Windows Laptop
Published: 2024-01-16 03:58:39

by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

# Highest-level Context

If you are into “**Shift-Left**” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

Objective: Towards enabling up to 5x developer-productivity by allowing developers to robustly SIMULATE the Cloud-environment on a Windows-Laptop.

# Quick Summary

Aiming for very simple single command, based on `python`-scripts --> to execute your python-code as a Glue Job’s build **locally on your Microsoft-Windows Laptop**.

Even if your company disallows AWS CLI credentials, I have a section on how to significantly raise your productivity, in debugging/developing your python-code.

To state the obvious, everything in here is 100% Python-Scripts.

**Note**: You should aim to have your software work on `arm64` containers, which invariably is cheapest compute on cloud.   More below.

# Problem Statements

1. Using Git for capturing ALL code-changes while simultaneously copying-n-pasting into AWS Glue-Studio (for testing/troubleshooting) is painful, error prone and frustrating.
2. As time progresses, your Glue Job will become complicated and require more than one Python-script.Worse, you have one or more folder-hierarchies, all of which contain `PY` files that you need to `import`!
3. Many developers prefer to develop/test/troubleshoot their code as “**plain python**”, and Not as a Glue-Job.There is No good reason to deny such developers from doing just that.. .. while ensuring that code will work without any issues inside AWS Glue running locally on Windows-Laptop, and eventually work without issues inside Glue on AWS.
    1. When running as “plain python”, there should be No runtime dependencies (like “`import awsglue`”).
    1. When running as “plain python”, there should be No `spark`-dependency.
    1. When running as “plain python”, all inputs/files should be on local laptop’s filesystem.  All output should be written to local filesystem only.
    1. When running as “plain python”, all information from Glue-Catalog should be available **OFF**line (as a Python `Dict` object)
1. How to proactively ensure the Glue Job will work on all chip-architectures - without having to scramble later? How to explicitly utilize all `x86_64/amd64/arm64` architectures locally on your Windows-Laptop?
1. If the Enterprise does Not allow Laptops to have AWS-Credentials (in `~/.aws/credentials` file);Even so, how can I **EFFICIENTLY** test/debug the python-code file locally on my laptop, even as it needs access to Glue Catalog and/or S3 buckets?

# Get started!

Based on your needs on AWS choose between these 2 lines below:
```powershell
$env:BUILDPLATFORM="linux/amd64"
$env:BUILDPLATFORM="linux/arm64"
```

Then run these commands in a `Powershell` window:

```powershell
$env:DOCKER_DEFAULT_PLATFORM="${BUILDPLATFORM}"
$env:TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

$WORK_AREA=~
cd $WORK_AREA
git clone https://gitlab.com/tosarma/macbook-m1.git
```

## To try out a sample ..

```powershell
cd macbook-m1
cd AWS-Glue/src

pip3 install docker
python3 $WORK_AREA/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.py  sample-glue-job.py
```

**WARNING**: Without the benefit of “`docker` cli”, you get **ZERO** visibility into the progress of docker-activity. This is due to use of un-friendly Docker’s Python APIs, because of which the python-code _ WILL _  _ HANG _ for a long time!

To repeat, “`run-glue-job-LOCALLY.py`” will hang with NO output, for roughly 2-to-5 minutes (depending on how much CPU and MEMORY you have allocated to the Docker-Desktop, as well as speed of your internet connection).

Important: I only tested using `Python3.11`; No other Python version tested.

# Ready to use it for your own Glue-Script?

1. First, read the full details within the [`macbook-m1/AWS-Glue/README.md`](https://gitlab.com/tosarma/macbook-m1/-/blob/main/AWS-Glue/README.md?ref_type=heads) file.
1.  Copy the `*.py` files in the `macbook-m1/AWS-Glue/src/common` subfolder into --> YOUR project’s TOPMOST-folder.
    *  ATTENTION: the files `./src/common/*.py` must exist in your project, after you are done copying.
1. Make sure to edit your `PY` file, to look like the example file provided ([`sample-glue-job.py`](https://gitlab.com/tosarma/macbook-m1/-/blob/main/AWS-Glue/src/sample-glue-job.py?ref_type=heads))
1.  From your project root, run:

```powershell
cd <your-project-ROOT-folder>

python3 $WORK_AREA/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.py `
          path/to/your/file.py
```

If you are having problems importing the new files under `./src/common`, then try adding this command below and then retry the above command.

```powershell
$env:PYTHONPATH="your-project's-root-folder"
```

Is your Python code-base consisting of multiple files across multiple folder-hierarchies?
See section below titled “_Complex folder-hierarchies?_”

# Want to change the CLI-arguments?
Three simple steps:

1. Edit the file  `macbook-m1/AWS-Glue/src/common/cli_utils.py`
2. Look inside “`process_all_argparse_cli_args()`” and make changes in that function.
3. Look inside “`process_std_glue_cli_args()`” and make changes inside that function.
    *   Note: make sure to make similar changes in above steps 2 & 3.

Example:-

1. You would like to support a new cli-arg as:
    * `       --JOB_NAME   123_ABC`

1. Insert a new line at (say) **line # 125** for `JOB_NAME`.
    *   This will ensure your code will get the value `123_ABC` when running __ INSIDE __ AWS-Glue !!
1. Insert a new line at (say) **line # 78** for `--JOB_NAME`
    *  This will allow you to run your python-code as a PLAIN python-command and read this CLI-arg.
    *  See more re: this in a following section titled “_running as a PLAIN python-command_”

Never edit the file:
`    macbook-m1/AWS-Glue/src/common/glue_utils.py `

The files “`common.py`” and “`names.py`” in that same folder can be edited.   Feel free to play around with them.

## Tips, Issues & Errors
See **Appendix** sections, for tips on configuring Docker-DESKTOP.

**Question**: Want to automatically cleanup/delete  the Docker-containers - after they exit?
**Answer**: INSERT the cli-arg “`--cleanup`” BEFORE the python-script-name, to that “`run-glue-job-LOCALLY.py`” script.

# Advanced User - Complex folder-hierarchies?
Is your Python code-base consisting of **multiple files across** multiple folder-hierarchies?
**Are you aware** that Glue requires you to **ZIP up all** those OTHER python-files into a single Zip-file?
FYI only - this requirement is driven by Spark!

That script “`run-glue-job-LOCALLY.py`” will **automatically** do that for you --> that is, it will **automatically look UNDER** the current-working-directory, and find all `**/*.py` files and put them in a **temporary** ZIP-file.
The script will then automatically pass it on to _Glue-inside-Docker_ (running on your laptop).

If you need to import `PY` files in parent/ancestor levels, you are in an UN-fortunate NO-Solution sitution.  FYI: Unlike on MacOS/Linux, you are _ NOT _ able to take advantage of MacOS/Linux “_symlink_” (“`ln -s`”) to those parent/ancestor files, where `git` CLI can preserve these “_symlinks_” as exactly just that.

# No AWS Credentials on your laptop?
For security-reasons, many companies are denying developers the AWS-credentials for AWS-CLI use.

That means you have a showstopper -> re: locally testing/debugging your python-code, for scenarios like:

1. COPY INPUT-files from S3-buckets --> into the “current-working directory”.
2. COPY OUTPUT-files from the “current-working directory” --> into S3-buckets.
3. Lookup Glue Catalog.
4. .. etc ..

To workaround this restriction ..
You need to write code that detects whether its running on a Windows-Laptop -_<u>versus</u>_- actually running inside AWS-Cloud.
In other words, you need to “_Short-Circuit_” all that code that interacts with AWS-APIs (Glue-Catalog, S3, ..) and mock the expected response from those AWS-APIs.

If you use my script “`run-glue-job-LOCALLY.py`”, it **automatically** sets an _environment-variable called_ “`running_on_LAPTOP`” when running your python-code inside a Docker-Glue container **on your laptop**!!

**How-To** “_short-circuit_”:

```
if ( os.environ.get('running_on_LAPTOP') ):
    print( "!!!!!!!!!!!!!!!!! running on laptop !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" )
    .. ### assume S3-get is already done and file is available in current-directory
    .. ### assume Glue-Catalog-Query is already done and ..
       ###      the "JSON-Response" is available in current-directory as a JSON-file
    ..
else:
    ..
    ..
```

To state the obvious, on AWS-Cloud, AWS-GLUE does __ NOT __ support environment-variables.
So, this environment-variable called “`running_on_LAPTOP`” will be **UN**-defined when running inside AWS-Cloud.

# Running as a plain python command
EXAMPLE:
I’m going to use the same “`macbook-m1/AWS-Glue/src/sample-glue-job.py`” file, **to show how to** run as PLAIN Python-program.

FYI: My python-code in that `sample-glue-job.py` expects the following 6 CLI-arguments (with their values).

If you do _ NOT _ like this list of CLI-args, see section above titled “_Want to change the CLI-args?_”

```
python3 sample-glue-job.py --ENV sandbox ^
     --commonDatabaseName MyCOMMONDATABASENAME ^
     --glueCatalogEntryName MyDICT ^
     --rawBucket MyRAWBUCKET ^
     --processedBucket MyPROCESSEDBUCKET  ^
     --finalBucket MyFINALBUCKET
```

If you want the ability to run both as a plain python-script as well as run it inside AWS-glue, you __ MUST __ replicate the structure and code within this “`macbook-m1/AWS-Glue/src/sample-glue-job.py`”.

# APPENDIX

## Running out of Disk-space or Memory?
Screenshot below shows the recommended “high” settings.
After building images, you can reduce:

- “CPU” can be lowered to “2”.
- “Memory” can be lowered to “4GB”.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wvy0t07w4vfsxvbgst4c.png)

End of Article.