#CloudGuruChallenge – Event-Driven Python on AWS
Tags: aws, etl, python, serverless
Published: 2020-10-09 16:18:00

It was a great experience participating in [Forrest Brazeal's](https://aws.amazon.com/developer/community/heroes/forrest-brazeal/) Cloud Resume [Challenge](https://cloudresumechallenge.dev/instructions/) in July and now have a resume [website](https://resume.logan-toler.dev) that I am proud to share. I am excited that he introduced another [challenge](https://acloudguru.com/blog/engineering/cloudguruchallenge-python-aws-etl) in September. I would recommend anyone, whether for fun or to learn, a fresher or seasoned professional, to participate.
 
Goal: Automate an ETL processing pipeline for COVID-19 data using Python and cloud services.
---

![Project Architecture](https://raw.githubusercontent.com/tolerl1/EventDrivenPython/master/covid-etl-architecture.png)

- - - - 
I was able to modify my template from the previous challenge to quickly jumpstart this project. I made a few templates to try out different scenarios but decided on using Lambda and DynamoDB for the ETL pipeline. The decision was based on cost-efficiency and reducing the complexity through the number of services used.

Visualizing the application was simple but it was tricky translating that into code. Transforming the dataframe into a DynamoDB compliant-format was by far my biggest pain-point. Fortunately, I learned about Jupyter, which helped me speed up the testing of different functions and packages. 

I use SNS to alert me of any run-time failures in both functions and the number of rows added when the ETL job completes. I added CI/CD using GitHub Actions. I prefer this method because it's free, supports the aws-cli, and introduces an extra layer of automation. 

Lastly, data isn't meaningful without a way to visualize it. BI tools are expensive and many have no native support for DynamoDB. I extended my template to accommodate a custom real-time dashboard and created an additional template to provision a CloudFront distribution with an S3 origin. API Gateway is used to trigger a second Lambda function that uses [Matplot](https://matplotlib.org/api/index.html) to create graphs; It also calculates the recent totals of cases, recoveries, and deaths. The graphs are stored in an S3 bucket and the API response returns the calculated totals in JSON format. 

![matplot-chart](https://covidetl-coviddatastorage-lnvxxen5igp3.s3.amazonaws.com/Screen+Shot+2020-10-09+at+11.51.37.png)
![api-response](https://covidetl-coviddatastorage-lnvxxen5igp3.s3.amazonaws.com/Screen+Shot+2020-10-09+at+11.58.00.png)
While I have not built the dashboard yet, the infrastructure is available for when I do. I learned about Redash, a SaaS BI tool, that supports DynamoDB. I decided to use the free-trial and host my [dashboard](https://app.redash.io/logan-toler/public/dashboards/bQv4OpTwd8oZ2NHjT31gtjtIFiDayXu1XPK0NbTM) there for the time being.



- - - -
Each project I do makes the next one easier. I completed this in roughly two weeks, whereas the previous challenge took about 1.5 months.  
 
A few things that I would like to improve upon:
- Restructure the src folder to remove duplicate code
- Error handling/logging
- Make use of Lambda layers

The source code and additional details are on [GitHub](https://github.com/tolerl1/EventDrivenPython).