September Cloud Guru Challenge - Event-Driven Python on AWS
Published: 2020-10-12 21:26:13

I found out about Forrest Brazeal  and https://acloud.guru last month as I was looking for some hands-on projects to start a Cloud / DevOps portfolio
I could show recruiters.

I'm a seasoned embedded software engineer but with little to no Cloud knowledge. As I know I can pick up new technology and tools qui fast,
recruiters may think otherwise until they see what I can do. This is why I was delighted to find about the Cloud Resume Challenge to start
working on my portfolio when I read about the September Cloud Challenge. The September Challenge is about building an ETL process (Extract Transform Load)
in Python that will read 2 CSVs online, manipulate them and store them in a database, with various error detection / coping mechanisms. That process is
to be scheduled to be run once by day. 
The produced data would then be visualized using a BI Dashboard.


##The ETL Job:

I'm moderately experienced in Python so this was a great opportunity to learn by doing. 
I won't be ashamed to say that before starting the Challenge I had no idea what a Lambda Function and Serverless were about.
This plus a lot of test cases to cover made the coder much bigger than one could expect. Various messages are to be sent according to the way the data is
treated. A "retry the next day" mechanism was to be implemented, but how the hell can one do that when the Lambda function memory is wiped after it returned?
S3 is the answer. Data (be it list, json, dictionary) can be serialized then saved to a S3 bucked, to be downloaded, deserialized and process the next day.
Does that make it a stateful serveless function?

Another thing that surprised me was the way one is supposed to package the lambda function and dependencies. Very smart actually, although some complications
are to be expected if the package is a bit too large, yet another thing to consider.

To sum up, this was a great learning experience but I'm not proud of the lack of optimization of my code. I'll hopefully refactor it sometime when I have more 
free time. I will just say I'm more used to read/write driver registers than handling json.

##Database:

Not much to say here as the ETL is not very complex DB wise. I first went for DynamoDB because it's a NoSQL DB and wanted to see how easy it was to implement it.
Oh, easy it was indeed but when I noticed how complex it would be to get my data fed to QuickSight I went for PostgresSQL instead.
Little did I know that someC other services provide easy plugin to DynamoDB! Wish I didn't read it!

##Cloud + IAC:

I had never logged in any Cloud service console prior to this challenge so this was another great opportunity to learn and grow.
I get why AWS is so popular and has such a big market share: although there are are a lot of services, getting started is made easy thanks to all the
available documentation and tutorials are all the Internet. Plus the Free Tier made my September Challenger free.
The easy way when it comes to IAC here would be to use CloudFormation but I don't want to be locked in to a Vendor. Instead I opted for Terraform
that seems to be the industry standard. It's quite amazing to be able to deploy or destroy a complete infrastructure in a matter of minutes.
Most AWS services are available so it's made very easy, no excused not to go IAC once you know how powerful it is.

##Testing:

Unit testing of the transformation process was made very easy due to the breaking down of the ETL in modules.
The tricky part was the Smoke Test of the whole process. Before deploying the Lambda Function to production, we need to make sure that our last change in GIT
didn't break anything and we had to do that without touching the production infrastructure. It would not be smart to duplicate the IAC and Lambda source code :
any change in either of them would mean we would have to merge it manually to the duplicate, not so DevOpsy right?
Instead a temporary folder for Lambda Function and IAC code would be created and a sneaky sed command would rename all the production items to smoke items and voilà,
a new infrastructure is ready to be deployed, tested and destroyed.
My approach then was to replace the data csv urls with static one so I would know how many rows I would get, and then compare them and fail if the figures didn't match.

##CI/CD:

I picked GitLab for its simplicity and 400 minutes of free pipeline usage, that was more than enough to play around a bit and set up my pipeline.
For small to medium sized projects GitLab CI is probably the safe choice. After headaches with Jenkins I was amazed to be able to set up the whole pipeline in a day!

##Security:

All credentials are stored in the Amazon Secure Store so I paid a special attention not to leak any information. Who wants to wake up and see that you just got a 10 000$ bill.

##Rework:

Alas the implementation is far from perfect. My Python code could be refactored for readability and data manipulation could be improved for a shorter processing time.
The packaging size of my Lambda Function could be much smaller by stripping some libraries and removing unused dependencies.
I also assume that my IAC code could be more elegant.

It's been a hectic month but being able to start coding past 9PM every night and enjoying it in spite of the fatigue really showed that I want to pursue this career evolution.
Thanks again Forrest & ACG.

For those who might be interested to take the challenge after reading this, here you go : https://acloudguru.com/blog/engineering/cloudguruchallenge-python-aws-etl , you won't regret it.

By the way, here is my dashboard. Let's say I have never been the artistic or powerpoint kind of guy ;)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/61pvquodlfgwpnaivsvi.png)