Outreachy ended and here is my experience in Apache Airflow Open Source
Tags: outreachy, apacheairflow, opensource
Published: 2022-03-27 18:00:50

My [Outreachy](https://www.outreachy.org/) internship ended, It was fast and I learned a lot of things in this period. I've never programmed in Python like I did in [Apache Airflow](https://github.com/apache/airflow). My mentor [Jarek Potiuk](https://github.com/potiuk) guide me write my first lines of code for Open Source, I am very grateful for his patience 🙌.
This is basically what encompasses my learning, on the technical skills side. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2p4z723h27bnu84h7sb4.png)
**Apache Airflow** is an open source workflow management platform for data engineering pipelines by creating workflows as directed acyclic graphs (DAGs) of tasks written in Python. It is considered one of the most robust platforms used by data engineers.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6k2b8xwvznornhex1y4o.png)
 
**Apache Airflow** has a very complete and complex local development environment written mainly in _Bash_, it's called **Breeze**. However, most project contributors are proficient in **Python**, so the local development environment is not good for maintenance and debugging.

🎯 Our goal **Convert Airflow Local Development environment (Breeze) from Bash-based to Python-based and also make it possible for Windows users to run the Breeze environment.**

My mentors detailed each task in the backlog using **GitHub** Projects. ShoutOut to my mentors: [Jarek Potiuk](https://github.com/potiuk), [Elad Kalif](https://github.com/eladkal), **Nasser Kaze** for giving me as much detail to get started.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44m66gk5ulvcfk1tx8ec.png)
 

> Backlog of our tasks on GitHub

## **Technical Skills**

To start working on a feature, we create a separate branch `git checkout -b <new-branch>`. We always “rebase”, to place our changes at the end of the latest updates that were in the original repository.

```bash
 git remote add apache git@github.com:apache/airflow.git
 git fetch apache
 git fetch --all
 # This will print the HASH of the base commit which you 
 # should use to rebase your feature from
 git merge-base my-branch apache/main
 git rebase HASH --onto apache/main
 git push --force-with-lease
```
One of the challenges for me was understanding how to resolve conflicts with **Git**, and **rebasing** was one of the techniques that worked quite well for me.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a6vuzebxfdz0kmj4vv8j.png) 

> [Image by Ryan Hodson](https://code.tutsplus.com/es/tutorials/rewriting-history-with-git-rebase--cms-23191)

Through discussions, my mentor and I analyze the bash script code and we defined what is the best way to approach it. Before I can rewrite it in Python, there are things that can definitely be done differently because of the flexibility of Python.

Before pushing, we use **pre-commits**, to review the code before it goes to the main repository of the version control system

```bash
pre-commit install
pre-commit --version
pre-commit uninstall
```
After finishing the implementation, we can create a Pull Request, if it is not ready, we can set it as a **DRAFT** to indicate that it is Work in Progress and that it is not yet ready for review.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7n1quz1vgjh2lcn5diiv.png) > [Image by Luke Hefson](https://github.blog/2019-02-14-introducing-draft-pull-requests/)

Once our Pull Request is created, test flows in **GitHub Actions** are executed, they validates many things, depending on the content that is modified. **_Example_**:

- If the change is related to documentation, validation is quite simple and awaits the approval of our mentor.

- If the core of Apache Airflow is changed, it runs all the necessary tests with different versions of Python.

This is an amazing flowchart that describes all the test cases workflow.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yx21sxyt7a7dbsy9odhy.png)

> Mermaind flowchart created by [Jarek Potiuk](https://github.com/apache/airflow/blob/main/SELECTIVE_CHECKS.md)
 
Once all the Jobs finish successfully, our Pull Request is ready to be approved and merged.

During my contribution stage I have used different Python libraries, these are some of them:

- **PSutil**
Library to  retrieve information on running processes and system utilization (CPU, memory, disks, network, sensors) in Python. 

- **Click**
Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.

- **Subprocess**
  module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. 

- **Data Class**
 Data classes are just regular classes that are geared towards storing state, rather than containing a lot of logic. 

## **Soft Skills**

These were some of the Soft Skills I focused on, I'm still learning from this.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n1uniu4bys92sod83gxj.png)

One of the great challenges for me was to "communicate" effectively orally and in writing. I have been able to perform well in writing, as I have taken the time to detail my thoughts and let my mentor know. Orally, the weekly meetings and the one-on-ones with my mentor were a great opportunity to practice. **I am quite happy because it is my first professional experience in an English-Speaking Environment.**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vk16jiv6yoi0xz3k6hme.gif)
 

---

I have tried to summarize all my experience in one post, I have left many without mentioning, but I will do it little by little through other posts.

Some technical things I am exploring these days are Docker, Kubernetes, Helm Chart,GHCR.io in Apache Airflow environment.

I am still working in Open Source in Apache Airflow, to change the script which executes all the tests in GitHub actions, it is called Selective Check which is in Bash Script. My mentor and I wanted to make it easier in Python.


🌿 **My lasts words:**

**Thanks Outreachy, and thanks to my mentors to gave me this opportunity in Apache Airflow.** 🥲🙏😊🙋🙈

---

I hope that you have learned something from this, and if you have questions about these topics please do not hesitate to write to me.
🌱 [Edi LinkedIn](https://www.linkedin.com/in/edithpuclla/)
🌱 [Edi GitHub](https://github.com/edithturn)