Navigating Week Two: Insights and Experiences from My Tublian Internship Journey
Tags: opensource, internships, softwareengineering, programming
Published: 2023-12-31 20:03:14

## Introduction

In the second week of my **Tublian internship**, the spotlight shifts to the dynamic realm of open source. Delving into the intricacies of project structures, community dynamics, and the innovative integration of AI Copilot, this week promises an enriching exploration of the open-source ecosystem.

In week Two, I contributed to the [Apache Airflow](https://github.com/apache/airflow) repository.

Undoubtedly, the second week at Tublian proved to be a more formidable terrain than the initial foray. However, within the crucible of challenges, I unearthed invaluable learning experiences that have not only enriched my skill set but also brought a profound sense of satisfaction in conquering hurdles.

###  Week 2 Contribution at Tublian Internship- Navigating the Azure Synapse Seas


*Contribution to Apache Airflow and Azure Synapse Integration*

In the immersive world of my Tublian internship's second week, the spotlight shone on orchestrating tasks within the intricate Azure Synapse environment. My journey led me to a key contribution in the form of enhancing a Directed Acyclic Graph (DAG) in Apache Airflow. Here's a glimpse into my endeavors

#### Understanding the Landscape

The DAG in question, named `example_synapse_run_pipeline`, acts as a choreographer for tasks related to Azure Synapse. With tasks ranging from running Spark jobs to executing specific pipelines, the DAG was a dynamic canvas awaiting my contributions.

Below is The code snippet for the `AzureSynapseRunPipelineOperator` I worked on:

    
    begin = EmptyOperator(task_id="begin")

    run_pipeline1 = AzureSynapseRunPipelineOperator(
        task_id="run_pipeline1",
        azure_synapse_conn_id="azure_synapse_connection",
        pipeline_name="Pipeline 1",
        azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint",
    )

    begin >> run_pipeline1

    from tests.system.utils.watcher import watcher`

#### Contributions Made

1. Optimizing Spark Job Execution

The heartbeat of the DAG pulses through the `AzureSynapseRunPipelineOperator`, and my first contribution involved optimizing the execution of Spark jobs. By delving into the intricacies of the existing codebase, I identified opportunities to enhance efficiency, resulting in a more streamlined Spark job execution process.

2.  Enhancing Pipeline Execution

The DAG orchestrates the execution of specific pipelines within Azure Synapse, and I took charge of refining this process. Leveraging the `AzureSynapseRunPipelineOperator`, I enhanced the DAG to seamlessly execute designated pipelines, contributing to a more robust and efficient workflow.

## Challenges Faced

Contributions seldom come without challenges. Navigating the complexities of the Azure Synapse environment and understanding the nuances of Apache Airflow tasks presented hurdles. However, each challenge became an opportunity for growth. Debugging intricacies, ensuring compliance with coding standards, and aligning with the collaborative nature of open source were all part of the learning journey.


![I encountered challenges while pushing new changes to the same branch where I made the initial changes on my end. I was able to fix it by `git pull` and then pushing it back to the branch.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3j4nzhy4jga75e1ufnwq.png)

![This is where a maintainer asked me to create a new example for `AzureSynapseRunPipelineOperator` followed by referencing it and then utilizing it. With this, I was able to fix it.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7fm5wczxwd5xr39auz1s.png)

## The Collaborative Spirit

The open-source nature of Apache Airflow thrives on collaboration. My contributions underwent meticulous code reviews, where experienced maintainers provided constructive feedback. This iterative process not only refined the codebase but also deepened my understanding of best practices in collaborative coding.

## Testing and Integration

The integration of a `watcher` task from the system testing module was pivotal. It not only validated the success and failure scenarios but also emphasized the importance of comprehensive testing in the world of DAG orchestration.

![The code passed through testing and integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zc2bhe5z1w3e8fywqs2v.png)

![Yaay! The PR got merged!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5wzvgt6z6vkvmda5q7f2.png)

## Looking Ahead

As Week 2 concludes, my contributions to the `example_synapse_run_pipeline` DAG stand as markers of progress. The journey through Apache Airflow and Azure Synapse has been an immersive learning experience, and I eagerly anticipate the upcoming weeks filled with new challenges, contributions, and continued growth.

................

Thank you for taking the time to delve into my blog post; your attention is incredibly valued! If you enjoyed the journey, a round of applause with a 👏 would mean the world to me. Share your thoughts and insights with a comment 💬, and let's continue this conversation.

Connect with me on [GithHub](https://github.com/Lymah123), [Medium](https://medium.com/@Lymah_real), [Twitter](https://twitter.com/CodesLymah), and [LinkedIn](https://www.linkedin.com/in/harlimat-odunola-2ab261235/) for the latest updates and to stay in the loop on upcoming ventures. Let's make this journey together an ongoing exploration into the vast realms of discovery.

Hold up, if you haven't seen week one of this series, please check it out [here](https://dev.to/lymah/tublian-internship-journey-navigating-the-internship-landscape-in-week-one-3dj7). 

Stay tuned for more insights from the Tublian internship saga!

If you want to get started with your open-source journey, check out this [OpenSauced](https://intro.opensauced.pizza/#/01-intro) intro course.

Once again, thank you for being a part of this exciting dawn of discovery!