10 Benefits to using Airflow
Tags: datascience, data, airflow, bigdataetl
Published: 2019-10-16 22:58:40

---
title: 10 Benefits to using Airflow
published: true
tags: data-science,data,airflow,big-data etl
canonical_url: https://medium.com/analytics-and-data/10-benefits-to-using-airflow-33d312537bae
---

![](https://cdn-images-1.medium.com/max/845/1*VdzG4IrO_zKbh4_Sz2dP7A.png)

In continuation from a series of posts, where I have explained the [basics of airflow](https://medium.com/analytics-and-data/airflow-the-easy-way-f1c26859ee21) and how to setup [airflow on azure](https://medium.com/analytics-and-data/setting-up-airflow-on-azure-connecting-to-ms-sql-server-8c06784a7e2b?source=your_stories_page---------------------------), and what [considerations to have when using airflow](https://dev.to/julienkervizic/5-considerations-to-have-when-using-airflow-5c81-temp-slug-8477774), I wanted to cover in details what makes airflow a great tool to use for data processing.

#### **1. DAGs** :

Dags are a way to setup workflows, they can setup a sequence of operations that can be individually retried on failure and restarted where the operation failed. Dags provide a nice abstraction to a series of operations.

#### 2. Programmatic Workflow Management:

Airflow provide a way to setup programmatic workflows, Tasks for instances can be generated on [fly within a dag](https://www.data-essential.com/create-dynamic-workflow-in-apache-airflow/). While Sub-DAGs and [XCom](https://airflow.apache.org/concepts.html?highlight=xcom), allows to create complex dynamic workflows.

Dynamics Dags can for instance be setup based on variables or connections defineed within the [Airflow UI](https://www.astronomer.io/guides/dynamically-generating-dags/).

#### 3. Automate your Queries, Python Code or Jupyter Notebook

Airflow has a lot of operators setup to run code. Airflow has operator for most databases and being setup in python it has a PythonOperator that allow for quickly porting python code to production.

[Papermilll](https://airflow.readthedocs.io/en/latest/papermill.html) is an extension to jupyter notebook, allowing the parametrization and execution of notebooks, it is supported through airflow PapermillOperator. Netflix notably has suggested a combination or airflow and [papermill](https://papermill.readthedocs.io/en/latest/) to automate and deploy notebook in production:

[Part 2: Scheduling Notebooks at Netflix](https://medium.com/netflix-techblog/scheduling-notebooks-348e6c14cfd6)

#### **4. Task Dependency management:**

It is extremely good at managing different sort of dependencies, be it a task completion, dag runs status, file or partition presence through specific sensor. Airflow also handles task dependency concept such as [branching](http://airflow.apache.org/concepts.html?highlight=branch#branching).

[Use conditional tasks with Apache Airflow](https://medium.com/@guillaume_payen/use-conditional-tasks-with-apache-airflow-98bab35f1846)

#### **5. Extendable model:**

It is fully extendable through the development of custom sensors, hooks and operators. Airflow notably benefits from a large amount of [community contributed operators](https://github.com/apache/airflow/tree/master/airflow/contrib/operators).

Operators in different programming languages such as R [[AIRFLOW-2193]](https://github.com/apache/airflow/pull/3115) are being built in using python wrappers, in the future other programing language such as Javascript which also have [python wrapper (pyv8)](https://code.google.com/archive/p/pyv8/) could also be created.

#### **6. Monitoring and management interface:**

![](https://cdn-images-1.medium.com/max/1024/1*FdCmNWA-rKgmsZshGpaj2Q.png)

Airflow provides a monitoring and managing interface, where it is possible to have a quick overview of the status of the different tasks, as well as have the possibility to trigger and clear tasks or DAGs runs.

#### 7. Retry policy built in:

![](https://cdn-images-1.medium.com/max/853/1*oDnmyS_c-HUZduhZ28b8Hw.png)

It has an auto-retry policy built-in, configurable through :

- **retries:** number of retries before failing the task
- **retry\_delays: (**timedelta) delay between retries
- **retry\_exponential\_backoff: (**Boolean) to setup an exponential backoff between retries
- **max\_retry\_delay:** Maximum delay (timedelta) between retgries

These arguments can be passed through the context to any operator, as they are supported by the BaseOperator class.

#### **8. Easy interface to interact with logs:**

![](https://cdn-images-1.medium.com/max/1024/1*nNl14Q4P0BSQiz-HJzWkAw.png)

Airflow provides an easy access to the logs of each of the different tasks run through its web-ui, making it easy to debug tasks in production.

#### **9. Rest API:**

[Airflow’s API](https://airflow.apache.org/api.html) allows to create workflows from external sources, and to be data product on top of it:

[Using Airflow Experimental Rest API on Google Cloud Platform: Cloud Composer and IAP](https://medium.com/google-cloud/using-airflow-experimental-rest-api-on-google-cloud-platform-cloud-composer-and-iap-9bd0260f095a)

The rest API, allows to use the same paradigm used to built pipelines, to create asynchronous workflows, such as[custom machine learning training operations](https://medium.com/adobetech/adobe-experience-platform-orchestration-service-with-apache-airflow-952203723c0b).

#### **10. Alerting system:**

It provides a default alerting system on tasks failed, email is the default, but alerting through slack can be set up using a callback and the slack operator:

[Integrating Slack Alerts in Airflow](https://medium.com/datareply/integrating-slack-alerts-in-airflow-c9dcd155105)

More from me on [Hacking Analytics](https://medium.com/analytics-and-data):

- [One the evolution of Data Engineering](https://medium.com/analytics-and-data/on-the-evolution-of-data-engineering-c5e56d273e37)
- [Overview of efficiency concepts in Big Data Engineering](https://medium.com/analytics-and-data/overview-of-efficiency-concepts-in-big-data-engineering-418995f5f992)
- [Setting up Airflow on Azure & connecting to MS SQL Server](https://medium.com/analytics-and-data/setting-up-airflow-on-azure-connecting-to-ms-sql-server-8c06784a7e2b?source=friends_link&sk=c9c7c39100e7eaef45c7793caaa265b7)
- [Airflow, the easy way](https://medium.com/analytics-and-data/airflow-the-easy-way-f1c26859ee21)

* * *