Tip: set depends_on_past=True in Airflow when creating a forecasting model pipeline
Tags: airflow, dataengineering
Published: 2023-08-21 18:17:24

---
title: Tip: set depends_on_past=True in Airflow when creating a forecasting model pipeline
published: true
date: 2023-07-27 03:00:00 UTC
tags: airflow,dataengineering
canonical_url: https://www.xboard.dev/posts/2023_07_27_tip_set_dependsonpast_in_airflow_when_creating_forecasting_model_pipeline/index.html
---

---

💡 When creating forecasting model pipeline in Airflow set `depends_on_past=True`.

---

<section id="why">
<h2 data-anchor-id="why">Why?</h2>
<p>Forecasting models help us predict future data based on patterns and trends observed in historical data. These models surprise, surprise… <strong>depend on past</strong> data, hence any deviation from the expected sequence can affect their accuracy.</p>
<p>One way to ensure the integrity of the sequence of data while creating forecasting models in Apache Airflow is to set the <code>depends_on_past</code> argument to ‘True’. This way each subsequent task in the DAG will only run once its preceding task has completed successfully. This approach guarantees that the forecasting model is fed with the correct inputs in the correct order and that the historical data is correctly sequenced.</p>
<p>If you don’t set <code>depends_on_past=True</code> and you have a problem with one of the partitions of the data used by your model, one of two things will happen in the following days:</p>
<ul>
<li><p>Worst case scenario: your workflow will run without any issues being reported. This is a sign that you forgot to add sensors to test the availability of historical data and, chances are, your features are using incomplete data, to say the least.</p></li>
<li><p>Alternatively, subsequent executions of your workflow will also fail, bombarding your oncall colleague’s pagerduty with messages. And when the problem is finally fixed, you still have to remember to clear and restart all the previous failed executions.</p></li>
</ul>
</section><section id="how">
<h2 data-anchor-id="how">How?</h2>
<p>For a specific task, most probably a sensor:</p>
<div id="cb1" style="background: #f1f3f5;"><pre><code><span id="cb1-1">task <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> SomeSensor(</span>
<span id="cb1-2"> task_id<span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span style="color: #20794D;
background-color: null;
font-style: inherit;">'task'</span>,</span>
<span id="cb1-3"> depends_on_past<span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="cb1-4"> <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ...</span></span>
<span id="cb1-5">)</span></code></pre></div>
<p>or in your dag <code>default_args</code> to enable <code>depends_on_past</code> for all your tasks</p>
<div id="cb2" style="background: #f1f3f5;"><pre><code><span id="cb2-1">default_args <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb2-2"> <span style="color: #20794D;
background-color: null;
font-style: inherit;">'depends_on_past'</span>: <span style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="cb2-3"> <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ...</span></span>
<span id="cb2-4">}</span>
<span id="cb2-5"></span>
<span id="cb2-6">dag <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> DAG(</span>
<span id="cb2-7"> dag_id<span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span style="color: #20794D;
background-color: null;
font-style: inherit;">"time_series_pipeline"</span>,</span>
<span id="cb2-8"> default_args<span style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>default_args,</span>
<span id="cb2-9"> <span style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ...</span></span>
<span id="cb2-10">)</span></code></pre></div>

</section>