collapsible-doc-dag 0.1rc1
An Airflow AddOn package for centralized collapsible documentation for DAGs and their Tasks.
Keywords: Airflow,UI,UX,Front-end,data governance,documentation,clean code

# CDocDAG
**CDocDAG** is an AddOn python package for **_Apache Airflow_**.

It exposes the **CDocDAG** class which extends the standard DAG class.

The difference is that with **CDocDAG** (standing for Collapsible Documentation DAG),
the DAG `doc_md` string is in a collapsed DOM element
on the Airflow UI and can be click-expanded (this part became "standard" with Airflow 2).

In addition, **CDocDAG** appends a sub-section to the DAG documentation.
This sub-section, itself also click-expandable, is a topologically-ordered list
of all individual `doc_md` for individual tasks.

The CDocDAG documentation shows in the three tabs which natively display
the DAG documentation, namely&nbsp;:
"Graph", "Tree", "Calendar". It is to be noted that, in the DAG "Graph" view, the tasks list is animated with :
  - click events
    (showing the usual task pluri-buttons window: "Task Instance Details", "View Log", etc.)
  - mouse-over events
    (synchronized with the D3 DAG graphic representation,
    displaying upstream and downstream tasks with colored borders)

Here's how it renders on Airflow 2&nbsp;:<br />
![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/CDocDAG_anim.gif)

**CDocDAG** is optimized against Airflow 1.10.12 and Airflow 2.1.2 on Chrome browser.
To extend that coverage, if need be, pull requests are encouraged ;)

---

## HowTo :

To use **CDocDAG** for your DAGs, simply ```pip install collapsible_doc_dag```
on the python env running your instance of Airflow or add ```collapsible_doc_dag``` to your Airflow ```requirement.txt```
prior to restarting your _airflow-webserver_. It's that easy.

> below is a side-by-side view of the 'tutorial' DAG that comes together
with the Airflow 1 distribution.
On the left column, we can see the original version.
On the right column, the **CDocDAG** version
(with screenshots of both `code` and `UI rendering` on Airflow 1) :
>
Tutorial DAG             |  Tutorial **CDocDAG**
:-------------------------:|:-------------------------:
code | code
![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/tutorial_BEFORE/tutorial_BEFORE_code.png)  | ![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/tutorial_AFTER/tutorial_AFTER_code.png)
>As can be seen above, the only differences are on line 30 and 66 (class import and class instantiation)
and the addition line 123 of a call to the 'compute_doc()' class method ; once everything has been declared.
>
>The result on Airflow 1 is as shown below :

UI rendering | UI rendering
:-------------------------:|:-------------------------:
![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/tutorial_BEFORE/tutorial_BEFORE_ui.jpg)  |  ![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/tutorial_AFTER/tutorial_AFTER_ui.jpg)


> Below we look into how to get to the same **CDocDAG** final result as above,
except that this time, in the `code`, the instantiation of the DAG
is made 'contextually'. Note that the final call to 'compute_doc' is no more necessary,
as it is performed (internally) at context-exit time :

>![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/tutorial_CONTEXT/tutorial_CONTEXT_code.png)

> The third way to instantiate a DAG (this is new with Airflow 2) is thru the usage of python decorators.
To do that and still enjoy the benefits of **CDocDAG**, simply change the standard import
for the **CDocDAG** equivalent and that's it, you're good to go&nbsp;:<br />
- ~~```from airflow.decorators import dag, task```~~
- ```from collapsible_doc_dag.decorators import dag, task```
---
If you like this Apache Airflow AddOn python package, I'd appreciate you spread the word and
give the GitHub repo a star as well
<a href="#" style="cursor: default;">
![](https://github.com/aurelienmorgan/collapsible_doc_dag/raw/master/images/github_star.png)</a>.
Thank you for your support&nbsp;!

---
<br /><br /><br />
## Extra
### The `collapsible_doc_dag.resources_utils.png_to_img_tag` function

The **CDocDAG** has originally been developed to be used either in production envirnonment on Airflow over GCP
(thru a `Composer` instance), or locally, or locally thru an Airflow Docker container.
For the purpose of easing the handling of png images located within a subfolder to the Composer 'data' directory
(#see [https://cloud.google.com/composer/docs/concepts/cloud-storage](https://cloud.google.com/composer/docs/concepts/cloud-storage)),
the **CDocDAG** project also exposes the `collapsible_doc_dag.resources_utils.png_to_img_tag` function,
which shall be used as follows :
```python
dag.doc_md = """\
A dag documentation markdown... """ + png_to_img_tag('relative_path_to_png_image') + """
...some more dag documentation markdown.
"""
```
where `relative_path_to_png_image` is the path to the png image,
relative to the Airflow `data/resources` directory.
This ensures that the image is properly displayed, notwithstanding which of
the three above-mentioned situations we're in (gcp, local, local thru Docker).