airflow-bootstrap-utils 0.2.1
Collection of Python tools to generate Airflow DAGs from control and configuration files.
Requires Python: >=3.10
Keywords: airflow_bootstrap_utils

=======================
Airflow Bootstrap Utils
=======================

Collection of Python tools to generate Airflow DAGs from control and configuration files.



Exported Console Scripts
------------------------

* airflow-generate-dag: Generate Airflow DAGs from control and configuration files.
* airflow-validate-control-file: Validate control files.

Installation
------------

```shell
pip install airflow-bootstrap-utils
```

Usage
-----

.. code-block:: shell

    generate-airflow-dag-script --control_file /tmp/airflow-demo/simple_workflow.yaml --outdir /tmp/airflow-demo
    --config_file was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/conf/config.yaml'
    --template_dir was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/templates'
    --logfile was not specified and therefore was set to '/tmp/airflow-demo/generate_dag.log'
    Wrote Airflow DAG Python script list to file '/tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt'

Contents of airflow_dag_scripts.txt:

.. code-block:: text

    cat /tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt               
    ## method-created: /tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/manager.py
    ## date-created: 2025-02-15-101314
    ## created-by: sundaram
    ## control-file: /tmp/airflow-demo/simple_workflow.yaml
    ## logfile: /tmp/airflow-demo/generate_dag.log
    /tmp/sundaram/airflow-bootstrap-utils/user_processing_L1/2025-02-15-101314/L1/user_processing_L1.airflow.dag.py
    /tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/L2/user_processing_L2.airflow.dag.py

Contents of L1 airflow script:

.. code-block:: python

    from airflow import DAG
    from airflow.operators.bash_operator import BashOperator
    from datetime import datetime

    #------------------------------
    # Airflow DAG definition
    #------------------------------
    dag = DAG(
        'user_processing_L1',
        description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
        schedule_interval='“@weekly”',
        start_date=datetime(2024,3,22),
        catchup=False,
        )

    #------------------------------
    # Airflow task defintions
    #------------------------------

    """
    The do_something executable is going to do something.
    It is going to use the lab number argument to do
    something.
    The lab number is a required parameter.
    """
    do_something = BashOperator(
        task_id='do_something',
        bash_command='bash /opt/do_something.sh --lab_number L1',
        dag=dag
    )


    """
    The do_something_else executable is going to do something else.
    It is going to use the lab number argument to do
    something else.
    The lab number is a required parameter.
    """
    do_something_else = BashOperator(
        task_id='do_something_else',
        bash_command='bash /opt/do_something_else.sh --lab_number L1',
        dag=dag
    )

    #------------------------------
    # Define the task dependencies
    #------------------------------
    do_something >> do_something_else

Contents of L2 airflow script:

.. code-block:: python

    from airflow import DAG
    from airflow.operators.bash_operator import BashOperator
    from datetime import datetime

    #------------------------------
    # Airflow DAG definition
    #------------------------------
    dag = DAG(
        'user_processing_L2',
        description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
        schedule_interval='“@weekly”',
        start_date=datetime(2024,3,22),
        catchup=False,
        )

    #------------------------------
    # Airflow task defintions
    #------------------------------

    """
    The do_something executable is going to do something.
    It is going to use the lab number argument to do
    something.
    The lab number is a required parameter.
    """
    do_something = BashOperator(
        task_id='do_something',
        bash_command='bash /opt/do_something.sh --lab_number L1',
        dag=dag
    )


    """
    The do_something_else executable is going to do something else.
    It is going to use the lab number argument to do
    something else.
    The lab number is a required parameter.
    """
    do_something_else = BashOperator(
        task_id='do_something_else',
        bash_command='bash /opt/do_something_else.sh --lab_number L1',
        dag=dag
    )

    #------------------------------
    # Define the task dependencies
    #------------------------------
    do_something >> do_something_else


=======
History
=======

0.1.0 (2024-03-20)
------------------

* First release on PyPI.