ka-air-dfs 2023.2.1
Airflow Dag Source Factory
Requires Python: >=3.10

##########
ka_air_dfs
##########

Overview
========

**ka_air_dfs** is a Command Line Package with Application Sub Package **dfs** and Utility Sub Package uts.

- dfs is used to create dags in ``Python source format`` from files 
  in ``portable Workflow DSL yaml format``

Installation
============

.. start installation

``ka_air_dfs`` can be installed from PyPI or Anaconda.

To install with ``pip``:

.. code-block:: shell

        $ python -m pip install ka_uts

To install with ``conda``:

.. code-block:: shell

        $ conda install -c conda-forge ka_uts

.. end installation

Sub Packages
============

``ka_air_dfs`` contains the following Sub Packages.

.. _sub-packages-label:
.. list-table:: *Sub Packages*
   :widths: auto
   :header-rows: 1

   * - Sub Package
     - Description
     - Type
   * - dfs
     - Dag Source Factory Processing
     - Application
   * - uts
     - Dag Source Factory Utilities
     - Utility

Modules
=======

-----------
dfs Modules
-----------

Sub Package ``dfs`` of Package ``ka_air_dfs`` contains the following Modules.

.. _dfs-modules-label:
.. list-table:: *dfs Modules*
   :widths: auto
   :header-rows: 1

   * - Module
     - Description
   * - do.py
     - dfs processing
   * - parms.py
     - parameter definition
   * - __init__.py
     -
   * - __main__.py
     - Command line interface for Sub Package

-----------
uts Modules
-----------

Sub Package ``uts`` of Package ``ka_air_dfs`` contains the following Modules.

.. _uts-modules-label:
.. list-table:: *uts Modules*
   :widths: auto
   :header-rows: 1

   * - Module
     - Description
   * - fac.py
     - Factory Utilities
   * - src.py
     - Source Processing Utilities
   * - uts.py
     - Package Utilities
   * - __init__.py
     - 

Configuration
=============

-----------------------------------
Configuration Files of Sub Packages
-----------------------------------

The Configuration of Application Sub Packages are defined in Yaml Configuration Files
with the Sub Package name located in the data directory of the Package.

.. _application-sub-package-configuration-file-label:
.. list-table:: *Sub Package Configuration File*
   :widths: auto
   :header-rows: 1

   * - Sub Package
     - Configuration File Path
   * - dfs
     - ka_air_dfs/data/dfs.yml

The Configuration file could be changed according to the Rules described inherently.

Calling Convention
==================

------------------
Command Line Usage 
------------------

  - **Call:** *run* ``module as a python script``

    .. code-block:: shell

     /usr/bin/env python3 -m ka_air_dfs.dfs \
       tenant=<tenant> \
       \
       path_in_mask='/data/<tenant>/prj/<project>/PWF/wfl/<task-type>/<tasks>/<files> \
       dir_out_dag_src='/data/<tenant>/prj/<project>/AIR/dags/<task-type>/<tasks>' \
       a_tags=<a_tags> \
       sw_cmd_ix=<``True``|False> \
       \
       sw_async=<``False``|True> \
       sw_debug=<``False``|True> \
       sw_run_pid_ts=<``False``|True> \
       \
       1>dfs_tsk_all.1 2>dfs_tsk_all.2
 
  - **Call:** *run* ``module executable``

    .. code-block:: shell

     /opt/airflow/dev1/latest/python/bin/ka_air_dfs-dfs \
       tenant=<tenant> \
       \
       path_in_mask='/data/<tenant>/prj/<project>/PWF/wfl/<task-type>/<tasks>/<files> \
       dir_out_dag_src='/data/<tenant>/prj/<project>/AIR/dags/<task-type>/<tasks>' \
       a_tags=<a_tags> \
       sw_cmd_ix=<``True``|False> \
       \
       sw_async=<``False``|True> \
       sw_debug=<``False``|True> \
       sw_run_pid_ts=<``False``|True> \
       \
       1>dfs_tsk_all.1 2>dfs_tsk_all.2
 
----------------------------
Command Line Usage: Examples
----------------------------

  - **Example:** *run* ``module as a python script``
    *for tenant = DB, project = FDW, tasks = all, files = \*.yaml*

    .. code-block:: shell

     /usr/bin/env python3 -m ka_air_dfs.dfs \
       tenant='DB' \
       \
       path_in_mask='/data/DB/prj/FDW/PWF/wfl/tsk/all/*.yaml' \
       dir_out_dag_src='/data/DB/prj/FDW/AIR/dags/tsk/all' \
       a_tags='[FDW_TSK]' \
       sw_cmd_ix=True \
       \
       sw_async=False \
       sw_debug=False \
       sw_run_pid_ts=False
       \
       1>dfs_tsk_all.1 2>dfs_tsk_all.2
 
  - **Example:** *run* ``module executable``
    *for tenant = DB, project = FDW, tasks = all, files = \*.yaml*

    .. code-block:: shell
    
     /opt/airflow/dev1/latest/python/bin/ka_air_dfs-dfs \
       tenant='DB' \
       \
       path_in_mask='/data/DB/prj/FDW/PWF/wfl/tsk/all/*.yaml' \
       dir_out_dag_src='/data/DB/prj/FDW/AIR/dags/tsk/all' \
       a_tags='[FDW_TSK]' \
       sw_cmd_ix=True \
       \
       sw_async=False \
       sw_debug=False \
       sw_run_pid_ts=False
       \
       1>dfs_tsk_all.1 2>dfs_tsk_all.2
 
----------
Parameters
----------

The Parameters are entered on the call as **Equates**: <Parameter Name>=<Parameter Value>

Application specific Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. _application-specific-parameter-label:
.. list-table:: *Application specific Parameters*
   :widths: auto
   :header-rows: 1

   * - Parameter
     - Description
     - Type
     - Values
     - Default
   * - path_in_mask
     - input path mask which defines the set of workflow files in portable Workflow
       DSL yaml format which has to be converted into dags in python source format
     - path mask string
     -
     -
   * - dir_out_dag_src
     - output directory for the converted dags
     - directory string
     -
     -
   * - a_tags
     - tags array of the converted dags
     - array string 
     -
     -
   * - sw_cmd_ix
     - if the switch is true then the workflow tasks are relatively
       numbered per task group otherwise per workflow
     - boolean
     - False
     - True

Special Parameters
^^^^^^^^^^^^^^^^^^

.. _special-parameter-label:
.. list-table:: *Special Parameters*
   :widths: auto
   :header-rows: 1

   * - Parameter
     - Description
     - Type
     - Values
     - Default
   * - tenant
     - application tenant
     - string
     -
     - DB
   * - sw_async
     - enables asynchronous processing
     - boolean
     -
     - False 
   * - sw_debug
     - enables the output of debug messages
     -
     - True
     - False
   * - sw_run_pid_ts
     - if the switch is True then the Run Filenames contains 
       pid and timestamp otherwise no pid and timestamp
     - boolean
     -
     - True

---------------------------------
Input Project Directory Structure
---------------------------------

Example
^^^^^^^

.. code-block::

 /data/DB/prj
 │   └── PWF
 │       ├── var
 │       │   ├── rbk
 │       │   │   ├── r_mon.json
 │       │   │   ├── ...
 │       │   ├── tsk
 │       │   │   ├── all
 │       │   │   │   ├── t_1000000.json
 │       │   │   │   ├── t_1000000.yaml
 │       │   │   │   ├── ...
 │       │   │   ├── mon
 │       │   │   │   ├── t_1000000.json
 │       │   │   │   ├── t_1000000.yaml
 │       │   │   │   ├── ...
 │       │   │   └── str
 │       │   │       ├── t_100023.json
 │       │   │       ├── t_100023.yaml
 │       │   │       ├── ...
 │       └── wfl
 │           ├── rbk
 │           │   ├── r_mon10.yaml
 │           │   ├── ...
 │           ├── tsk
 │           │   ├── all
 │           │   │   ├── t_1000000.json
 │           │   │   ├── t_1000000.yaml
 │           │   │   ├── ...
 │           │   ├── ctm
 │           │   │   ├── t_M_CHK_GLOB_PARM.yaml
 │           │   │   ├── t_M_EPE_CHK_IF_FULL_PRECALC.yaml
 │           │   │   ├── t_M_INTE1_CHK_GLOB_PARM.yaml
 │           │   │   ├── t_M_INTE1_FW_ECB.yaml
 │           │   │   ├── t_M_INTE1_IF_1.yaml
 │           │   │   ├── t_M_INTE1_IF_INTERIM_HYPERSTAR.yaml
 │           │   │   ├── t_M_LOAD_CHK_GLOB_PARM.yaml
 │           │   │   ├── t_M_PREP_CHK_GLOB_PARM.yaml
 │           │   │   ├── t_M_PREP_CHK_IF_RCV_MXR.yaml
 │           │   │   └── t_M_PREP_FW_LOAD_SWWR.yaml
 │           │   ├── mon
 │           │   │   ├── t_1000000.json
 │           │   │   ├── t_1000000.yaml
 │           │   │   ├── ...

----------------------------------
Output Project Directory Structure
----------------------------------

Example
^^^^^^^

.. code-block::

 /data/DB/prj
 ├── FDW
 │   ├── AIR
 │   │   ├── dags
 │   │   │   ├── rbk
 │   │   │   │   ├── r_mon10.py
 │   │   │   │   ├── ...
 │   │   │   ├── tsk
 │   │   │   │   ├── all
 │   │   │   │   │   ├── t_1000000.py
 │   │   │   │   │   ├── ...
 │   │   │   │   ├── mon
 │   │   │   │   │   ├── t_1000000.py
 │   │   │   │   │   ├── ...
 │   │   │   │   ├── str
 │   │   │   │   │   ├── t_100023.py
 │   │   │   │   │   ├── ...

-----------------------
Run Directory Structure
-----------------------

Directory Tree if sw_filename_pid_ts=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block::

  data/
  ├── <tenant>/
  │   └── RUN/
  │       └── ka_air_dfs/
  │           └── dfs/
  │               ├── debs/
  │               │   └── debs_<pid>_<timestamp>.log
  │               ├── errs/
  │               │   └── errs_<pid>_<timestamp>.log
  │               ├── logs/
  │               │   └── logs_<pid>_<timestamp>.log
  │               └── reps/
  │                   └── <email>_logs_<pid>_<timestamp>.txt
  ...

Directory Tree if sw_filename_pid_ts=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block::

  data/
  ├── <tenant>/
  │   └── RUN/
  │       └── ka_air_dfs/
  │           └── dfs/
  │               ├── debs/
  │               │   └── debs.log
  │               ├── errs/
  │               │   └── errs.log
  │               ├── logs/
  │               │   └── logs.log
  │               └── reps/
  │                   └── <email>.txt
  ...

Rationale
=========

---------------
Library Purpose
---------------

The purpose of this library is to migrate files in portable Workflow DSL yaml file format into Dags in Python source format.

# .. include:: ka_air_dfs/data/dfs.yml.rst
# .. include:: KA-Architecture.rst

Appendix
========

.. contents:: **Table of Content**