dhis2.py 2.3.0
Python wrapper for DHIS2
License: MIT
Requires Python: >=3.6
Keywords: dhis2

dhis2.py
########

|Latest version| |Downloads| |Build| |BuildWin| |Coverage| |LGTM| |CodeClimate|

A Python library for `DHIS2 <https://dhis2.org>`_ wrapping `requests <http://docs.python-requests.org/en/master/user/quickstart/>`_
for **general-purpose API interaction** with DHIS2. It attempts to be **useful for any data/metadata import and export tasks**
including various utilities like file loading, UID generation and logging. A strong focus is on JSON.

Supported and tested on Linux/macOS, Windows and DHIS2 versions >= 2.25. Python 3.6+ is required.

.. contents::
.. section-numbering::


Installation
=============

Python 3.6+ is required.

.. code:: bash

    pip install dhis2.py

For instructions on installing Python / pip for your operating system see `realpython.com/installing-python <https://realpython.com/installing-python>`_.


Quickstart
==========

Create an ``Api`` object:

.. code:: python

    from dhis2 import Api

    api = Api('play.dhis2.org/demo', 'admin', 'district')

Then run requests on it:

.. code:: python

    r = api.get('organisationUnits/Rp268JB6Ne4', params={'fields': 'id,name'})

    print(r.json())
    # { "name": "Adonkia CHP", "id": "Rp268JB6Ne4" }

    r = api.post('metadata', json={'dataElements': [ ... ] })
    print(r.status_code) # 200


- ``api.get()``
- ``api.post()``
- ``api.put()``
- ``api.patch()``
- ``api.delete()``

see below for more methods.

They all return a *Response* object from `requests <http://docs.python-requests.org/en/master/user/quickstart/>`_
except noted otherwise. This means methods and attributes are equally available
(e.g. ``Response.url``, ``Response.text``, ``Response.status_code`` etc.).

Usage
=====

Api instance creation
-----------------------

Authentication in code
^^^^^^^^^^^^^^^^^^^^^^

Create an API object

.. code:: python

    from dhis2 import Api

    api = Api('play.dhis2.org/demo', 'admin', 'district')

optional arguments:

- ``api_version``: DHIS2 API version
- ``user_agent``: submit your own User-Agent header. This is useful if you need to parse e.g. Nginx logs later.


Authentication from file
^^^^^^^^^^^^^^^^^^^^^^^^^

Load from a auth JSON file in order to not store credentials in scripts.
Must have the following structure:

.. code:: json

    {
      "dhis": {
        "baseurl": "http://localhost:8080",
        "username": "admin",
        "password": "district"
      }
    }

.. code:: python

    from dhis2 import Api

    api = Api.from_auth_file('path/to/auth.json', api_version=29, user_agent='myApp/1.0')


If no file path is specified, it tries to find a file called ``dish.json`` in:

1. the ``DHIS_HOME`` environment variable
2. your Home folder


Get info about the DHIS2 instance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

API version as a string:

.. code:: python

    print(api.version)
    # '2.30'

API version as an integer:

.. code:: python

    print(api.version_int)
    # 30

API revision / build:

.. code:: python

    print(api.revision)
    # '17f7f0b'

API URL:

.. code:: python

    print(api.api_url)
    # 'https://play.dhis2.org/demo/api/30'

Base URL:

.. code:: python

    print(api.base_url)
    # 'https://play.dhis2.org/demo'

system info (this is persisted across the session):

.. code:: python

    print(api.info)
    # {
    #   "lastAnalyticsTableRuntime": "11 m, 51 s",
    #   "systemId": "eed3d451-4ff5-4193-b951-ffcc68954299",
    #   "contextPath": "https://play.dhis2.org/2.30",
    #   ...


Getting things
--------------

Normal method: ``api.get()``, e.g.

.. code:: python

    r = api.get('organisationUnits/Rp268JB6Ne4', params={'fields': 'id,name'})
    data = r.json()

Parameters:

- `timeout`: to override the timeout value (default: 5 seconds) in order to prevent the client to wait indefinitely on a server response.


Paging
^^^^^^

Paging for larger GET requests via ``api.get_paged()``

Two possible ways:

a) Process every page as they come in:

.. code:: python

    for page in api.get_paged('organisationUnits', page_size=100):
        print(page)
        # { "organisationUnits": [ {...}, {...} ] } (100 organisationUnits)

b) Load all pages before proceeding (this may take a long time) - to do this, do not use ``for`` and add ``merge=True``:

.. code:: python

    all_pages = api.get_paged('organisationUnits', page_size=100, merge=True):
    print(all_pages)
    # { "organisationUnits": [ {...}, {...} ] } (all organisationUnits)

*Note:* Returns directly a JSON object, not a requests.Response object unlike normal GETs.


SQL Views
^^^^^^^^^^

Get SQL View data as if you'd open a CSV file, optimized for larger payloads, via ``api.get_sqlview()``

.. code:: python

    # poll a sqlView of type VIEW or MATERIALIZED_VIEW:
    for row in api.get_sqlview('YOaOY605rzh', execute=True, criteria={'name': '0-11m'}):
        print(row)
        # {'code': 'COC_358963', 'name': '0-11m'}

    # similarly, poll a sqlView of type QUERY:
    for row in api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}):
        print(row)

    # if you want a list directly, cast it to a ``list`` or add ``merge=True``:
    data = list(api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}))
    # OR
    # data = api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}, merge=True)

*Note:* Returns directly a JSON object, not a requests.response object unlike normal GETs.

Beginning of 2.26 you can also use normal filtering on sqlViews. In that case, it's recommended
to use the ``stream=True`` parameter of the ``Dhis.get()`` method.



GET other content types
^^^^^^^^^^^^^^^^^^^^^^^

Usually defaults to JSON but you can get other file types:

.. code:: python

    r = api.get('organisationUnits/Rp268JB6Ne4', file_type='xml')
    print(r.text)
    # <?xml version='1.0' encoding='UTF-8'?><organisationUnit ...

    r = api.get('organisationUnits/Rp268JB6Ne4', file_type='pdf')
    with open('/path/to/file.pdf', 'wb') as f:
        f.write(r.content)



Updating / deleting things
--------------------------

Normal methods:

* ``api.post()``
* ``api.put()``
* ``api.patch()``
* ``api.delete()``


Post partitioned payloads
^^^^^^^^^^^^^^^^^^^^^^^^^^

If you have such a large payload (e.g. metadata imports) that you frequently get a HTTP Error:
``413 Request Entity Too Large`` response e.g. from Nginx you might benefit from using
the following method that splits your payload in partitions / chunks and posts them one-by-one.
You define the amount of elements in each POST by specifying a number in ``thresh`` (default: ``1000``).

Note that it is only possible to submit one key per payload (e.g. ``dataElements`` only, not additionally ``organisationUnits`` in the same payload).

``api.post_partitioned()``

.. code:: python
    
    import json
    
    data = {
        "organisationUnits": [
            {...},
            {...} # very large number of org units
        ]
    {
    for response in api.post_partitioned('metadata', json=data, thresh=5000):
        text = json.loads(response.text)
        print('[{}] - {}'.format(text['status'], json.dumps(text['stats'])))


Multiple params with same key
-----------------------------

If you need to pass multiple parameters to your request with the same key, you may submit as a list of tuples instead when e.g.:

.. code:: python

    r = api.get('dataValueSets', params=[
            ('dataSet', 'pBOMPrpg1QX'), ('dataSet', 'BfMAe6Itzgt'),
            ('orgUnit', 'YuQRtpLP10I'), ('orgUnit', 'vWbkYPRmKyS'),
            ('startDate', '2013-01-01'), ('endDate', '2013-01-31')
        ]
    )

alternatively:

.. code:: python

    r = api.get('dataValueSets', params={
        'dataSet': ['pBOMPrpg1QX', 'BfMAe6Itzgt'],
        'orgUnit': ['YuQRtpLP10I', 'vWbkYPRmKyS'],
        'startDate': '2013-01-01',
        'endDate': '2013-01-31'
    })


Utilities
---------

Load JSON file
^^^^^^^^^^^^^^^^^

.. code:: python

    from dhis2 import load_json

    json_data = load_json('/path/to/file.json')
    print(json_data)
    # { "id": ... }


Load CSV file
^^^^^^^^^^^^^^^^

Via a Python generator:

.. code:: python

    from dhis2 import load_csv

    for row in load_csv('/path/to/file.csv'):
        print(row)
        # { "id": ... }

Via a normal list, loaded fully into memory:

.. code:: python

    data = list(load_csv('/path/to/file.csv'))

Generate UID
^^^^^^^^^^^^

Create a DHIS2 UID:

.. code:: python

    uid = generate_uid()
    print(uid)
    # 'Rp268JB6Ne4'

To create a list of 1000 UIDs:

.. code:: python

    uids = [generate_uid() for _ in range(1000)]


Validate UID
^^^^^^^^^^^^

Check if something is a valid DHIS2 UID:

.. code:: python

    uid = 'MmwcGkxy876'
    print(is_valid_uid(uid))
    # True

    uid = 25329
    print(is_valid_uid(uid))
    # False

    uid = 'MmwcGkxy876 '
    print(is_valid_uid(uid))
    # False


Clean an object
^^^^^^^^^^^^^^^^

Useful for deep-removing certain keys in an object,
e.g. remove all sharing by recursively removing all ``user`` and ``userGroupAccesses`` fields.

.. code:: python

    from dhis2 import clean_obj

    metadata = {
        "dataElements": [
            {
                "name": "ANC 1st visit",
                "id": "fbfJHSPpUQD",
                "publicAccess": "rw------",
                "userGroupAccesses": [
                    {
                        "access": "r-r-----",
                        "userGroupUid": "Rg8wusV7QYi",
                        "displayName": "HIV Program Coordinators",
                        "id": "Rg8wusV7QYi"
                    },
                    {
                        "access": "rwr-----",
                        "userGroupUid": "qMjBflJMOfB",
                        "displayName": "Family Planning Program",
                        "id": "qMjBflJMOfB"
                    }
                ]
            }
        ],
        "dataSets": [
            {
                "name": "ART monthly summary",
                "id": "lyLU2wR22tC",
                "publicAccess": "rwr-----",
                "userGroupAccesses": [
                    {
                        "access": "r-rw----",
                        "userGroupUid": "GogLpGmkL0g",
                        "displayName": "_DATASET_Child Health Program Manager",
                        "id": "GogLpGmkL0g"
                    }
                ]
            }
        ]
    }


    cleaned = clean_obj(metadata, ['userGroupAccesses', 'publicAccess'])
    pretty_json(cleaned)

Which would eventually recursively remove all keys matching to ``userGroupAccesses`` or ``publicAccess``:

.. code:: json

    {
      "dataElements": [
        {
          "name": "ANC 1st visit",
          "id": "fbfJHSPpUQD"
        }
      ],
      "dataSets": [
        {
          "name": "ART monthly summary",
          "id": "lyLU2wR22tC"
        }
      ]
    }


Print pretty JSON
^^^^^^^^^^^^^^^^^

Print easy-readable JSON objects with colors, utilizes `Pygments <http://pygments.org/>`_.

.. code:: python

    from dhis2 import pretty_json

    obj = {"dataElements": [{"name": "Accute Flaccid Paralysis (Deaths < 5 yrs)", "id": "FTRrcoaog83", "aggregationType": "SUM"}]}
    pretty_json(obj)

... prints (in a terminal it will have colors):

.. code:: json

    {
      "dataElements": [
        {
          "aggregationType": "SUM",
          "id": "FTRrcoaog83",
          "name": "Accute Flaccid Paralysis (Deaths < 5 yrs)"
        }
      ]
    }


Check import response
^^^^^^^^^^^^^^^^^^^^^^

Check the importSummary response from e.g. `/api/dataValues` or `/api/metadata` import. Returns true if import went well,
false if there are ignored values or the status reports not a OK or SUCCESS.
This can be useful if the response returns a 200 OK but the summary reports ignored data.

.. code:: python

    from dhis2 import import_response_ok

    # response as e.g. from response = api.post('metadata', data=payload).json()
    response = {
        "description": "The import process failed: java.lang.String cannot be cast to java.lang.Boolean",
        "importCount": {
            "deleted": 0,
            "ignored": 1,
            "imported": 0,
            "updated": 0
        },
        "responseType": "ImportSummary",
        "status": "WARNING"
    }

    import_successful = import_response_ok(response)
    print(import_successful)
    # False

Logging
-------

Logging utilizes `logzero <https://github.com/metachris/logzero>`_.

- Color output depending on log level
- DHIS2 log format including the line of the caller
- optional ``logfile=`` specifies a rotating log file path (20 x 10MB files)


.. code:: python

    from dhis2 import setup_logger, logger

    setup_logger(logfile='/var/log/app.log')

    logger.info('my log message')
    logger.warning('missing something')
    logger.error('something went wrong')
    logger.exception('with stacktrace')

::

    * INFO  2018-06-01 18:19:40,001  my log message [script:86]
    * ERROR  2018-06-01 18:19:40,007  something went wrong [script:87]

Use ``setup_logger(include_caller=False)`` if you want to remove ``[script:86]`` from logs.

Exceptions
----------

There are two exceptions:

- ``RequestException``: DHIS2 didn't like what you requested. See the exception's ``code``, ``url`` and ``description``.
- ``ClientException``: Something didn't work with the client not involving DHIS2.

They both inherit from ``Dhis2PyException``.


Examples
========

* Real-world script examples can be found in the ``examples`` folder.
* dhis2.py is used in `dhis2-pk <https://github.com/davidhuser/dhis2-pk>`_ (dhis2-pocket-knife)

Changelog
==========

Versions `changelog <https://github.com/davidhuser/dhis2.py/blob/master/CHANGELOG.rst>`_

Contribute
==========

Feedback welcome!

- Add `issue <https://github.com/davidhuser/dhis2.py/issues/new>`_
- Install the dev environment (see below)
- Fork, add changes to *master* branch, ensure tests pass with full coverage and add a Pull Request

.. code:: bash

    pip install pipenv
    git clone https://github.com/davidhuser/dhis2.py
    cd dhis2.py
    pipenv install --dev
    pipenv run tests

    # install pre-commit hooks
    pipenv run pre-commit install

    # run type annotation check
    pipenv run mypy dhis2

    # run flake8 style guide enforcement
    pipenv run flake8

License
=======

dhis2.py's source is provided under MIT license.
See LICENCE for details.

* Copyright (c), 2020, David Huser


.. |Latest version| image:: https://img.shields.io/pypi/v/dhis2.py.svg?label=PyPi
   :target: https://pypi.org/project/dhis2.py
   :alt: PyPi version
   
.. |Downloads| image:: https://static.pepy.tech/badge/dhis2-py/month
   :target: https://pepy.tech/project/dhis2.py
   :alt: Downloads

.. |Build| image:: https://img.shields.io/circleci/project/github/davidhuser/dhis2.py/master.svg?label=Linux%20build
   :target: https://circleci.com/gh/davidhuser/dhis2.py
   :alt: CircleCI build

.. |BuildWin| image:: https://img.shields.io/appveyor/ci/davidhuser/dhis2-py.svg?label=Windows%20build
   :target: https://ci.appveyor.com/project/davidhuser/dhis2-py
   :alt: Appveyor build

.. |Coverage| image:: https://img.shields.io/codecov/c/github/davidhuser/dhis2.py.svg?label=Coverage
   :target: https://codecov.io/gh/davidhuser/dhis2.py
   :alt: Test coverage

.. |LGTM| image:: https://img.shields.io/lgtm/grade/python/g/davidhuser/dhis2.py.svg?label=Code%20quality
   :target: https://lgtm.com/projects/g/davidhuser/dhis2.py
   :alt: Code quality

.. |CodeClimate| image:: https://img.shields.io/codeclimate/maintainability/davidhuser/dhis2.py.svg?label=Maintainability
   :target: https://codeclimate.com/github/davidhuser/dhis2.py/maintainability
   :alt: Code maintainability