metadata-client 4.1.2
Python Client for European XFEL Metadata Catalogue Web App available at https://in.xfel.eu/metadata
License: MIT
Requires Python: >=3.7

Metadata Catalogue Client
=========================

MyMdC is the Web App design for Data Management at European XFEL.

This library (metadata_client) is a client for the RESTful APIs exposed
by the European XFEL Metadata Catalogue Web Application - myMdC
(https://in.xfel.eu/metadata).

*Repository:*

- https://git.xfel.eu/ITDM/metadata_client

*Dependencies:*

- oauthlib (https://pypi.python.org/pypi/oauthlib)
- requests (https://github.com/psf/requests)
- requests-oauthlib (https://github.com/requests/requests-oauthlib)
- oauth2_xfel_client (https://git.xfel.eu/ITDM/oauth2_xfel_client)
- pytz (https://pypi.org/project/pytz/)

Installation
------------

Python project
""""""""""""""

1. Install requirements, if never done before

 1.1. For OS X distributions::

  1.1.1. Homebrew

        brew install python3

  1.1.2 Port

        sudo port install python36

        sudo port select --set python3 python36

        sudo port install py36-pip
        sudo port select --set pip pip36

 1.2. For Linux distributions::

    sudo apt-get update
    sudo apt-get install python3.9


2. Make metadata_client library available in your python environment

 2.1. Install it via pip::

    # Use venv
     python3.10 -m venv .venv
     source .venv/bin/activate

    # Install dependencies from the pypi
    pip install .

    # Force re-installation of packages
    pip install . --ignore-installed

    # Install dependencies from local wheels files is possible,
    # if manually generated:
    # ```pip wheel --wheel-dir=./external_dependencies .```
    pip install . --no-index --find-links ./external_dependencies/

 Installing it will place two folders under the current Python installation
 site-packages folder:

 - `metadata_client` with the sources;
 - `metadata_client-4.1.2.dist-info/` with Wheels configuration files.

 To identify your Python site-packages folder run::

    python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"


Usage
-----

To use this project you need to import it::

    from metadata_client import MetadataClient


1. Connection to the MdC (Metadata Catalog)::

    from metadata_client import MetadataClient

    # Necessary configuration variables to establish a connection
    # Go to https://in.xfel.eu/metadata/oauth/applications to make a token for
    # the metadata catalogue.
    user_id = '201ed15ff071a63e76cb0b91a1ab17b36d5f92d24b6df4497aa646e39c46a324'
    user_secret = 'a8ae80f5e96531f19bf2d2b6102f5a537196aca44a673ad36533310e07529757'
    user_email = 'luis.maia@xfel.eu'
    #
    metadata_web_app_url = 'https://in.xfel.eu/metadata'
    token_url = 'https://in.xfel.eu/metadata/oauth/token'
    refresh_url = 'https://in.xfel.eu/metadata/oauth/token'
    auth_url = 'https://in.xfel.eu/metadata/oauth/authorize'
    scope = ''
    base_api_url = 'https://in.xfel.eu/metadata/api/'

    # Generate the connection (example with minimum parameter options)
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url)

    # Generate the connection (example with all parameter options)
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url,
                                 session_token=None,
                                 max_retries=3,
                                 timeout=12,
                                 ssl_verify=True)


2. Interaction with the MyMdC (Metadata Catalog):

 2.1 Example data_group_types::

    all_group_types = client_conn.get_all_data_group_types()

    all_group_types
    # >>> {'success': True,
    #      'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
    #      'data': [{'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1},
    #               {'description': '', 'identifier': 'CAL', 'name': 'Calibration', 'flg_available': True, 'id': 2},
    #               {'description': '', 'identifier': 'PROC', 'name': 'Processed', 'flg_available': True, 'id': 3},
    #               {'description': '', 'identifier': 'REDU', 'name': 'Reduced', 'flg_available': True, 'id': 4},
    #               {'description': '', 'identifier': 'SIM', 'name': 'Simulation', 'flg_available': True, 'id': 5},
    #               {'description': '', 'identifier': 'UNK', 'name': 'Unknown', 'flg_available': True, 'id': 6}],
    #      'app_info': {},
    #      'info': 'Got data_group_type successfully'}

    all_group_types['success']
    # >>> True

    all_group_types['pagination']
    # >>> {'Date': 'Wed, 11 May 2022 09:55:34 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'}

    all_group_types['data'][0]
    # >>> {'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1}

    all_group_types['data'][0]['name']
    # >>> 'Raw'

 2.2 Example instruments::

    all_xfel_instruments = client_conn.get_all_xfel_instruments()

    >>> for instrument in all_xfel_instruments['data']:
    ...   print('id = {0} | name = {1}'.format(instrument['id'], instrument['name']))
    ...
    # id = -1 | name = test-instrument
    # id = 1 | name = SPB/SFX SASE1
    # id = 2 | name = FXE SASE1
    # id = 3 | name = SQS SASE3
    # id = 4 | name = SCS SASE3
    # id = 5 | name = MID SASE2
    # id = 6 | name = HED SASE2
    # id = 7 | name = Hera South Detector Test Stand
    # id = 8 | name = SASE1 Test Stand
    # id = 9 | name = SASE2 Test Stand
    # id = 10 | name = SASE3 Test Stand

    all_xfel_instruments = client_conn.get_all_xfel_instruments(page=1, page_size=1)
    all_xfel_instruments

    # >>> {'success': True,
    #      'info': 'Got instrument successfully',
    #      'app_info': {},
    #      'pagination': {'Date': 'Wed, 11 May 2022 09:57:45 GMT', 'X-Total-Pages': '21', 'X-Count-Per-Page': '1', 'X-Current-Page': '1', 'X-Total-Count': '21'},
    #      'data': [{'id': 1, 'name': 'SPB/SFX SASE1', 'identifier': 'SPB', 'url': 'https://www.xfel.eu/facility/instruments/spb_sfx', 'instrument_leader_id': 230, 'deputy_instrument_leader_id': 1018, 'facility_id': 1, 'instrument_type_id': 2, 'repository_id': 103, 'topic_id': 1, 'dsg_host': None, 'system_user': None, 'flg_online_resource': True, 'online_script': 'make_online', 'flg_available': True, 'description': 'The Single Particles, Clusters, and Biomolecules & Serial Femtosecond Crystallography (SPB/SFX) instrument of the European XFEL is primarily concerned with three-dimensional diffractive imaging, and three-dimensional structure determination, of micrometre-scale and smaller objects, at atomic or near-atomic¿resolution.', 'doi': None, 'techniques': [{'id': 250, 'identifier': 'PaNET01168', 'name': 'serial femtosecond crystallography', 'url': 'http://purl.org/pan-science/PaNET/PaNET01168', 'flg_available': True, 'description': None}, {'id': 259, 'identifier': 'PaNET01188', 'name': 'small angle x-ray scattering', 'url': 'http://purl.org/pan-science/PaNET/PaNET01188', 'flg_available': True, 'description': None}, {'id': 364, 'identifier': 'PaNET01101', 'name': 'x-ray powder diffraction', 'url': 'http://purl.org/pan-science/PaNET/PaNET01101', 'flg_available': True, 'description': None}, {'id': 28, 'identifier': 'PaNET01174', 'name': 'coherent diffraction imaging', 'url': 'http://purl.org/pan-science/PaNET/PaNET01174', 'flg_available': True, 'description': None}]}]}

 2.3 Get instrument active proposal::

    active_proposal = client_conn.get_active_proposal_by_instrument(1)

 2.4 Register Run replica::

    # (e.g. proposal_number == 1234)
    # (e.g. proposal_number == 12)
    # (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

    resp = client_conn.register_run_replica(
        proposal_number, run_number, repository_identifier
    )
    # resp = {'success': True,
    #         'info': 'Run replica registered successfully',
    #         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
    #         'data': {'experiment_id': '-1',
    #                  'sample_id': '-1',
    #                  'run_id': '1588',
    #                  'data_group_id': '777'},
    #         'app_info': {}}

 2.5 Unregister Run replica::

    # (e.g. proposal_number == 1234)
    # (e.g. proposal_number == 12)
    # (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

    resp = client_conn.unregister_run_replica(
        proposal_number, run_number, repository_identifier
    )
    # resp = {'success': True,
    #         'info': 'Run replica unregistered successfully',
    #         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
    #         'data': {'data_group_id': '-1',
    #                  'repository_id': '1',
    #                  'flg_available': 'false'},
    #         'app_info': {}}

 2.6 Get proposal's runs::

    # (e.g. proposal_number == 1234)
    # (e.g. page == 1 | Default == 1)
    # (e.g. page_size == 5 | Default == 100 | Limit: 500)

    resp = client_conn.get_proposal_runs(proposal_number, page=1, page_size=5)
    # RESPONSE example
    #
    # resp = {'info': 'Got proposal successfully',
    #         'success': True,
    #         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT',
    #                        'X-Total-Pages': '1',
    #                        'X-Count-Per-Page': '100',
    #                        'X-Current-Page': '1',
    #                        'X-Total-Count': '6'},
    #         'data': {
    #           'proposal': {
    #               'id': -1,
    #               'number': 0,
    #               'title': 'Proposal Title 001'
    #                  },
    #           'runs': [
    #               {
    #               'id': -1,
    #               'run_number': 1,
    #               'flg_status': 1,
    #               'flg_run_quality': -1,
    #               'size': null,
    #               'num_files': 0,
    #               'repositories': {
    #                   'XFEL_TESTS_REPO': {
    #                       'name": 'XFEL Tests Repository',
    #                       'mount_point': '/webstorage/XFEL',
    #                       'data_groups': 1
    #                       }
    #                   }
    #               }
    #            ]
    #          },
    #         'app_info': {}}

2.7 Get proposal's samples::

    # (e.g. proposal_number == 1234)
    # (e.g. page == 1 | Default == 1)
    # (e.g. page_size == 50 | Default == 100 | Limit: 500)

    resp = client_conn.get_proposal_samples(proposal_number, page=1, page_size=50)
    #
    # RESPONSE example
    #
    # resp = {'info': 'Got sample successfully',
    #         'success': True,
    #         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT',
    #                        'X-Total-Pages': '1',
    #                        'X-Count-Per-Page': '100',
    #                        'X-Current-Page': '1',
    #                        'X-Total-Count': '6'},
    #         'data': [{'id': -1,
    #                   'name': 'TestSample DO NOT DELETE!',
    #                   'proposal_id': -1,
    #                   'sample_type_id': 1,
    #                   'flg_available': True,
    #                   'url': '',
    #                   'description': ''}],
    #         'app_info': {}}

For additional examples, please take a look in the tests/ folder.


Development & Testing
---------------------

When developing, and before commit changes, please validate that:

1. All tests continue passing successfully (to validate that run *pytest*)::

    # Go to the source code directory
    cd metadata_client

    # Use venv
    python3.10 -m venv .venv
    source .venv/bin/activate

    # Upgrade package and all its required packages
    pip install . -U --upgrade-strategy eager

    # Install test dependencies
    pip install '.[test]' -U --upgrade-strategy eager

    # Run all tests using pytest
    pytest metadata_client/tests

    # When running all tests against the standard http application
    OAUTHLIB_INSECURE_TRANSPORT=1 pytest metadata_client/tests

    # Run all tests and get information about coverage for all files inside metadata_client package
    pytest --cov metadata_client/tests --cov-report term-missing

2. Code keeps respecting pycodestyle code conventions (to validate that run **pycodestyle**)::

    pycodestyle .
    pycodestyle . --exclude venv

3. To generate all the wheels files for the dependencies, execute::

    # Generate Wheels to itself and dependencies
    pip wheel --wheel-dir=./external_dependencies .
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .

4. Check that you have the desired dependency versions in ``external_dependencies`` folder, since no versions are now set in ``setup.py``.


Registering library on https://pypi.org
---------------------------------------

To register this python library, the following steps are necessary::

    # Install twine
    python -m pip install --upgrade twine

    # Generates source distribution (.tar.gz) and wheel (.whl) files in the dist/ folder
    python setup.py sdist
    python setup.py bdist_wheel

    # Upload new version .egg and .whl files
    twine upload dist/*

    # In case a test is necessary, it is possible to test it against test.pypi.org
    twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose