ms-graph-exporter 0.1.0rc4
A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.
License: MIT
Requires Python: >=3.6,<4.0
Keywords: development,monitoring,microsoft graph

# MsGraphExporter

[![Python 3.6+](https://img.shields.io/badge/Python-3.6+-blue.svg)][PythonRef] [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)][BlackRef] [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)][MITRef]
[![ReadTheDocs](https://readthedocs.org/projects/msgraphexporter/badge/?version=latest)][DocsRef] [![Build Status](https://dev.azure.com/undp/MsGraphExporter/_apis/build/status/MsGraphExporter_Github?branchName=develop)][BuildStatusRef]

[PythonRef]: https://docs.python.org/3.6/
[BlackRef]: https://github.com/ambv/black
[MITRef]: https://opensource.org/licenses/MIT
[DocsRef]: https://msgraphexporter.readthedocs.io/en/latest/
[BuildStatusRef]: https://dev.azure.com/undp/MsGraphExporter/_build/latest?definitionId=21&branchName=develop

`MsGraphExporter` is an application that performs periodic export of time-domain data like Azure AD user signins from [Microsoft Graph API][MsGraphApiDoc] into a buffer key-value store (currently supports [Redis][RedisRef]) for subsequent processing. It uses [Celery][CeleryProjectRef] task queue for parallel processing, [gevent][GeventRef] greenlets for concurrent uploads, relies on the [Graph API pagination][MsGraphApiPage] to control memory footprint and respects [Graph API throttling][MsGraphApiThrottle]. The application could be deployed as a single-container worker or as a set of multiple queue-specific workers for high reliability and throughput.

[MsGraphApiDoc]: https://docs.microsoft.com/en-us/graph/overview
[MsGraphApiPage]: https://docs.microsoft.com/en-us/graph/paging
[MsGraphApiThrottle]: https://docs.microsoft.com/en-us/graph/throttling

## Getting Started

Follow these instructions to use the application.

### Installing

`MsGraphExporter` is distributed through the [Python Package Index][PyPIRef] as [ms-graph-exporter][PyPIProjRef]. Run the following command to:

[PyPIRef]: https://pypi.org
[PyPIProjRef]:https://pypi.org/project/ms-graph-exporter/

* install a specific version

    ```sh
    pip install "ms-graph-exporter==0.1"
    ```

* install the latest version

    ```sh
    pip install "ms-graph-exporter"
    ```

* upgrade to the latest version

    ```sh
    pip install --upgrade "ms-graph-exporter"
    ```

* install optional DEV dependencies like `pytest` framework and plugins necessary for performance and functional testing

    ```sh
    pip install "ms-graph-exporter[test]"
    ```

`MsGraphExporter` is also distributed as the Alpine-based docker container and published on [Docker Hub][DockerHubRef] as [undp/ms-graph-exporter][DockerHubProjRef].

[DockerHubRef]: https://hub.docker.com/
[DockerHubProjRef]: https://hub.docker.com/r/undp/ms-graph-exporter

`Service Principal` credentials with correct access rights to `Graph API` must be defined for `docker/docker-compose.yaml`example deployment to work. Following `.env` file should be placed at the root of the project folder, or following ENV variables defined in other way:

```sh
GRAPH_CLIENT_ID=<clientId>
GRAPH_CLIENT_SECRET=<clientSecret>
GRAPH_TENANT=<tenant>
```

Then, docker deployment could be initiated with:

```sh
docker-compose --file docker/docker-compose.yaml up
```

### Requirements

* Python >= 3.6

## Built using

* [Celery][CeleryProjectRef] - Distributed task queue
* [gevent][GeventRef] - concurrent data upload to [Redis][RedisRef]
* [redis-py][RedisPyGithub] - Python interface to [Redis][RedisRef]

[RedisRef]: https://redis.io/
[CeleryProjectRef]:http://www.celeryproject.org/
[GeventRef]: http://www.gevent.org
[RedisPyGithub]: https://github.com/andymccurdy/redis-py

## Versioning

We use [Semantic Versioning Specification][SemVer] as a version numbering convention.

[SemVer]: http://semver.org/

## Release History

For the available versions, see the [tags on this repository][RepoTags]. Specific changes for each version are documented in [CHANGELOG.md][ChangelogRef].

Also, conventions for `git commit` messages are documented in [CONTRIBUTING.md][ContribRef].

[RepoTags]: https://github.com/undp/MsGraphExporter/tags
[ChangelogRef]: CHANGELOG.md
[ContribRef]: CONTRIBUTING.md

## Authors

* **Oleksiy Kuzmenko** - [OK-UNDP@GitHub][OK-UNDP@GitHub] - *Initial design and implementation*

[OK-UNDP@GitHub]: https://github.com/OK-UNDP

## Acknowledgments

* Hat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.

## License

Unless otherwise stated, all authors (see commit logs) release their work under the [MIT License][MITRef]. See [LICENSE.md][LicenseRef] for details.

[LicenseRef]: LICENSE.md

## Contributing

There are plenty of ways you could contribute to this project. Feel free to:

* submit bug reports and feature requests
* outline, fix and expand documentation
* peer-review bug reports and pull requests
* implement new features or fix bugs

See [CONTRIBUTING.md][ContribRef] for details on code formatting, linting and testing frameworks used by this project.