concordia 0.1.1
Automated monitoring of machine learning models in production. Tracks and finds discrepancies in features, predictions, and labels
License: MIT
Keywords: machine learning,data science,automated machine learning,deploying,machine learning in production,productionizing machine learning,tracking,feature discrepancies,train/serve skew,train serve skew,train-serve skew,model accuracy,alerts,monitoring,production ready,test coverage

Concordia
=========

|Build Status| |Coverage Status| |License|

Concordia is a part of a suite of open-source machine learning packages
that allow organizations to more rapidly develop and deploy machine
learning models.

Installation
------------

``pip install concordia``

Description
-----------

Concordia is a tracking and analytics tool for machine learning models
running in production.

Using Concordia, you should be able to rapidly have confidence in your
shipped ML models.

If everything's working as expected, you should have quick proof that
you're in a position to scale up this model.

If things are not going according to plan, you should be able to see
that rapidly, and have a suite of information to hone in on the root
cause of those discrepancies.

Basic Setup
-----------

::


    from concordia import Concordia
    concord = Concordia()

    ml_predictor = load_ml_model()
    concord.add_model(model=model, model_id='model123')

Basic Usage
-----------

In your training environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The goal here is to save the features and predictions as you calcate
them in your training environment. Then, we can compare these to the
features and predictions coming from your live environment. We use the
row\_ids to match rows across the two environments.

::


    df = load_my_data()
    ml_predictor = train_ml_model()

    predictions = ml_predictor.predict(df)

    concord.add_data_and_predictions(model_id='model123', features=df, predictions=predictions, row_ids=df['my_row_identifier'])

In your live environment
^^^^^^^^^^^^^^^^^^^^^^^^

::


    from concordia import load_concordia
    concord = load_concordia()

    data = get_live_data()

    prediction = concord.predict(model_id='model123', features=data, row_id=data['my_row_identifier'])

In your analytics environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::


    from Concordia import load_concordia
    concord = load_concordia()

    concord.analyze_prediction_discrepancies(model_id='model123')

    concord.analyze_feature_discrepancies(model_id='model123')

Infrastructure Assumptions
--------------------------

Concordia relies on MongoDB and Redis. These can be either local, or in
the cloud. You can specify DB credentials and connection options when
creating and loading Concordia.

Database Configuration
----------------------

You can easily specify your own DB connection. You'll need to do this
both when creating the Concordia instance in the first place, as well as
when you ``load_concordia()`` to get access to that same Concordia
instance later.

::


    persistent_db_config = {
        'db': '__concordia_test_env'
        , 'host': 'localhost'
        , 'port': 27017
    }

    in_memory_db_config = {
        'db': 8
        , 'host': 'localhost'
        , 'port': 6379
    }

    concord = Concordia(in_memory_db_config=in_memory_db_config, persistent_db_config=persistent_db_config)

    # To load this instance of Concordia later, use that same persistent_db_config
    concord = load_concordia(persistent_db_config=persistent_db_config)

What does Concordia do, under the hood?
---------------------------------------

-  It ensures a consistent db schema
-  It ensures a consistent way of saving data (predictions and
   features), so that data can be matched up and compared later
-  It builds in a suite of analytics tools for analyzing discrepancies
-  All of this happens automatically for each model that you ship
   |Analytics|

.. |Build Status| image:: https://travis-ci.org/ClimbsRocks/Concordia.svg?branch=master
   :target: https://travis-ci.org/ClimbsRocks/Concordia
.. |Coverage Status| image:: https://coveralls.io/repos/github/ClimbsRocks/Concordia/badge.svg?branch=master&cacheBuster=1
   :target: https://coveralls.io/github/ClimbsRocks/Concordia?branch=master
.. |License| image:: https://img.shields.io/github/license/mashape/apistatus.svg
   :target: (https://img.shields.io/github/license/mashape/apistatus.svg)
.. |Analytics| image:: https://ga-beacon.appspot.com/UA-58170643-5/concordia/pypi
   :target: https://github.com/igrigorik/ga-beacon