destimator 0.1.4
A metadata-saving proxy for scikit-learn etimators.
License: MIT
Keywords: scikit-learn,machine-learning,data,science

==========
destimator
==========

destimator makes it easy to store trained ``scikit-learn`` estimators together with their metadata (training data, package versions, performance numbers etc.). This makes it much safer to store already-trained classifiers/regressors and allows for better reproducibility (see `this talk <https://www.youtube.com/watch?v=7KnfGDajDQw>`_ by `Alex Gaynor <https://alexgaynor.net/>`_ for some rationale).

Specifically, the ``DescribedEstimator`` class proxies most calls to the original ``Estimator`` it is wrapping, but also contains the following information:

* training and test (validation) data (``features_train``, ``labels_train``, ``features_test``, ``labels_test``)
* creation date (``created_at``)
* feature names (``feature_names``)
* performance numbers on the test set (``precision``, ``recall``, ``fscore``, ``support`` via `sklearn <http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html>`_)
* distribution info (``distribution_info``; python distribution and versions of all installed packages)
* VCS hash (``vcs_hash``, if used inside a git repository, otherwise and empty string).

An instantiated ``DescribedEstimator`` can be easily serialized using the ``.save()`` method and deserialized using either ``.from_file()`` or ``.from_url()``. Did you ever want to store your models in S3? Now it's easy!

``DescribedEstimator`` can be used as follows:

.. code-block:: python

  import numpy as np
  from sklearn.datasets import load_iris
  from sklearn.ensemble import RandomForestClassifier
  from sklearn.cross_validation import train_test_split
  from sklearn.metrics import precision_recall_fscore_support

  from destimator import DescribedEstimator


  # get some data
  iris = load_iris()
  features = iris.data
  labels = iris.target
  features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.1)

  # train an estimator as usual (in this case a RandomForestClassifier)
  clf = RandomForestClassifier(n_estimators=10, max_depth=None, min_samples_split=10, random_state=0)
  clf.fit(features_train, labels_train)

  # wrap the estimator in the DescribedEstimator class giving it all the training and test (validation) data
  dclf = DescribedEstimator(
      clf,
      features_train,
      labels_train,
      features_test,
      labels_test,
      iris.feature_names
  )

Now you can use the classifier as usual:

.. code-block:: python

  print(dclf.predict(features_test))
  > [2 1 2 2 0 1 0 2 2 1 2 0 2 1 2]

and you can also access a bunch of other properties, such as the training data you supplied:

.. code-block:: python

  print(dclf.feature_names)
  > ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

  print(dclf.features_test)
  > [[ 6.3  2.8  5.1  1.5]
     [ 5.6  3.   4.5  1.5]
     [ 6.7  3.1  5.6  2.4]
     [ 6.   2.7  5.1  1.6]
     [ 4.9  3.1  1.5  0.1]
     [ 6.2  2.2  4.5  1.5]
     [ 4.7  3.2  1.6  0.2]
     [ 6.9  3.1  5.1  2.3]
     [ 7.7  2.6  6.9  2.3]
     [ 5.8  2.6  4.   1.2]
     [ 7.2  3.   5.8  1.6]
     [ 5.4  3.7  1.5  0.2]
     [ 7.2  3.2  6.   1.8]
     [ 6.3  3.3  4.7  1.6]
     [ 6.8  3.2  5.9  2.3]]

  print(dclf.labels_test)
  > [2 1 2 1 0 1 0 2 2 1 2 0 2 1 2]

the performance numbers:

.. code-block:: python

  print('precision: %s' % (dclf.precision))
  > precision: [1.0, 1.0, 0.875]

  print('recall:    %s' % (dclf.recall))
  > recall:    [1.0, 0.8, 1.0]

  print('fscore:    %s' % (dclf.fscore))
  > fscore:    [1.0, 0.888888888888889, 0.9333333333333333]

  print('support:   %s' % (dclf.support))
  > support:   [3, 5, 7]

  print('roc_auc:   %s' % (dclf.roc_auc))
  > roc_auc:   0.5

or information about the Python distribution used for training:

.. code-block:: python

  from pprint import pprint
  pprint(dclf.distribution_info)

  > {'packages': ['appnope==0.1.0',
                  'decorator==4.0.4',
                  'destimator==0.0.0.dev3',
                  'gnureadline==6.3.3',
                  'ipykernel==4.2.1',
                  'ipython-genutils==0.1.0',
                  'ipython==4.0.1',
                  'ipywidgets==4.1.1',
                  'jinja2==2.8',
                  'jsonschema==2.5.1',
                  'jupyter-client==4.1.1',
                  'jupyter-console==4.0.3',
                  'jupyter-core==4.0.6',
                  'jupyter==1.0.0',
                  'markupsafe==0.23',
                  'mistune==0.7.1',
                  'nbconvert==4.1.0',
                  'nbformat==4.0.1',
                  'notebook==4.0.6',
                  'numpy==1.10.1',
                  'path.py==8.1.2',
                  'pexpect==4.0.1',
                  'pickleshare==0.5',
                  'pip==7.1.2',
                  'ptyprocess==0.5',
                  'pygments==2.0.2',
                  'pyzmq==15.1.0',
                  'qtconsole==4.1.1',
                  'requests==2.8.1',
                  'scikit-learn==0.17',
                  'scipy==0.16.1',
                  'setuptools==18.2',
                  'simplegeneric==0.8.1',
                  'terminado==0.5',
                  'tornado==4.3',
                  'traitlets==4.0.0',
                  'wheel==0.24.0'],
     'python': '3.5.0 (default, Sep 14 2015, 02:37:27) \n'
               '[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]'}

Finally, the object can be serialized to a `zip` file containing all the above data:

.. code-block:: python

    dclf.save('./classifiers', 'dclf')

and deserialized either from a file,

.. code-block:: python

    dclf = DescribedEstimator.from_file('./classifiers/dclf.zip')

or from a URL:

.. code-block:: python

    dclf = DescribedEstimator.from_url('http://localhost/dclf.zip')