semvec/pypi-packages huggingface dataset:
knit 0.2.4
Python wrapper for YARN Applications
License: BSD
Keywords: yarn

knit
====

|Build Status| |Coverage Status|

The ``knit`` library provides a Python interface to Scala for interacting
with the YARN resource manager.

View the documentation_ for ``knit``.

Overview
--------

``knit`` allows you to use python in conjunction with YARN, the most common resource
manager for Hadoop systems.
It provides to following high-level entry-points:

- ``CondaCreator``, a way to create zipped conda environments, so that they can be uploaded to
  HDFS and extracted for use in YARN containers
- ``YARNAPI``, an interface to the YARN resource manager to get application/container statuses,
  logs, and to kill running jobs
- ``Knit``, a YARN application runner, which generates an instance of a scala-based YARN client,
  and launches an application on YARN, which in turn runs commands in YARN containers
- ``DaskYARNCluster``, launches a Dask distributed cluster on YARN, one worker process
  per container.

The intent is to use ``knit`` from a cluster edge-node, i.e.,
with YARN configuration and the CLI available locally.

Quickstart
----------

Install from conda-forge

> conda install -c conda-forge knit

or with pip

> pip install knit

If installing from source, you must first build the java library (requires java and maven)

> python setup.py install mvn

To run an arbitrary command on the yarn cluster

.. code-block:: python

   import knit
   k = knit.Knit()
   k.start('env')  # wait some time
   k.logs()

To start a dask cluster on YARN

.. code-block:: python

   import dask_yarn
   cluster = dask_yarn.DaskYARNCluster()
   cluster.start(nworkers=4, memory=1024, cpus=2)


.. _documentation: http://knit.readthedocs.io/en/latest/


.. |Build Status| image:: https://travis-ci.org/dask/knit.svg?branch=master
   :target: https://travis-ci.org/dask/knit
.. |Coverage Status| image:: https://coveralls.io/repos/github/dask/knit/badge.svg
   :target: https://coveralls.io/github/dask/knit
dask-searchcv 0.2.0
Tools for doing hyperparameter search with Scikit-Learn and Dask
License: BSD

dask-searchcv
=============

|Travis Status| |Doc Status| |Conda Badge| |PyPI Badge|

Tools for performing hyperparameter search with
`Scikit-Learn <http://scikit-learn.org>`_ and `Dask <http://dask.pydata.org>`_.

Highlights
----------

- Drop-in replacement for Scikit-Learn's ``GridSearchCV`` and
  ``RandomizedSearchCV``.

- Hyperparameter optimization can be done in parallel using threads, processes,
  or distributed across a cluster.

- Works well with Dask collections. Dask arrays, dataframes, and delayed can be
  passed to ``fit``.

- Candidate estimators with identical parameters and inputs will only be fit
  once. For composite-estimators such as ``Pipeline`` this can be significantly
  more efficient as it can avoid expensive repeated computations.


For more information, check out the `documentation <http://dask-searchcv.readthedocs.io>`_.


Install
-------

Dask-searchcv is available via ``conda`` or ``pip``:

::

   # Install with conda
   $ conda install dask-searchcv -c conda-forge

   # Install with pip
   $ pip install dask-searchcv


Example
-------

.. code-block:: python

    from sklearn.datasets import load_digits
    from sklearn.svm import SVC
    import dask_searchcv as dcv
    import numpy as np

    digits = load_digits()

    param_space = {'C': np.logspace(-4, 4, 9),
                   'gamma': np.logspace(-4, 4, 9),
                   'class_weight': [None, 'balanced']}

    model = SVC(kernel='rbf')
    search = dcv.GridSearchCV(model, param_space, cv=3)

    search.fit(digits.data, digits.target)


.. |Travis Status| image:: https://travis-ci.org/dask/dask-searchcv.svg?branch=master
   :target: https://travis-ci.org/dask/dask-searchcv
.. |Doc Status| image:: http://readthedocs.org/projects/dask-searchcv/badge/?version=latest
   :target: http://dask-searchcv.readthedocs.io/en/latest/index.html
   :alt: Documentation Status
.. |PyPI Badge| image:: https://img.shields.io/pypi/v/dask-searchcv.svg
   :target: https://pypi.python.org/pypi/dask-searchcv
.. |Conda Badge| image:: https://anaconda.org/conda-forge/dask-searchcv/badges/version.svg
   :target: https://anaconda.org/conda-forge/dask-searchcv