semvec/pypi-packages huggingface dataset:
spark-optimizer 0.1.8
Optimize AWS EMR spark settings (spark-config-cheatsheet)
License: MIT
Keywords: spark emr aws cluster yarn

# Spark-optimizer

[![Build Status](https://api.travis-ci.org/delijati/spark-optimizer.svg?branch=master)](https://travis-ci.org/delijati/spark-optimizer)

Optimize spark settings (for cluster aka yarn run)

Original source: http://c2fo.io/c2fo/spark/aws/emr/2016/07/06/apache-spark-config-cheatsheet/

## Usage

Install:

    $ virtualenv env
    $ env/bin/pip install spark-optimizer

Dev install:

    $ virtualenv env
    $ env/bin/pip install -e .


Generate settings for `c4.4xlarge` with `4` nodes:

    $ env/bin/spark-optimizer c4.4xlarge 4
    Optimal numPartitions: 162 
    {'spark.default.parallelism': '108',
     'spark.driver.cores': '2',
     'spark.driver.maxResultSize': '3481m',
     'spark.driver.memory': '3481m',
     'spark.driver.memoryOverhead': '614m',
     'spark.executor.cores': '2',
     'spark.executor.instances': '27',
     'spark.executor.memory': '3481m',
     'spark.executor.memoryOverhead': '614m'}

Update instance info:

    $ env/bin/python spark_optimizer/emr_update.py


# CHANGES

0.1.8 (2020-02-14)
------------------

- add calculation for `numPartitions`
- dropping pypy and python3.4


0.1.7 (2019-12-09)
------------------

- set `long_description_content_type="text/markdown"`


0.1.6 (2019-12-09)
------------------

- fix docs


0.1.5 (2019-12-09)
------------------

- update ``emr_instance.yaml``


0.1.4 (2019-03-10)
------------------

- add ec2 and emr cost to yaml


0.1.3 (2019-03-08)
------------------

- add emr cost to yaml
- export load yaml file
- make ``memory_overhead_coefficient`` editable


0.1.2 (2019-02-20)
------------------

- unpin the versions
- rename cli from _ to -


0.1.1 (2018-09-12)
------------------

- fix email


0.1.0 (2018-09-12)

- initial release
xgbtune 1.1.0
Automated XGBoost tunning
License: MIT

==========
XGBTune
==========


.. image:: https://badge.fury.io/py/xgbtune.svg
    :target: https://badge.fury.io/py/xgbtune

.. image:: https://github.com/mainro/xgbtune/workflows/Python%20package/badge.svg
    :target: https://github.com/mainro/xgbtune/actions?query=workflow%3A%22Python+package%22
    :alt: Github WorkFlows

.. image:: https://readthedocs.org/projects/xgbtune/badge/?version=latest
    :target: https://xgbtune.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status


XGBTune is a library for automated XGBoost model tuning. Tuning an XGBoost
model is as simple as a single function call.

Get Started
============

.. code:: python

    from xgbtune import tune_xgb_model

    params, round_count = tune_xgb_model(params, x_train, y_train)


Install
========

XGBTune is available on PyPi and can be installed with pip:

.. code:: console

    pip install xgbtune


Tuning steps
=============

The tuning is done in the following steps:

* compute best round
* tune max_depth and min_child_weight
* tune gamma
* re-compute best round
* tune subsample and colsample_bytree
* fine tune subsample and colsample_bytree
* tune alpha and lambda
* tune seed

This steps can be repeated several times. By default, two passes are done.
mush 2.8.1
Type-based dependency injection for scripts.
License: MIT

Mush
====

|CircleCI|_ |Docs|_

.. |CircleCI| image:: https://circleci.com/gh/Simplistix/mush/tree/master.svg?style=shield
.. _CircleCI: https://circleci.com/gh/Simplistix/mush/tree/master

.. |Docs| image:: https://readthedocs.org/projects/mush/badge/?version=latest
.. _Docs: http://mush.readthedocs.org/en/latest/

Mush is a light weight dependency injection framework aimed at
enabling the easy testing and re-use of chunks of code that make up
scripts.

This is done by combining several callables into a re-usable
runner. Those callables may produce or require resource objects which
mush passes between them based on the type of the object. The
callables are called in the order they are added to the
runner, while labels may be used to insert callables at specific points 
in the runner.