semvec/pypi-packages huggingface dataset:
alexflow 1.1.4
ALEXFlow is a python workflow library built for reproducible complex workflow
License: MIT
Requires Python: >=3.7

# alexflow
ALEXFlow is a python workflow library built for reproducible complex workflow, mainly for machine learning training.


## Get Started

For the installation from pypi, simply install via pip.

`pip install alexflow` 


## Remarks

##### Support of type hints with `dataclasses`

luigi does not work well with type hints, which makes it difficult to build workflow when it is complex. With use of dataclasses, we'd like to gain benefit of type hints.

##### Build workflow by composition, rather than parameter bucket relies.

Parameter bucket rely finally build a huge global state at the entrypoint of workflow, which is pretty difficult to maintain in general as it is works similarly with global variables... Instead, we've decided to compose workflow with compositions. With this architecture we can gain the benefit of divide and conquer strategy.

##### Focus of reproducibility with immutability tasks

Task class is designed to be a immutable dataclass object, for distributed execution, strong consistency, and reproducibility. And also those `Task` objects can be serialized as json object, and you can easily trace the exact parameters used to generate the `Output`. 

##### Dependency via Outputs, rather than Tasks

Description of workflow dependency by `Output` makes it easy to run partially graph.

## A exmaple of Task construction

Also you can see the example workflow at `examples/workflow.py`.

```python
from typing import Tuple
from sklearn import linear_model
from dataclasses import dataclass, field
from alexflow import Task, no_default, NoDefaultVar, Output, BinaryOutput


@dataclass(frozen=True)
class Train(Task):
    # Here you can write parameter of task as dataclass fields. Task's unique id will be 
    # generated from given parameters' and each task is executed at once while the entire
    # graph computation.
    X: NoDefaultVar[Output] = no_default
    y: NoDefaultVar[Output] = no_default
    model_type: NoDefaultVar[str] = no_default
    # Here you can describe in-significant parameter with compare=False, with following
    # dataclass' object equality. Even you changed those variables, Task's unique id is
    # consistent.
    verbose: bool = field(default=True, compare=False)

    def input(self):
        """Here describes the dependent output of your task"""
        return self.X, self.y

    def output(self):
        """Here describes the dependent output of your task"""
        return self.build_output(BinaryOutput, key="model.pkl")

    def run(self, input: Tuple[BinaryOutput, BinaryOutput], output: BinaryOutput):
        # Dependent output you defined in `input()` method is available as input variable.
        X = input[0].load()
        y = input[1].load()

        model_class = getattr(linear_model, self.model_type)

        cls = model_class().fit(X, y)

        # And you can store what you want to output in following manner.
        output.store(cls)
```
slurmpter 1.0.0
A package to build Slurm submit files of a workflow of jobs easily.
License: MIT
Requires Python: >=3.5

|pipeline| |coverage| |pypi| |docs| |license|

.. image:: docs/source/_static/logo.svg

========================================

Slurmpter (Slurm Scripter) is a package to build Slurm submit files of a workflow of jobs easily. This package uses PyCondor_ as the backend. The user interface of slurmpter is very similar to that of PyCondor_ except for some arguments dedicated for Slurm.

Documentation
=============

Documentation of ``slurmpter`` can be found at https://slurmpter.readthedocs.io/en/latest/index.html.

Installation
============

PyPI
----

The latest release of ``slurmpter`` can be installed with ``pip``:

.. code-block:: bash

    pip install slurmpter

Conda
-----

The latest release of ``slurmpter`` can be installed with ``conda``:

.. code-block:: bash

    conda install -c isaac-cfwong slurmpter

Useful Links
============

`slurmpter @ GitLab <https://gitlab.com/isaac-cfwong/slurmpter>`_

`slurmpter mirror @ GitHub <https://github.com/isaac-cfwong/slurmpter>`_

`Issue tracker <https://gitlab.com/isaac-cfwong/slurmpter/-/issues>`_

Copyright (c) 2020 Isaac Chun Fung WONG

.. _PyCondor: https://github.com/jrbourbeau/pycondor

.. |pipeline| image:: https://gitlab.com/isaac-cfwong/slurmpter/badges/master/pipeline.svg
    :target: https://gitlab.com/isaac-cfwong/slurmpter/commits/master

.. |coverage| image:: https://gitlab.com/isaac-cfwong/slurmpter/badges/master/coverage.svg
    :target: https://codecov.io/gl/isaac-cfwong/slurmpter/

.. |pypi| image:: https://badge.fury.io/py/slurmpter.svg
    :target: https://pypi.org/project/slurmpter/
    :alt: Package on PyPI

.. |docs| image:: https://readthedocs.org/projects/sphinx/badge/?version=master
    :target: https://slurmpter.readthedocs.io/en/latest/

.. |license| image:: https://img.shields.io/badge/License-MIT-blue.svg
    :target: https://gitlab.com/isaac-cfwong/slurmpter/-/blob/master/LICENSE