semvec/pypi-packages huggingface dataset:
ml-pipeline-orchestrator 0.1.2
A lightweight ML pipeline orchestrator
License: MIT
Requires Python: >=3.8

# ML Pipeline Orchestrator

A lightweight, ML pipeline orchestrator that transforms your individual ML scripts into reliable, automated workflows. Built specifically for robust Machine Learning pipeline management without the complexity of cloud systems.

## ✨ Key Features

- **🔄 Smart Dependency Management** - Automatic task ordering with DAG validation
- **💾 State Persistence** - Resume interrupted pipelines from where they left off
- **🔁 Robust Error Handling** - Configurable retry logic and graceful failure recovery
- **⚙️ CLI & Python API** - Use from command line or integrate into your code
- **📊 Real ML Examples** - MNIST computer vision pipeline with PyTorch included
- **🎯 Zero Configuration** - Works with your existing Python scripts

## 🚀 Quick Start

### Installation

```bash
pip install ml-pipeline-orchestrator
```

### Basic Usage

1. **Create a pipeline config** (`my_pipeline.yml`):

```yaml
name: "ml_training_pipeline"
tasks:
  - name: "fetch_data"
    command: "python scripts/fetch_data.py --output data/"
    
  - name: "train_model"
    command: "python scripts/train.py --data data/ --model model.pkl"
    depends_on: ["fetch_data"]
    retry_count: 3
    
  - name: "evaluate"
    command: "python scripts/evaluate.py --model model.pkl"
    depends_on: ["train_model"]
```

2. **Run your pipeline**:

```bash
# Command line
ml-orchestrator run my_pipeline.yml

# Python API
from ml_orchestrator import Pipeline
pipeline = Pipeline.from_config('my_pipeline.yml')
pipeline.run()
```

3. **Monitor and manage**:

```bash
ml-orchestrator status ml_training_pipeline
ml-orchestrator resume ml_training_pipeline  # Resume if failed
ml-orchestrator list  # Show all pipelines
```

## 📋 Configuration Reference

### Task Options

| Option | Description | Example |
|--------|-------------|---------|
| `name` | Unique task identifier | `"preprocess_data"` |
| `command` | Shell command to execute | `"python train.py"` |
| `depends_on` | Task dependencies | `["fetch_data", "validate"]` |
| `retry_count` | Number of retries on failure | `3` |
| `timeout` | Max execution time (seconds) | `3600` |
| `working_dir` | Execution directory | `"/path/to/project"` |
| `environment` | Environment variables | `{"GPU": "0"}` |

### Pipeline Options

```yaml
name: "my_pipeline"
description: "ML training pipeline"
version: "1.0"
author: "Data Team"
tags: ["training", "production"]

tasks:
  # ... task definitions
```

## 🎓 Use Cases

- **Model Training Pipelines** - Automate data → preprocess → train → evaluate workflows
- **Data Processing** - Multi-step ETL pipelines with dependency management
- **Model Deployment** - Training → validation → deployment → monitoring chains
- **Batch Inference** - Scheduled prediction pipelines
- **Hyperparameter Tuning** - Coordinate multiple training experiments
- **A/B Testing** - Manage parallel model training and comparison

## 📜 License

MIT License - feel free to use in commercial and open-source projects.

## 🔗 Links

- **PyPI**: [ml-pipeline-orchestrator](https://pypi.org/project/ml-pipeline-orchestrator/)

---
arxify 1.3.2
Pack a latex project into an arXiv compatible archive.
License: MIT
Requires Python: >=3.10

# arxify - arXiv Packaging Tool
`arxify` is a command line tool that packages latex projects into an arXiv-compatible archives.
It removes comments from you *.tex files and removes any files that are not needed during the compile process.
The latter is achieved by compiling the project once and observing which files are being accessed by the compiler using [watchdog](https://github.com/gorakhargosh/watchdog).

## Installation
Install `arxify` via
```bash
pip install arxify
```

## Usage
Call with your latex main file and a target filename for the resulting zip archive:
```bash
arxify /path/to/root/main.tex /path/to/output.zip
```

By default `arxify` will assume that the root directory of your project is the parent directory of your main *.tex file.
Should that not be the case, specify the root directory with the `-r` option:
```bash
arxify /path/to/root/subdir/main.tex /path/to/output.zip -r /path/to/root/
```

By default `arxify` will attempt to compile your project using `pdflatex` and `bibtex`.
Currently supported are also `lualatex` and `biber`, which can be selected via
```bash
arxify /path/to/root/main.tex /path/to/output.zip -c lualatex -b biber
```

If you wish to include files into the archive that are not used by the compiler, specify them with the `-i` option:
```bash
arxify /path/to/root/main.tex /path/to/output.zip -i /path/to/root/a_file.txt /path/to/root/another_file.txt
```