semvec/pypi-packages huggingface dataset:
lilcom 1.8.2
Lossy-compression utility for sequence data in NumPy
License: MIT
Requires Python: >=3.6
Keywords: compression numpy

# lilcom


This package lossily compresses floating-point NumPy arrays
into byte strings, with an accuracy specified by the user.
The main anticipated use is in machine learning applications, for
storing things like training data and models.

This package requires Python 3 and is not compatible with Python 2.

## Installation with PyPi

From PyPi you can install this with just
```
pip3 install lilcom
```
## Installation with conda

```bash
conda install -c lilcom lilcom
```

### How to use

The most common usage pattern will be as follows (showing Python code):
```
import numpy as np
import lilcom

a = np.random.randn(300,500)
a_compressed = lilcom.compress(a)
# a_compressed is of type `bytes`, a byte string.
# In this case it will use about 1.3 bytes per element.

# decompress a
a_decompressed = lilcom.decompress(a_compressed)
```
The compression is lossy so `a_decompressed` will not be exactly the same
as `a`.  The amount of error (absolute, not relative!)  is determined by the
optional `tick_power` argument to lilcom.compress() (default: -8), which is the
power of 2 used for the step size between discretized values.  The maximum error
per element is 2**(tick_power-1), e.g.  for tick_power=-8, it is 1/512.



### Installation from Github

To install lilcom from github, first clone the repository;
```
git clone https://github.com/danpovey/lilcom.git
```
then run setup with `install` argument.
```
python3 setup.py install
```
(you may need to add the `--user` flag if you don't have system privileges).
You need to make sure a C++ compiler is installed, e.g. g++ or clang.
To test it, you can then cd to `test` and run:

```
python3 test_lilcom.py
```


## Technical details

The algorithm regresses each element on the previous element (for a 1-d array)
or, for general n-d arrays, it regresses on the previous elements along each of
the axes, i.e.  we regress element `a[i,j]` on `a[i-1,j]` and `a[i,j-1]`.  The
regression coefficients are global and written as part of the header in the
string.

The elements are then integerized and the integers are compressed using
an algorithm that gives good compression when successive elements tend to
have about the same magnitude (the number of bits we're transmitting
varies dynamically acccording to the magnitudes of the elements).

The core parts of the code are implemented in C++.
ezmpi 0.1.0
A simple MPI processing pool for Python
License: MIT
Requires Python: >=3.8
Keywords: mpi, parallel, processing, pool, hpc

# ezMPI

A simple MPI-based processing pool for Python that distributes tasks across multiple processes using MPI (Message Passing Interface).

## Features

- Simple API similar to Python's built-in `map()` function
- Automatic worker process management
- Support for complex objects via optional `dill` integration
- Context manager support for automatic cleanup
- Lightweight and easy to integrate into existing projects

## Requirements

- Python 3.8+
- MPI implementation (OpenMPI, MPICH, or similar)
- `mpi4py` library

## Installation

### Install from source

```bash
git clone https://github.com/minask/ezmpi.git
cd ezmpi
pip install -e .
```

### Install with development dependencies

```bash
pip install -e ".[dev]"
```

### Install with dill support

```bash
pip install -e ".[dill]"
```

## Quick Start

### Basic Usage

```python
from ezmpi import MPIPool

def square(x):
    return x * x

# Create a pool (must be run with mpiexec)
with MPIPool() as pool:
    results = pool.map(square, [1, 2, 3, 4, 5])
    print(results)  # [1, 4, 9, 16, 25]
```

Run with multiple processes:
```bash
mpiexec -n 4 python your_script.py
```

### Using dill for Complex Objects

```python
from ezmpi import MPIPool

def process_data(data):
    # Complex processing logic
    return sum(data) / len(data)

# Enable dill for better pickling support
with MPIPool(use_dill=True) as pool:
    results = pool.map(process_data, [[1, 2, 3], [4, 5, 6], [7, 8, 9]])
    print(results)  # [2.0, 5.0, 8.0]
```

## MPI Execution

ezMPI requires at least 2 MPI processes (1 master + 1+ workers):

```bash
# Minimum: 2 processes
mpiexec -n 2 python script.py

# Typical: 4-8 processes
mpiexec -n 4 python script.py

# On HPC systems
srun -n 32 python script.py
```

**Note**: The master process (rank 0) coordinates tasks, all other processes are workers.

## API Reference

### MPIPool

```python
class MPIPool(comm=None, use_dill=True)
```

**Parameters:**
- `comm`: MPI communicator (optional, defaults to MPI.COMM_WORLD)
- `use_dill`: Use dill for pickling instead of standard pickle (default: True)

**Methods:**
- `map(worker, tasks)`: Execute worker function on each task in parallel
- `close()`: Shutdown worker processes
- `is_master()`: Check if current process is the master
- `is_worker()`: Check if current process is a worker

## Development

### Setup Development Environment

```bash
git clone https://github.com/minask/ezmpi.git
cd ezmpi
pip install -e ".[dev]"
```

### Running Tests

```bash
# Run basic tests pytest -v

# Run MPI tests
mpiexec -n 4 pytest -v

# Run with coverage
pytest --cov=ezmpi --cov-report=html
```

### Code Quality

```bash
# Lint code
ruff check .

# Format code
ruff format .

# Type checking
mypy src/ezmpi

# Import sorting
isort src/
```

## License

MIT License - see LICENSE file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Acknowledgments

This implementation is inspired by and adapted from similar MPI pool implementations in the scientific Python community.