semvec/pypi-packages huggingface dataset:
data-laundry 0.1
A small python package to clean dirty data
Requires Python: >=3.8

# data_laundry

Your one stop solution for cleaning dirty data in pandas dataframes.

# Features 

1. Automatic column renaming to snake_case 
2. Read from csv, xlsx and parquet files into pandas dataframe
3. Selecting subset of columns from the function call itself
4. Uses `dtype_backend='pyarrow'`
5. Convert dataframe to SQL Server compatible code. This is useful if you don't have `LOAD FILE` privileges on the database

# Usage 

```python
import data_laundry as dl

df = dl.launder("/path/to/file", select_columns="columns to select", **default_pandas_kwargs)
df = dl.launder(df, select_columns="columns to select", **default_pandas_kwargs)

dl.convert_to_sql(
    df,
    table_name="name of sql table",
    output_path="file path where sql code will be written",
    step=1000
)
```

# Installation

```python
pip install data_laundry
```
lzdata 0.0.1
Lazy data validation and transformation.
Requires Python: <4.0,>=3.10
Keywords: python

# lzdata

[![Release](https://img.shields.io/github/v/release/lzkit/lzdata)](https://img.shields.io/github/v/release/lzkit/lzdata)
[![Build status](https://img.shields.io/github/actions/workflow/status/lzkit/lzdata/main.yml?branch=main)](https://github.com/lzkit/lzdata/actions/workflows/main.yml?query=branch%3Amain)
[![codecov](https://codecov.io/gh/lzkit/lzdata/branch/main/graph/badge.svg)](https://codecov.io/gh/lzkit/lzdata)
[![Commit activity](https://img.shields.io/github/commit-activity/m/lzkit/lzdata)](https://img.shields.io/github/commit-activity/m/lzkit/lzdata)
[![License](https://img.shields.io/github/license/lzkit/lzdata)](https://img.shields.io/github/license/lzkit/lzdata)

Lazy data validation and transformation.

- **Github repository**: <https://github.com/lzkit/lzdata/>
- **Documentation** <https://lzkit.com/lzdata/>
jp2-remediator 1.1.0
A module to validate jp2images
Requires Python: >=3.10

# jp2_remediator

<!-- test and coverage badge from https://github.com/harvard-lts/etd-base-template/wiki --><a href="https://github.com/harvard-lts/jp2_remediator/actions/workflows/test.yml"><img src="https://github.com/harvard-lts/jp2_remediator/actions/workflows/test.yml/badge.svg"></a> <a href="https://github.com/harvard-lts/jp2_remediator/actions/workflows/test.yml"><img src="https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/kimpham54/67d4eba1556653d896d2d36fcb3e5c7c/raw/covbadge.json"></a>
<!-- pypi badge from https://badge.fury.io/for/py/jp2_remediator --><a href="https://pypi.org/project/jp2_remediator"><img src="https://badge.fury.io/py/jp2-remediator.svg" alt="PyPI version" height="18"></a>

A Python module to validate jp2 images:
- reads the bytes of a jp2 file or directory of files or S3 bucket
- looks for presence of ICC profile
- maps and validates curv values according to the ISO/IEC 15444-1:2019 (E) and ICC.1:2022 Specifications
- corrects values where the rTRC/gTRC/bTRC tag size in the Tag table structure (ICC.1:2022 Table 24 tag size) is not the same as the count value (gamma, where n == 1) in the curveType encoding (ICC.1:2022 Table 35, n value)
- flag and (optionally) log values where the count value n != 1 for future review

<a href="https://pypi.org/project/jp2_remediator/">jp2_remediator on PyPI</a>

## Installation

```bash
pip install jp2_remediator
python3 -m pip install jp2_remediator

pip install jp2_remediator==NN.NN.NN (version number)
```

## Usage

```bash
python3 src/jp2_remediator/main.py  -h

usage: main.py [-h] {file,directory,bucket} ...

JP2 file processor

options:
  -h, --help            show this help message and exit

Input source:
  {file,directory,bucket}
    file                Process a single JP2 file
    directory           Process all JP2 files in a directory
    bucket              Process all JP2 files in an S3 bucket
```

### Process one file
```bash
python3 src/jp2_remediator/main.py file tests/test-images/7514499.jp2

python3 src/jp2_remediator/main.py file tests/test-images/481014278.jp2
```

### Process directory
```bash
python3 src/jp2_remediator/main.py directory tests/test-images/
```

### Process all .jp2 files in an S3 bucket:
```bash
python3 src/jp2_remediator/main.py bucket remediation-folder
```

### Process only files with a specific prefix (folder):
```bash
python3 src/jp2_remediator/main.py bucket remediation-folder --prefix testbatch_20240923`
```

## Run tests

### Run integration tests
```bash
pytest src/jp2_remediator/tests/integration/
```

### Run unit tests
```bash
pytest src/jp2_remediator/tests/unit/
```

## Docker environment

Build Docker image
```bash
./bin/docker-build.sh
```

Run Docker container as executable
```bash
./bin/docker-run.sh -h
```

## Development environment
```bash
python3 -m venv myenv
source myenv/bin/activate
export PYTHONPATH="${PYTHONPATH}:src"
pip install -r requirements.txt

python src/jp2_remediator/main.py -h
```
leadindicator 0.1.1
A powerful tool for analyzing leading indicators and setting optimal thresholds
Requires Python: >=3.8

# LeadIndicator

A Python package for analyzing leading indicators and finding optimal thresholds.

## Features

- Analyze time series data to identify leading indicators
- Find optimal thresholds using multiple metrics
- Generate comprehensive analysis reports
- Support for categorical analysis
- Beautiful visualizations
- Polars-based for high performance

## Installation

```bash
pip install leadindicator
```

## Quick Start

```python
import polars as pl
from leadindicator import ThresholdAnalyzer

# Load your data
df = pl.read_excel("your_data.xlsx")

# Create analyzer
analyzer = ThresholdAnalyzer(
    score_column="score",
    target_column="event",
    category_column="category"  # Optional
)

# Run analysis
results = analyzer.analyze(df)

# Generate report
analyzer.save_report("analysis_report.html")
```

## Example: Fruit Spoilage Analysis

The package includes an example that demonstrates how to use LeadIndicator to analyze fruit spoilage data:

```python
from leadindicator import ThresholdAnalyzer
import polars as pl

# Load example data
df = pl.read_excel("Test Data.xlsx")

# Create analyzer
analyzer = ThresholdAnalyzer(
    score_column="Ethylene",
    target_column="Spoiled",
    category_column="Fruit Type"
)

# Run analysis
results = analyzer.analyze(df)

# Save report
analyzer.save_report("fruit_spoilage_analysis.html")

# Access results programmatically
best_threshold = results[0]
print(f"Best threshold: {best_threshold.threshold}")
print(f"Balanced accuracy: {best_threshold.balanced_accuracy:.2%}")
print(f"Capture rate: {best_threshold.capture_rate:.1f}%")
print(f"Event probability: {best_threshold.event_probability:.1f}%")
```

## Documentation

For detailed documentation, visit [docs/](docs/).

## Contributing

We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.