semvec/pypi-packages huggingface dataset:
pipelineV 0.1.0
ETL pipeline project
Requires Python: >=3.12

# ETL Pipeline Project

This project implements an Extract, Transform, Load (ETL) pipeline designed to facilitate data processing from various sources into a target database or files. The project is structured to support multiple environments, including development, testing, and production.

## Project Structure

- **config/**: Contains configuration files for different environments.
  - `dev.yaml`: Development environment settings.

- **src/**: Core ETL pipeline code.
  - `pipeline.py`: Orchestrator for running ETL steps, parameterized by environment.
  - `etlBase.py` : Base class of the ETL program
  - `TestETL.py` : Example ETL program that query data from Protex and write data to dahsboard sql server

- **tests/**: Unit and integration tests.

- **scripts/**: SQL query scripts

- **requirements.txt**: Lists Python dependencies required for the project.

- **.gitignore**: Specifies files and directories to be ignored by Git.

## Setup Instructions

1. Configure the pyproject.toml as needed.

2. Configure the environment settings in the `config/` directory as needed.

3. Run pipeline for Windows
   - call uv_setup.bat

   Run pipeline for Macos/Linux
   - .\uv_setup.bat
steganan 0.2.2
Steganography in floating point data with NaN payloads
Requires Python: >=3.9
Keywords: python

# steganan

[![PyPI - Version](https://img.shields.io/pypi/v/steganan)](https://pypi.org/project/steganan/)
[![Release](https://img.shields.io/github/v/release/kjordahl/steganan)](https://github.com/kjordahl/steganan/releases)
[![Build status](https://img.shields.io/github/actions/workflow/status/kjordahl/steganan/main.yml?branch=main)](https://github.com/kjordahl/steganan/actions/workflows/main.yml?query=branch%3Amain)
[![License](https://img.shields.io/github/license/kjordahl/steganan)](https://github.com/kjordahl/steganan/blob/main/LICENSE)

Steganography in floating point data with NaN payloads

- **Github repository**: <https://github.com/kjordahl/steganan/>

Do you have not-a-number values in your floating point data that
you're not making full use of?  Have you ever pondered what to do with
all of those wasted bits? Steganan may be for you!  Encode whatever
data you like into those spare bits! Most programs won't even notice
the difference. Save them out to a file if you like. As long as your
file format preserves the full floating point values, you can get your
hidden data back!

Want to add compression and/or encryption of your data? That is
certainly possible, but is left as an exercise for the user.

## Examples

### Storing image data in an array of NaN values

```
>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from skimage.data import astronaut
>>> import steganan
>>> img = astronaut()
>>> print(img.shape)
(512, 512, 3)
>>> a = steganan.encode_array(img, stack=True, dtype=np.float64)
>>> print(a)
[[nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 ...
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]]
>>> decoded = steganan.decode_array(a, stack=True, depth=3)
>>> plt.figure(figsize=(4, 4))
>>> plt.imshow(decoded)
```
![decoded.png](https://github.com/kjordahl/steganan/raw/refs/heads/main/data/decoded.png)

### Hiding data in an existing floating point array

```
>>> import rasterio as rio
>>> src = rio.open("data/modis_aod_06_2025.tif")
>>> a = src.read(1)
>>> steganan.write_str_to_nans(a, "I have a secret!")
>>> plt.imshow(a, cmap='inferno_r', vmin=0, vmax=1.0)
```
![encoded.png](https://github.com/kjordahl/steganan/raw/refs/heads/main/data/encoded.png)
```
>>> message = steganan.retrieve_string_from_payloads(a)
>>> print(message)
I have a secret!
```

### Notebook example

See [lightning talk slides](https://github.com/kjordahl/steganan/blob/main/slides/talk.ipynb).

---

Repository initiated with [fpgmaas/cookiecutter-uv](https://github.com/fpgmaas/cookiecutter-uv).
cogent-base 0.1.17
Base and shared modules for cogents
Requires Python: <3.12,>=3.10

# Cogent Base

[![PyPI version](https://img.shields.io/pypi/v/cogent-base)](https://pypi.python.org/pypi/cogent-base)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/mirasurf/cogent-base/actions/workflows/ci.yml/badge.svg)](https://github.com/mirasurf/cogent-base/actions/workflows/ci.yml)

A shared Python module for agentic cognitive computing frameworks, providing extensible configuration management, logging utilities, and core components.

## Features

- **Extensible Configuration System**: Register custom configurations with TOML support
- **Flexible Logging**: Basic logging utilities that can be overridden by downstream libraries
- **Provider Abstraction**: Unified interfaces for LLM, embedding, reranking, and vector store providers
- **Sensory Processing**: Document parsing and text chunking capabilities
- **Modular Design**: Clean separation of concerns with extensible architecture

## Installation

**Requirements**: Python 3.11+

```bash
pip install cogent-base
```

To use advanced vendor's features, install with

```bash
pip install cogent-base[extensions]
```

to enable features:

- vector stores of weaviate
- FlagEmbedding
- video parser with assemblyai
- smart frame with pandasai
- smart voice with ailyun lingjie NLS

For development:

```bash
git clone https://github.com/mirasurf/cogent-base.git
cd cogent-base
make install-dev
```

## Quick Start

```python
from cogent_base.config import get_cogent_config

# Get the global configuration
config = get_cogent_config()

# Access built-in configurations
llm_config = config.llm
vector_store_config = config.vector_store
```

## Documentation

- **[Examples](examples/)** - Practical usage examples for all features
- **[Development Guide](DEVELOPMENT.md)** - Setup, testing, and deployment procedures
- **[Testing Guide](tests/README.md)** - Comprehensive testing documentation

## Core Components

### Configuration System
- Layered configuration loading (Class Defaults → Package TOML → User TOML)
- Custom configuration class registration
- TOML-based configuration files

### Provider Interfaces
- **LLM Providers**: LiteLLM-based completion models
- **Embedding Providers**: Text embedding generation
- **Vector Stores**: Weaviate integration for vector operations
- **Reranking**: Document reranking capabilities

### Sensory Processing
- **Document Parsing**: Multi-format document text extraction
- **Text Chunking**: Configurable text splitting with overlap

### Smart Processing
- **Smart DataFrame**: dataframe talks to LLM with pandasai
- **Smart Voice**: voice-to-text transcription based on Aliyun Lingji AI.

## License

MIT