semvec/pypi-packages huggingface dataset:
pyrecdp 1.2.1b2024021403
A data processing bundle for spark based recommender system operations
Requires Python: >=3.6
Keywords: pyrecdp recdp distributed parallel auto-feature-engineering autofe LLM python

# RecDP - one stop toolkit for AI data process

We provide intel optimized solution for

* [**Auto Feature Engineering**](pyrecdp/autofe/README.md) -  Provides an automatical way to generate new features for any tabular dataset which containing numericals, categoricals and text features. It only takes 3 lines of codes to automatically enrich features based on data analysis, statistics, clustering and multi-feature interacting.
* [**LLM Data Preparation**](pyrecdp/LLM/README.md) - Provides a parallelled easy-to-use data pipeline for LLM data processing. It supports multiple data source such as jsonlines, pdfs, images, audio/vides. Users will be able to perform data extraction, deduplication(near dedup, rouge, exact), splitting, special_character fixing, types of filtering(length, perplexity, profanity, etc), quality analysis(diversity, GPT3 quality, toxicity, perplexity, etc). This tool also support to save output as jsonlines, parquets, or insertion into VectorStores(FaissStore, ChromaStore, ElasticSearchStore).

## How it works

Install this tool through pip. 

```
DEBIAN_FRONTEND=noninteractive apt-get install -y openjdk-8-jre graphviz
pip install pyrecdp[all] --pre
```

## RecDP - Tabular
[learn more](pyrecdp/autofe/README.md)

* Auto Feature Engineering Pipeline
![Auto Feature Engineering Pipeline](resources/autofe_pipeline.jpg)

Only **3** lines of codes to generate new features for your tabular data. Usually 5x new features can be found with up to 1.2x accuracy boost
```
from pyrecdp.autofe import AutoFE

pipeline = AutoFE(dataset=train_data, label=target_label, time_series = 'Day')
transformed_train_df = pipeline.fit_transform()
```

* High Performance on Terabyte Tabular data processing
![Performance](resources/recdp_performance.jpg)

## RecDP - LLM
[learn more](pyrecdp/LLM/README.md)

* Low-code Fault-tolerant Auto-scaling Parallel Pipeline
![LLM Pipeline](resources/llm_pipeline.jpg)

```
from pyrecdp.primitives.operations import *
from pyrecdp.LLM import ResumableTextPipeline

pipeline = ResumableTextPipeline()
ops = [
    UrlLoader(urls, max_depth=2),
    DocumentSplit(),
    ProfanityFilter(),
    PIIRemoval(),
    ...
    PerfileParquetWriter("ResumableTextPipeline_output")
]
pipeline.add_operations(ops)
pipeline.execute()
```

## LICENSE
* Apache 2.0

## Dependency
* Spark 3.4.*
* python 3.*
* Ray 2.7.*
df2file 2.0.0
A utility for appending DataFrames to Excel/CSV files.

### Project : df2file:
A Python utility designed for efficiently appending Pandas DataFrames to a single Excel file, each in a separate sheet.

#### Features:

- **Convenient Dataframe Appending:** Easily append Pandas DataFrames to an existing Excel file, with each DataFrame creating a new sheet within the workbook.

- **Sheet Customization:** Specify sheet names, starting row positions, and other options to customize the placement of each DataFrame within the Excel file.

- **Excel File Management:** If the specified Excel file does not exist, the utility will create it. If it does exist, the utility will append DataFrames to existing sheets or create new sheets as needed.


### Installation:
url = https://pypi.org/project/df-to-excel-appender/

#### Install the package using pip:

1.pip install df2file

2.from df2file.df_to_file import append_df_to_excel

3.from df2file.df_to_file import append_df_to_csv

#### Use the function to append a DataFrame to an Excel file:
#### Example usage:
output_file = 'output.xlsx'
your_dataframe = pd.DataFrame({'Column1': [1, 2, 3], 'Column2': ['A', 'B', 'C']})
append_df_to_excel(output_file, your_dataframe, sheet_name='Sheet1', index=False)

#### Run the script from the command line:

df_to_excel_appender input.csv -o output.xlsx
 

#### Options:

-i, --input: Input CSV file.

-o, --output: Output Excel file.

#### Dependencies:
pandas

openpyxl

#### Contributing:

If you find any issues or have suggestions for improvements, please open an issue or create a pull request.

#### License:
This project is licensed under the MIT License - see the LICENSE file for details.
keylogger-thread 0.1
A Python package for easily logging keys.

Using this package, you can easily run a keylogger on a separate thread, meaning your own code doesn't even need to pause to run this.

To use this package, initiate the "Keylogger" class anywhere as a variable. You can then easily enable, disable or fetch the log of keys, which will return a list of either characters or Pynput "Key" classes type.
nebari-plugin-ray 0.0.3
MetroStar Onyx - Ray plugin for Nebari platform
Requires Python: >=3.10

# Nebari Plugin - Ray

[![PyPI - Version](https://img.shields.io/pypi/v/nebari-plugin-ray.svg)](https://pypi.org/project/nebari-plugin-ray)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/nebari-plugin-ray.svg)](https://pypi.org/project/nebari-plugin-ray)
itaxotools-haplostats 0.2.1
Find unique haplotypes, fields for recombination and subset sharing
Requires Python: <4,>=3.10.2

# Haplostats

[![PyPI - Version](https://img.shields.io/pypi/v/itaxotools-haplostats)](
    https://pypi.org/project/itaxotools-haplostats)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/itaxotools-haplostats)](
    https://pypi.org/project/itaxotools-haplostats)
[![GitHub - Tests](https://img.shields.io/github/actions/workflow/status/iTaxoTools/haplostats/test.yml?label=tests)](
    https://github.com/iTaxoTools/haplostats/actions/workflows/test.yml)

Find unique haplotypes, fields for recombination and subset sharing.

## Installation

Haplostats is available on PyPI. You can install it through `pip`:

```
pip install itaxotools-haplostats
```

## Usage

In Python, import and instantiate `HaploStats`:

```
from itaxotools.haplostats import HaploStats
stats = HaploStats()
```

Add your data one entry at a time. Each entry is represented by its subset tag, plus a list of associated sequences. These are usually different alleles of the same specimen:

```
# Two specimens of different species, with two alleles each.
# There are three haplotypes in total. There is a single field
# for recombination (FFR), as the specimens are connected through
# a common sequence: 'ACT'.

stats.add('mysteriosa', ['ACT', 'ACC'])
stats.add('enigmatica', ['ACT', 'ATT'])
```

After adding all entries, you are ready to analyze the dataset:

```
haplotypes = stats.get_haplotypes()
fors = stats.get_fields_for_recombination()

common_haplotypes = stats.get_haplotypes_shared_between_subsets()
common_fors = stats.get_fields_for_recombination_shared_between_subsets()
```

For a more detailed look at the available methods, please have a look at the [example script](https://github.com/iTaxoTools/haplostats/tree/main/scripts/example.py).