semvec/pypi-packages huggingface dataset:
data-prep-toolkit-transforms 1.1.7
Data Preparation Toolkit Transforms using Ray
License: Apache-2.0
Requires Python: <3.14,>=3.10
Keywords: transforms, data preprocessing, data preparation, llm, generative, ai, fine-tuning, llmapps

# DPK Python Transforms

## installation

The [transforms](https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/README.md) are delivered as a standard pyton library available on pypi and can be installed using uv pip install:
`pip install uv`

`python -m uv pip install data-prep-toolkit-transforms[all]`
or
`python -m uv pip install data-prep-toolkit-transforms[ray, all]`
or
`python -m uv pip install data-prep-toolkit-transforms[language]`


installing the python transforms will also install  `data-prep-toolkit`

installing the ray transforms will also install  `data-prep-toolkit[ray]`

## Release notes:

### 1.1.1.dev1
	Include all code transforms as extra [code]

### 1.1.1.dev0
	Refactored code transforms (code_uality, code2parquet, header_cleanser, license select, proglang_select)
	Added ml-filter and enrichment
	renamed PDF2Parquet to Docling2Paruqet 

### 1.0.1.dev1
	Added Gneissweb transforms
	fdedup fix for windows
### 1.0.1.dev0
	PR #979 (code_profiler)
### 1.0.0.a6
	Added Profiler
	Added Resize
### 1.0.0.a5
	Added Pii Redactor
	Relax fasttext requirement >= 0.9.2
### 1.0.0.a4
	Added missing ray implementation for lang_id, doc_quality, tokenization and filter
	Added ray notebooks for lang id, Doc Quality, tokenization, and Filter
### 1.0.0.a3
	Added code_profiler
### 1.0.0.a2
   Relax dependencies on pandas (use latest or whatever is installed by application)
   Relax dependencies on requests (use latest or whatever is installed by application)
data-prep-connector 0.2.4
Scalable and Compliant Web Crawler
License: Apache-2.0
Requires Python: <3.14,>=3.10
Keywords: data, data acquisition, crawler, web crawler, llm, generative, ai, fine-tuning, llmapps

# DPK Connector

DPK Connector is a scalable and compliant web crawler developed for data acquisition towards LLM development. It is built on [Scrapy](https://scrapy.org/).
For more details read [the documentation](doc/overview.md).

## Virtual Environment

The project uses `pyproject.toml` and a Makefile for operations.
To do development you should establish the virtual environment
```shell
make venv
```
and then either activate
```shell
source venv/bin/activate
```
or set up your IDE to use the venv directory when developing in this project

## Library Artifact Build and Publish

To test, build and publish the library
```shell
make test build publish
```

To up the version number, edit the Makefile to change VERSION and rerun the above. This will require committing both the `Makefile` and the autotmatically updated `pyproject.toml` file.

## How to use

See [the overview](doc/overview.md).