semvec/pypi-packages huggingface dataset:
data-prep-toolkit-idiud 1.1.0
Subset of Data Preparation Toolkit Transforms
License: Apache-2.0
Requires Python: <3.13,>=3.10
Keywords: transforms, data preprocessing, data preparation, llm, generative, ai, fine-tuning, llmapps

# DPK Python Transforms

## installation

The [transforms](https://github.com/IBM/data-prep-kit/blob/dev/transforms/README.md) are delivered as a standard pyton library available on pypi and can be installed using pip install:

`python -m pip install data-prep-toolkit-idiud[all]`

## List of Transforms in current package

Note: This list includes the transforms that were part of the wheel.

* [doc_quality](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/doc_quality/README.md)
* [lang_id](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/lang_id/README.md)
* [ededup](https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/ededup/README.md)
* [filter](https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/filter/README.md)
* [resize](https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/resize/README.md)
* [doc_id](https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/doc_id/README.md)
* [extrem tokenized](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/extreme_tokenized/README.md)
* [readablity](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/readability/README.md)
* [repetition removal](https://github.com/IBM/data-prep-kit/blob/dev/transforms/universal/rep_removal/README.md)