semvec/pypi-packages huggingface dataset:
inclitoken 0.1.0
Inclitoken is implementation of Byte Pair Encoding Tokenizer from scratch.
Requires Python: >=3.14
Keywords: bpetokenizer, tokenizer, byte pair encoding

# IncliToken

A simple Byte Pair Encoding (BPE) tokenizer implementation from scratch in Python.

## Installation

```bash
uv add inclitoken
```

Or you can use `pip`:
```bash
pip install inclitoken
```

## Usage

```python
from inclitoken.tokenizer import BPETokenizer

# Initialize tokenizer
tokenizer = BPETokenizer()

# Train on your text
text = "Hello world! This is a simple example."
tokenizer.train(text, turns=100, verbose=False)

# Encode text to token IDs
ids = tokenizer.encode("Hello world!")
print(ids)

# Decode token IDs back to text
decoded = tokenizer.decode(ids)
print(decoded)
```

## Features

- Train custom BPE tokenizers on your text
- Encode text into token IDs
- Decode token IDs back into text
- Track merge operations and vocabulary

## Requirements

- Python >= 3.14
- tqdm

## Author

Built by [Adarsh Dubey](https://twitter.com/inclinedadarsh)