semvec/pypi-packages huggingface dataset:
TokenizerChanger 1.0.5
Library for manipulating the existing tokenizer.
Requires Python: >=3.10
Keywords: tokenizer deletion tokens

# Tokenizer-Changer

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.

# Installation

Installation from PyPI:

```bash
pip install tokenizerchanger
```

# Requirements

- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4

# Docs

[https://tokenizer-changer.readthedocs.io/en/latest/](https://tokenizer-changer.readthedocs.io/en/latest/)