semvec/pypi-packages huggingface dataset:
rag-tokenizer-test 0.0.4
rag tokenizer for RAGFlow
Requires Python: <3.15,>=3.10

# rag-tokenizer

# Upload to pypi.org 

```shell
uv build;
uv publish --token $PYPI_API_TOKEN
```

# Install
```shell
pip install rag-tokenizer
```

# Usage

```python
import rag_tokenizer
res = rag_tokenizer.tokenize("Hello world!")
print(res)
```