semvec/pypi-packages huggingface dataset:
tokenizer-hub 0.0.1
Yoctol Natural Language Tokenizer
License: MIT
Requires Python: >=3.5

# tokenizer-hub

yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂

Tokenizers have the same interface of Jieba:

```python
from tokenizer_hub import XXX_tokenizer
tokenizer = XXX_tokenizer()
tokenizer.lcut('我来到北京清华大学')
> ['我', '来到', '北京', '清华大学']
```