semvec/pypi-packages huggingface dataset:
soco-tokenizer 1.2
Fast tokenizer

# Tokenizer
Tokenizer different language fast.


## Build a package
    python setup.py bdist_wheel
    twine upload dist/*

## Use Locally
    x1 = '<a>刘强东是一个著名企业家。</a> 他创建了京东。'
    t = EncoderLoader.load_tokenizer('bert-base-chinese-zh_v4-10K')
    print(t.tokenize(x1, mode='char'))
    print(t.tokenize(x1, mode='word'))
    print(t.tokenize(x1, mode='all'))