semvec/pypi-packages huggingface dataset:
tokengeex 1.1.0
TokenGeeX is an efficient tokenizer for code based on UnigramLM and TokenMonster.
License: Apache-2.0
Requires Python: >=3.8
Keywords: tokenizer, nlp

# TokenGeeX - Efficient Tokenizer for CodeGeeX

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.10959).