Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
Spaces:
yhavinga
/
dutch-tokenizer-arena
like
6
Running
App
Files
Files
Community
1
1b7fc74
dutch-tokenizer-arena
/
vocab
/
gpt_nexo_20b
2 contributors
History:
5 commits
xu-song
add compression leaderboard
1b7fc74
5 months ago
tokenizer
update
about 1 year ago
20B_tokenizer.json
2.47 MB
update
about 1 year ago
20B_tokenizer.zh.json
2.11 MB
update
about 1 year ago
README.md
1.69 kB
add compress rate
5 months ago
__init__.py
114 Bytes
add compression leaderboard
5 months ago
convert_vocab_to_txt.py
459 Bytes
update
about 1 year ago
test_gpt_neox_20b.py
2.96 kB
update
about 1 year ago
test_hf_gpt_neox.py
487 Bytes
update
about 1 year ago
test_oov.py
505 Bytes
update
about 1 year ago
test_special_token.py
340 Bytes
update
about 1 year ago
test_tokenizer.py
3.39 kB
add compress rate
5 months ago
test_tokenizer_HF.py
1.3 kB
update
about 1 year ago
test_zh_coding_len.py
447 Bytes
update
about 1 year ago
vocab.zh.txt
9.34 kB
update
about 1 year ago