Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /gpt_nexo_20b

2 contributors

History: 4 commits

xu-song's picture

add compress rate

814ee6b 7 months ago

tokenizer
update about 1 year ago
20B_tokenizer.json

2.47 MB

update about 1 year ago
20B_tokenizer.zh.json

2.11 MB

update about 1 year ago
README.md

1.69 kB

add compress rate 7 months ago
__init__.py

717 Bytes

update about 1 year ago
convert_vocab_to_txt.py

459 Bytes

update about 1 year ago
test_gpt_neox_20b.py

2.96 kB

update about 1 year ago
test_hf_gpt_neox.py

487 Bytes

update about 1 year ago
test_oov.py

505 Bytes

update about 1 year ago
test_special_token.py

340 Bytes

update about 1 year ago
test_tokenizer.py

3.39 kB

add compress rate 7 months ago
test_tokenizer_HF.py

1.3 kB

update about 1 year ago
test_zh_coding_len.py

447 Bytes

update about 1 year ago
vocab.zh.txt

9.34 kB

update about 1 year ago