Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /gpt_35_turbo

2 contributors

History: 12 commits

xu-song's picture

fix tiktoken

a6c67ec 9 months ago

README.md

1.65 kB

fix chatglm; new feature about add_special_tokens; 9 months ago
__init__.py

389 Bytes

fix tiktoken 9 months ago
cl100k_base.tiktoken

1.68 MB

update about 1 year ago
decode_test.py

291 Bytes

fix chatglm; new feature about add_special_tokens; 9 months ago
test_arena.py

199 Bytes

add more tokenizers 11 months ago
test_tiktoken.py

1.09 kB

fix chatglm; new feature about add_special_tokens; 9 months ago
tiktoken_decode_test.py

965 Bytes

add more tokenizers 11 months ago
vocab.jsonl

3.78 MB

add more tokenizers 11 months ago