Hugging Face's logo

stabilityai
/
arcade100k

Model card Files Community
arcade100k

Commit History

fix: update license
7e4138f

jon-tow commited on

fix: store `cl100k_base` mergeable ranks in `arcade100k.tiktoken`
1587587

jon-tow commited on

Merge branch 'main' of hf.co:jon-tow/stablelm-tokenizer-v1 into main
d8301d9

jon-tow commited on

fix: re-ordering special tokens
f221007

jon-tow commited on

Create tokenizer_config.json
af51a49

jon-tow commited on

Delete tokenizer_config.json
beeb2b0

jon-tow commited on

Delete special_tokens_map.json
83aaca7

jon-tow commited on

fix: pass in `vocab_file` to tiktoken loader
a06afc6

jon-tow commited on

Upload tokenizer
f3e546a

jon-tow commited on

fix: remove `print` debug statements
3c66e0d

jon-tow commited on

Update tokenization_arcade100k.py
8593ce9

jon-tow commited on

Update README.md
be6493b

jon-tow commited on

fix: create final list of special tokens
9b47601

jon-tow commited on

init: first commit
aceea5b

jon-tow commited on

initial commit
99cb326

jon-tow commited on