tokenizer-arena / requirements.txt
xu-song's picture
add more tokenizers
f4973d4
raw
history blame
60 Bytes
transformers==4.33.1
sentencepiece
tiktoken
icetk
torch
zhon