Spaces:

eson
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / requirements.txt

eson's picture

add olmo tokenizer

bbefe94 5 months ago

No virus

72 Bytes

	transformers
	sentencepiece
	tiktoken
	icetk
	torch
	zhon
	nltk
	boto3
	ai2-olmo