evo-1-131k-base / tokenizer_config.json

Commit History

Fix auto tokenizer import reference format in auto map as list for slow and fast.
a9a132f

Pragaash commited on

Replace tokenizer class with auto map for relative import of byte tokenizer.
72e0b3e
verified

pragaash commited on

Update byte tokenizer to be compatible with auto tokenizer and clean-up.
158e9d1
verified

pragaash commited on

Create tokenizer_config.json
2c3da52
verified

Zymrael commited on