col-minilm / tokenizer.json

Commit History

clean up tokenizer and add [unused0] and [unused1] as special tokens
85b42e5

Jo Kristian Bergum commited on

Upload tokenizer.json
4eafbac

bergum commited on