fix(README): remove `trust_remote_code` requirement from tokenizer snippet db5a120 verified jon-tow commited on Mar 1
fix: make `eos_token`/`pad_token` overridable and add `pickle` support 4c846d7 jon-tow commited on Jan 23
feat(tokenizer): expose merge ranks and special tokens for GGUF a8f2f28 verified jon-tow commited on Jan 19