NovelAI Tokenizer v1
This repository is exactly the same as NovelAI/nerdstash-tokenizer-v1, but the config has been changed to address the following points (the sp model itself is not changed).
- Load as T5Tokenizer
- Enable to decode digits (In the original, digits are registered as
additional_special_tokens, so ifskip_special_tokens=Truewhen decoding, the digits are also skipped.)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mkshing/novelai-tokenizer-v1", use_fast=False)
text = "1+1=3"
tokenizer.decode(tokenizer.encode(text), skip_special_tokens=True)
# '1+1=3'
- Downloads last month
- 0
Unable to determine this model's library. Check the
docs
.