tokenizer-arena / utils /convert_sp_to_json.py
xu-song's picture
update
751936e
raw
history blame
54 Bytes
from vocab.baichuan_7b import tokenizer
tokenizer.sp