Spaces:

eson
/

tokenizer-arena

Running

tokenizer-arena / utils /convert_sp_to_json.py

update

751936e 10 months ago

No virus

54 Bytes


	from vocab.baichuan_7b import tokenizer

	tokenizer.sp