Spaces:

yhavinga
/

dutch-tokenizer-arena

Running

dutch-tokenizer-arena / utils /convert_sp_to_json.py

update

751936e about 1 year ago

54 Bytes


	from vocab.baichuan_7b import tokenizer

	tokenizer.sp