GuaPeTe-2-tiny / tokenizer_config.json
system's picture
system HF staff
Update tokenizer_config.json
705ace6
{"max_len": 1024, "bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>"}