Spaces:

xu-song
/

tokenizer-arena

Running

File size: 168 Bytes

751936e

{
    "pad_token": "<|endoftext|>",
    "unk_token": "<|endoftext|>",
    "bos_token": "<|startoftext|>",
    "eos_token": "<|endoftext|>",
    "do_clean_text": false
}