xlrs-53-finnish / tokenizer.json
vneralla's picture
Latest
f0a8522
PreTrainedTokenizer(name_or_path='', vocab_size=36, model_max_len=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>')