incoder-1B / tokenizer_config.json

add tokenizer

00e3bf6 about 2 years ago

No virus

212 Bytes

{"model_max_length": 2048, "bos_token": "<|endoftext|>", "special_tokens_map_file": "tokenizer/v2_trfs/special_tokens_map.json", "name_or_path": "tokenizer/v2_trfs/", "tokenizer_class": "PreTrainedTokenizerFast"}