latin-bert / tokenizer_config.json

fix: apply do_lower_case to root-level HF-facing files

f25becd about 2 months ago

389 Bytes

	{
	"tokenizer_class": "LatinBertTokenizer",
	"auto_map": {
	"AutoTokenizer": ["tokenization_latin_bert.LatinBertTokenizer", "tokenization_latin_bert_fast.LatinBertTokenizerFast"]
	},
	"pad_token": "[PAD]",
	"unk_token": "[UNK]",
	"cls_token": "[CLS]",
	"sep_token": "[SEP]",
	"mask_token": "[MASK]",
	"eos_token": "<EOS>_",
	"do_lower_case": true,
	"model_max_length": 512
	}