bert-base-japanese-char-v3 / tokenizer_config.json

Add model files

ccddee5 about 1 year ago

No virus

251 Bytes

	{
	"tokenizer_class": "BertJapaneseTokenizer",
	"model_max_length": 512,
	"do_lower_case": false,
	"word_tokenizer_type": "mecab",
	"subword_tokenizer_type": "character",
	"mecab_kwargs": {
	"mecab_dic": "unidic_lite"
	}
	}