pkshatech
/

simcse-ja-bert-base-clcmlp

Sentence Similarity sentence-transformers PyTorch Transformers Japanese bert feature-extraction

Model card Files Files and versions Community

simcse-ja-bert-base-clcmlp / tokenizer_config.json

akiFQC's picture

update tokenizer

63782c6 over 1 year ago

raw history blame contribute delete

No virus

456 Bytes

	{
	"do_lower_case": false,
	"do_subword_tokenize": true,
	"do_word_tokenize": true,
	"mask_token": "[MASK]",
	"mecab_kwargs": {
	"mecab_dic": "unidic_lite"
	},
	"model_max_length": 512,
	"pad_token": "[PAD]",
	"cls_token": "[CLS]",
	"sep_token": "[SEP]",
	"special_tokens_map_file": null,
	"name_or_path": "cl-tohoku/bert-base-japanese-v2",
	"subword_tokenizer_type": "wordpiece",
	"unk_token": "[UNK]",
	"word_tokenizer_type": "mecab"
	}