Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
ybelkada
/
japanese-dummy-tokenizer
like
1
Transformers
snow_simplified_japanese_corpus
English
Japanese
japanese
tokenizer
Inference Endpoints
License:
mit
Model card
Files
Files and versions
Community
1
Train
Deploy
Use this model
main
japanese-dummy-tokenizer
File size: 65 Bytes
f342985
1
{
"eos_token"
:
"</s>"
,
"unk_token"
:
"<unk>"
,
"pad_token"
:
"<pad>"
}