Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
ybelkada
/
japanese-dummy-tokenizer
like
1
Transformers
snow_simplified_japanese_corpus
English
Japanese
japanese
tokenizer
Inference Endpoints
License:
mit
Model card
Files
Files and versions
Community
1
Train
Deploy
Use this model
f2449c8
japanese-dummy-tokenizer
2 contributors
History:
3 commits
Younes Belkada
add python file
f2449c8
over 2 years ago
japanese-dummy-tokenizer
Add tokenizer file
over 2 years ago
.gitattributes
1.22 kB
initial commit
over 2 years ago
tokenizer.py
927 Bytes
add python file
over 2 years ago