README.md

metadata

language:
  - en
  - ja
license: mit
datasets:
  - snow_simplified_japanese_corpus
tags:
  - ja
  - japanese
  - tokenizer
widget:
  - text: 誰が一番に着くか私には分かりません。

Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on snow_simplified_japanese_corpus dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner.

Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences.

How to use it

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ybelkada/japanese-dummy-tokenizer")

How to train the tokenizer

Check the file tokenizer.py, you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from csebuetnlp/mT5_multilingual_XLSum.