BertJapanese
Overview
The BERT models trained on Japanese text.
There are models with two different tokenization methods:
- Tokenize with MeCab and WordPiece. This requires some extra dependencies, fugashi which is a wrapper around MeCab.
- Tokenize into characters.
To use MecabTokenizer, you should pip install transformers["ja"]
(or pip install -e .["ja"]
if you install
from source) to install dependencies.
See details on cl-tohoku repository.
Example of using a model with MeCab and WordPiece tokenization:
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer
>>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")
>>> ## Input Japanese Text
>>> line = "εΎθΌ©γ―η«γ§γγγ"
>>> inputs = tokenizer(line, return_tensors="pt")
>>> print(tokenizer.decode(inputs['input_ids'][0]))
[CLS] εΎθΌ© γ― η« γ§ γγ γ [SEP]
>>> outputs = bertjapanese(**inputs)
Example of using a model with Character tokenization:
>>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese-char")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-char")
>>> ## Input Japanese Text
>>> line = "εΎθΌ©γ―η«γ§γγγ"
>>> inputs = tokenizer(line, return_tensors="pt")
>>> print(tokenizer.decode(inputs['input_ids'][0]))
[CLS] εΎ θΌ© γ― η« γ§ γ γ γ [SEP]
>>> outputs = bertjapanese(**inputs)
Tips:
- This implementation is the same as BERT, except for tokenization method. Refer to the documentation of BERT for more usage examples.
This model was contributed by cl-tohoku.
BertJapaneseTokenizer
( vocab_file do_lower_case = False do_word_tokenize = True do_subword_tokenize = True word_tokenizer_type = 'basic' subword_tokenizer_type = 'wordpiece' never_split = None unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' mecab_kwargs = None **kwargs )
Parameters
-
vocab_file (
str
) — Path to a one-wordpiece-per-line vocabulary file. -
do_lower_case (
bool
, optional, defaults toTrue
) — Whether to lower case the input. Only has an effect when do_basic_tokenize=True. -
do_word_tokenize (
bool
, optional, defaults toTrue
) — Whether to do word tokenization. -
do_subword_tokenize (
bool
, optional, defaults toTrue
) — Whether to do subword tokenization. -
word_tokenizer_type (
str
, optional, defaults to"basic"
) — Type of word tokenizer. -
subword_tokenizer_type (
str
, optional, defaults to"wordpiece"
) — Type of subword tokenizer. -
mecab_kwargs (
str
, optional) — Dictionary passed to theMecabTokenizer
constructor.
Construct a BERT tokenizer for Japanese text, based on a MecabTokenizer.