update: README.md

README.md

@@ -14,7 +14,7 @@ metrics:
   - accuracy
 mask_token: "[MASK]"
 widget:
-    - text: "京都 大学 で 自然 言語 処理 を [MASK] する 。"
+    - text: "京都大学で自然言語処理を[MASK]する。"
 ---
 
 # Model Card for Japanese DeBERTa V2 base
@@ -29,10 +29,10 @@ You can use this model for masked language modeling as follows:
 
 ```python
 from transformers import AutoTokenizer, AutoModelForMaskedLM
-tokenizer = AutoTokenizer.from_pretrained('ku-nlp/deberta-v2-base-japanese')
+tokenizer = AutoTokenizer.from_pretrained('ku-nlp/deberta-v2-base-japanese', trust_remote_code=True)
 model = AutoModelForMaskedLM.from_pretrained('ku-nlp/deberta-v2-base-japanese')
 
-sentence = '京都 大学 で 自然 言語 処理 を [MASK] する 。'  # input should be segmented into words by Juman++ in advance
+sentence = '京都大学で自然言語処理を[MASK]する。'
 encoding = tokenizer(sentence, return_tensors='pt')
 ...
 ```
@@ -41,7 +41,9 @@ You can also fine-tune this model on downstream tasks.
 
 ## Tokenization
 
-The input text should be segmented into words by [Juman++](https://github.com/ku-nlp/jumanpp) in advance. [Juman++ 2.0.0-rc3](https://github.com/ku-nlp/jumanpp/releases/tag/v2.0.0-rc3) was used for pre-training. Each word is tokenized into subwords by [sentencepiece](https://github.com/google/sentencepiece).
+~~The input text should be segmented into words by [Juman++](https://github.com/ku-nlp/jumanpp) in advance. [Juman++ 2.0.0-rc3](https://github.com/ku-nlp/jumanpp/releases/tag/v2.0.0-rc3) was used for pre-training. Each word is tokenized into subwords by [sentencepiece](https://github.com/google/sentencepiece).~~
+
+UPDATE: The input text is internally segmented by [Juman++](https://github.com/ku-nlp/jumanpp) within `DebertaV2JumanppTokenizer(Fast)`, so there's no need to segment it in advance. To use `DebertaV2JumanppTokenizer(Fast)`, you need to install [Juman++ 2.0.0-rc3](https://github.com/ku-nlp/jumanpp/releases/tag/v2.0.0-rc3) and [rhoknp](https://github.com/ku-nlp/rhoknp).
 
 ## Training data