Update README.md

README.md

@@ -1,3 +1,83 @@
 ---
+language: ja
 license: cc-by-sa-4.0
+library_name: transformers
+tags:
+  - bert
+  - fill-mask
+datasets:
+  - wikipedia
+mask_token: "[MASK]"
+widget:
+  - text: "[MASK] 大学 で 自然 言語 処理 を 専攻 する 。"
 ---
+
+# ku-accms/bert-base-japanese-ssuw
+## Model description
+This is a Japanese BERT base model pre-trained on a Japanese Wikipedia dump with super short unit words (SSUW).
+
+## Pre-processing
+The input text should be converted to full-width (zenkaku) characters and segmented into super short unit words in advance (e.g., by KyTea).
+
+## How to use
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='ku-accms/bert-base-japanese-ssuw')
+>>> unmasker("[MASK] 大学 で 自然 言語 処理 を 専攻 する 。")
+[{'sequence': 'スタンフォード 大学 で 自然 言語 処理 を 専攻 する 。',
+  'score': 0.13041487336158752,
+  'token': 26978,
+  'token_str': 'スタンフォード'},
+ {'sequence': '早稲田 大学 で 自然 言語 処理 を 専攻 する 。',
+  'score': 0.05302431806921959,
+  'token': 17048,
+  'token_str': '早稲田'},
+ {'sequence': 'ハーバード 大学 で 自然 言語 処理 を 専攻 する 。',
+  'score': 0.048841025680303574,
+  'token': 21731,
+  'token_str': 'ハーバード'},
+ {'sequence': '筑波 大学 で 自然 言語 処理 を 専攻 する 。',
+  'score': 0.04634753614664078,
+  'token': 20287,
+  'token_str': '筑波'},
+ {'sequence': '東京 大学 で 自然 言語 処理 を 専攻 する 。',
+  'score': 0.030050478875637054,
+  'token': 13949,
+  'token_str': '東京'}]
+```python
+import zenhan
+import Mykytea
+kytea_model_path = "somewhere"
+kytea = Mykytea.Mykytea("-model {} -notags".format(kytea_model_path))
+def preprocess(text):
+    return " ".join(kytea.getWS(zenhan.h2z(text)))
+
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('ku-accms/bert-base-japanese-ssuw')
+model = BertModel.from_pretrained("ku-accms/bert-base-japanese-ssuw")
+text = "京都大学で自然言語処理を専攻する。"
+encoded_input = tokenizer(preprocess(text), return_tensors='pt')
+output = model(**encoded_input)
+```
+
+## Training data
+We used a Japanese Wikipedia dump (as of 20230101, 3.7GB).
+
+## Training procedure
+We first segmented the texts into words by KyTea and then tokenized the words into subwords using WordPiece with a vocabulary size of 32,000. We pre-trained the BERT model using [transformers](https://github.com/huggingface/transformers) library. The training took about 8 days using 4 NVIDIA A100-SXM4-80GB GPUs.
+
+The following hyperparameters were used for the pre-training.
+
+- learning_rate: 2e-4
+- weight decay: 1e-2
+- per_device_train_batch_size: 80
+- num_devices: 4
+- gradient_accumulation_steps: 3
+- total_train_batch_size: 960
+- max_seq_length: 512
+- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-06
+- lr_scheduler_type: linear schedule with warmup
+- training_steps: 500,000
+- warmup_steps: 10,000