Update README.md

README.md

@@ -19,7 +19,6 @@ This is the set of 5 Chinese word-based RoBERTa models pre-trained by [UER-py](h
 
 You can download the 5 Chinese RoBERTa miniatures either from the [UER-py Github page](https://github.com/dbiir/UER-py/), or via HuggingFace from the links below:
 
-
 |          |           Link           |
 | -------- | :-----------------------: |
 | **Tiny**  | [**2/128 (Tiny)**][2_128] |
@@ -28,10 +27,6 @@ You can download the 5 Chinese RoBERTa miniatures either from the [UER-py Github
 | **Medium**  | [**8/512 (Medium)**][8_512] |
 | **Base** | [**12/768 (Base)**][12_768] |
 
-
-We use sentencepiece model to segment Chinese word and train this RoBERTa base model. You can download the model via HuggingFace from the link [roberta-base-word-chinese-cluecorpussmall](https://huggingface.co/uer/roberta-base-word-chinese-cluecorpussmall).
-
-We found some bugs when using Hosted inference API. If the target character is a single word, the entire sentence will be displayed. If the target character is multiple words, only the target character will be displayed. In order to display correctly ,we recommend using the JSON Output in the lower left corner of the Hosted inference API.
 ## How to use
 
 You can use this model directly with a pipeline for masked language modeling:
@@ -68,16 +63,12 @@ output = model(encoded_input)
 
 ## Training data
 
-[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. 
-
-## Training procedure
-
-We use google's **[sentencepiece](https://github.com/google/sentencepiece)** to train the sentencepiece model.
+[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. Google's [sentencepiece](https://github.com/google/sentencepiece) is used for word segmentation. The sentencepiece model is trained on CLUECorpusSmall corpus:
 
 ```
 >>> import sentencepiece as spm
->>> spm.SentencePieceTrainer.train(input='CLUEsmall_shuf.txt',
-             model_prefix='clue_6',
+>>> spm.SentencePieceTrainer.train(input='cluecorpussmall.txt',
+             model_prefix='cluecorpussmall_spm',
              vocab_size=100000,
              max_sentence_length=1024,
              max_sentencepiece_length=6,
@@ -96,13 +87,17 @@ We use google's **[sentencepiece](https://github.com/google/sentencepiece)** to
             )
 ```
 
-The model is pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud TI-ONE](https://cloud.tencent.com/product/tione/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. 
+## Training procedure
+
+Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent Cloud TI-ONE](https://cloud.tencent.com/product/tione/). We pre-train 1,000,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512. We use the same hyper-parameters on different model sizes.
+
+Taking the case of word-based RoBERTa-Medium
 
 Stage1:
 
 ```
 python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
-                      --spm_model_path models/clue_6.model \
+                      --spm_model_path models/cluecorpussmall_spm.model \
                       --dataset_path cluecorpussmall_word_seq128_dataset.pt \
                       --processes_num 32 --seq_length 128 \
                       --dynamic_masking --target mlm
@@ -111,13 +106,12 @@ python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
 ```
 python3 pretrain.py --dataset_path cluecorpussmall_word_seq128_dataset.pt \
                     --spm_model_path models/cluecorpussmall_spm.model \
-                    --config_path models/bert/base_config.json \
-                    --output_model_path models/cluecorpussmall_word_roberta_base_128.bin \
+                    --config_path models/bert/medium_config.json \
+                    --output_model_path models/cluecorpussmall_word_roberta_medium_seq128_model.bin \
                     --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                     --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                     --learning_rate 1e-4 --batch_size 64 \
-                    --embedding word_pos_seg --encoder transformer --mask fully_visible \
-                    --target mlm --tie_weights
+                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
 ```
 
 Stage2:
@@ -132,21 +126,20 @@ python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
 
 ```
 python3 pretrain.py --dataset_path cluecorpussmall_word_seq512_dataset.pt \
-                    --pretrained_model_path models/cluecorpussmall_word_roberta_base_128.bin-1000000 \
+                    --pretrained_model_path models/cluecorpussmall_word_roberta_medium_seq128_model.bin-1000000 \
                     --spm_model_path models/cluecorpussmall_spm.model \
-                    --config_path models/bert/base_config.json \
-                    --output_model_path models/cluecorpussmall_word_roberta_base_512.bin \
+                    --config_path models/bert/medium_config.json \
+                    --output_model_path models/cluecorpussmall_word_roberta_medium_seq512_model.bin \
                     --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                     --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                     --learning_rate 5e-5 --batch_size 16 \
-                    --embedding word_pos_seg --encoder transformer --mask fully_visible \
-                    --target mlm --tie_weights
+                    --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
 ```
 
 Finally, we convert the pre-trained model into Huggingface's format:
 
 ```
-python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_word_roberta_base_512.bin-250000 \
+python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_word_roberta_medium_seq128_model.bin-250000 \
                                                         --output_model_path pytorch_model.bin \
                                                         --layers_num 12 --target mlm
 ```