Update README.md
Browse files
README.md
CHANGED
@@ -147,7 +147,7 @@ python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
|
|
147 |
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
|
148 |
--total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
|
149 |
--learning_rate 5e-5 --batch_size 16 \
|
150 |
-
--
|
151 |
```
|
152 |
|
153 |
Finally, we convert the pre-trained model into Huggingface's format:
|
|
|
147 |
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
|
148 |
--total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
|
149 |
--learning_rate 5e-5 --batch_size 16 \
|
150 |
+
--embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm --tie_weights
|
151 |
```
|
152 |
|
153 |
Finally, we convert the pre-trained model into Huggingface's format:
|