Hugging Face's logo

junnyu
/
roformer_small_generator

Fill-Mask
Transformers
PyTorch
English
roformer
electra
masked-lm
rotary position embedding
Inference Endpoints
Model card Files Community
2
junnyu commited on
Commit
d528ac3
•
1 Parent(s): e2db733

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -2
README.md CHANGED
@@ -11,11 +11,13 @@ datasets:
11
  - openwebtext
12
  ---
13
  # 一、 个人在openwebtext数据集上添加rotary-position-embedding，训练得到的electra-small模型
 
14
  # 二、 复现结果(dev dataset)
15
  |Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
16
  |---|---|---|---|---|---|---|---|---|---|
17
  |ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
18
- |**ELECTRA-Small-OWT (this)**| xx|xx|xx|xx|xx|xx|xx|xx|xx|
 
19
  # 三、 训练细节
20
  - 数据集 openwebtext
21
  - 训练batch_size 256
@@ -26,7 +28,9 @@ datasets:
26
  - 训练时间总共耗费55h
27
 
28
  ### wandb日志
29
- https://wandb.ai/junyu/electra_rotary_small_pretrain?workspace=user-junyu
 
 
30
 
31
  # 四、安装
32
  ```bash
 
11
  - openwebtext
12
  ---
13
  # 一、 个人在openwebtext数据集上添加rotary-position-embedding，训练得到的electra-small模型
14
+
15
  # 二、 复现结果(dev dataset)
16
  |Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
17
  |---|---|---|---|---|---|---|---|---|---|
18
  |ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
19
+ |**ELECTRA-RoFormer-Small-OWT (this)**|55.76|90.45|87.3|86.64|89.61|81.17|88.85|62.71|80.31|
20
+
21
  # 三、 训练细节
22
  - 数据集 openwebtext
23
  - 训练batch_size 256
 
28
  - 训练时间总共耗费55h
29
 
30
  ### wandb日志
31
+ [预训练日志](https://wandb.ai/junyu/electra_rotary_small_pretrain?workspace=user-junyu)
32
+
33
+ [GLUE微调日志](https://wandb.ai/junyu/electra_rotary_glue_100?workspace=user-junyu)
34
 
35
  # 四、安装
36
  ```bash