junnyu commited on
Commit
27c46a7
1 Parent(s): 8dbb1e5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +37 -9
README.md CHANGED
@@ -17,31 +17,59 @@ https://github.com/JunnYu/RoFormer_pytorch
17
  - pip install roformer==0.4.3
18
 
19
  ## 评测对比
20
- ### CLUE-dev榜单分类任务结果,base版本。
21
 
22
  | | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl |
23
  | :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
24
  | BERT | 60.06 | 56.80 | 72.41 | 79.56 | 73.93 | 78.62 | 83.93 |
25
  | RoBERTa | 60.64 | 58.06 | 74.05 | 81.24 | 76.00 | 87.50 | 84.50 |
26
  | RoFormer | 60.91 | 57.54 | 73.52 | 80.92 | 76.07 | 86.84 | 84.63 |
27
- | RoFormer-pytorch(本仓库代码) | 60.60 | 57.51 | 74.44 | 80.79 | 75.67 | 86.84 | 84.77 |
28
  | RoFormerV2<sup>*</sup> | 60.87 | 56.54 | 72.75 | 80.34 | 75.36 | 80.92 | 84.67 |
29
- | RoFormerV2<sup>*</sup>-pytorch(本仓库代码) | **62.87** | 59.03 | **76.20** | 80.85 | 79.73 | 87.82 | **91.87** |
30
  | GAU-α | 61.41 | 57.76 | 74.17 | 81.82 | 75.86 | 79.93 | 85.67 |
 
 
31
  | GAU-α-pytorch(Adafactor) | 61.18 | 57.52 | 73.42 | 80.91 | 75.69 | 80.59 | 85.5 |
32
- | GAU-α-pytorch(AdamW wd0.01 warm_decay0.1) | 60.68 | 57.95 | 73.08 | 81.02 | 75.36 | 81.25 | 83.93 |
33
- | RoFormerV2<sup>*</sup>-large-pytorch(本仓库代码) | 61.75 | **59.21** | 76.14 | **82.35** | **81.73** | **91.45** | 91.5 |
 
34
 
35
 
36
- ### CLUE-1.0-test榜单分类任务结果,base版本。
37
 
38
  | | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl |
39
  | :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
40
  | RoFormer-pytorch(本仓库代码) | 59.54 | 57.34 | 74.46 | 80.23 | 73.67 | 80.69 | 84.57 |
41
- | RoFormerV2<sup>*</sup>-pytorch(本仓库代码) | **63.15** | 58.24 | 75.42 | 80.59 | 74.17 | 83.79 | 83.73 |
42
  | GAU-α-pytorch(Adafactor) | 61.38 | 57.08 | 74.05 | 80.37 | 73.53 | 74.83 | **85.6** |
43
- | GAU-α-pytorch(AdamW wd0.01 warm_decay0.1) | 60.54 | 57.67 | 72.44 | 80.32 | 72.97 | 76.55 | 84.13 |
44
- | RoFormerV2-large<sup>*</sup>-pytorch(本仓库代码) | 61.85 | **59.13** | **76.38** | **80.97** | **76.23** | **85.86** | 84.33 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
 
46
  ## pytorch & tf2.0使用
47
  ```python
17
  - pip install roformer==0.4.3
18
 
19
  ## 评测对比
20
+ ### CLUE-dev榜单分类任务结果,base+large版本。
21
 
22
  | | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl |
23
  | :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
24
  | BERT | 60.06 | 56.80 | 72.41 | 79.56 | 73.93 | 78.62 | 83.93 |
25
  | RoBERTa | 60.64 | 58.06 | 74.05 | 81.24 | 76.00 | 87.50 | 84.50 |
26
  | RoFormer | 60.91 | 57.54 | 73.52 | 80.92 | 76.07 | 86.84 | 84.63 |
 
27
  | RoFormerV2<sup>*</sup> | 60.87 | 56.54 | 72.75 | 80.34 | 75.36 | 80.92 | 84.67 |
 
28
  | GAU-α | 61.41 | 57.76 | 74.17 | 81.82 | 75.86 | 79.93 | 85.67 |
29
+ | RoFormer-pytorch(本仓库代码) | 60.60 | 57.51 | 74.44 | 80.79 | 75.67 | 86.84 | 84.77 |
30
+ | RoFormerV2-pytorch(本仓库代码) | **62.87** | 59.03 | **76.20** | 80.85 | 79.73 | 87.82 | **91.87** |
31
  | GAU-α-pytorch(Adafactor) | 61.18 | 57.52 | 73.42 | 80.91 | 75.69 | 80.59 | 85.5 |
32
+ | GAU-α-pytorch(AdamW wd0.01 warmup0.1) | 60.68 | 57.95 | 73.08 | 81.02 | 75.36 | 81.25 | 83.93 |
33
+ | RoFormerV2-large-pytorch(本仓库代码) | 61.75 | **59.21** | 76.14 | 82.35 | **81.73** | **91.45** | 91.5 |
34
+ | Chinesebert-large-pytorch | 61.25 | 58.67 | 74.70 | **82.65** | 79.63 | 87.83 | 84.97 |
35
 
36
 
37
+ ### CLUE-1.0-test榜单分类任务结果,base+large版本。
38
 
39
  | | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl |
40
  | :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
41
  | RoFormer-pytorch(本仓库代码) | 59.54 | 57.34 | 74.46 | 80.23 | 73.67 | 80.69 | 84.57 |
42
+ | RoFormerV2-pytorch(本仓库代码) | **63.15** | 58.24 | 75.42 | 80.59 | 74.17 | 83.79 | 83.73 |
43
  | GAU-α-pytorch(Adafactor) | 61.38 | 57.08 | 74.05 | 80.37 | 73.53 | 74.83 | **85.6** |
44
+ | GAU-α-pytorch(AdamW wd0.01 warmup0.1) | 60.54 | 57.67 | 72.44 | 80.32 | 72.97 | 76.55 | 84.13 |
45
+ | RoFormerV2-large-pytorch(本仓库代码) | 61.85 | **59.13** | **76.38** | 80.97 | 76.23 | **85.86** | 84.33 |
46
+ | Chinesebert-large-pytorch | 61.54 | 58.57 | 74.8 | **81.94** | **76.93** | 79.66 | 85.1 |
47
+
48
+ ### 注:
49
+ - 其中RoFormerV2<sup>*</sup>表示的是未进行多任务学习的RoFormerV2模型,该模型苏神并未开源,感谢苏神的提醒。
50
+ - 其中不带有pytorch后缀结果都是从[GAU-alpha](https://github.com/ZhuiyiTechnology/GAU-alpha)仓库复制过来的。
51
+ - 其中带有pytorch后缀的结果都是自己训练得出的。
52
+ - 苏神代码中拿了cls标签后直接进行了分类,而本仓库使用了如下的分类头,多了2个dropout,1个dense,1个relu激活。
53
+
54
+ ```python
55
+ class RoFormerClassificationHead(nn.Module):
56
+ def __init__(self, config):
57
+ super().__init__()
58
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
59
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
60
+ self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
61
+
62
+ self.config = config
63
+
64
+ def forward(self, features, **kwargs):
65
+ x = features[:, 0, :] # take <s> token (equiv. to [CLS])
66
+ x = self.dropout(x)
67
+ x = self.dense(x)
68
+ x = ACT2FN[self.config.hidden_act](x) # 这里是relu
69
+ x = self.dropout(x)
70
+ x = self.out_proj(x)
71
+ return x
72
+ ```
73
 
74
  ## pytorch & tf2.0使用
75
  ```python