Hugging Face's logo

junnyu
/
electra_small_discriminator

Transformers
PyTorch
English
electra
pretraining
Inference Endpoints
Model card Files Community
junnyu commited on
Commit
0c41186
•
1 Parent(s): 06da743

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -4
README.md CHANGED
@@ -10,21 +10,22 @@ datasets:
10
  - openwebtext
11
 
12
  ---
13
- # 在openwebtext数据集上训练得到的electra-small
14
- # 复现结果
 
15
  |Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
16
  |---|---|---|---|---|---|---|---|---|---|
17
  |ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
18
  |**ELECTRA-Small-OWT (this)**| 55.82 |89.67|87.0|86.96|89.28|80.08|87.50|66.07|80.30|
19
 
20
- # 训练细节
21
  - 数据集 openwebtext
22
  - 训练batch_size 256
23
  - 学习率lr 2e-4
24
  - 最大句子长度max_seqlen 128
25
  - 训练total step 625000
26
 
27
- # 使用
28
  ```python
29
  import torch
30
  from transformers.models.electra import ElectraModel, ElectraTokenizer
 
10
  - openwebtext
11
 
12
  ---
13
+ # 一、 个人在openwebtext数据集上训练得到的electra-small模型
14
+
15
+ # 二、 复现结果
16
  |Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
17
  |---|---|---|---|---|---|---|---|---|---|
18
  |ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
19
  |**ELECTRA-Small-OWT (this)**| 55.82 |89.67|87.0|86.96|89.28|80.08|87.50|66.07|80.30|
20
 
21
+ # 三、 训练细节
22
  - 数据集 openwebtext
23
  - 训练batch_size 256
24
  - 学习率lr 2e-4
25
  - 最大句子长度max_seqlen 128
26
  - 训练total step 625000
27
 
28
+ # 四、 使用
29
  ```python
30
  import torch
31
  from transformers.models.electra import ElectraModel, ElectraTokenizer