File size: 984 Bytes
072e3cd b3e6c62 072e3cd 905d956 b3e6c62 072e3cd 30c5aa2 072e3cd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
language: en
thumbnail: https://github.com/junnyu
tags:
- pytorch
- electra
- masked-lm
license: mit
datasets:
- openwebtext
---
# 一、 个人在openwebtext数据集上训练得到的electra-small模型
# 二、 复现结果(dev dataset)
|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
|---|---|---|---|---|---|---|---|---|---|
|ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
|**ELECTRA-Small-OWT (this)**| 55.82 |89.67|87.0|86.96|89.28|80.08|87.50|66.07|80.30|
# 三、 训练细节
- 数据集 openwebtext
- 训练batch_size 256
- 学习率lr 5e-4
- 最大句子长度max_seqlen 128
- 训练total step 62.5W
- GPU RTX3090
- 训练时间总共耗费2.5天
# 四、 使用
```python
from transformers import pipeline
fill_mask = pipeline(
"fill-mask",
model="junnyu/electra_small_generator",
tokenizer="junnyu/electra_small_generator"
)
print(
fill_mask("HuggingFace is creating a [MASK] that the community uses to solve NLP tasks.")
)
``` |