Create README.md

README.md

@@ -0,0 +1,37 @@
+---
+language: "en"
+thumbnail: "https://github.com/junnyu"
+tags:
+- pytorch
+- electra
+- openwebtext
+license: "MIT"
+datasets:
+- openwebtext
+
+---
+# 在openwebtext数据集上训练得到的electra-small
+# 复现结果
+|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg. of Avg.|
+|---|---|---|---|---|---|---|---|---|---|
+|ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
+|**ELECTRA-Small-OWT (this)**| 55.82 |89.67|87.0|86.96|89.28|80.08|87.50|66.07|80.30|
+
+# 训练细节
+- 数据集 openwebtext
+- 训练batch_size 256
+- 学习率lr  2e-4
+- 最大句子长度max_seqlen  128
+- 训练total step  625000
+
+# 使用
+```python
+import torch
+from transformers.models.electra import ElectraModel, ElectraTokenizer
+tokenizer = ElectraTokenizer.from_pretrained("junnyu/electra_small_discriminator")
+model = ElectraModel.from_pretrained("junnyu/electra_small_discriminator")
+inputs = tokenizer("Beijing is the capital of China.", return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+    print(outputs[0].shape)
+```