update

README.md

@@ -0,0 +1,38 @@
+---
+language: "en"
+thumbnail: "https://github.com/junnyu"
+tags:
+- pytorch
+- electra
+license: "MIT"
+datasets:
+- openwebtext
+
+---
+# 一、 个人在openwebtext数据集上训练得到的electra-small模型
+
+# 二、 复现结果(dev dataset)
+|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
+|---|---|---|---|---|---|---|---|---|---|
+|ELECTRA-Small-OWT(original)|56.8|88.3|87.4|86.8|88.3|78.9|87.9|68.5|80.36|
+|**ELECTRA-Small-OWT (this)**| 55.82 |89.67|87.0|86.96|89.28|80.08|87.50|66.07|80.30|
+
+# 三、 训练细节
+- 数据集 openwebtext
+- 训练batch_size 256
+- 学习率lr  5e-4
+- 最大句子长度max_seqlen  128
+- 训练total step  62.5W
+- GPU RTX3090
+- 训练时间总共耗费2.5天
+
+# 四、 使用
+```python
+import torch
+from transformers.models.electra import ElectraModel, ElectraTokenizer
+tokenizer = ElectraTokenizer.from_pretrained("junnyu/electra_small_generator")
+model = ElectraModel.from_pretrained("junnyu/electra_small_generator")
+inputs = tokenizer("Beijing is the capital of [MASK].", return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+```

config.json

@@ -0,0 +1,21 @@
+{
+    "architectures": [
+        "ElectraForMaskedLM"
+    ],
+    "attention_probs_dropout_prob": 0.1,
+    "embedding_size": 128,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 64,
+    "initializer_range": 0.02,
+    "intermediate_size": 256,
+    "layer_norm_eps": 1e-12,
+    "max_position_embeddings": 512,
+    "model_type": "electra",
+    "num_attention_heads": 4,
+    "num_hidden_layers": 12,
+    "pad_token_id": 0,
+    "position_embedding_type": "absolute",
+    "type_vocab_size": 2,
+    "vocab_size": 30522
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18acdae4d103a64f0b70d551f0638fdeb846f15519bff584f898508bd4acb532
+size 54273423

tokenizer_config.json

@@ -0,0 +1,4 @@
+{
+  "do_lower_case": true
+}
+