update readme
Browse files
README.md
CHANGED
@@ -5,20 +5,17 @@ license: cc-by-4.0
|
|
5 |
# DictBERT model (uncased)
|
6 |
|
7 |
-- This is the model checkpoint of our [ACL 2022](https://www.2022.aclweb.org/) paper "*Dict-BERT: Enhancing Language Model Pre-training with Dictionary*" [\[PDF\]](https://aclanthology.org/2022.findings-acl.150/).
|
8 |
-
In this paper, we propose DictBERT, a novel pre-trained language model by leveraging rare word definitions in English dictionaries (e.g., Wiktionary). DictBERT is based on the BERT architecture, trained under the same setting as BERT. Please refer more details in our paper.
|
9 |
|
10 |
## Evaluation results
|
11 |
|
12 |
-
When fine-tuned DictBERT on
|
13 |
-
|
14 |
-
CoLA is evaluated by matthews; STS-B is evaluated by pearson; others are evaluated by accuracy.
|
15 |
|
16 |
| | MNLI | QNLI | QQP | SST-2 | CoLA | MRPC | RTE | STS-B | Average |
|
17 |
|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
|
18 |
| BERT(HF) | 84.12 | 90.69 | 90.75 | 92.52 | 58.89 | 86.17 | 68.67 | 89.39 | 82.65 |
|
19 |
| DictBERT | 84.36 | 91.02 | 90.78 | 92.43 | 61.81 | 87.25 | 72.90 | 89.40 | 83.74 |
|
20 |
|
21 |
-
|
22 |
HF: huggingface checkpoint for BERT-base uncased
|
23 |
|
24 |
### BibTeX entry and citation info
|
|
|
5 |
# DictBERT model (uncased)
|
6 |
|
7 |
-- This is the model checkpoint of our [ACL 2022](https://www.2022.aclweb.org/) paper "*Dict-BERT: Enhancing Language Model Pre-training with Dictionary*" [\[PDF\]](https://aclanthology.org/2022.findings-acl.150/).
|
8 |
+
In this paper, we propose DictBERT, which is a novel pre-trained language model by leveraging rare word definitions in English dictionaries (e.g., Wiktionary). DictBERT is based on the BERT architecture, trained under the same setting as BERT. Please refer more details in our paper.
|
9 |
|
10 |
## Evaluation results
|
11 |
|
12 |
+
When fine-tuned BERT and our DictBERT on GLEU benchmarks tasks. CoLA is evaluated by matthews, STS-B is evaluated by pearson, and others are evaluated by accuracy. DictBERT model achieves the following results:
|
|
|
|
|
13 |
|
14 |
| | MNLI | QNLI | QQP | SST-2 | CoLA | MRPC | RTE | STS-B | Average |
|
15 |
|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
|
16 |
| BERT(HF) | 84.12 | 90.69 | 90.75 | 92.52 | 58.89 | 86.17 | 68.67 | 89.39 | 82.65 |
|
17 |
| DictBERT | 84.36 | 91.02 | 90.78 | 92.43 | 61.81 | 87.25 | 72.90 | 89.40 | 83.74 |
|
18 |
|
|
|
19 |
HF: huggingface checkpoint for BERT-base uncased
|
20 |
|
21 |
### BibTeX entry and citation info
|