racai-andrei
commited on
Commit
•
bc5520e
1
Parent(s):
8e3369a
Update README.md
Browse files
README.md
CHANGED
@@ -11,6 +11,9 @@ datasets:
|
|
11 |
|
12 |
This repository contains the uncased Romanian DistilBERT (named Distil-RoBERT-base in the paper). The teacher model used for distillation is: [readerbench/RoBERT-base](https://huggingface.co/readerbench/RoBERT-base).
|
13 |
|
|
|
|
|
|
|
14 |
## Usage
|
15 |
|
16 |
```python
|
@@ -54,3 +57,14 @@ We evaluated the model in comparison with the RoBERT-base on 5 Romanian tasks:
|
|
54 |
|--------------------------------|:----:|:----:|:---:|:----:|:---:|:--:|:---:|
|
55 |
| RoBERT-base | 98.02 | 97.15 | 85.14 | 98.30 | 79.40 | 96.07 | 81.18 |
|
56 |
| distilbert-base-romanian-uncased | 97.12 | 95.79 | 83.11 | 98.01 | 79.58 | 96.11 | 79.80 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
11 |
|
12 |
This repository contains the uncased Romanian DistilBERT (named Distil-RoBERT-base in the paper). The teacher model used for distillation is: [readerbench/RoBERT-base](https://huggingface.co/readerbench/RoBERT-base).
|
13 |
|
14 |
+
The model was introduced in [this paper](https://arxiv.org/abs/2112.12650). The adjacent code can be found
|
15 |
+
[here](https://github.com/racai-ai/Romanian-DistilBERT).
|
16 |
+
|
17 |
## Usage
|
18 |
|
19 |
```python
|
|
|
57 |
|--------------------------------|:----:|:----:|:---:|:----:|:---:|:--:|:---:|
|
58 |
| RoBERT-base | 98.02 | 97.15 | 85.14 | 98.30 | 79.40 | 96.07 | 81.18 |
|
59 |
| distilbert-base-romanian-uncased | 97.12 | 95.79 | 83.11 | 98.01 | 79.58 | 96.11 | 79.80 |
|
60 |
+
|
61 |
+
### BibTeX entry and citation info
|
62 |
+
```bibtex
|
63 |
+
@article{avram2021distilling,
|
64 |
+
title={Distilling the Knowledge of Romanian BERTs Using Multiple Teachers},
|
65 |
+
author={Andrei-Marius Avram and Darius Catrina and Dumitru-Clementin Cercel and Mihai Dascălu and Traian Rebedea and Vasile Păiş and Dan Tufiş},
|
66 |
+
journal={ArXiv},
|
67 |
+
year={2021},
|
68 |
+
volume={abs/2112.12650}
|
69 |
+
}
|
70 |
+
```
|