racai-andrei
commited on
Commit
•
568ad9d
1
Parent(s):
875f10a
Update README.md
Browse files
README.md
CHANGED
@@ -10,6 +10,9 @@ datasets:
|
|
10 |
|
11 |
This repository contains the uncased Romanian DistilBERT (named Distil-BERT-base-ro in the paper). The teacher model used for distillation is: [dumitrescustefan/bert-base-romanian-cased-v1](https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1).
|
12 |
|
|
|
|
|
|
|
13 |
## Usage
|
14 |
|
15 |
```python
|
@@ -53,3 +56,14 @@ We evaluated the model in comparison with its teacher on 5 Romanian tasks:
|
|
53 |
|--------------------------------|:----:|:----:|:---:|:----:|:---:|:--:|:---:|
|
54 |
| bert-base-romanian-cased-v1 | 98.00 | 96.46 | 85.88 | 98.07 | 79.61 | 95.58 | 79.11 |
|
55 |
| distilbert-base-romanian-cased | 97.97 | 97.08 | 83.35 | 98.40 | 83.01 | 96.31 | 80.57 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
10 |
|
11 |
This repository contains the uncased Romanian DistilBERT (named Distil-BERT-base-ro in the paper). The teacher model used for distillation is: [dumitrescustefan/bert-base-romanian-cased-v1](https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1).
|
12 |
|
13 |
+
The model was introduced in [this paper](https://arxiv.org/abs/2112.12650). The adjacent code can be found
|
14 |
+
[here](https://github.com/racai-ai/Romanian-DistilBERT).
|
15 |
+
|
16 |
## Usage
|
17 |
|
18 |
```python
|
|
|
56 |
|--------------------------------|:----:|:----:|:---:|:----:|:---:|:--:|:---:|
|
57 |
| bert-base-romanian-cased-v1 | 98.00 | 96.46 | 85.88 | 98.07 | 79.61 | 95.58 | 79.11 |
|
58 |
| distilbert-base-romanian-cased | 97.97 | 97.08 | 83.35 | 98.40 | 83.01 | 96.31 | 80.57 |
|
59 |
+
|
60 |
+
### BibTeX entry and citation info
|
61 |
+
```bibtex
|
62 |
+
@article{avram2021distilling,
|
63 |
+
title={Distilling the Knowledge of Romanian BERTs Using Multiple Teachers},
|
64 |
+
author={Andrei-Marius Avram and Darius Catrina and Dumitru-Clementin Cercel and Mihai Dascălu and Traian Rebedea and Vasile Păiş and Dan Tufiş},
|
65 |
+
journal={ArXiv},
|
66 |
+
year={2021},
|
67 |
+
volume={abs/2112.12650}
|
68 |
+
}
|
69 |
+
```
|