|
--- |
|
language: |
|
- en |
|
thumbnail: https://github.com/karanchahal/distiller/blob/master/distiller.jpg |
|
tags: |
|
- question-answeringlicense: Apache 2.0 |
|
datasets: |
|
- squad |
|
metrics: |
|
- squad |
|
--- |
|
|
|
# DistilBERT with a second step of distillation |
|
|
|
## Model description |
|
|
|
This model replicates the "DistilBERT (D)" model from Table 2 of the [DistilBERT paper](https://arxiv.org/pdf/1910.01108.pdf). In this approach, a DistilBERT student is fine-tuned on SQuAD v1.1, while a fine-tuned BERT model acts as a teacher for a second step of task-specific distillation. |
|
|
|
In this version, the following pre-trained models were used: |
|
|
|
* Student: `distilbert-base-uncased` |
|
* Teacher: `maroo93/squad1.1`## Intended uses & limitations |
|
|
|
## Training data |
|
|
|
This model was trained on the SQuAD v1.1 dataset which can be obtained from the `datasets` library as follows: |
|
|
|
```python |
|
from datasets import load_dataset |
|
squad = load_dataset('squad') |
|
``` |
|
|
|
## Training procedure |
|
|
|
|
|
## Eval results |
|
|
|
Exact Match | F1 |
|
-------|--------- |
|
78.05 | 86.09 |
|
|
|
|
|
The score was calculated using the `squad` metric from `datasets`. |
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@misc{sanh2020distilbert, |
|
title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}, |
|
author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf}, |
|
year={2020}, |
|
eprint={1910.01108}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
