File size: 1,441 Bytes
d988483 debc7b9 d988483 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
---
language:
- en
thumbnail: https://github.com/karanchahal/distiller/blob/master/distiller.jpg
tags:
- question-answeringlicense: Apache 2.0
datasets:
- squad
metrics:
- squad
---
# DistilBERT with a second step of distillation
## Model description
This model replicates the "DistilBERT (D)" model from Table 2 of the [DistilBERT paper](https://arxiv.org/pdf/1910.01108.pdf). In this approach, a DistilBERT student is fine-tuned on SQuAD v1.1, while a fine-tuned BERT model acts as a teacher for a second step of task-specific distillation.In this version, the following pre-trained models were used:* Student: `distilbert-base-uncased`* Teacher: `maroo93/squad1.1`## Intended uses & limitations
#### How to use
## Training data
This model was trained on the SQuAD v1.1 dataset which can be obtained from the `datasets` library as follows:```pythonfrom datasets import load_datasetsquad = load_dataset('squad')```
## Training procedure
## Eval results
Exact Match | F1
-------|---------
78.05 | 86.09
The score was calculated using the `squad` metric from `datasets`.
### BibTeX entry and citation info
```bibtex
@misc{sanh2020distilbert,
title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
year={2020},
eprint={1910.01108},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
``` |