autoevaluator
HF staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator
dee980a
metadata
language:
- en
license: apache-2.0
tags:
- question-answering
datasets:
- squad
metrics:
- squad
thumbnail: https://github.com/karanchahal/distiller/blob/master/distiller.jpg
model-index:
- name: osanseviero/distilbert-base-uncased-finetuned-squad-d5716d28
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: adversarial_qa
type: adversarial_qa
config: adversarialQA
split: train
metrics:
- type: loss
value: 4.052208423614502
name: Loss
verified: true
verifyToken: >-
eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTE0YjhkNjk3ZDk0MzMyZmQyOTYxNjI3OGNmMjkwZGVjZjA1MWI3OGVmOWM4MmI5YjUwNTUxOTkwNzY2NmU5MyIsInZlcnNpb24iOjF9.p3QHHdy3BLWWpweJ0iQFBu0vFCPTylofdqTAnBhgjKvU-wATwSUqO7z9EYosTZSfIXQbnGTA2HLBey2j9sENDQ
DistilBERT with a second step of distillation
Model description
This model replicates the "DistilBERT (D)" model from Table 2 of the DistilBERT paper. In this approach, a DistilBERT student is fine-tuned on SQuAD v1.1, but with a BERT model (also fine-tuned on SQuAD v1.1) acting as a teacher for a second step of task-specific distillation.
In this version, the following pre-trained models were used:
- Student:
distilbert-base-uncased
- Teacher:
lewtun/bert-base-uncased-finetuned-squad-v1
Training data
This model was trained on the SQuAD v1.1 dataset which can be obtained from the datasets
library as follows:
from datasets import load_dataset
squad = load_dataset('squad')
Training procedure
Eval results
Exact Match | F1 | |
---|---|---|
DistilBERT paper | 79.1 | 86.9 |
Ours | 78.4 | 86.5 |
The scores were calculated using the squad
metric from datasets
.
BibTeX entry and citation info
@misc{sanh2020distilbert,
title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
year={2020},
eprint={1910.01108},
archivePrefix={arXiv},
primaryClass={cs.CL}
}