File size: 3,160 Bytes
0881747 89db1c3 0881747 89db1c3 0881747 89db1c3 ae2b9e9 0881747 3733c6d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
---
widget:
- context: While deep and large pre-trained models are the state-of-the-art for various
natural language processing tasks, their huge size poses significant challenges
for practical uses in resource constrained settings. Recent works in knowledge
distillation propose task-agnostic as well as task-specific methods to compress
these models, with task-specific ones often yielding higher compression rate.
In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
that leverages the advantage of task-specific methods for learning a small universal
model that can be applied to arbitrary tasks and languages. To this end, we study
the transferability of several source tasks, augmentation resources and model
architecture for distillation. We evaluate our model performance on multiple tasks,
including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
question answering dataset and a massive multi-lingual NER dataset with 41 languages.
example_title: xtremedistil q1
text: What is XtremeDistil?
- context: While deep and large pre-trained models are the state-of-the-art for various
natural language processing tasks, their huge size poses significant challenges
for practical uses in resource constrained settings. Recent works in knowledge
distillation propose task-agnostic as well as task-specific methods to compress
these models, with task-specific ones often yielding higher compression rate.
In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
that leverages the advantage of task-specific methods for learning a small universal
model that can be applied to arbitrary tasks and languages. To this end, we study
the transferability of several source tasks, augmentation resources and model
architecture for distillation. We evaluate our model performance on multiple tasks,
including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
question answering dataset and a massive multi-lingual NER dataset with 41 languages.
example_title: xtremedistil q2
text: On what is the model validated?
datasets:
- squad_v2
metrics:
- f1
- exact
tags:
- question-answering
model-index:
- name: nbroad/xdistil-l12-h384-squad2
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_v2
type: squad_v2
config: squad_v2
split: validation
metrics:
- name: Exact Match
type: exact_match
value: 75.4591
verified: true
- name: F1
type: f1
value: 79.3321
verified: true
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- name: Exact Match
type: exact_match
value: 81.8604
verified: true
- name: F1
type: f1
value: 89.6654
verified: true
---
xtremedistil-l12-h384 trained on SQuAD 2.0
"eval_exact": 75.45691906005221
"eval_f1": 79.32502968532793 |