---
widget:
- context: While deep and large pre-trained models are the state-of-the-art for various
    natural language processing tasks, their huge size poses significant challenges
    for practical uses in resource constrained settings. Recent works in knowledge
    distillation propose task-agnostic as well as task-specific methods to compress
    these models, with task-specific ones often yielding higher compression rate.
    In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
    that leverages the advantage of task-specific methods for learning a small universal
    model that can be applied to arbitrary tasks and languages. To this end, we study
    the transferability of several source tasks, augmentation resources and model
    architecture for distillation. We evaluate our model performance on multiple tasks,
    including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
    question answering dataset and a massive multi-lingual NER dataset with 41 languages.
  example_title: xtremedistil q1
  text: What is XtremeDistil?
- context: While deep and large pre-trained models are the state-of-the-art for various
    natural language processing tasks, their huge size poses significant challenges
    for practical uses in resource constrained settings. Recent works in knowledge
    distillation propose task-agnostic as well as task-specific methods to compress
    these models, with task-specific ones often yielding higher compression rate.
    In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
    that leverages the advantage of task-specific methods for learning a small universal
    model that can be applied to arbitrary tasks and languages. To this end, we study
    the transferability of several source tasks, augmentation resources and model
    architecture for distillation. We evaluate our model performance on multiple tasks,
    including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
    question answering dataset and a massive multi-lingual NER dataset with 41 languages.
  example_title: xtremedistil q2
  text: On what is the model validated?
datasets:
- squad_v2
metrics:
- f1
- exact
tags:
- question-answering
model-index:
- name: nbroad/xdistil-l12-h384-squad2
  results:
  - task:
      type: question-answering
      name: Question Answering
    dataset:
      name: squad_v2
      type: squad_v2
      config: squad_v2
      split: validation
    metrics:
    - name: Exact Match
      type: exact_match
      value: 75.4591
      verified: true
    - name: F1
      type: f1
      value: 79.3321
      verified: true
  - task:
      type: question-answering
      name: Question Answering
    dataset:
      name: squad
      type: squad
      config: plain_text
      split: validation
    metrics:
    - name: Exact Match
      type: exact_match
      value: 81.8604
      verified: true
    - name: F1
      type: f1
      value: 89.6654
      verified: true
---

xtremedistil-l12-h384 trained on SQuAD 2.0

"eval_exact": 75.45691906005221  
"eval_f1": 79.32502968532793