|
--- |
|
language: en |
|
tags: |
|
- deberta |
|
- deberta-v3 |
|
- mdeberta |
|
thumbnail: https://huggingface.co/front/thumbnails/microsoft.png |
|
license: mit |
|
--- |
|
|
|
## DeBERTa: Decoding-enhanced BERT with Disentangled Attention |
|
|
|
[DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. |
|
|
|
Please check the [official repository](https://github.com/microsoft/DeBERTa) for more details and updates. |
|
|
|
In DeBERTa V3, we replaced the MLM objective with the RTD(Replaced Token Detection) objective introduced by ELECTRA for pre-training, as well as some innovations to be introduced in our upcoming paper. Compared to DeBERTa-V2, our V3 version significantly improves the model performance in downstream tasks. You can find a simple introduction about the model from the appendix A11 in our original [paper](https://arxiv.org/abs/2006.03654), but we will provide more details in a separate write-up. |
|
|
|
mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was trained with CC100 multilingual data. |
|
|
|
The mDeBERTa V3 base model comes with 12 layers and a hidden size of 768. Its total parameter number is 280M since we use a vocabulary containing 250K tokens which introduce 190M parameters in the Embedding layer. This model was trained using the 2.5T CC100 data as XLM-R. |
|
|
|
|
|
#### Fine-tuning on NLU tasks |
|
|
|
We present the dev results on XNLI with zero-shot crosslingual transfer setting, i.e. training with english data only, test with other languages. |
|
|
|
| Model | en | fr| es | de | el | bg | ru |tr |ar |vi | th | zh | hi | sw | ur | avg | |
|
|-------------------|----|----|---- |-- |-- |-- | -- |-- |-- |-- | -- | -- | -- | -- | -- | ----| |
|
| XLM-R-base |85.8|79.7|80.7 |78.7 |77.5 |79.6 |78.1 |74.2 |73.8 |76.5 |74.6 |76.7| 72.4| 66.5| 68.3|75.6 | |
|
| mDeBERTa-base |88.2|82.6|84.4 |82.7 |82.3 |82.4 |80.8 |79.5 |78.5 |78.1 |76.4 |79.5| 75.9| 73.9| 72.4|79.8 +/- 0.2| |
|
|
|
#### Fine-tuning with HF transformers |
|
|
|
```bash |
|
#!/bin/bash |
|
|
|
cd transformers/examples/pytorch/text-classification/ |
|
|
|
pip install datasets |
|
|
|
output_dir="ds_results" |
|
|
|
num_gpus=8 |
|
|
|
batch_size=4 |
|
|
|
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \ |
|
run_xnli.py \ |
|
--model_name_or_path microsoft/deberta-v3-base \ |
|
--task_name $TASK_NAME \ |
|
--do_train \ |
|
--do_eval \ |
|
--train_language en \ |
|
--language en \ |
|
--evaluation_strategy steps \ |
|
--max_seq_length 256 \ |
|
--warmup_steps 3000 \ |
|
--per_device_train_batch_size ${batch_size} \ |
|
--learning_rate 2e-5 \ |
|
--num_train_epochs 6 \ |
|
--output_dir $output_dir \ |
|
--overwrite_output_dir \ |
|
--logging_steps 1000 \ |
|
--logging_dir $output_dir |
|
|
|
``` |
|
|
|
### Citation |
|
|
|
If you find DeBERTa useful for your work, please cite the following paper: |
|
|
|
``` latex |
|
@inproceedings{ |
|
he2021deberta, |
|
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION}, |
|
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen}, |
|
booktitle={International Conference on Learning Representations}, |
|
year={2021}, |
|
url={https://openreview.net/forum?id=XPZIaotutsD} |
|
} |
|
``` |
|
|
