Add mDeBERTa base model

README.md

@@ -0,0 +1,82 @@
+---
+language: en
+tags: 
+  - deberta
+  - deberta-v3
+	- mdeberta
+thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
+license: mit
+---
+
+## DeBERTa: Decoding-enhanced BERT with Disentangled Attention
+
+[DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. 
+
+Please check the [official repository](https://github.com/microsoft/DeBERTa) for more details and updates.
+
+In DeBERTa V3, we replaced the MLM objective with the RTD(Replaced Token Detection) objective introduced by ELECTRA for pre-training, as well as some innovations to be introduced in our upcoming paper. Compared to DeBERTa-V2,  our V3 version significantly improves the model performance in downstream tasks.  You can find a simple introduction about the model from the appendix A11 in our original [paper](https://arxiv.org/abs/2006.03654),  but we will provide more details in a separate write-up.
+
+mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was trained with CC100 multilingual data.
+
+The mDeBERTa V3 base model comes with 12 layers and a hidden size of 768. Its total parameter number is 280M since we use a vocabulary containing 250K tokens which introduce 190M parameters in the Embedding layer.  This model was trained using the 2.5T CC100 data as XLM-R.
+
+
+#### Fine-tuning on NLU tasks
+
+We present the dev results on XNLI with zero-shot crosslingual transfer setting, i.e. training with english data only, test with other languages.
+
+| Model             | en |  fr| es  | de  | el  | bg  | ru  |tr   |ar   |vi   | th  | zh | hi  | sw  | ur  | avg |
+|-------------------|----|----|---- |--   |--   |--   | --  |--   |--   |--   | --  | -- | --  | --  | --  | ----|
+| XLM-R-base        |85.8|79.7|80.7 |78.7 |77.5 |79.6 |78.1 |74.2 |73.8 |76.5 |74.6 |76.7| 72.4| 66.5| 68.3|75.6 |
+| mDeBERTa-base     |88.2|82.6|84.4 |82.7 |82.3 |82.4 |80.8 |79.5 |78.5 |78.1 |76.4 |79.5| 75.9| 73.9| 72.4|79.8 +/- 0.2|
+
+#### Fine-tuning with HF transformers
+
+```bash
+#!/bin/bash
+
+cd transformers/examples/pytorch/text-classification/
+
+pip install datasets
+
+output_dir="ds_results"
+
+num_gpus=8
+
+batch_size=4
+
+python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
+  run_xnli.py \
+  --model_name_or_path microsoft/deberta-v3-base \
+  --task_name $TASK_NAME \
+  --do_train \
+  --do_eval \
+	--train_language en \
+	--language en \
+  --evaluation_strategy steps \
+  --max_seq_length 256 \
+  --warmup_steps 3000 \
+  --per_device_train_batch_size ${batch_size} \
+  --learning_rate 2e-5 \
+  --num_train_epochs 6 \
+  --output_dir $output_dir \
+  --overwrite_output_dir \
+  --logging_steps 1000 \
+  --logging_dir $output_dir
+
+```
+
+### Citation
+
+If you find DeBERTa useful for your work, please cite the following paper:
+
+``` latex
+@inproceedings{
+he2021deberta,
+title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
+author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
+booktitle={International Conference on Learning Representations},
+year={2021},
+url={https://openreview.net/forum?id=XPZIaotutsD}
+}
+```

config.json

@@ -0,0 +1,22 @@
+{
+		"model_type": "deberta-v2",
+		"attention_probs_dropout_prob": 0.1,
+		"hidden_act": "gelu",
+		"hidden_dropout_prob": 0.1,
+		"hidden_size": 768,
+		"initializer_range": 0.02,
+		"intermediate_size": 3072,
+		"max_position_embeddings": 512,
+		"relative_attention": true,
+		"position_buckets": 256,
+		"norm_rel_ebd": "layer_norm",
+		"share_att_key": true,
+		"pos_att_type": "p2c|c2p",
+		"layer_norm_eps": 1e-7,
+		"max_relative_positions": -1,
+		"position_biased_input": false,
+		"num_attention_heads": 12,
+		"num_hidden_layers": 12,
+		"type_vocab_size": 0,
+		"vocab_size": 251000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05c748186a22f523505099ce137f90dd4e55f875a4035c11350aaa125932230c
+size 560166373

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13c8d666d62a7bc4ac8f040aab68e942c861f93303156cc28f5c7e885d86d6e3
+size 4305025

tokenizer_config.json

@@ -0,0 +1,4 @@
+{
+  "do_lower_case": false,
+  "vocab_type": "spm"
+}