Pengcheng He commited on
Commit
ff3ef10
1 Parent(s): acded10

Add DeBERTa v3 large model

Browse files
Files changed (5) hide show
  1. README.md +42 -0
  2. config.json +22 -0
  3. pytorch.model.bin +3 -0
  4. spm.model +3 -0
  5. tokenizer_config.json +4 -0
README.md ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - deberta
5
+ - deberta-v3
6
+ thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
7
+ license: mit
8
+ ---
9
+
10
+ ## DeBERTa: Decoding-enhanced BERT with Disentangled Attention
11
+
12
+ [DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.
13
+
14
+ Please check the [official repository](https://github.com/microsoft/DeBERTa) for more details and updates.
15
+
16
+ This is the DeBERTa V3 large model with 24 layers, 1024 hidden size. Total parameters is 418M while Embedding layer takes about 131M due to the usage of 128k vocabulary. It's trained with 160GB data.
17
+ For more details of our V3 model, please check appendix A11 in our paper.
18
+
19
+ #### Fine-tuning on NLU tasks
20
+
21
+ We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.
22
+
23
+ | Model | SQuAD 1.1 | SQuAD 2.0 | MNLI-m |
24
+ |-------------------|-----------|-----------|--------|
25
+ | RoBERTa-base | 91.5/84.6 | 83.7/80.5 | 87.6 |
26
+ | XLNet-base | -/- | -/80.2 | 86.8 |
27
+ | **DeBERTa-v3-large** | -/- | 91.5/89.0 | 92.0 |
28
+
29
+ ### Citation
30
+
31
+ If you find DeBERTa useful for your work, please cite the following paper:
32
+
33
+ ``` latex
34
+ @inproceedings{
35
+ he2021deberta,
36
+ title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
37
+ author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
38
+ booktitle={International Conference on Learning Representations},
39
+ year={2021},
40
+ url={https://openreview.net/forum?id=XPZIaotutsD}
41
+ }
42
+ ```
config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "deberta-v2",
3
+ "attention_probs_dropout_prob": 0.1,
4
+ "hidden_act": "gelu",
5
+ "hidden_dropout_prob": 0.1,
6
+ "hidden_size": 1024,
7
+ "initializer_range": 0.02,
8
+ "intermediate_size": 4096,
9
+ "max_position_embeddings": 512,
10
+ "relative_attention": true,
11
+ "position_buckets": 256,
12
+ "norm_rel_ebd": "layer_norm",
13
+ "share_att_key": true,
14
+ "pos_att_type": "p2c|c2p",
15
+ "layer_norm_eps": 1e-7,
16
+ "max_relative_positions": -1,
17
+ "position_biased_input": false,
18
+ "num_attention_heads": 16,
19
+ "num_hidden_layers": 24,
20
+ "type_vocab_size": 0,
21
+ "vocab_size": 128100
22
+ }
pytorch.model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a5c566fe4de56ca94f7f9ebb607b21e872c67f73019ceb68385cb2e6ef52b04
3
+ size 875777713
spm.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
3
+ size 2464616
tokenizer_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "do_lower_case": false,
3
+ "vocab_type": "spm"
4
+ }