Pengcheng He commited on
Commit
26cca7c
1 Parent(s): d3a3d9c

Create deberta base mnli fine-tuned model

Browse files

Files changed (5) hide show
  1. README.md +37 -0
  2. bpe_encoder.bin +3 -0
  3. config.json +18 -0
  4. pytorch_model.bin +3 -0
  5. tokenizer_config.json +3 -0
README.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
3
+ license: mit
4
+ ---
5
+
6
+ ## DeBERTa: Decoding-enhanced BERT with Disentangled Attention
7
+
8
+ [DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.
9
+
10
+ Please check the [official repository](https://github.com/microsoft/DeBERTa) for more details and updates.
11
+
12
+ This model is the base DeBERTa model fine-tuned with MNLI task
13
+
14
+ #### Fine-tuning on NLU tasks
15
+
16
+ We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.
17
+
18
+ | Model | SQuAD 1.1 | SQuAD 2.0 | MNLI-m |
19
+ |-------------------|-----------|-----------|--------|
20
+ | RoBERTa-base | 91.5/84.6 | 83.7/80.5 | 87.6 |
21
+ | XLNet-Large | -/- | -/80.2 | 86.8 |
22
+ | **DeBERTa-base** | 93.1/87.2 | 86.2/83.1 | 88.8 |
23
+
24
+ ### Citation
25
+
26
+ If you find DeBERTa useful for your work, please cite the following paper:
27
+
28
+ ``` latex
29
+ @misc{he2020deberta,
30
+ title={DeBERTa: Decoding-enhanced BERT with Disentangled Attention},
31
+ author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
32
+ year={2020},
33
+ eprint={2006.03654},
34
+ archivePrefix={arXiv},
35
+ primaryClass={cs.CL}
36
+ }
37
+ ```
bpe_encoder.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e7c6f9eecb461c01e09c00656ccf3e27944b9e74bfe29e51632b13d3cd9d6c8e
3
+ size 3917897
config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "attention_probs_dropout_prob": 0.1,
3
+ "hidden_act": "gelu",
4
+ "hidden_dropout_prob": 0.1,
5
+ "hidden_size": 768,
6
+ "initializer_range": 0.02,
7
+ "intermediate_size": 3072,
8
+ "max_position_embeddings": 512,
9
+ "relative_attention": true,
10
+ "pos_att_type": "c2p|p2c",
11
+ "layer_norm_eps": 1e-7,
12
+ "max_relative_positions": -1,
13
+ "position_biased_input": false,
14
+ "num_attention_heads": 12,
15
+ "num_hidden_layers": 12,
16
+ "type_vocab_size": 0,
17
+ "vocab_size": 50265
18
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f82b3a368bf628768567c1ce03b354eea1299b87c42ded71a56e4cdd3ea0039
3
+ size 556811547
tokenizer_config.json ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ {
2
+ "do_lower_case": false
3
+ }