julien-c HF staff commited on
Commit
04cd60a
1 Parent(s): 35f404f

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/huawei-noah/DynaBERT_MNLI/README.md

Files changed (1) hide show
  1. README.md +20 -0
README.md ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## DynaBERT: Dynamic BERT with Adaptive Width and Depth
2
+
3
+ * DynaBERT can flexibly adjust the size and latency by selecting adaptive width and depth, and
4
+ the subnetworks of it have competitive performances as other similar-sized compressed models.
5
+ The training process of DynaBERT includes first training a width-adaptive BERT and then
6
+ allowing both adaptive width and depth using knowledge distillation.
7
+
8
+ * This code is modified based on the repository developed by Hugging Face: [Transformers v2.1.1](https://github.com/huggingface/transformers/tree/v2.1.1), and is released in [GitHub](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT).
9
+
10
+ ### Reference
11
+ Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu.
12
+ [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://arxiv.org/abs/2004.04037).
13
+ ```
14
+ @inproceedings{hou2020dynabert,
15
+ title = {DynaBERT: Dynamic BERT with Adaptive Width and Depth},
16
+ author = {Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu},
17
+ booktitle = {Advances in Neural Information Processing Systems},
18
+ year = {2020}
19
+ }
20
+ ```