Deep Incubation

This repository contains the pre-trained models for Deep Incubation.

Title: Deep Incubation: Training Large Models by Divide-and-Conquering
Authors:  Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, Gao Huang (Corresponding Author)
Institute: Tsinghua University and Beijing Academy of Artificial Intelligence (BAAI)
Publish:   arXiv preprint (arXiv 2212.04129)
Contact:  nzl22 at mails dot tsinghua dot edu dot cn

Models

model	image size	#param.	top-1 acc.	checkpoint
ViT-B	224x224	87M	82.4%	🤗 HF link
ViT-B	384x384	87M	84.2%	🤗 HF link
ViT-L	224x224	304M	83.9%	🤗 HF link
ViT-L	384x384	304M	85.3%	🤗 HF link
ViT-H	224x224	632M	84.3%	🤗 HF link
ViT-H	392x392	632M	85.6%	🤗 HF link

Data Preparation

The ImageNet dataset should be prepared as follows:

data
├── train
│   ├── folder 1 (class 1)
│   ├── folder 2 (class 1)
│   ├── ...
├── val
│   ├── folder 1 (class 1)
│   ├── folder 2 (class 1)
│   ├── ...

Citation

If you find our work helpful, please star🌟 this repo and cite📑 our paper. Thanks for your support!

@article{Ni2022Incub,
  title={Deep Incubation: Training Large Models by Divide-and-Conquering},
  author={Ni, Zanlin and Wang, Yulin and Yu, Jiangwei and Jiang, Haojun and Cao, Yue and Huang, Gao},
  journal={arXiv preprint arXiv:2212.04129},
  year={2022}
}

Acknowledgements

Our implementation is mainly based on deit. We thank to their clean codebase.

Contact

If you have any questions or concerns, please send mail to nzl22@mails.tsinghua.edu.cn.