| # Deep Incubation | |
| This repository contains the pre-trained models for [Deep Incubation](https://arxiv.org/abs/2212.04129). | |
| > **Title**:  [**Deep Incubation: Training Large Models by Divide-and-Conquering**](https://arxiv.org/abs/2212.04129) | |
| > **Authors**: [Zanlin Ni](https://scholar.google.com/citations?user=Yibz_asAAAAJ&hl=en&oi=ao), [Yulin Wang](https://scholar.google.com/citations?hl=en&user=gBP38gcAAAAJ), Jiangwei Yu, [Haojun Jiang](https://scholar.google.com/citations?hl=en&user=ULmStp8AAAAJ), [Yue Cao](https://scholar.google.com/citations?hl=en&user=iRUO1ckAAAAJ), [Gao Huang](https://scholar.google.com/citations?user=-P9LwcgAAAAJ&hl=en&oi=ao) (Corresponding Author) | |
| > **Institute**: Tsinghua University and Beijing Academy of Artificial Intelligence (BAAI) | |
| > **Publish**: *arXiv preprint ([arXiv 2212.04129](https://arxiv.org/abs/2212.04129))* | |
| > **Contact**: nzl22 at mails dot tsinghua dot edu dot cn | |
| ## Models | |
| | model | image size | #param. | top-1 acc. | checkpoint | | |
| | ----- | ---------- | ------- | ---------- | ----------------------------------------------------------------------------------------------------------- | | |
| | ViT-B | 224x224 | 87M | 82.4% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_base.pth) | | |
| | ViT-B | 384x384 | 87M | 84.2% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_base.pth) | | |
| | ViT-L | 224x224 | 304M | 83.9% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_large.pth) | | |
| | ViT-L | 384x384 | 304M | 85.3% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_large.pth) | | |
| | ViT-H | 224x224 | 632M | 84.3% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_huge.pth) | | |
| | ViT-H | 392x392 | 632M | 85.6% | [🤗 HF link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_huge.pth) | | |
| ## Data Preparation | |
| - The ImageNet dataset should be prepared as follows: | |
| ``` | |
| data | |
| ├── train | |
| │ ├── folder 1 (class 1) | |
| │ ├── folder 2 (class 1) | |
| │ ├── ... | |
| ├── val | |
| │ ├── folder 1 (class 1) | |
| │ ├── folder 2 (class 1) | |
| │ ├── ... | |
| ``` | |
| ## Citation | |
| If you find our work helpful, please **star🌟** this repo and **cite📑** our paper. Thanks for your support! | |
| ``` | |
| @article{Ni2022Incub, | |
| title={Deep Incubation: Training Large Models by Divide-and-Conquering}, | |
| author={Ni, Zanlin and Wang, Yulin and Yu, Jiangwei and Jiang, Haojun and Cao, Yue and Huang, Gao}, | |
| journal={arXiv preprint arXiv:2212.04129}, | |
| year={2022} | |
| } | |
| ``` | |
| ## Acknowledgements | |
| Our implementation is mainly based on [deit](https://github.com/facebookresearch/deit). We thank to their clean codebase. | |
| ## Contact | |
| If you have any questions or concerns, please send mail to [nzl22@mails.tsinghua.edu.cn](mailto:nzl22@mails.tsinghua.edu.cn). | |