Repository: transformers
Issue: Community contribution: enabling `device_map="auto"` support for more vision and multimodal models
Labels: Good Second Issue

### Feature request

# Feature Request 

`transformers` models can be easily loaded across multiple devices using `device_map="auto"`. This will automatically allocate weights across available devices e.g. GPUs and offload any weights onto CPU, then disk as necessary. This is useful when doing inference with large models. 

To enable this, `_no_split_modules` has to be defined in the model's pretrained model class e.g. [like here for LLaMa](https://github.com/huggingface/transformers/blob/fadb053379b3ef24c4ec8e6d7d58555af21f58db/src/transformers/models/llama/modeling_llama.py#L793). This defines layers which should not be split across devices, and should contain as few layers as possible.

### Steps to add
* Pick a model to work on and open a PR - comment on this issue to say which model you're working on
* Define `_no_split_modules` in the PreTrainedModel subclass. Try with `_no_split_modules = []` first
* Enable testing
    * Ensure the following tests are not skipped for the model: `test_disk_offload_bin`, `test_disk_offload_safetensors`, `test_cpu_offload`, `test_model_parallelism`, `test_model_parallel_beam_search`
    * Run the tests in a multi-gpu environment `pytest tests/models/{MODEL_NAME}/test_modeling_{MODEL_NAME}.py -vv -k "offload or parallelism"`

## Models
- [ ] [Align](https://github.com/huggingface/transformers/blob/main/src/transformers/models/align/modeling_align.py)
- [ ] [Altclip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/altclip/modeling_altclip.py)
- [x] [Beit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/beit/modeling_beit.py) #30379
- [ ] [Bit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bit/modeling_bit.py)
- [ ] [Blip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip/modeling_blip.py)
- [ ] [Chinese_clip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/chinese_clip/modeling_chinese_clip.py)
- [x] [Convnext](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnext/modeling_convnext.py) #30207
- [x] [Convnextv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnextv2/modeling_convnextv2.py) #30207
- [x] [Cvt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/cvt/modeling_cvt.py) #30207
- [ ] [Data2vec](https://github.com/huggingface/transformers/blob/main/src/transformers/models/data2vec/modeling_data2vec_vision.py)
- [ ] [Depth_anything](https://github.com/huggingface/transformers/blob/main/src/transformers/models/depth_anything/modeling_depth_anything.py)
- [ ] [Dinat](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dinat/modeling_dinat.py)
- [ ] [Dinov2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dinov2/modeling_dinov2.py)
- [ ] [Donut](https://github.com/huggingface/transformers/blob/main/src/transformers/models/donut/modeling_donut_swin.py)
- [ ] [Dpt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dpt/modeling_dpt.py)
- [ ] [Efficientformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/efficientformer/modeling_efficientformer.py)
- [x] [Efficientnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/efficientnet/modeling_efficientnet.py) #29989
- [ ] [Flava](https://github.com/huggingface/transformers/blob/main/src/transformers/models/flava/modeling_flava.py)
- [ ] [Focalnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/focalnet/modeling_focalnet.py) #30207
- [ ] [Git](https://github.com/huggingface/transformers/blob/main/src/transformers/models/git/modeling_git.py)
- [x] [Glpn](https://github.com/huggingface/transformers/blob/main/src/transformers/models/glpn/modeling_glpn.py) #30207
- [ ] [Groupvit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/groupvit/modeling_groupvit.py)
- [x] [Imagegpt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/imagegpt/modeling_imagegpt.py) #30207
- [ ] [Layoutlmv3](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlmv3/modeling_layoutlmv3.py)
- [x] [Levit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/levit/modeling_levit.py) #30207
- [ ] [Mask2former](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mask2former/modeling_mask2former.py)
- [ ] [Maskformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/maskformer/modeling_maskformer.py)
- [ ] [Maskformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/maskformer/modeling_maskformer_swin.py)
- [x] [Mgp_str](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mgp_str/modeling_mgp_str.py) #30207
- [x] [Mobilenet_v1](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilenet_v1/modeling_mobilenet_v1.py) #30207
- [x] [Mobilenet_v2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilenet_v2/modeling_mobilenet_v2.py) #30207
- [x] [Mobilevit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilevit/modeling_mobilevit.py) #30207
- [ ] [Mobilevitv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilevitv2/modeling_mobilevitv2.py)
- [ ] [Nat](https://github.com/huggingface/transformers/blob/main/src/transformers/models/nat/modeling_nat.py)
- [ ] [Oneformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/oneformer/modeling_oneformer.py)
- [ ] [Perceiver](https://github.com/huggingface/transformers/blob/main/src/transformers/models/perceiver/modeling_perceiver.py)
- [x] [Poolformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/poolformer/modeling_poolformer.py) #30207
- [ ] [Pvt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pvt/modeling_pvt.py)
- [x] [Regnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/regnet/modeling_regnet.py) #30207
- [ ] [Resnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/resnet/modeling_resnet.py) #30207
- [x] [Sam](https://github.com/huggingface/transformers/blob/main/src/transformers/models/sam/modeling_sam.py) #30207
- [ ] [Segformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/segformer/modeling_segformer.py)
- [x] [Swiftformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swiftformer/modeling_swiftformer.py) #30207
- [x] [Swin](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin/modeling_swin.py) #30207
- [ ] [Swin2sr](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin2sr/modeling_swin2sr.py)
- [x] [Swinv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swinv2/modeling_swinv2.py) #30207
- [ ] [Timesformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/timesformer/modeling_timesformer.py)
- [ ] [Timm_backbone](https://github.com/huggingface/transformers/blob/main/src/transformers/models/timm_backbone/modeling_timm_backbone.py)
- [x] [Trocr](https://github.com/huggingface/transformers/blob/main/src/transformers/models/trocr/modeling_trocr.py) #30207
- [ ] [Tvlt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/tvlt/modeling_tvlt.py)
- [ ] [Tvp](https://github.com/huggingface/transformers/blob/main/src/transformers/models/tvp/modeling_tvp.py)
- [x] [Upernet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/upernet/modeling_upernet.py) #30207
- [ ] [Videomae](https://github.com/huggingface/transformers/blob/main/src/transformers/models/videomae/modeling_videomae.py)
- [ ] [Vit_mae](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit_mae/modeling_vit_mae.py)
- [ ] [Vit_msn](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit_msn/modeling_vit_msn.py)
- [x] [Vitmatte](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vitmatte/modeling_vitmatte.py) #30379
- [x] [Vivit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vivit/modeling_vivit.py) #30379
- [ ] [X_clip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/x_clip/modeling_x_clip.py)
- [x] [Yolos](https://github.com/huggingface/transformers/blob/main/src/transformers/models/yolos/modeling_yolos.py) #30207

### Motivation

Enable a powerful HF feature for all of our vision models

### Your contribution

Ping me for review 🤗