Update model config and README

d7f6603 about 1 year ago

4.09 kB

	---
	tags:
	- image-classification
	- timm
	library_name: timm
	license: apache-2.0
	datasets:
	- imagenet-1k
	---
	# Model card for vit_base_patch16_rpn_224.sw_in1k

	A Vision Transformer (ViT) image classification model. This is a `timm` specific variation of the architecture with residual post normalization blocks. Trained on ImageNet-1k in `timm` using recipe template described below.

	Recipe details:
	* Based on Swin Transformer train / pretrain recipe with modifications (related to both DeiT and ConvNeXt recipes)
	* AdamW optimizer, gradient clipping, EMA weight averaging
	* Cosine LR schedule with warmup


	## Model Details
	- Model Type: Image classification / feature backbone
	- Model Stats:
	- Params (M): 86.5
	- GMACs: 16.8
	- Activations (M): 16.4
	- Image size: 224 x 224
	- Papers:
	- Swin Transformer V2: Scaling Up Capacity and Resolution: https://arxiv.org/abs/2111.09883
	- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929v2
	- Dataset: ImageNet-1k
	- Original: https://github.com/huggingface/pytorch-image-models

	## Model Usage
	### Image Classification
	```python
	from urllib.request import urlopen
	from PIL import Image
	import timm

	img = Image.open(urlopen(
	'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
	))

	model = timm.create_model('vit_base_patch16_rpn_224.sw_in1k', pretrained=True)
	model = model.eval()

	# get model specific transforms (normalization, resize)
	data_config = timm.data.resolve_model_data_config(model)
	transforms = timm.data.create_transform(**data_config, is_training=False)

	output = model(transforms(img).unsqueeze(0)) # unsqueeze single image into batch of 1

	top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)
	```

	### Image Embeddings
	```python
	from urllib.request import urlopen
	from PIL import Image
	import timm

	img = Image.open(urlopen(
	'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
	))

	model = timm.create_model(
	'vit_base_patch16_rpn_224.sw_in1k',
	pretrained=True,
	num_classes=0, # remove classifier nn.Linear
	)
	model = model.eval()

	# get model specific transforms (normalization, resize)
	data_config = timm.data.resolve_model_data_config(model)
	transforms = timm.data.create_transform(**data_config, is_training=False)

	output = model(transforms(img).unsqueeze(0)) # output is (batch_size, num_features) shaped tensor

	# or equivalently (without needing to set num_classes=0)

	output = model.forward_features(transforms(img).unsqueeze(0))
	# output is unpooled, a (1, 196, 768) shaped tensor

	output = model.forward_head(output, pre_logits=True)
	# output is a (1, num_features) shaped tensor
	```

	## Model Comparison
	Explore the dataset and runtime metrics of this model in timm [model results](https://github.com/huggingface/pytorch-image-models/tree/main/results).

	## Citation
	```bibtex
	@misc{rw2019timm,
	author = {Ross Wightman},
	title = {PyTorch Image Models},
	year = {2019},
	publisher = {GitHub},
	journal = {GitHub repository},
	doi = {10.5281/zenodo.4414861},
	howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
	}
	```
	```bibtex
	@inproceedings{liu2021swinv2,
	title={Swin Transformer V2: Scaling Up Capacity and Resolution},
	author={Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
	booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},
	year={2022}
	}
	```
	```bibtex
	@article{dosovitskiy2020vit,
	title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
	author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
	journal={ICLR},
	year={2021}
	}
	```