ybelkada commited on
Commit
f86beaa
1 Parent(s): 1b6340d

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +96 -0
README.md ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - vision
5
+ - image-classification
6
+ datasets:
7
+ - imagenet-1k
8
+ ---
9
+
10
+ # Vision Transformer (base-sized model) - Hybrid
11
+
12
+ ![Pull figure](https://s3.amazonaws.com/moonup/production/uploads/1670350379252-62441d1d9fdefb55a0b7d12c.png)
13
+
14
+
15
+ The hybrid Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining very good results compared to familiar convolutional architectures. ViT hybrid is a slight variant of the [plain Vision Transformer](vit), by leveraging a convolutional backbone (specifically, [BiT](bit)) whose features are used as initial "tokens" for the Transformer.
16
+
17
+ Disclaimer: The team releasing ViT did not write a model card for this model so this model card has been written by the Hugging Face team.
18
+
19
+ ## Model description
20
+
21
+ *While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.*
22
+
23
+ ## Intended uses & limitations
24
+
25
+ You can use the raw model for image classification. See the [model hub](https://huggingface.co/models?search=google/vit) to look for
26
+ fine-tuned versions on a task that interests you.
27
+
28
+ ### How to use
29
+
30
+ Here is how to use this model to classify an image of the COCO 2017 dataset into one of the 1,000 ImageNet classes:
31
+
32
+ ```python
33
+ from transformers import ViTHybridFeatureExtractor, ViTHybridForImageClassification
34
+ from PIL import Image
35
+ import requests
36
+
37
+ url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
38
+ image = Image.open(requests.get(url, stream=True).raw)
39
+
40
+ feature_extractor = ViTHybridFeatureExtractor.from_pretrained('google/vit-base-patch16-224')
41
+ model = ViTHybridForImageClassification.from_pretrained('google/vit-base-patch16-224')
42
+
43
+ inputs = feature_extractor(images=image, return_tensors="pt")
44
+ outputs = model(**inputs)
45
+ logits = outputs.logits
46
+ # model predicts one of the 1000 ImageNet classes
47
+ predicted_class_idx = logits.argmax(-1).item()
48
+ print("Predicted class:", model.config.id2label[predicted_class_idx])
49
+ >>> tabby, tabby cat
50
+ ```
51
+
52
+ For more code examples, we refer to the [documentation](https://huggingface.co/transformers/model_doc/vit.html#).
53
+
54
+ ## Training data
55
+
56
+ The ViT-Hybrid model was pretrained on [ImageNet-21k](http://www.image-net.org/), a dataset consisting of 14 million images and 21k classes, and fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/), a dataset consisting of 1 million images and 1k classes.
57
+
58
+ ## Training procedure
59
+
60
+ ### Preprocessing
61
+
62
+ The exact details of preprocessing of images during training/validation can be found [here](https://github.com/google-research/vision_transformer/blob/master/vit_jax/input_pipeline.py).
63
+
64
+ Images are resized/rescaled to the same resolution (224x224) and normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).
65
+
66
+ ### Pretraining
67
+
68
+ The model was trained on TPUv3 hardware (8 cores). All model variants are trained with a batch size of 4096 and learning rate warmup of 10k steps. For ImageNet, the authors found it beneficial to additionally apply gradient clipping at global norm 1. Training resolution is 224.
69
+
70
+ ## Evaluation results
71
+
72
+ For evaluation results on several image classification benchmarks, we refer to tables 2 and 5 of the original paper. Note that for fine-tuning, the best results are obtained with a higher resolution (384x384). Of course, increasing the model size will result in better performance.
73
+
74
+ ### BibTeX entry and citation info
75
+
76
+ ```bibtex
77
+ @misc{wu2020visual,
78
+ title={Visual Transformers: Token-based Image Representation and Processing for Computer Vision},
79
+ author={Bichen Wu and Chenfeng Xu and Xiaoliang Dai and Alvin Wan and Peizhao Zhang and Zhicheng Yan and Masayoshi Tomizuka and Joseph Gonzalez and Kurt Keutzer and Peter Vajda},
80
+ year={2020},
81
+ eprint={2006.03677},
82
+ archivePrefix={arXiv},
83
+ primaryClass={cs.CV}
84
+ }
85
+ ```
86
+
87
+ ```bibtex
88
+ @inproceedings{deng2009imagenet,
89
+ title={Imagenet: A large-scale hierarchical image database},
90
+ author={Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li},
91
+ booktitle={2009 IEEE conference on computer vision and pattern recognition},
92
+ pages={248--255},
93
+ year={2009},
94
+ organization={Ieee}
95
+ }
96
+ ```