lysandre HF staff adirik commited on
Commit
b03589d
1 Parent(s): f8095a6

Update README.md (#1)

Browse files

- Update README.md (832b1c314f90a7e65731239a7e8b0b2f373579b2)
- Update README.md (b3135b51a3eb792e088ab564e42b20f0841f14e4)


Co-authored-by: Alara Dirik <adirik@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +89 -0
README.md CHANGED
@@ -1,3 +1,92 @@
1
  ---
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ tags:
4
+ - vision
5
+ - object-detection
6
  ---
7
+
8
+ # Model Card: OWL-ViT
9
+
10
+ ## Model Details
11
+
12
+ The OWL-ViT (short for Vision Transformer for Open-World Localization) was proposed in [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230) by Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. OWL-ViT is a zero-shot text-conditioned object detection model that can be used to query an image with one or multiple text queries.
13
+
14
+ OWL-ViT uses CLIP as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.
15
+
16
+
17
+ ### Model Date
18
+
19
+ May 2022
20
+
21
+ ### Model Type
22
+
23
+ The model uses a CLIP backbone with a ViT-L/14 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. The CLIP backbone is trained from scratch and fine-tuned together with the box and class prediction heads with an object detection objective.
24
+
25
+
26
+ ### Documents
27
+
28
+ - [OWL-ViT Paper](https://arxiv.org/abs/2205.06230)
29
+
30
+
31
+ ### Use with Transformers
32
+
33
+ ```python3
34
+ import requests
35
+ from PIL import Image
36
+ import torch
37
+
38
+ from transformers import OwlViTProcessor, OwlViTForObjectDetection
39
+
40
+ processor = OwlViTProcessor.from_pretrained("google/owlvit-large-patch14")
41
+ model = OwlViTForObjectDetection.from_pretrained("google/owlvit-large-patch14")
42
+
43
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
44
+ image = Image.open(requests.get(url, stream=True).raw)
45
+ texts = [["a photo of a cat", "a photo of a dog"]]
46
+ inputs = processor(text=texts, images=image, return_tensors="pt")
47
+ outputs = model(**inputs)
48
+
49
+ # Target image sizes (height, width) to rescale box predictions [batch_size, 2]
50
+ target_sizes = torch.Tensor([image.size[::-1]])
51
+ # Convert outputs (bounding boxes and class logits) to COCO API
52
+ results = processor.post_process(outputs=outputs, target_sizes=target_sizes)
53
+
54
+ i = 0 # Retrieve predictions for the first image for the corresponding text queries
55
+ text = texts[i]
56
+ boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
57
+
58
+ # Print detected objects and rescaled box coordinates
59
+ score_threshold = 0.1
60
+ for box, score, label in zip(boxes, scores, labels):
61
+ box = [round(i, 2) for i in box.tolist()]
62
+ if score >= score_threshold:
63
+ print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
64
+ ```
65
+
66
+
67
+ ## Model Use
68
+
69
+ ### Intended Use
70
+
71
+ The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, text-conditioned object detection. We also hope it can be used for interdisciplinary studies of the potential impact of such models, especially in areas that commonly require identifying objects whose label is unavailable during training.
72
+
73
+ #### Primary intended uses
74
+
75
+ The primary intended users of these models are AI researchers.
76
+
77
+ We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.
78
+
79
+ ## Data
80
+
81
+ The CLIP backbone of the model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet. The prediction heads of OWL-ViT, along with the CLIP backbone, are fine-tuned on publicly available object detection datasets such as [COCO](https://cocodataset.org/#home) and [OpenImages](https://storage.googleapis.com/openimages/web/index.html).
82
+
83
+ ### BibTeX entry and citation info
84
+
85
+ ```bibtex
86
+ @article{minderer2022simple,
87
+ title={Simple Open-Vocabulary Object Detection with Vision Transformers},
88
+ author={Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby},
89
+ journal={arXiv preprint arXiv:2205.06230},
90
+ year={2022},
91
+ }
92
+ ```