nielsr HF staff commited on
Commit
f71cf10
1 Parent(s): f087f3a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +66 -0
README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: mit
4
+ tags:
5
+ - vision
6
+ - image-to-text
7
+ model_name: microsoft/git-large-textvqa
8
+ ---
9
+
10
+ # GIT (GenerativeImage2Text), large-sized, fine-tuned on TextVQA
11
+
12
+ GIT (short for GenerativeImage2Text) model, large-sized version, fine-tuned on TextVQA. It was introduced in the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Wang et al. and first released in [this repository](https://github.com/microsoft/GenerativeImage2Text).
13
+
14
+ Disclaimer: The team releasing GIT did not write a model card for this model so this model card has been written by the Hugging Face team.
15
+
16
+ ## Model description
17
+
18
+ GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a lot of (image, text) pairs.
19
+
20
+ The goal for the model is simply to predict the next text token, giving the image tokens and previous text tokens.
21
+
22
+ The model has full access to (i.e. a bidirectional attention mask is used for) the image patch tokens, but only has access to the previous text tokens (i.e. a causal attention mask is used for the text tokens) when predicting the next text token.
23
+
24
+ ![GIT architecture](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg)
25
+
26
+ This allows the model to be used for tasks like:
27
+
28
+ - image and video captioning
29
+ - visual question answering (VQA) on images and videos
30
+ - even image classification (by simply conditioning the model on the image and asking it to generate a class for it in text).
31
+
32
+ ## Intended uses & limitations
33
+
34
+ You can use the raw model for visual question answering (VQA). See the [model hub](https://huggingface.co/models?search=microsoft/git) to look for
35
+ fine-tuned versions on a task that interests you.
36
+
37
+ ### How to use
38
+
39
+ For code examples, we refer to the [documentation](https://huggingface.co/transformers/main/model_doc/git.html).
40
+
41
+ ## Training data
42
+
43
+ From the paper:
44
+
45
+ > We collect 0.8B image-text pairs for pre-training, which include COCO (Lin et al., 2014), Conceptual Captions
46
+ (CC3M) (Sharma et al., 2018), SBU (Ordonez et al., 2011), Visual Genome (VG) (Krishna et al., 2016),
47
+ Conceptual Captions (CC12M) (Changpinyo et al., 2021), ALT200M (Hu et al., 2021a), and an extra 0.6B
48
+ data following a similar collection procedure in Hu et al. (2021a).
49
+
50
+ => however this is for the model referred to as "GIT" in the paper, which is not open-sourced.
51
+
52
+ This checkpoint is "GIT-large", which is a smaller variant of GIT trained on 20 million image-text pairs.
53
+
54
+ Next, the model was fine-tuned on TextVQA.
55
+
56
+ See table 11 in the [paper](https://arxiv.org/abs/2205.14100) for more details.
57
+
58
+ ### Preprocessing
59
+
60
+ We refer to the original repo regarding details for preprocessing during training.
61
+
62
+ During validation, one resizes the shorter edge of each image, after which center cropping is performed to a fixed-size resolution. Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.
63
+
64
+ ## Evaluation results
65
+
66
+ For evaluation results, we refer readers to the [paper](https://arxiv.org/abs/2205.14100).