L INEARLY M APPING FROM I MAGE TO T EXT S PACE
A BSTRACT
The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question.
Prior work has shown that pretrained LMs can be taught to caption images when a vision model’s parameters are optimized to encode images in the language space.
We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map.
We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection.
Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model).
We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions).
We ﬁnd that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform signiﬁcantly better on benchmark language-and-vision tasks.
Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images.
Code is available here: https://github.com/jmerullo/limber
I NTRODUCTION
Much recent work in NLP has revolved around studying the limits on representational capacity incurred by training on form-only text data, as discussed in Bender & Koller (2020).
Tied to this argument is the idea that without explicit grounding , language models are not inclined to learn conceptual representations of language that reﬂect the rich conceptual knowledge that humans gain from interacting with the physical, non-linguistic world.
Despite this, there have been remarkable ﬁndings in large language models’ abilities to generalize to and reason about non-linguistic phenomena (Tsimpoukelli et al., 2021; Eichenberg et al., 2021; Li et al., 2021; Patel & Pavlick, 2022).
Thus, an open question in the ﬁeld is to what extent (if at all) a language model trained on text-only data can learn aspects of the physical world.
In this paper, we test a speciﬁc hypothesis about the relationship
[Figure]
between language model and image encoder representations: that these conceptual representations can be approximately mapped to one another through a linear transformation.
To do this, we train a single linear layer to project from the representation space of images into the language space of a generative LM without tuning any other model parameters, which we call LiMBeR :
Linearly Mapping Between Representation spaces.
That is, we linearly transform an image representation into “soft prompts”–vector(s) in the embedding space that do not correspond to discrete language tokens (Lester et al., 2021).
The weights of this linear projection are tuned for an image captioning task (illustrated in Figure 1).
We can then evaluate its performance on vision-language (VL) tasks at test time by exploring the text the LM generates.
Because of the simplicity of the linear transformation, we would expect that if the conceptual representation spaces of the two models are structured similarly, this transfer will be successful and the LM will have little trouble describing the contents of images.
We use three different image encoders with increasing levels of linguistic supervision in pretraining:
BEIT (Bao et al., 2021), Normalizer Free Resnet50 (NFRN50) (Brock et al., 2021), and CLIP (Radford et al., 2021) to train different projections into the LM.
By linguistic supervision , we refer to the extent to which the image encoder was exposed to language data during its pretraining, thus inﬂuencing the expected representational similarity between it and an LM.
While CLIP was pretrained to align images with full natural language captions in a shared image-text representation space, BEIT had no exposure to language and was trained by predicting the contents of masked out sections of images.
NFRN50 falls somewhere between these extremes: having been pretrained on an image classiﬁcation task for identifying the subject of an image over the set of classes in ImageNet1k Russakovsky et al.
Although there is no natural language in this task, the pretraining objective encourages the model to map visual features along lexical categorical concepts (the image classes) derived from the WordNet hierarchy (Miller, 1995).
We show that prompting an LM with any of the three image encoders effectively transfers semantic content in the image that the LM describes with natural language.
However, performance also appears proportional to the strength of the linguistic supervision the image encoder had.
While CLIP and NFRN50 perform competitively with tuning the models freely (e.g., Tsimpoukelli et al.
Eichenberg et al.
, BEIT appears to transfer mostly coarse-grained visual properties and struggles with encouraging the LM to generate exact lexical categories .
We interpret this as evidence that models trained on either language or vision data learn conceptual spaces that are structurally similar to each other, but that the exact degree of similarity depends on the type of supervision the image encoder receives.
In summary, we show: (1) that visual semantic information can be linearly mapped to language models in the form of soft prompts without tuning any model parameters.
(2) That this mapping allows generative models to describe images and answer questions about images at a level that is comparable to what is achieved by multimodal models which tune image and language representations jointly.
And (3) by training our prompting pipeline with different image encoder backbones, we demonstrate that linguistic supervision in pretraining plays a key role in concept formation in models and thus, the transferability of visual features from vision to text spaces.
R ELATED W ORK
Our approach takes inspiration from recent work in adapting pretrained language models for accepting representations of images as inputs.
Particularly, the Frozen and MAGMA models (Tsimpoukelli et al., 2021; Eichenberg et al., 2021), as well as Sung et al.
; Alayrac et al.
Mokady et al.
; Luo et al. (2022); Lin et al.
; Zhai et al.
(2022), which show that pretrained image and text networks can be tuned together on an image captioning task and applied to downstream vision-language (VL) tasks.
These approaches either ﬁne-tune the pretrained models, or train non-linear MLP projection/fusion networks between modalities, making interpretation of the representations difﬁcult compared to our approach.
Scialom et al.
(2020) show a learned linear transformation is sufﬁcient for BERT to encode image region representations which are then fed to a text decoder to generate questions about the image, but it is not well understood what abstractions
LMs are able to transfer from a transformation of this type, or if a text decoder can operate on linear transformations of visual encodings directly.
Pretrained/from scratch LMs have typically been used in the past for image captioning applications in which an image representation is fed into the LM as input (Desai & Johnson, 2021;
Shen et al., 2021; Devlin et al., 2015).
Gui et al.
; Yuan et al.
pretrain vision-language models from scratch using image-caption data.
Zeng et al.
; Xie et al. (2022); Wang et al. (2022) augment multimodal performance by feeding text prompts derived from VL models into an LM, in order to incorporate knowledge learned by LM training.
These show LMs can interface with visual inputs described in text; our work questions whether the visual input can be fed directly into the LM, without bridging through language ﬁrst.
The success of aforementioned models on VL tasks indicates there is a representational similarity learned by text and image models independently, which we investigate in this paper.
Our work is also highly related to the idea of model “stitching” (Lenc & Vedaldi, 2015) in which two different models are attached at a certain layer.
LiMBeR can be described as stitching the output of an image encoder to the input of an LM in the form of soft prompts (Lester et al., 2021).
Stitching offers distinct advantages in evaluating the representational similarity between two models, as described in Bansal et al.
(2021), over other conventional methods like RSA and CKA (Kriegeskorte et al., 2008; Kornblith et al., 2019).
For example, LiMBeR allows us to show not just that CLIP encodings are more similar to text encodings than BEIT representations, but that BEIT representations are nevertheless able to transfer visual property information to the LM (§5.3).
There has been considerable interest in recent work in establishing if LMs model aspects of the nonlinguistic world in order to model language.
Lu et al.
(2021) show that the weights of a pretrained
LM can generalize to tasks with different modalities.
Hao et al.
(2022) similarly show that LMs can act as interfaces for multiple modalities.
Li et al.
(2021) show that models of entities and situations can be derived from contextual word representations.
Patel & Pavlick (2022) show that very large LMs (GPT-3 scale (Brown et al., 2020)) can learn in-context non-linguistic conceptual domains depicted in text.
Our work differs from these in that we have an LM interface directly with non-text data without changing model weights and show that, although fundamentally different, the representation space of a text-only LM shares non-trivial similarities to that of several vision-based models.
M ETHOD : L INEARLY M APPING FROM I MAGE TO T EXT R EPRESENTATIONS
While previous work has shown success in mapping images to language model soft prompts as a method for multimodal pretraining (e.g., Frozen, Magma; see Section 2), there have been no attempts to restrict the mechanism behind this mapping and understand how it works.
Our basic approach is to train a single linear layer P to project from the hidden size h
I of a pretrained image encoder into the input space e L of a generative language model for an image captioning task.
The projected inputs do not correspond to discrete language tokens, and can be thought of as soft prompts (Lester et al., 2021) representing the image.
For brevity, we refer to training P as Li nearly M apping
Be tween R epresentation spaces (i.e., LiMBeR ) 1 .
Our approach can also be viewed as paring down the method used in Tsimpoukelli et al.
(2021) and Eichenberg et al.
(2021), such that the only trained parameters reside in the projection P .
By freezing the image encoder E and LM on either side of the projection, we can examine the similarities between the representation spaces of the two as a function of the ability of the LM to describe an image input or perform some task relating to it.
We expect that, if a language model represents visual conceptual information structurally similarly to that learned by a vision encoder, then a simple linear transformation to the language space is all that is required to transfer visual features into the language model.
Before describing the training procedure, we will describe the basic components of the model, and the variations we chose.
Language Model LM & Image Encoders E
We hypothesize that the conceptual representations learned by an LM are equivalent, up to a linear transformation, to the representations from an image encoder E .
The language model used is the 6 billion parameter decoder-only GPT-J model (Wang & Komatsuzaki, 2021).
P is trained to project from h I to the input space e L = 4096 of the LM.
We train several models with different E ’s to determine the compatibility between encodings from
E and the LM.
We also test how the choice of E inﬂuences performance on this task, speciﬁcally, with regards to the degree of linguistic supervision E saw in pretraining, as described in Section 1.
From E we extract an image encoding of dimensionality
h
I representing the image.
We then project that encoding to a e L ∗ k sequence of soft prompts, which we hereafter refer to as image prompts .
k is determined by the architecture of the E .
For example, for consistency with the MAGMA model, we use the 12x12x3072d feature map before pooling from CLIP, which we ﬂatten to k = 12 ∗ 12 = 144 .
The encoders we experiment with are (1) CLIP RN50x16 (Radford et al., 2021), k
= 144 , h I = 3072 .
Because CLIP is trained to learn multimodal image-text embeddings, we expect that it will be easier for the model to learn a projection into language space than a vision only encoder.
(2) NFRN
(Brock et al., 2021), k
= 2 , h I = 2048 .
We train three variants using NFResnet50: one pretrained and frozen during caption training (NFRN50), one tuned during caption training (NFRN50 Tuned; note that the LM is still frozen), and one randomly initialized (NFRN
Random).
The NFRN50 models are pretrained on an image classiﬁcation task on data that is labeled according to WordNet hypo/hypernym structure.
This signal trains the model to separate object classes according to these words.
For this reason, we consider it to have indirect access to linguistic supervision.
(3) BEIT-Large (Bao et al., 2021), k = 196 , h I = 1024 .
BEIT is pretrained using a self-supervised masked visual token modeling task and does not have access to any labeled data which may give the model an inductive bias towards a linguistic structure.
We use the 16-pixel patch version that was pretrained only on ImageNet22k.
We additionally test two variants of this model:
BEIT Random, is randomly initialized, and BEIT FT, which was pretrained on the same task and then ﬁnetuned for image classiﬁcation on the same dataset.
We use this model to show that it is indeed the linguistic supervision of the pretraining objective which induces better performance in the captioning task.
3. T RAINING P ROCEDURE
Following the MAGMA and Frozen models (Eichenberg et al., 2021; Tsimpoukelli et al., 2021), we train a projection on an image captioning task so that we can learn to align the representation spaces of E and the LM.
All models are trained with the same basic hyperparameters and settings as described in the MAGMA paper (see Appendix A for details) on the Conceptual Captions 3M dataset (CC3M, Sharma et al.
for 15,000 training steps.
Baselines As baselines, we use NFRN50 Random, NFRN50 Tuned, and train our own instance of
MAGMA base .
Please note that NFRN50 Tuned is a stand-in for the Frozen model: it is architecturally the same, but differs in that we use the hyperparameters used to train the MAGMA model.
NFRN
Random allows us to test the efﬁcacy of LiMBeR when the image encoder backbone has not learned any useful visual features.
The MAGMA we train uses the CLIP RN50x16 image encoder (Radford et al., 2021), GPT-J as the LM, and adapters in sequence in the attention blocks with a downsample factor of 4.
3. L IMITATIONS
Due to computational constraints, we did not control for the prompt length ( k ) for each image encoder.
Tsimpoukelli et al.
(2021) experiment with a small range for the value of k for the Frozen model and show that while there are some differences, k is mostly a factor in hyperparameter tuning and should not strongly affect the comparison between models.
We use much higher values of k for
CLIP and BEIT, and this is therefore is not strongly controlled for in our study.
We consider LM runoff another potential confound.
In some cases, if the LM recognizes and generates a relevant word for one concept (e.g., “the beach”), it might continue generating relevant information due to a strong linguistic prior for that info showing up (e.g., “building a sandcastle”), giving the illusion it is recognizing every element in an image (even if it never saw “the sandcastle”).
Regardless, the scope of this problem is very limited, and across multiple large datasets our results show that recovery of any image information is still possible, even if the full and precise extent of which is impossible to know.
We also include a ‘blind’ model in visual question answering analysis to further control for this.
Q: What is the person doing?
A: surfing
CLIP
He is surﬁng
[Figure] [Figure]
He
[Figure]
BEIT
He is
[Figure]
the
[Figure]
NFRN
[Figure]
Q: What is the person holding?
A: tennis racket
CLIP
A tennis racket
NFRN
A tennis racket
BEIT
A baseball bat.
NFRN
Random
A tree
CLIP a gira ﬀ e in the lobby of the building
NFRN50 the gira ﬀ e in the zoo.
BEIT a peacock in the garden
NFRN
Random
[Table]
and
[Table]
woman in
[Table]
of ﬂowers
CLIP tennis player in action
NFRN50 tennis player at the tennis tournament.
BEIT tennis player during a tennis match.
NFRN
Random the new logo for the team
[Figure]
P ERFORMANCE ON V ISION -L ANGUAGE T ASKS
We ﬁrst verify that image representations that are linearly projected into the input space of the LM carry semantic information about the content of the image that the LM can make sense of.
Since we only tune a single projection between the image encoder and text decoder, the prompt tokens in the
LM are equivalent to the image representation up to that linear transformation.
If LMs are learning a conceptual space that reﬂects that of the non-linguistic, purely visually grounded space of the image encoder, the LM should be able to capture the image information and describe it in text.
Data We evaluate on image prompts generated by each image encoder on multiple image captioning datasets: MSCOCO (Lin et al., 2014) and NoCaps (Agrawal et al., 2019), as well as the VQA (Goyal et al., 2017) visual question-answering dataset.
Following convention from SimVLM and
MAGMA, we input the preﬁx “A picture of” after every image to prompt the model.
Like in previous work, we ﬁnd that this is a favorable prompt which tends to increase performance.
Metrics For image captioning, we report CIDEr-D (Vedantam et al., 2015), CLIPScore, and RefCLIPScore (Hessel et al., 2021).
CIDEr-D rewards generating accurate words which are more likely to be visually informative, and CLIPScore can evaluate similarity between an image and caption without references, which helps us give credit for captions that vary greatly from the ground truth, but similar in semantic content (e.g. describing a pool as a lake).
We report additional captioning metrics in Appendix B. For visual question answering, we follow the few-shot procedure used in
Eichenberg et al.
(2021) in which we prompt the models with the “[image] Q: [q] A:” format.
We take the ﬁrst word of the generation and, like in the MAGMA paper, truncate to the length of the longest ground truth answer.
We also use the normalization procedure and accuracy metric described in the VQA repo
Results
Our main results can be seen in Table 1.
As evidenced by comparing MAGMA and CLIP, and NFRN50 tuned and frozen, we ﬁnd that there is relatively little beneﬁt in training parameters in either encoder or decoder.
Note that the MAGMA model we implemented is identical to the frozen
CLIP model, with the exceptions that MAGMA tunes the image encoder and LM.
On captioning and VQA tasks, performance of the jointly-tuned models (MAGMA, NFRN50 Tuned) is not consistently better, and is often worse, than just training the projection with frozen models.
This trend persists across over 10 automatic captioning metrics, which are described in
Appendix B.
Our results indicate that there is in fact a relationship between the linguistic supervision of the pretraining task
Image Captioning
NoCaps - CIDEr-D
NoCaps (All)
CoCo CoCo
In
Out
Near
All
CLIP-S
Ref-S
CIDEr-D
CLIP-S
Ref-S
NFRN50 Tuned 20.
MAGMA (released) 18.
MAGMA (ours) 30.
BEIT Random 5.
NFRN
Random 5.
BEIT 20.
NFRN 21.
BEIT FT.
CLIP 34.
VQA n-shots
Blind 20.
NFRN50 Tuned 27.
MAGMA (ours) 24.
MAGMA (reported) 32.
[Table]
Notably we ﬁnd that even vanilla BEIT, which has no linguistic supervision in pretraining, still transfers well to the LM for captioning, far outperforming random NFRN50 across the board, which had no pretraining to learn visual features.
We do ﬁnd that BEIT captions using vaguer language, and/or semantically related-but-incorrect descriptions of objects (Figure 2; more examples in Appendix B).
We see this reﬂected in the CLIPScores of the captions as well, which reward semantic similarity rather than precise lexical overlap with a reference caption.
BEIT captions score 62 and 63.6 for NoCaps and COCO respectively; on average only 4.5 points behind NFRN50 but 14.3 ahead of random NFRN50.
Perhaps we see the greatest failure of BEIT prompts in the inability to transfer details that the LM can use to answer questions about images (At 4-shot VQA, BEIT scores 31.72% while a ‘blind’ LM with no image input scores 36.99%).
We hypothesize this is because BEIT representations do not encode visual information that corresponds well to lexical categories.
In Section 5, we provide evidence in favor of this hypothesis, and investigate the granularity of detail prompts from each frozen encoder transfer to the LM.
T RANSFER OF V ISUAL C ONCEPTS
Examining the conditions that cause an image prompt to succeed or fail to transfer to the LM can help us understand the differences between the text and image representation spaces.
Doing so can also help us understand why BEIT prompts perform so poorly for VQA despite performing decently for captioning.
In Section 5.1, we analyze the ability to accurately generate speciﬁc lexical categories in captions when they appear in images (e.g., mentioning “squirrel” when given a picture of one).
Following that, in Section 5.3 we focus on mistakes the models make: when the LM generates a bad caption, does it generate a caption that describes entities with similar visual properties ?
For example, a caption generated from an image of a “small”, “woodland”, and “furry” animal might not mention the actual animal depicted (e.g., a squirrel); but does it instead mention a different but similar furry animal (e.g., a rabbit)?
We ﬁnd that only linguistically informed image encoders (NFRN50, CLIP) tend to strongly encode concepts aligning to lexical categories , but all pretrained models including BEIT encode property information approximately equally well, and far better than a randomly initialized image encoder baseline.
5. T RANSFER OF L EXICAL C ATEGORICAL C ONCEPTS
Using the COCO validation set, we count the top 50 nouns, modiﬁers (e.g., adjectives), and relations (e.g., verbs, prepositional phrases) that appear in the ground truth captions and calculate how often they appear in the generated captions that were used to calculate the scores in Table 1.
Metrics We calculate the precision/recall/F1 for each word, broken down along conceptual categories.
To test our hypothesis that BEIT transfers coarser information, we also report the Wu-Palmer similarity (Wup) (Wu & Palmer, 1994) between the ground truth word and the most similar word in the generated caption.
The Wup score works by calculating the distance between the ground truth word and the generated word in the WordNet taxonomy, offering a way to measure ‘how close’ a word was to the correct answer.
Results In Figure 3, we show that BEIT’s recall for nouns in categories like ‘people’, ‘environment’, ‘vehicles’, and ‘objects’ is lower than NFRN50 or CLIP, but is comparable in terms of Wup similiarity in many categories.
Unlike NFRN50 and CLIP’s pretraining, BEIT’s pretraining does not encourage it to learn conceptual differences between two similar looking objects that use different words.
Compared to prompts from a randomly initialized NFRN50, for which very few consistent patterns emerge, the LM can still extract the broad conceptual meaning behind BEIT prompts, as evidenced by high Wup similarity (and CLIPScore results in Table 1).
We interpret these results as supporting the hypothesis that BEIT prompts transfer conceptual information from the purely visual to purely text space, but only in terms of coarse-grained conceptual information corresponding to visual properties, not lexical categories.
Our full analysis, including additional metrics and results for each individual word from the top 50 nouns, modiﬁers, and relations can be found in Appendix
B.
5. P ROBING
To rule out the possibility that BEIT representations are encoding lexical concept information, but are merely unable to linearly transfer it to the LM due to representational differences, we train linear probes on several datasets for image classiﬁcation.
We ﬁnd that BEIT typically does not encode ﬁne- grained information as well as NFRN50 or CLIP, though it far outperforms the randomly initialized
NFRN50 baseline.
We discuss training details and results in Appendix E.
5. T RANSFER OF C OARSE -G RAINED P ERCEPTUAL C ONCEPTS
To better understand what BEIT encodes, if not word category information, we further investigate where errors arise, and how the structures of the embedding spaces for each frozen image encoder differ.
For the sake of this analysis we constrain the task to generating captions for pictures of animals.
The reason for this narrower scope is that the captions are easier to analyze: the caption describing a picture of an animal should virtually always mention the name of that animal and the word used to describe the animal is mostly unambiguous.
[Figure]
[Figure]
Data For this task we use the Animals With Attributes 2 (AWA) dataset (Xian et al., 2019) which contains 37k total images covering 50 animal classes.
Each animal class also comes with annotations for 85 properties describing the animals (e.g., ‘claws’, ‘stripes’, ‘jungle’), which allow us to analyze if prompts from certain encoders consistently make mistakes along any of these dimensions.
Metrics When an image prompt produces a caption, we can measure the similarity of any animals mentioned to the WordNet synset of the ground truth animal label.
We can also measure similarity using the annotated properties provided by the AWA dataset.
For a given animal (e.g., “squirrel”), we can look at the other animal in the dataset that it is most often mistaken for (e.g., “rabbit”) and compare the proportion of properties that they share.
Results We generate captions for each image using prompts from each frozen image encoder.
We consider a caption to be ‘correct’ if it contains the name of the animal the image depicts.
CLIP and
NFRN50 are correct most often: 59% and 43% of the time respectively.
BEIT and the randomly initialized NFRN50 only achieve 13% and 0.4% accuracy, respectively.
This aligns with previous observations that BEIT struggles with encoding ﬁne-grained lexical level concepts.
By looking at failure cases for each model, we can establish whether each model is predicting the presence of a similar animal or not.
In Figure 4a, we show that when captions generated from each model mistake one animal for another, the mistaken animals are highly similar to the ground truth animal when measuring both Wu-Palmer similarity (Averages: BEIT: 0.8, NFRN50: 0.81, CLIP: 0.8) and overlap of AWA properties (Averages: BEIT: 0.62, NFRN50: 0.68, CLIP: 0.59).
Although BEIT prompts do not transfer the exact animal concept to the LM, the coarse grained perceptual information is transferred and ‘understood’ by the LM.
In Figure 4b we create UMAP projections of the encodings for each image in AWA and indeed ﬁnd that NFRN50 and CLIP cluster according to tight lexical categories (the animal types), BEIT clusters most tightly by perceptual features, such as habitat, having ﬂippers, etc.
D ISCUSSION & F UTURE W ORK
We connect image representations as inputs to an LM with an extremely simple transformation: a linear projection.
We interpret the success of generating text relevant to an image with this approach as indicative of an underexplored representational similarity between language and vision representations.
Depending on the linguistic guidance in the pretraining task used to pretrain the image encoder, we see varying performance.
Using a vision-only encoder (BEIT) leads to generations from the LM that are often incorrect but close under measures of perceptual relatedness.
Unless ﬁnetuned with image classiﬁcation BEIT has no inductive bias in pretraining to distinguish concepts that we might normally distinguish with language, especially when they are perceptually very similar.
The fact that only image encoders trained with linguistic supervision can do this suggests interesting future work on the role of language in category formation.
Despite strong performance with a linear transformation, the representation spaces of these models seem to contain differences that cannot be approximated in the language space.
Multimodal models, ideally, will learn richer representations by taking advantage of these differences.
It is useful to think about how current multimodal pretraining objectives succeed and/or fail at doing this.
LiMBeR can serve as a strong baseline for future multimodal models, as it provides a point of comparison for a minimal mapping between vision and language.
We see LiMBeR as a useful tool for understanding how representations trained from different modalities can be similar or different.
For concepts that are represented similarly, can we take advantage of this fact to reduce the amount of data required for learning good text representations?
Where they are different, can multimodal models learn richer representations by incorporating information from both?
For example, vision data may help with reporting bias in text corpora (Paik et al., 2021).
Answering these questions can help us understand the limits of text-only pretraining, as well as how to better ground LMs to non-linguistic data.
C ONCLUSION
In this paper, we test how similar pretrained image and text representations are by training a linear map to project image representations into the input space of a language model (LM), such that the LM can accurately describe the contents of the images.
We show that models trained through
LiMBeR (Linearly Mapping Between Representation spaces) are competitive on image captioning and visual question answering benchmarks with similar models like MAGMA that tune both image and text networks.
However, we also ﬁnd that such transfer is highly dependant on the amount of linguistic supervision the image encoder backbone had during its pretraining phase.
BEIT, which is a vision-only image encoder underperforms compared to a Resnet model trained on image classiﬁcation, which in turn underperforms compared to CLIP, which was pretrained with natural language captions.
We explore what conceptual information transfers successfully, and ﬁnd through analysis of generated text, clustering, and probing that the representational similarity between LMs and vision-only image representations is mostly restricted to coarse-grained concepts of perceptual features, while linguistically supervised vision models can transfer lexical concepts.
Our ﬁndings indicate that LMs and vision models learn conceptually similar representation spaces, such that a minimal linear transformation is an adequate approximation for transferring information about an image.
The extent of this representational similarity is not well understood and is an interesting direction for future work.
R EPRODUCIBILITY S TATEMENT
We are committed to making all of our results reproducible.
Code for training all models, as well as weights of the linear projections can be found here: https://github.com/jmerullo/ limber .
We also release the weights of the linear projections that were trained for LiMBeR .
Because we froze the image and text models attached to the projection, the weights can be used to quickly reproduce our results with the corresponding off-the-shelf pretrained models with no other tuning necessary.
We use the default data splits for all datasets we used and release the random seeds used for all tasks that require generation from the LM in our codebase as well.
For the AWA tasks that require matching Wordnet synsets, we document the exact animal synsets that we used as the ‘ground truth’ for the animal labels in Appendix D, Table 6.
A CKNOWLEDGMENTS
We would like to thank Aaron Traylor and Nihal Nayak for thoughtful discussions and feedback on this work as well as StabilityAI for donating compute resources for model training.
This research is supported in part by ODNI and IARPA via the BETTER program (2019-19051600004).
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies, either expressed or implied, of ODNI, IARPA or the
U.S. Government
A L I MB E R T RAINING D ETAILS A. T RAINING D ETAILS
We mimic the MAGMA pretraining process for each of our models as outlined in Eichenberg et al.
As described above, the training task is caption generation.
For each training example, an image and caption pair ( x, y ) is fed into the model.
The image encoder E encodes the image into a i 1 , ..., i k of dimensionality
h I and length k .
For the CLIP Encoder, for example, we extract (12,12,3072) feature patches, which we resize to (144,3072) and feed into the projection layer P .
The output of P is fed as tokens representing the image into the language model LM .
The caption y is tokenized as t 1 , ..., t m where m is the variable length of the caption.
LM is given the encoded image tokens and starting with t 1 learns to minimize the next token log probability of t i for timestep i conditioned on i 1 , ..., i k and t 1 , ...
t i − 1 .
During training, we minimize the loss with the AdamW (Loshchilov & Hutter, 2018) optimizer per mini-batch, with the help of ZeRO stage 2 (Rajbhandari et al., 2019).
We use a dropout probability of 0.1, a weight decay of 0, betas = (0.9, 0.95), and gradient clipping = 1.0.
All models are trained for 15,000 training steps across 16 A100 GPUs for approximately 1.75 days.
Our effective batch size was 2048.
We use a learning rate of 8 ∗ 10 − 4 for the projection layer P .
For models where we tune E as well, we tune its parameters with a learning rate of 2 ∗ 10 − 6 .
[Table]
B C APTIONING P ERFORMANCE
To get a better idea of the kinds of captions LiMBeR produces, see Figure 6, which includesimages that were randomly selected from the COCO validation set (2017) and the generated captions for all models we test.
We include a greater range captioning metric results in Table 3 for the COCO dataset.
Overall, we ﬁnd the same trends that we report in the main paper, which are has very low that 1.)
the greater amount of linguistic supervision that an image encoder has, the better its captioning performance and 2.)
unfreezing the image encoder does not seem to lead to consistently signiﬁcant improvements.
We also include the breakdown of the SPICE metric across the associated subcategories such as relations, attributes, and objects.
Of the LiMBeR models, we ﬁnd that
CLIP based models do the best across the board (12.1 for CLIP vs. 9.28 for NFRN50).
Besides the random baseline, BEIT performs the worst overall except in the color category (0.45 vs. 0.42 for
NFRN50) also include heatmaps with recall/precision/F1/
-Palmer similarity metrics comparing the captions generated by each model and the top 50 nouns objects), modiﬁers, and from the from the COCO validation set (Figures 7, 8, 9).
We also use COCO images and captions to measure the similarity between vision and text encodings of the same concepts.
That is, if there is a structural similarity between image and text encodings, do typical representational similarity metrics reﬂect this?
For this experiment we want to encode a large number of images and captions that depict a set of concepts, and compare the relative representational similarity between the concepts for the image and text representations.
The intuition is that if image and text models are representing concepts in a similar structure, then two concepts that are represented similarly in an image model should be similar within text representations as well.
We sample a small subset of images depicting 10 different animals from the COCO dataset, and encode each image with each of our four image encoders (without a LiMBeR projection), and the ground truth captions with GPT-J.
We use the last hidden state of the last token as the representation of the caption.
We choose animals because it is easy for humans to intuitively compare how similar two animals are.
In total, we collect 939 images and captions for each animal class and compare the representational similarity of the subset of encodings using the representational similarity analysis (RSA) technique from Kriegeskorte et al.
The similarity matrix between each representation for each set is calculated, and then the upper triangular matrix for each similarity matrix is used to calculate the Pearson correlation.
The results of this can be seen in Figure 5.
We zero out the diagonal for visibility of the other values, since the diagonal is always equal to 1.
RSA does not seem to capture similarity in a way that reﬂects the transfer performance to the LM.
For example, similarity between the BEIT and randomly initialized NFRN50 model are unusually high given differences in performance in our experiments.
Further analysis on the geometry of these representation spaces is required to make stronger statements about how (dis)similar they are.
We leave this for future work.
C V ISUAL Q UESTION A NSWERING P ERFORMANCE
The VQA accuracies are broken down by question type and model for the 4-shot case in Table 5.
We don’t notice any signiﬁcant patterns in the per question type breakdown.
It does seem perhaps
[Table] [Table]
NFRN50 is better at
[Table]
is the
[Table]
what is the 15.
none of the above 38.
37.4 is this 62.
57.1 is this a 63.
58.5 what is 9.
what kind of  22.
are the  64.
58.9 is there a 60.
58.6 what type of 16.
where is the  1.
is it 63.
58.3 what are the 13. does the 66.
64.6 is 63.
58.5 is there 62.
54.5 what color are the  36.
are these 62.
56.6 are there 60.
55.4 what is the man  16.
is the man 61.
59.2 which 23.
26 how 12.
9.8 are 60.
56.8 does this 66.
61.3 what is on the 11.
how many people are 27.
18.7 what does the  7.
what time 15.
what is in the 9.
[Table]
12.4 how many people are in 23.
17.4 what is the person 8.
is this an 65.
59.2 can you 59.
59.5 what is the woman 8.
what animal is 10.
what is the color of the 35.
was 62.
59.5 is the person 61.
57.1 what is the name 3.
what room is  43.
is this person  61.
57.5 do you 61.
58.3 is that a 60.
59.6 what number is 8.
[Table]
why is the
[Table]
Note that NFRN50, NFRN Random, and NFRN Tuned are renamed to save space.
MAGMA refers to our version of the model.
D A NIMALS WITH A TTRIBUTES
AWA2 The Animals with Attributes 2 Dataset contains 37k images of 50 animal classes annotated with 85 properties (e.g. ‘stripes’, ‘tough’).
For each image encoder we generate captions using image projections into the LM for each image in the dataset and identify any animals mentioned using overlapping WordNet synsets with the ground truth labels.
We report the animal synsets that we used as the ‘ground truth’ synsets.
For example, to calculate accuracy, etc.
we split generated captions on white space and check the possible synsets of each word and check if there is any overlap with our list.
Each label synset is listed in Table 6.
We report two AWA experiments here: (1) the ﬁrst appears in the main paper, and investigates how similar the most similar words mentioned in the generated captions are to the ground truth animal in terms of Wu-Palmer similarity for the generated captions in which the ground truth animal does not appear – i.e., mistakes.
The per animal results can be found in Table 7.
(2) We also report the average precision (AP) of the average animal properties vector for the animals predicted to the ground truth for each model.
This experiment is similar to the properties-overlap experiment reported in the paper but is a measure of how often the predicted animal properties are similar vs. very different from the ground truth.
If the LM mentions an animal in the dataset, we add that animal’s property vector to a running list.
If the LM does not mention an animal in the dataset, we ignore that caption.
For each image-caption pair, we take the average of the properties vectors that correspond to the animals mentioned in the captions, and calculate the AP against the ground truth binary properties vector.
Results per animal class are in Table 8.
We exclude any entries for which a model makes fewer than 50 mistakes for an animal class.
For example, for all of the images labeled as depicting a tiger in the
AWA dataset, CLIP based captions fail to mention a tiger a single time.
[Table]
report all animal synsets that were used for experiments that require disambiguating words in captions to animal classes.
This allows us to correctly count a mention of “tigress” in a caption as a mention of the “tiger” animal type without relying on unreliable string matching techniques.
Random
[Table]
Random
BEIT
NFRN
CLIP polar
[Table]
bear
[Table] [Table]
[Table] [Table]
buffalo
[Table]
[Table] [Table]
[Table] [Table]
bobcat
[Table] [Table]
[Table] [Table]
[Table]
grizzly+bear
[Table]
[Table] [Table]
[Table] [Table]
blue
[Table]
whale
[Table]
[Table] [Table]
dalmatian
[Table] [Table]
[Table] [Table]
chihuahua
[Table] [Table]
[Table] [Table]
leopard
[Table] [Table]
[Table]
collie
[Table] [Table]
[Table]
german+shepherd
[Table] [Table]
[Table] [Table]
siamese+cat
[Table] [Table]
[Table] [Table]
persian
[Table]
cat
[Table] [Table]
[Table]
ox
[Table]
[Table] [Table]
seal 0.
[Table]
spider+monkey
[Table]
[Table] [Table]
[Table]
antelope 0.
skunk 0.
mole 0.
lion 0.
rat 0. hamster 0.
walrus 0. wolf 0.
killer+whale 0. humpback+whale 0.
chimpanzee 0.
moose 0.
fox 0. weasel 0.
hippopotamus 0.
deer 0. otter 0.
pig 0.
cow 0. sheep 0.
horse 0.
giant+panda 0.
beaver 0.
– tiger 0.
– – elephant 0.
– – gorilla 0.
– squirrel 0.
– rhinoceros 0. – rabbit 0.
– bat
[Table] [Table]
bear
[Table]
dolphin
[Table]
leopard 0.
horse 0.
german+shepherd 0.
blue+whale 0.
siamese+cat 0.
killer+whale 0.
beaver 0.
hippopotamus 0.
persian+cat 0.
spider+monkey 0. humpback+whale 0.
elephant 0.
gorilla 0.
moose 0.
cow 0. fox 0.
seal 0.
sheep 0. pig 0.
otter 0.
mole 0.
walrus 0.
antelope 0.
ox 0.
collie 0.
weasel 0. buffalo 0.
skunk 0.
dalmatian 0.
Mean= 0.
[Table] [Table]
[Table]
giant+panda 0.
moose 0.
deer 0. pig 0.
leopard 0.
seal 0.
wolf 0.
hamster 0.
mouse 0.
beaver 0.
rat 0.
chimpanzee 0.
lion 0.
killer+whale 0.
otter 0.
mole 0.
walrus 0.
bobcat 0.
grizzly+bear 0.
chihuahua 0.
antelope 0.
collie 0.
weasel 0. buffalo 0.
dalmatian 0.
ox 0.
skunk 0.
siamese+cat 0.
german+shepherd 0.
[Table]
E P ROBING V ISUAL R EPRESENTATIONS
We train probes on the image encodings from each of our image encoders to classify ﬁne-grained lexical and coarse grained categorical concepts on several datasets: COCO (Lin et al., 2014), and
CC3M (Sharma et al., 2018), and CIFAR-100 (Krizhevsky et al., 2009).
The architecture is a single linear layer which takes an image encoding of dimension
h
I (see Table 2) and projects to the number of classes for the classiﬁcation task.
For single label classiﬁcation, we use a softmax activation on the logits and train with cross entropy as the loss function.
For multilabel classiﬁcation tasks, we use a sigmoid activation layer on top of the logits and train with a binary cross entropy loss function.
We consider a certain class predicted if the value of the class after the sigmoid is ¿0.5.
Hyperparameters For simplicity, all probes are trained with the same hyperparameters (with a few exceptions for the CC3M probes): learning rate: 1e-4, optimizer: AdamW (Loshchilov &
Hutter, 2018), betas=(0.9, 0.999), batch size: 48 for CC3M probes; 32 for all others, max epochs: 300 for CC3M probes; 300 for all others.
Object Supercategory Objects within class accessory
[Table]
E. COCO O BJECT C LASSES P ROBE
The COCO dataset contains labels for 80 objects in images, typically used for object detection or segmentation tasks.
Because one image can be labeled for multiple objects, we train the probe as a multi-label classiﬁcation task.
In addition to having ﬁne-grained labels for each type, COCO provides labels for each broad category that each object belongs to, called the supercategory .
In Table
[Table]
we show which supercategory each object label falls under.
E.1. C OARSE -G RAINED S UPERCATEGORY P ROBES
We train a multilabel classiﬁcation probe to classify the object category seen in a given image.
We report F1 for each LiMBeR model in Figure 11.
We ﬁnd that BEIT does a bit worse than NFRN and CLIP overall, but is able to classify some categories (images with accessories, vehicles, and ‘outdoor’ objects) well.
E.1. F INE -G RAINED O BJECT L ABEL P ROBES
Next, we look at the probe results for probes trained to identify individual objects by type.
Our results can be found in Figure 12d.
We ﬁnd the same pattern emerges, and that BEIT does not seem to be signiﬁcantly closer to the other pretrained models in terms of F1 on the coarse-grained vs. ﬁne-grained patterns as we might expect.
However, we do show that BEIT encodes strong, but weaker lexical concept categories than the other two models, and that the ﬁnding that BEIT transfers coarser grained information is not due to irreconcilable representational differences between BEIT space and the LM space.
E. CC3M P ROBES
We also train the same set of probes on image data from CC3M, but evaluate on the same validation set from COCO (i.e., the same evaluation as used in Section E.1.2).
The purpose of this experiment is to create a setting for a linear probe that better matches the LiMBeR setup we use in the main paper.
If there are concepts that the probe has no trouble with, but rarely appear in captions, that could be an indicator that the LM and the image encoder represent that concept very differently in representation space.
Data To align the CC3M images with the object labels in COCO, we create labels by looking for exact string matches of the object label words (e.g. “teddy bear”) in CC3M captions.
Of the 80 object classes, we cut out any that have fewer than 1000 images.
This leaves us with 782, training images for the probes across 53 object classes; fewer than the CC3M dataset used to train
LiMBeR , but far more than the previous datasets we used for probing.
Results Because, in this setting, we train our probes on the same distribution as the LiMBeR models, we compare the F1 of the probes identifying objects in images to the F1 of those concepts appearing in the generated captions.
Our results can be seen in Figure 13.
It appears that if the image encoder encodes the lexical concept, it generally also transfers to the LM with LiMBeR .
Limitations of this approach are that (1) the BEIT probe appears much worse at the domain shift from CC3M to
COCO (e.g., F1 for animals drops 0.3 compared to when the probes are trained on COCO) and (2) some words in the label space are often substituted for more common words in generated captions (e.g. “person” could be generated as “man”, “woman”, etc.).
This makes it difﬁcult to recognize cases where the probe succeeds but the transfer fails.
An interesting problem for future work is better understanding which concepts are encoded in an image encoder’s representations, but do not transfer well with a linear map to the LM.
E. CIFAR-100 P ROBES
CIFAR-100 (Krizhevsky et al., 2009) is a dataset of 60,000 32x32 images balanced across 100 object classes.
Like COCO, the 100 object labels are also annotated for coarse-grained object categories including ‘aquatic mammals’ and ‘household furniture’.
For CIFAR data, we train a classiﬁer which classiﬁes an image for a single object label.
Like with COCO, we train a set of probes for the ﬁne and coarse labels.
We were surprised to ﬁnd that for CIFAR images, BEIT representations tended to better than NFRN50 in terms of average F1 (for the ﬁne-grained probe, BEIT: 0.57, NFRN50: 0.47); a ﬁrst for any of the experiments we ran.
Given the majority of evidence shows NFRN encodes lexical category information stronger than BEIT, we hypothesize this is not because of BEIT encoding lexical concepts more strongly, but due to the small resolution of images: because BEIT uses visual tokens, it may be more robust to extremely blurry images of distribution
E.3. C OARSE -G RAINED P ROBES
In Figure 14 we show the F1 results for the coarse-grained image labels probes trained on image encodings from each model.
Each coarse-grained class in the testing set has 500 images each.
E.3. F INE -G RAINED P ROBES
In Figure 15d we show the per object class F1 results for the image encodings from each model.
Each class in the testing set has 100 images each.
Similar to the COCO probe we do not see signiﬁcant intra-model changes between the coarse-grained and ﬁne-grained probe results.
Ground Truth: A black Honda motorcycle parked in front of a garage.
NFRN50 Tuned : my motorcycle
MAGMA (Ours) : my bike
MAGMA (Rel.) : a 1973 Honda CB
NFRN Random: the car on the road
BEIT: the bike.'
NFRN50: a motorcycle on a motorcycle.
CLIP: my motorcycle.
Ground Truth: A bicycle is standing next to a bed in a room.
NFRN50 Tuned : my bike with a basket.
MAGMA (Ours) : my bike
MAGMA (Rel.) : a vintage bike vintage bike
NFRN Random: a young boy in a blue shirt and a red hat on a bicycle
BEIT: the bike.
NFRN50: my bicycle with a basket on the front.
CLIP: my bike, which is a fixed gear bike.
Ground Truth: A man riding a wave on top of a white surfboard.
NFRN50 Tuned : a surfer in the water
MAGMA (Ours) : a surfer riding a wave.
MAGMA (Rel.) : a surfer riding a wave
NFRN Random: the view from the top of the mountain
BEIT: a surfer riding a wave.
NFRN50: a surfer in the water.
CLIP: a surfer riding a wave.
Ground Truth: A person standing on top of a beach flying a kite.
NFRN50 Tuned : a man standing on the beach with a surfboard.
MAGMA (Ours) : a kite on a beach.
MAGMA (Rel.)
: a man running on the beach
NFRN Random: the new car in the garage
BEIT: a man standing on a rock in the sea.
NFRN50: a man surfing on the beach.
CLIP: a man flying a kite on the beach.
Ground Truth:
A lady explains the process of milking a cow.
NFRN50 Tuned : a horse and rider at the fair.
MAGMA (Ours) : a woman on a stage with kids in the audience.
MAGMA (Rel.) : a woman milking a cow in front of a group of children.
NFRN Random: the new restaurant.
BEIT: the crowd at the event.
NFRN50: a crowd of people at a festival
CLIP: the dairy cows.
Ground Truth:
A woman is taking notes in front of her laptop
NFRN50 Tuned : the man who was shot and killed in the lobby of the building.
MAGMA (Ours) : a student studying.
MAGMA (Rel.) : a student working on a paper
NFRN Random: the man who was killed in the attack.
BEIT: the room where the meeting took place.
NFRN50: the room where the meeting took place.
CLIP: a student working on his laptop.
Ground Truth:
A couple of men standing on top of a field together.
NFRN50 Tuned : the team before the match
MAGMA (Ours) : men playing a game of sport.
MAGMA (Rel.) : a men's lacrosse game at the University of
[Table]
England
NFRN Random: the new home of the football team
BEIT: football player and football player during a training session.
NFRN50: the team before the game.
CLIP: the team during a practice session.
[Table]
MAGMA (Rel.) : a toddler boy standing up and holding his head.
NFRN Random: the artist with his son.
BEIT: a boy in a wheelchair.
NFRN50: a little girl with a toy airplane CLIP: person with his new toy
Ground Truth: A white toilet sitting in a corner of a room.
NFRN50 Tuned : the toilet in the bathroom.
MAGMA (Ours) : actor and a toilet
MAGMA (Rel.) : the bathroom in the <PERSON>'s suite.
NFRN Random: the bride and groom at the wedding
BEIT: a white and black cat on a black background.
NFRN50: a toilet in a public restroom.
CLIP: the toilet in the room
Ground Truth: Several different pieces of electronic equipment and computers on a desk.
NFRN50 Tuned : my desk at work.
MAGMA (Ours) : my desk with a laptop and various devices
MAGMA (Rel.) :
my desk
NFRN Random: a man and a woman sitting on a bench in a park.
BEIT: the computer i use.
NFRN50: my computer desk at work.
CLIP: my desk as it looked after a few days of working.
Ground Truth: A living room filled with furniture and a table.
NFRN50 Tuned : the living room.
MAGMA (Ours) : the living room
MAGMA (Rel.) : the living room.
NFRN Random: the living room in the house
BEIT: the living room.
NFRN50: the living room in the house.
CLIP: the living room.
[Figure]
[Figure] [Figure]
Answer: microwave
NFRN50 Tuned : A stove.
MAGMA (Ours) : microwave oven.
MAGMA (Rel.) :
A vent hood.
NFRN Random:
A hob.
BEIT: A stove.
NFRN50: The stove.
CLIP: The oven.
Answer: 1/
NFRN50 Tuned : Enough to make them sick.
MAGMA (Ours) : enough to fill a lunch box.
MAGMA (Rel.) :
Enough to make them sick.
NFRN Random:
They will eat as much as they want.
BEIT: They will eat about a cup of pickle.
NFRN50: They will eat as much as they want.
CLIP: All of it!
Answer: 60 NFRN50 Tuned : ___ MAGMA (Ours) : it took.
MAGMA (Rel.) :
It took her about 10 minutes.
NFRN Random:
It took her about 10 minutes to make the kite fly BEIT:
It took her about five minutes to make it fly.
NFRN50: ___ CLIP:
It took her about a minute to make it fly.
Answer:
no NFRN50 Tuned : Yes, there are people in the photo.
MAGMA (Ours) :
yes MAGMA (Rel.)
: Yes, there are people in the photo.
NFRN Random: Yes, there are people in the photo.
BEIT: yes, there are people in the photo.
NFRN50:
Yes, there are.
CLIP:
Yes, there are people in the photo.
Answer: cloudy
NFRN50 Tuned : Clear
MAGMA (Ours
) : it rains most of the time and the clouds are dark
MAGMA (
Rel.)
: The skies are cloudy
NFRN Random:
They are clear and blue
BEIT:
The sky is clear and the sun is shining
NFRN50: Very cloudy
CLIP:
The skies are cloudy and the winds are strong.
Answer: Gray
NFRN50 Tuned :
The dog is brown
MAGMA (Ours) :
he's blue
MAGMA (Rel
It's a brown dog
NFRN Random
: The dog is black.
BEIT:
It's a brown dog
NFRN
: It is a black and white dog.
[Table]
NFRN50 Tuned :
[Table]
a
[Table]
MAGMA (Ours) : he
[Table]
probably
[Table]
a street
[Table]
MAGMA (Rel.) :
No, he's on his way to a skateboard
NFRN Random:
He is a very smart guy.
BEIT: No
[Table]
he'
[Table]
his
[Table]
to a
[Table]
interview
NFRN50:
He
[Table]
on his
[Table]
to
[Table]
corporate job
CLIP:
No, he
[Table]
s on his
[Table]
to a
[Table]
NFRN50 Tuned :
[Table] [Table]
MAGMA (Ours) : no.
MAGMA (Rel.) :
Yes.
NFRN Random:
Yes, it is a two-way street.
BEIT:
Yes, it is.
NFRN50: Yes, it is.
CLIP:
Yes, it is.
[Table]
Yes
NFRN50 Tuned : No, it looks like it is about to eat you
MAGMA (Ours) : he looks like he is trying to be happy.
MAGMA (Rel.)
: It's a zebra, so it's always happy
NFRN Random:
Yes, he is enjoying the sunshine.
BEIT:
Yes, it is happy.
NFRN50: Yes!
CLIP:
Yes, he is enjoying the grass.
[Table]
What color is the dog on the table?
Is this picture taken inside or outside?
Is he on his way to a corporate job?
Is the street a two-way street?
Does the zebra look happy?
What activity caused him to fall?
How many minutes did it take for the little girl to make her kite ﬂy?
What brand shirt is man wearing?
What is the zebra doing?
Is the player wearing helmet?
What is the cat sitting on?
[Table]
CLIP:
[Table] [Table]
Tuned : It is eating grass.
MAGMA (Ours) : he is eating grass!
MAGMA (Rel.)
It is eating grass.
NFRN Random:
It is trying to get out of the water.
BEIT: It is eating grass.
NFRN50: Eating grass
CLIP:
Eating grass.
Answer: no
NFRN50 Tuned : No, he is not.
MAGMA (Ours) :
no
MAGMA (Rel.) :
No.
NFRN Random:
Yes, he is wearing a helmet.
BEIT:
No, he is not.
NFRN50: No, he is not.
CLIP: No.
Answer: suitcase NFRN50 Tuned : It's a book.
MAGMA (Ours) : a suitcase.
MAGMA (Rel.) :
A suitcase.
NFRN Random:
It is a cat.
BEIT:
It's a cat.
NFRN50: It's a laptop!
CLIP: A suitcase.
How much pickle will they eat?
What is the thing above the stove called?
How are the skies?
What color is the dog on the table?
Is this picture taken inside or outside?
Is he on his way to a corporate job?
Does the zebra look happy?
What activity caused him to fall?
How many minutes did it take for the little girl to make her kite ﬂy?
What brand shirt is man wearing?
What is the zebra doing?
Is the player wearing helmet?
What is the cat sitting on?
Are there people in the photo?
[Figure] [Figure]