IMAGINATOR : Pre-Trained I mage+ T ext Joint Embeddings using Word-Level Grounding of Images (2023)

Abstract: Word embeddings, i.e., semantically meaningful vector representation of words, are largely influenced by the distributional hypothesis You shall know a word by the company it keeps (Harris, 1954), whereas modern prediction- based neural network embeddings rely on de- sign choices and hyperparameter optimization. Word embeddings like Word2Vec, GloVe etc. well capture the contextuality and real-world analogies but contemporary convolution-based image embeddings such as VGGNet, AlexNet, etc. do not capture contextual knowledge. The popular king-queen analogy does not hold true for most commonly used vision embeddings. In this paper, we introduce a pre-trained joint embedding (JE), named IMAGINATOR, trained on 21K distinct image objects. JE is a way to encode multimodal data into a vec- tor space where the text modality serves as the grounding key, which the complementary modality (in this case, the image) is anchored with. IMAGINATOR encapsulates three in- dividual representations: (i) object-object co- location, (ii) word-object co-location, and (iii) word-object correlation. These three ways cap- ture complementary aspects of the two modal- ities which are further combined to obtain the final object-word JEs. Generated JEs are intrinsically evaluated to assess how well they capture the contextual- ity and real-world analogies. We also evalu- ate pre-trained IMAGINATOR JEs on three downstream tasks: (i) image captioning, (ii) Im- age2Tweet, and (iii) text-based image retrieval. IMAGINATOR establishes a new standard on the aforementioned downstream tasks by out- performing the current SoTA on all the selected tasks. The code is available at https:// github.com/varunakk/IMAGINATOR.

## intro-joint modality and contextuality word embeddings are learned representations such that words with similar meanings are represented similarly. distribution-based compositional word embeddings like word2vec @xcite and glove @xcite are popular in modern nlp. these are used to extract the notion of relatedness across different words, and capture the overall semantic meaning of a text. consider the king-queen @xcite word vector analogy (figure 1 ), which shows how good these word embeddings are at capturing syntactic and semantic regularities in language. the notion of contextual similarity (i.e., words occurring together) is used in learning the representations, because of which vector arithmetic like king -man + @xmath0. see figure 1 @xcite . deriving an analogous representation using images is a challenging task since the concept of relatedness among images is not well-defined. motivated by this argument, we propose creating joint embeddings (jes) that can represent real-world analogies, which can aid in solving several multimodal tasks owing to their distributional semantics. the king-queen vs. man-woman analogy has been popularized by @xcite , whereas drawing a similar analogy in image vector space is rather difficult. we argue joint embedding is the alternative. ## related works: contemporary joint embedding methods canonical correlation analysis (cca) based methods use similarities to project two inputs onto a vector space. clip @xcite utilizes contrastive pre-training and encodes aligned image and text embeddings with the help of text and visual modality encoders. stanford's joint embedding (kolluru, 2019) uses vgg-19 @xcite and glove @xcite to generate the image and text encodings using a triplet loss. chen et al. (2020) proposed uniter, trained on a large dataset, which uses an image and text encoder and a transformer to generate the final embeddings. jia et al. (2021) use a noisy dataset of 1 billion (image, alt-text) pairs and propose a dual architecture for aligning and generating the visual and textual embeddings. this architecture uses contrastive loss for learning. tan @xcite proposed a framework to create a relation between visual and language modalities. this architecture consists of three encoders, one object relation encoder, a language encoder and a cross-modal encoder. compared to the aforementioned prior works, illustrated in appendix figure 8 , the unique differentiating factor with imagina-tor is that we focus on the word-level grounding @xcite of images while prior works perform embedding generation at the sentence level. our belief is that this will help us learn rich relational features, i.e., features that are rich encapsulations of words and the corresponding objects they represent via images. ## imaginator -learning joint embeddings off-the-shelf word embeddings like word2vec @xcite , glove @xcite and the embeddings generated by bert @xcite , gpt @xcite are used for tacking several downstream nlp tasks. the motivation behind creating imagina-tor is to have similar pre-trained embeddings for vision-language tasks. researchers can download pre-trained jes and utilize them for any visionlanguage task they have in hand. existing techniques have only explored jes from the sentencelevel perspective, which makes it less flexible to repurpose them for other tasks, but most importantly, demands a lot more data for the model to understand and derive meaningful relationships. we thus operate at the word level rather than sentence-level, to help improve the "sharpness" of the data, with the hope that this would, in turn, help synthesize higher relational features that can offer optimal performance on downstream tasks. to that end, we make some simple assumptions and posit arguments on their choice as better alternatives. ## lessons learnt from nlp word embeddings are learnt in two major ways: (i) classical count based methods, and (ii) neural network based prediction methods. @xcite argue that the performance gains of neural network based word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. furthermore, they show that these modifications can be transferred to traditional distributional models, yielding similar gains. in contrast to prior reports, they show mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others. therefore, we remain grounded to count-based distributional semantics methods. raw counts or normalized counts are not useful, rather we choose alternatives like pmi and svd. ## intrinsic evaluation of imaginator to be able to make vector arithmetic like king -man + @xmath1. contemporary image embeddings are devoid of contextuality, whereas text embeddings are much more meaningful, as shown in figure 1. with joint embeddings, we aim to add a contextual component to improve the semantic richness of the joint embeddings vector space. we use two kinds of intrinsic evaluation setup to evaluate imaginator: (i) word contextuality, and (ii) image similarity. ## imaginator for downstream tasks the downstream vision-language (vl) tasks chosen to test our pre-trained jes are: (i) image captioning, (ii) image2tweet @xcite , and (iii) text-based image retrieval. for all the downstream tasks, our models have around approx 160m parameters out of which 138m come from the vgg19. the vgg19 is frozen uring training. ## conclusion and futurework we proposed a new pre-trained joint embedding imaginator. our major contribution is on adopting count-based methods for joint modality, echoing the philosophy from @xcite . we present an in-depth intrinsic evaluation along with a new architecture bert im agin at or . imag-inator outperformed sota on three tasks: (i) image captioning, (ii) image2tweet, and (iii) textbased image retrieval. in the future, we would like to explore other multimodal tasks such as vqa.