Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training (2021)

Abstract: Language grounding aims at linking the symbolic representation of language (e.g., words) into the rich perceptual knowledge of the outside world. The general approach is to embed both textual and visual information into a common space -the grounded space- confined by an explicit relationship. We argue that since concrete and abstract words are processed differently in the brain, such approaches sacrifice the abstract knowledge obtained from textual statistics in the process of acquiring perceptual information. The focus of this paper is to solve this issue by implicitly grounding the word embeddings. Rather than learning two mappings into a joint space, our approach integrates modalities by implicit alignment. This is achieved by learning a reversible mapping between the textual and the grounded space by means of multi-task training. Intrinsic and extrinsic evaluations show that our way of visual grounding is highly beneficial for both abstract and concrete words. Our embeddings are correlated with human judgments and outperform previous works using pretrained word embeddings on a wide range of benchmarks. Our grounded embeddings are publicly available here.

## introduction the distributional hypothesis asserts that words occurring in similar contexts are semantically related @xcite . current state-of-the-art word embedding models @xcite , despite their successful application to various nlp tasks @xcite , suffer from the lack of grounding in general knowledge @xcite , such as captured by human perceptual and motor systems @xcite . to overcome this limitation, research has been directed to linking word embeddings to perceptual knowledge in visual scenes. most studies have attempted to bring visual and language representations into close vicinity in a common feature space @xcite @xcite . however, studies of human cognition indicate that the brain processes abstract and concrete words differently @xcite due to the difference in associated sensory perception. according to @xcite , similar activity for both categories are observed in the perirhinal cortext, a region related to memory and recognition, whereas in the parahippocampal cortex, associated with memory formation, higher activity only occurs for abstract words. we argue that forcing the textual and visual modalities to be represented in a shared space causes grounded embeddings to suffer from the bias towards concrete words as reported by @xcite ; @xcite . therefore, we propose a zero-shot approach that implicitly integrates perceptual knowledge into pre-trained textual embeddings @xcite and fasttext @xcite ) via multitask training. our approach learns multifaceted grounded embeddings which capture multiple aspects of words' meaning and are highly beneficial for both concrete and abstract words. figure 1 lays out the architecture of our model. it learns a reversible mapping from pre-trained textbased embeddings to grounded embeddings which maintains the linguistic co-occurrence statistics while integrating visual information. the architecture features a similar structure as an auto-encoder @xcite translating from words to grounded space and back. the training is carried out as multi-task learning by combining image captioning in two directions and image-sentence pair discrimination. at the core is a mapping matrix that acts as an intermediate representation between the grounded and textual space, which learns to visually ground the textual word vectors. this mapping is trained on a subset of words and then is the zero-shot mapping matrix m , shared by all the tasks, learns to visually ground the textual word vectors by learning a reversible mapping from textual space to grounded space. we evaluate our grounded embeddings on both intrinsic and extrinsic tasks @xcite and show that they outperform textual embeddings and previous related works in the majority of cases. overall, our contributions are the following: a) we design a language grounding framework that can effectively ground different pre-trained word embeddings in a zero-shot manner; b) we create visually grounded versions of two popular word embeddings and make them publicly available; c) unlike many previous works, our embeddings support both concrete and abstract words; d) we show that visual grounding has the potential to refine the irregularities of a text-based vector space. ## related works the many attempts to combine images and text in order to obtain visually grounded word/sentence representations can be grouped into the following categories. feature level fusion: where the grounded embedding is the result of combining the visual and textual features. combining strategies range from simple concatenation to adopting svd and gru gating mechanisms @xcite @xcite . mapping to perceptual space: this is usually a regression task predicting the image vector given its corresponding textual vector. the grounded embedding are extracted from an intermediate layer in auto-encoders @xcite , the output of an mlp @xcite or an rnn @xcite . another method is mapping both modalities into a common space in which their distance is minimized @xcite . equipping distributional semantic models with visual context: here images are treated as a context in the process of computing the word vectors. many of these approaches modify the word2vec @xcite and glove (pennington et al., 2014) models by incorporating image features to the context for concrete words @xcite @xcite ; minimizing the maxmargin loss between the image-vector and its corresponding word vectors @xcite ; providing social cues based on child-directed speech along with visual scenes @xcite ; or by extracting the relationship between words and images using multi-view spectral graphs @xcite . hybrid: this category covers the combination of previous methods and other strategies. here, the grounded word vectors are usually the results of updating the textual word vectors during training @xcite or the output of sentence encoders such as lstm @xcite . such methods include predicting the image vector along with training a language model @xcite or generating an alternative caption at the same time @xcite . other approaches such as using the coefficients of classifiers for grounded representation have also emerged @xcite . our model falls in the hybrid category as we take a multitasking approach. however, unlike some previous works @xcite @xcite we do not impose explicit constraints between the image features and their captions. our model learns the relationship indirectly via multitask training. ## multi-task visual grounding in this section, we present the details of the developed method. the training data set d consists of image-caption pairs, (s k , i k ) ∈ d, with s @xmath0, w 2 ...w n ] being a sentence with n words describing the image i k . we use the mi-crosoft_coco_2017 dataset @xcite in our experiments. let t e (w) ∈ r d be a pretrained textual embedding of the word w, which has been trained on textual data only (e.g., glove). the objective is to train a mapping matrix m to ground the word vector t e (w) visually, resulting in a grounded embedding g e (w) = t e (w) • m , where g e (w) ∈ r c . to do so, we train the matrix m to refine the textual vector space via two image-based language model tasks and a binary discrimination task on image-sentence pairs. for the language models, a gru @xcite is trained to predict the next word, given the previous words in the sentence provided as image caption, and its associated image vector. the transpose of the textual embedding t e is used to compute the probability distribution over the vocabulary (see figure1). we employ an identical scenario to form a second language model task using another gru, where the sentence is fed backward into the model. the image-sentence discrimination is a binary classification task predicting whether the given sentence s k represented in the grounded space matches the image i k . by training the model simultaneously on these three tasks confined by a linear transformation, we augment the visual information into the grounded embeddings (output of mapping matrix in figure 1 ) while preserving the underlying structure of the textual embeddings. ## experimental setup we use the microsoft_coco_2017 dataset @xcite for training. each sample contains an image with 5 captions. the dataset is split into 118k train and 5k validation samples. each batch includes 256 image vectors along with one of their captions. hence, multiple image vectors might occur in each batch. image vectors are obtained by transferring the penultimate layer of pre-trained inception-v3 @xcite trained on imagenet @xcite . a nn with one hidden layer and tanh activation is employed to project the image vectors into the initial hidden state of the grus: h @xmath1. we lowercase all the words, delete the punctuation marks, and only keep the top 10k most frequent words. two popular pre-trained textual word embeddings namely glove (crawl-300d-2.2m -cased) and fasttext (crawl -300d -2m -subw ) are used for initialization of the embedding t e . the mapping matrix m transforms the textual embeddings into the grounded space. we investigate the best dimension of this step and the improvement over pure textual embeddings in the next sections. batch normalization @xcite is applied after each gru. for the regularization, r(α = 0.001, β = 1) for glove and r(α = 0.01, β = 0) for fasttext yielded the best relative results by meta parameter search. this shows that fasttext embeddings require more deviation (β = 0 indicates 90 degree deviation) to adapt to the proposed tasks. we trained the model for 20 epochs with 5 epochs tolerance early stopping using nadam @xcite with a learning rate of 0.001. as we train a single mapping matrix m for projecting from textual to grounded space, it can be used after the training to transfer out-of-vocabulary (oov) word-vectors into the grounded space in a zero-shot manner. this way, visually grounded versions of both glove and fasttext are obtained despite being exposed to only 10k words. ## evaluations while the question of what is a good word embedding model is still open @xcite , there are two main categories of evaluation methods: intrinsic and extrinsic. intrinsic evaluators measure the quality of word embeddings independent of any downstream tasks. for instance, quality can be assessed by comparing similarities between embeddings with word similarities as perceived by human raters. extrinsic evaluators on the other hand assess the performance based on sentence-level downstream tasks. there is not necessarily a positive correlation between intrinsic and extrinsic methods for a word embedding model @xcite . nonetheless, we use both types of evaluators to compare our visually grounded embeddings with those presented in related works as well as to purely text-based embeddings. baselines: we considered two types of embeddings as baselines 1) the pre-trained textual embeddings t e , 2) t e refined based only on the captions without injecting any image information using a similar language modeling task l f w with a one-layer gru (h @xmath2. we refer to the second baseline as c_glove and c_fasttext for glove and fasttext trained only on captions. intrinsic evaluators: we evaluate on some of the common lexical semantic similarity benchmarks: men @xcite , simlex999 @xcite , rare-words @xcite , mturk771 @xcite , wordsim353 @xcite , and simverb3500 @xcite . the evaluation metric is the spearman correlation between the predicted cosine similarity vector and the ground truth. ## results intrinsic evaluation -baselines: ## model analysis we further analyze the performance of our model from different perspectives as follows. dependency on the encoding dimension c: we train our model with different dimensions of the ## conclusion we investigated the effect of integrating perceptual knowledge from images into word embeddings via multi-task training. we constructed the visually grounded versions of glove and fasttext by learning a zero-shot transformation from textual to grounded space trained on the mscoco dataset. results on intrinsic and extrinsic evaluation support that visual grounding benefits current textual word embedding models. the major findings in our experiments are as follows: a) our improvement of visual grounding is not limited to words with concrete meanings and covers highly abstract words as well. b) discrimination between relatedness and similarity is more precise when using grounded embeddings. c) perceptual knowledge can profitably be transferred to purely textual downstream tasks. moreover, we showed that visual grounding has the potential to refine the irregularities in textual vector spaces by aligning words with their realworld relations. this paves the way for future research on how visual grounding could resolve the problem of dissimilar words that occur frequently in the same context (e.g., small and big). in the future, we will investigate whether transformer blocks could profitably replace the gru cells since they lead the state-of-the-art in many downstream sentence tasks. moreover, while thus far our focus has been on words, a similar approach could be extended to obtain grounded sentence representations.