Visual Grounding of Inter-lingual Word-Embeddings (2022)

Abstract: Visual grounding of Language aims at enriching textual representations of language with multiple sources of visual knowledge such as images and videos. Although visual grounding is an area of intense research, inter-lingual aspects of visual grounding have not received much attention. The present study investigates the inter-lingual visual grounding of word embeddings. We propose an implicit alignment technique between the two spaces of vision and language in which inter-lingual textual information interacts in order to enrich pre-trained textual word embeddings. We focus on three languages in our experiments, namely, English, Arabic, and German. We obtained visually grounded vector representations for these languages and studied whether visual grounding on one or multiple languages improved the performance of embeddings on word similarity and categorization benchmarks. Our experiments suggest that inter-lingual knowledge improves the performance of grounded embeddings in similar languages such as German and English. However, inter-lingual grounding of German or English with Arabic led to a slight degradation in performance on word similarity benchmarks. On the other hand, we observed an opposite trend on categorization benchmarks where Arabic had the most improvement on English. In the discussion section, several reasons for those findings are laid out. We hope that our experiments provide a baseline for further research on inter lingual visual grounding.

## introduction distributional semantic models (dsms) have long been used to capture words' meaning. they estimate semantic representations from co-occurrences of words in text corpora. even though embeddings are the dominant method for large scale data, from a psychological and cognitive point of view, distributional models suffer from the problem referred to as the symbol grounding problem @xcite : the meaning of a symbol (word) is entirely accounted for in terms of other symbols without any links to the outside world. in the context of natural language processing (nlp), grounding is defined as " the process of linking the symbolic representation of language (e.g., words) into the rich perceptual knowledge of the outside world " @xcite . moreover, @xcite have proved that multi-modal learning outperforms uni-modal learning as it has access to a better quality latent space representation. many studies have addressed grounding of language in vision, typically focusing on grounding for english @xcite . as a consequence, inter-lingual visual grounding is still poorly understood. this study investigates whether monolingual textual embeddings benefit from the knowledge of other languages in the process of visual grounding. we extend a state-of-the-art model for monolingual visual grounding @xcite by considering different combinations of three languages, namely, english, german, and arabic. using various word categorization benchmarks, our experiments show that the three languages profitably exchange inter-lingual knowledge across a simple linear vector space. to the best of our knowledge, we are the first to investigate the problem of visual grounding of inter-lingual word embeddings. overall, our contributions are as follows: a) we propose a simple extension of a state-of-theart visual grounding model to integrate three different languages. b) we obtain zero-shot visually grounded embeddings in three languages. c) using various benchmarks, we reveal how visual grounding changes textual vector space across languages and show that inter-lingual knowledge transfers to downstream tasks. our paper is structured as follows: section 2 briefly highlights the related works. section 3 introduces our problem of interest. in section 4 our pro-posed model is elaborated. implementation details are covered in section 5. the results are presented in section 6, with further discussion in section 7. in section 8, we conclude our research, and finally, we point out the limitations and future directions of our work. ## related work there have been many studies on language grounding in vision most of which focus on monolingual visual grounding. there have been also other works on cross-modal and cross-lingual representations tailored for specific downstream applications. ## inter-lingual visual grounding multilingual-language models hold great promise for the development of embeddings for underresourced languages @xcite . the central idea in this line of research is that different languages bring different perspectives (e.g., cultural information and grammar) which can inform each other, resulting in a richer model that has a better understanding of words' meanings in any specific language. moreover, since typical visual scenes are thought to produce similar information across different languages, integrating visual knowledge (e.g., images) into a multilingual model can contribute to obtaining a better quality grounded embedding space. ## model architecture our model maps a textual description of an image into its corresponding image representation. it makes use of a linear alignment to preserve most of the textual knowledge in the word embeddings, allowing only subtle modifications by the error received from the image. it is trained using multilingual image captioning data. the model is given the task to match, for a given image, the multilingual captions to that image in such a way that language-specific features are preserved, and not overwhelmed by inter-lingual features, and image features. our model maps two (or three) languages to the grounded space using a shared linear alignment. for instance, figure 1 introduces the model for the combination of english and arabic languages. let d be the dataset consisting of triple samples of (i, s en , s ar ) ∈ d. here i refers to an image, s en and s ar denote matching captions of i in english and arabic respectively. as shown in figure 1 , the two captions are passed through a pre-trained embedding layer (glove) (pennington et al., 2014) to obtain their textual representations t en , t ar which are then mapped to a visually grounded space through a linear transformation. we refer to this linear transformation as the alignment layer. the alignment layer is used to extract grounded embeddings after training. during training, grounded word vectors of each caption are encoded as a single vector using an lstm layer as follows: v @xmath0, c 0 , h 0 |θ), v @xmath1, c 0 , h 0 |θ) where, g en , g ar denote the grounded word vectors of the english and arabic captions respectively. c 0 , h 0 and θ represent the initial cell state, initial hidden state, and the trainable parameters of the lstm. the parameters of the linear alignment and the lstm layer are optimized to match the sentence representations in both languages to the same image vector v i as follows: ## implementation details we used the microsoft coco 2017 dataset @xcite for our experiments. this dataset consists of 123,287 images with 5 captions each. it is split into 118k training images and 5k validation images. we experimented with three languages for the captions: english, arabic, and german. the original dataset provided by microsoft contains the english captions. we obtained the german captions from @xcite , who translated the english coco captions using the fairseq neural machine translator, and the arabic captions from @xcite , who generated the captions using google's advanced cloud translation api. for the arabic version of coco, we only had available to us translations of the captions for 82k samples, which we split into 77k samples for training and 5k samples for validation, and this is the set of images that we use for models that included arabic. for fair comparisons, we also investigated model performance for english and german using the same 82k images. for all the experiments, we used tensorflow as a development framework . the training environment is similar to the one used by @xcite . we used a batch size of 256 image-caption pairs. we trained for 20 epochs with 5 epochs as early stopping tolerance, using the nadam optimizer @xcite with a learning rate of 0.001. the image vectors were obtained using pre-trained vectors from inception-v3 @xcite , which are based on ima-genet @xcite . for pre-trained textual embeddings we used glove embeddings @xcite . the vocabulary considered for training english comprised the 10k most frequent words. for german and arabic, which have much richer inflectional systems compared to english, we took into account the 30k most frequent words. we set the dimension of grounded word embeddings to 1024 (g @xmath2, and matched the size of the lstm's output to that of the image vectors (both to 2048). both the embedding layer and the pre-trained cnn were frozen during training. ## results in this section, we explain our evaluation criteria and report the results of our experiments. we use various word similarity/relatedness and word categorization benchmarks and provide both quantitative and qualitative results. ## discussion we proposed an inter-lingual visual grounding model on textual word embeddings. our model thus far supports the benefit of visual grounding and inter-lingual visual grounding on various word similarity and word categorization benchmarks. some of the results in section 6 however are hard to interpret. in this section, we will discuss possible explanations for the model's behavior on different tasks across different languages. on the word similarity benchmarks (tables 1 , 2 , and 3) we observe that german and english seem to interact more efficiently than arabic with either. we believe the slight degradation in performance when adding arabic might be due to the fact that the arabic language structure is quite different: much more information is packed into its verbs, and pronouns are used differently and more sparingly. moreover, its orthography leaves out a lot of phonological information (hardly any vowels), so word embeddings are much more ambiguous relative to english or german. therefore, the semantic spaces that are constructed are much less similar to that in the two other languages. apart from the evident differences between arabic and the other two languages, it is worth mentioning that adding arabic is far from detrimental. that is, the resulting embeddings (arabic added) still outperform the textual embeddings significantly. this implies that there exists a linearly aligned common core between the three languages (vector spaces) which as observed in section 6.3, yielded the lowest variance and more pure vector space. ## conclusions the main purpose of this study is to shed light on the problem of inter-lingual visual grounding. we stated the importance of grounding in language understanding and the cognitive plausibility of text representations. we also suggested a baseline architecture for inter-lingual visual grounding and analyzed the performance of the resulting embeddings on word similarity and categorization benchmarks. our findings indicate that inter-lingual features lead to improvements on both similarity and categorization benchmarks with a more significant effect on categorization. our results on the similarity benchmarks indicate that inter-lingual visual grounding is more beneficial for related languages such as english and german, but can lead to reduced performance when unrelated languages, such as english and arabic, or german and arabic, are considered jointly. on the other hand, arabic provided the most improvement on categorization benchmarks for grounded english embeddings. we hope that these initial steps towards interlingual visual grounding inspire further research. low-resourced languages might benefit from joint processing with high-resourced languages in multilingual models but one has to make sure that their unique characteristics are not overwhelmed and masked by datasets acquired in different cultural settings.