Learning grounded word meaning representations on similarity graphs (2021)

Abstract: This paper introduces a novel approach to learn visually grounded meaning representations of words as low-dimensional node embeddings on an underlying graph hierarchy. The lower level of the hierarchy models modality-specific word representations, conditioned to another modality, through dedicated but communicating graphs, while the higher level puts these representations together on a single graph to learn a representation jointly from both modalities. The topology of each graph models similarity relations among words, and is estimated jointly with the graph embedding. The assumption underlying this model is that words sharing similar meaning correspond to communities in an underlying graph in a low-dimensional space. We named this model Hierarchical Multi-Modal Similarity Graph Embedding (HM-SGE). Experimental results validate the ability of HM-SGE to simulate human similarity judgments and concept categorization, outperforming the state of the art.

## introduction during the last decade, there has been an increasing interest in deriving semantic representations from text corpora in terms of word vector space for both words in context and words in isolation @xcite @xcite @xcite . however, semantic representations are tied to sensory experience, at least for concrete nouns @xcite , and deriving them by relying solely on text leads to lack of grounding on the extralanguage modality. for this reason, several works have addressed the problem of grounding perceptual information in the form of visual information approximated by feature norms elicited from humans @xcite , or extracted automatically from images @xcite @xcite , or a combination of them @xcite . furthermore, several integration mechanisms for the linguistic and perceptual modalities have been proposed. they include methods employing transformation and dimension reduction on the concatenation of unimodal representations @xcite ; generative probabilistic models @xcite @xcite ; deep learning methods such as autoencoders and recursive neural networks @xcite . while the approaches above take as input previously extracted unimodal representations, @xcite @xcite learn directly from raw data by integrating both modalities non-linearly in a low-dimensional space within a skip-gram model framework. these approaches typically lead to marginal improvements since only a small part of the vocabulary is covered by a corresponding visual information. recent work has focused on generating pre-trainable generic representations for visual-linguistic tasks by using an instance-level contextualized approach @xcite @xcite . such models have proved to be effective for several natural language applications such as visual question answering and visual commonsense reasoning, but their ability to simulate human behaviour phenomena has never been assessed so far. in this paper, we propose a novel graph-based model for learning word representations across vision and language modalities that can simulate human behaviour in many nlp tasks. our assumption is that words sharing similar meaning correspond to communities in an underlying hierarchy of graphs in low-dimensional spaces. the lower level of the hierarchy models modality-specific word represen-tations through modality-specific but communicating graphs, while the higher level puts these representations together on a single graph to learn a single representation jointly from both modalities. at each level, the graph topology models similarity relations among words, and it is estimated jointly with the graph embedding. to the best of our knowledge this is the first model that uses a graphbased approach to learn grounded word meaning representations. technically, our method is a joint feature learning and clustering approach. moreover, it is compatible with both associative and domain-general learning theories in experimental psychology, following which the learning of a visual (linguistic) output is triggered and mediated by a linguistic (visual) input @xcite , and these mediated representations are then further encoded in higher-level cortices @xcite . this work has applications in cognitive science, prominently in simulations of human behavior involving deep dyslexia, semantic priming and similarity judgments, among others. the contributions of this work are as follows: 1) we propose a novel technical approach to learn grounded semantic representations as lowdimensional embeddings on a hierarchy of similarity graphs, 2) we achieve state-of-the-art performance with respect to several baselines for simulating human behaviour in the tasks of similarity rating and categorization, 3) we demonstrate the ability of our model to perform inductive inference, 4) we validate the proposed approach through an extensive ablation study, and provide insights about the learnt representations through qualitative results and visualizations, 5) our model is compatible with associative and domain-general learning theories in experimental psychology. ## related work distributional semantic models (dsms) are based on the distributional hypothesis (harris, 2013), following which words that appear in similar linguistic contexts are likely to have related meanings. dsms associate each word with a vector (a.k.a. word embedding) that encodes information about its co-occurrence with other words in corpora @xcite . recently, instance-level contextualized word embeddings @xcite have emerged as a natural extension of type-level non-contextualized dsms and have demonstrated their effectiveness with respect to their counterpart in a wide variety of common nlp tasks @xcite @xcite . however, humans learn the verbal description of objects by hearing words while looking at /listening to/interacting with objects. therefore, in recent years there has been an increasing interest in developing linguistic models augmented with perceptual information. these are commonly called grounded semantic spaces. following the classification introduced by @xcite , we can distinguish two integration strategies: 1) a posteriori combination, where each modality is learnt separately and they are integrated afterwards, and 2) simultaneous learning, where a single representation is learnt from raw input data enriched with both modalities. a posteriori combination. several works aimed at projecting directly vision and language into a common space. among them, @xcite concatenate and project two independently constructed textual and visual spaces onto a lowerdimensional space using singular value decomposition (svd). other approaches along the same line build on extensions of topic models as latent dirichlet allocation (lda), where topic distributions are learnt from the observed variables (words and other perceptual units) @xcite @xcite . in @xcite an empirical improvement is obtained by using state-of-the-art convolutional neural networks to extract visual features, and the skip-gram model for textual features, that are simply concatenated. in @xcite a stacked auto-encoder framework is used to learn a representation by means of an unsupervised criterion (the minimization of the reconstruction error of the attribute-based representation) and then fine-tuned with a semi-supervised criterion (object classification of the input). in the approach proposed by @xcite the weights of the unimodal feature concatenation are learnable parameters that allow to dynamically fuse representations from different modalities according to different types of words. other works focus on learning bimodal representations that are task-specific, with the goal of reasoning about one modality given the other @xcite . for example, the aim of image retrieval is to find a mapping between two modalities to tackle an image based task such as zero-shot learning @xcite or caption generation/retrieval and figure 1 : hm-sge model overview. the first layer learns unimodal representations conditioned to the other modality via its corresponding similarity graph. the concatenation of the enhanced representations is then used to learn jointly a common representation. caption-based image retrieval @xcite . these models typically use a training criterion and an architecture suited to the task at hand. simultaneous learning. little work has explored the possibility of learning multimodal representations directly from raw input data, i.e, images and corpora, building on the skip-gram framework. @xcite treat perceptual input as a word linguistic context and has proved to be effective in propagating visual knowledge into abstract words. @xcite modify the skipgram objective function to predict both visual and linguistic features and is especially good in zeroshot image classification. @xcite contributed to this research line by leveraging the visual surroundings of objects to fulfill the distributional hypothesis for the visual modality. this class of approaches typically leads only to a small empirical improvement of linguistic vectors since words from the raw text corpus associated with images (and hence perceptual information) cover only a small portion of the training dataset. in the last few years, increasing efforts are being devoted to deriving generic pre-trainable representations for visual-linguistic tasks based on transformers @xcite @xcite . ## proposed approach model assumptions and psychological foundations. we assume that each modality, linguistic (x) and visual (y ), can be well represented by an unknown underlying graph in a low-dimensional space, in which nodes correspond to words and weighted edges to respective word similarity. the associative nature of human memory, a widely accepted theory in experimental psychology @xcite , suggests that beyond being complementary, there exists a certain degree of correlation between these modalities @xcite . therefore, in a first step, our model aims at exploiting these correlations to enhance each unimodal representation individually. this simulates the fact that when humans face, for instance, the task of visual similarity rating of a pair of images, their mental search necessarily includes both components, the visual and the semantic one. in a second step, these enhanced unimodal representations are mapped into a common embedding space by inducing semantic representations that integrate both modalities. this agrees with neuroscientific findings showing that it is unlikely that humans have separate representations for different aspects of word meaning @xcite . overall, the proposed two layer structure, where the first layer has two modality-specific branches that provide input to a second single-generic branch, is compatible with theories of both associative memory @xcite and domain-general learning @xcite . ## conclusion this paper has proposed a novel approach, named hm-sge, to learn grounded word meaning representations as low-dimensional node embeddings on a hierarchy of graphs. the first layer of the hierarchy encodes unimodal representations conditioned to the other modality, and the second layer integrates these enhanced unimodal representations into a single one. the proposed hm-sge approach is compatible with theories of associative memory @xcite and of domain-general learning @xcite . comparative results on word similarity simulation and word categorization show that our model outperforms baselines and related models trained on the same attribute-based input. our evaluation reveals that hm-sge is particularly good at learning enhanced unimodal representations that simulate how the response of our brain to semantic tasks involving a single modality is always triggered by other modalities. moreover, it succeeds in encoding these unimodal representations into a meaningful unified representation, compatible with the point of view of domain-general learning theory. the ablation study thoughtfully validates the proposed hierarchical architecture. beside quantitative results, we give several insights on the learnt grounded semantic space through visualization of nearest neighbors, clusters, and most similar pairs. these additional results corroborate the quality of the learnt multimodal representations. furthermore, the proposed approach is able to perform inductive inference for concepts for which only one modality is available.