Learning Functional Distributional Semantics with Visual Data (2022)

Abstract: Functional Distributional Semantics is a recently proposed framework for learning distributional semantics that provides linguistic interpretability. It models the meaning of a word as a binary classifier rather than a numerical vector. In this work, we propose a method to train a Functional Distributional Semantics model with grounded visual data. We train it on the Visual Genome dataset, which is closer to the kind of data encountered in human language acquisition than a large text corpus. On four external evaluation datasets, our model outperforms previous work on learning semantics from Visual Genome.

## introduction the target of distributional semantics models is to understand and represent the meanings of words from their distributions in large corpus. many approaches learn a numerical vector for each word, which encodes its distributional information. they can be roughly divided into two categories: frequency-based methods such as cooccurrence matrix @xcite , and predictionbased methods such as word2vec @xcite . more recently, progress has been made in learning word representations in a specific context, which are also called contextualized embeddings. examples include elmo @xcite and bert @xcite . functional distributional semantics is a framework that not only provides contextualized semantic representations, but also provides more interpretability. it was first proposed by @xcite , and it explicitly separates the modeling of words and the modeling of objects and events. this is a fundamental distinction in predicate logic. while logic is not necessary for all nlp tasks, it is an essential tool for modeling many semantic phenomena (for example, see @xcite @xcite . for semantic research questions, having a logical interpretation is a clear advantage over vector-based models. we will explain the framework in section 2.2. another issue with distributional semantic models, as discussed by @xcite , is the symbol grounding problem -if meanings of words are defined in terms of other words, the definitions are circular. during human language acquisition, words are learned while interacting with the physical world, rather than from text or speech alone. an important goal for a semantic theory is to explain how language relates to the world, and how this relationship is learned. we focus on the visual genome dataset, not only because it provides relatively fine-grained annotations, but also it is similar to realistic circumstance encountered during language acquisition, as we will explain in section 2.3. our main theoretical contribution is to adapt the functional distributional semantics framework to better suit visual data. this is a step approaching the completion of long-term goal: leveraging previous work @xcite , we could joint train the functional distributional semantics model with both textual and visual data. in order to make it compatible with modern techniques for machine vision, while retaining its logical interpretability, we replace the rbm of previous work with a gaussian mrf, as explained in section 3. our main empirical contribution is to demonstrate the effectiveness of the resulting model. in section 4.1, we intrinsically evaluate the major components of our model, to see how well they fit the training data. in section 4.2, we evaluate our model on four external evaluation datasets, comparing against previous approaches to learning from visual genome, as well as strong text-based baselines. not only do we confirm herbelot (2020)'s finding that learning from grounded data is more data-efficient than learning from text alone, but our model outperforms the previous approaches, demonstrating the value of our functional approach. ## model and methods we will explain the probabilistic structure of our model in section 3.1, and how we train the components in sections 3.2 and 3.3. in section 3.4, we present an inference model to infer latent pixies from words and the context. figure 2 : our probabilistic graphical model. the top blue box contains the world model, which learns the joint distribution of the observed pixies x, y and z from their corresponding images. the bottom red box shows the lexicon model, where each semantic function in the vocabulary v is applied to each pixie. for each pixie, one predicate is generated, with probability proportional to the probability of truth. ## evaluation to train our model, we follow the same preprocessing and filtering of visual genome as herbelot (2020). details of pre-processing and hyperparameters are given in the appendix. ## conclusion in this paper, we proposed a method to train a functional distributional semantics model with visual data. our model outperformed the previous works and achieved a new state of the art on learning natural language semantics from visual genome. further to this, our model achieved better performance than word2vec and glove on simlex-999 and matched word2vec-1b on men. this shows that our model can use parameters and data more efficiently than word2vec and glove. additionally, we also showed that our model can successfully be used to make contextual inferences. as future work, we could leverage previous work to jointly train the functional distributional semantics model with both visual and textual data, such that we could improve the vocabulary coverage and have better understanding of abstract words.