Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics (2020)

Abstract: Functional Distributional Semantics provides a linguistically interpretable framework for distributional semantics, by representing the meaning of a word as a function (a binary classifier), instead of a vector. However, the large number of latent variables means that inference is computationally expensive, and training a model is therefore slow to converge. In this paper, I introduce the Pixie Autoencoder, which augments the generative model of Functional Distributional Semantics with a graph-convolutional neural network to perform amortised variational inference. This allows the model to be trained more effectively, achieving better results on two tasks (semantic similarity in context and semantic composition), and outperforming BERT, a large pre-trained language model.

## introduction the aim of distributional semantics is to learn the meanings of words from a corpus @xcite @xcite . many approaches learn a vector for each word, including count models and embedding models (for an overview, see @xcite , and some recent approaches learn a vector for each token in a particular context (for example @xcite . however, such vector representations do not make a clear distinction between words and the things they refer to. this means that such models are challenging to interpret semantically. in contrast, functional distributional semantics @xcite aims to provide a framework which can be interpreted in terms of model theory, a standard approach to formal semantics. furthermore, this framework supports first-order logic, where quantifying over logical variables is replaced by marginalising out random variables @xcite . this connection to logic is a clear strength over vector-based models. even the linguistically inspired tensor-based framework of @xcite and @xcite cannot model quantifiers, as shown by @xcite . however, the linguistic interpretability of functional distributional semantics comes at a computational cost, with a high-dimensional latent variable for each token. training a model by gradient descent requires performing bayesian inference over these latent variables, which is intractable to calculate exactly. the main theoretical contribution of this paper is to present an amortised variational inference algorithm to infer these latent variables. this is done using a graph-convolutional network, as described in §3. the main empirical contribution of this paper is to demonstrate that the resulting system, the pixie autoencoder, improves performance on two semantic tasks, as described in §4. i also present the first published results of applying a large language model (bert) to these tasks, showing that results are sensitive to linguistic detail in how the model is applied. despite being a smaller model trained on less data, the pixie autoencoder outperforms bert on both tasks. while the proposed inference network is designed for functional distributional semantics, the proposed techniques should also be of wider interest. from a machine learning perspective, amortised variational inference with graph convolutions ( §3.3) could be useful in other tasks where the input data is a graph, and the use of belief propagation to reduce variance ( §3.4) could be useful for training other generative models. however, the most important contribution of this work is from a computational semantics perspective. this paper takes an important step towards truth-conditional distributional semantics, showing that truth-conditional functions can be efficiently learnt from a corpus. ## functional distributional semantics in this section, i summarise previous work on functional distributional semantics. i begin in §2.1 by introducing model-theoretic semantics, which motivates the form of the machine learning model. i then explain in §2.2 how the meaning of a word is represented as a binary classifier, and finally present the probabilistic graphical model in §2.3. ## the pixie autoencoder a practical challenge for functional distributional semantics is training a model in the presence of high-dimensional latent variables. in this section, i present the pixie autoencoder, which augments the generative model with an encoder that predicts these latent variables. for example, consider dependency graphs for the child cut the cake and the gardener cut the grass. these are true of rather different situations. although the same verb is used in each, the pixie for cut should be different, because they describe events with different physical actions and different tools (slicing with a knife vs. driving a lawnmower). training requires inferring posterior distributions for these pixies, but exact inference is intractable. in §3.1 and §3.2, i describe previous work: amortised variational inference is useful to efficiently predict latent variables; graph convolutions are useful when the input is a graph. in §3.3, i present the encoder network, to predict latent pixies in functional distributional semantics. it uses the tools introduced in §3.1 and §3.2, but modified to better suit the task. in §3.4, i explain how the encoder network can be used to train the generative model, since training requires the latent variables. finally, i summarise the architecture in §3.5, and compare it to other autoencoders in §3.6. ## experiments and evaluation i have evaluated on two datasets, chosen for two reasons. firstly, they allow a direct comparison with previous results @xcite . secondly, they require fine-grained semantic understanding, which starts to use the expressiveness of a functional model. more open-ended tasks such as lexical substitution and question answering would require combining my model with additional components such as a semantic parser and a coreference resolver. robust parsers exist which are compatible with my model (for example: @xcite , but this would be a non-trivial extension, particularly for incorporating robust coreference resolution, which would ideally be done hand-inhand with semantic analysis. incorporating finegrained semantics into such tasks is an exciting research direction, but beyond the scope of the current paper. when reporting results, significance tests follow @xcite . ## conclusion i have presented the pixie autoencoder, a novel encoder architecture and training algorithm for functional distributional semantics, improving on previous results in this framework. for gs2011, the pixie autoencoder achieves state-of-the-art results. for relpron, it learns information not captured by a vector space model. for both datasets, it outperforms bert, despite being a shallower model with fewer parameters, trained on less data. this points to the usefulness of building semantic structure into the model. it is also easy to apply to these datasets (with no need to tune query strings), as it has a clear logical interpretation.