Functional Distributional Semantics at Scale (2023)

Abstract: Functional Distributional Semantics is a linguistically motivated framework for modelling lexical and sentence-level semantics with truth-conditional functions using distributional information. Previous implementations of the framework focus on subject–verb–object (SVO) triples only, which largely limits the contextual information available for training and thus the capability of the learnt model. In this paper, we discuss the challenges of extending the previous architectures to training on arbitrary sentences. We address the challenges by proposing a more expressive lexical model that works over a continuous semantic space. This improves the flexibility and computational efficiency of the model, as well as its compatibility with present-day machine-learning frameworks. Our proposal allows the model to be applied to a wider range of semantic tasks, and improved performances are demonstrated from experimental results.

## introduction functional distributional semantics (fds; emerson and @xcite aims to capture the truth-conditional aspects of words through learning from distributional information of a corpus. whilst truth-conditional semantics deals with predications over discrete entities, fds aims to generalize about predications over a space of entity representations with probabilistic semantics. contrasted with most distributional methods which map words to vectors, fds can model various aspects of meaning in a linguistically rigorous manner. for example, vagueness is represented by the probabilistic nature of predications, and hypernymy, defined formally as the subsumption of the extensions between two word senses, can be represented by the subsumption of regions of space @xcite . going beyond simple vector spaces, some models of distributional semantics represent words as tensors for composition (e.g., @xcite , as static distributions for uncertainty and entailment (e.g., @xcite , as posterior distributions for contextspecific meaning (e.g., @xcite , and as regions for set-theoretic properties (e.g., @xcite . among them, only a regionbased approach favours logical interpretations (for a discussion, see @xcite . in order to be computationally tractable, most models of distributional semantics are trained based on instances defined by context windows (e.g., @xcite or incomplete linguistic structures such as immediate dependencies (e.g., @xcite . all previous instances of fds (further discussed in §2) are only trained on svo triples. consequently, these models underutilize much contextual information. we hope to extend fds learning to arbitrary sentences, but not larger linguistic units (e.g., paragraphs), for handling them requires non-trivial extensions such as robust coreference resolution, which is beyond the scope of this work. to this end, we propose to adopt a continuous semantic space and a more expressive lexical model in place of the previous world model on a discrete space. our new formulation provides a computationally efficient and linguistically principled solution to applying fds to arbitrary sentences. furthermore, this also situates the framework closer to modern machine learning models which are mostly built upon continuous latent spaces, thus favouring comparisons among and integration with them. for example, @xcite integrated a pre-trained computer vision model with a continuous space to fds, applying it to annotated images. joint learning of the visually-grounded and corpus-based models was however left as future work due to the incompatibility of latent spaces. in this paper, we first give an introduction to fds in §2, explaining why it is difficult for previous implementations to scale up. then, we present in detail the proposed formulation and how to train the model in §3- §4. finally, we demonstrate how our model can be applied to a number of semantic evaluation data sets and present the results in §5. ## functional distributional semantics the core idea of functional distributional semantics is that a sentence refers to a set of entities, and a word is a predicate that is true or false of entities. compared to other approaches to distributional semantics, it aligns more with model-theoretic semantics, which approaches meaning in the same way in terms of a model structure. however, fixing a specific set of entities would make it impossible to generalize to new situations. in order for the model to be learnable, predicates do not directly take entities as input, but rather entity representations, referred to as pixies for brevity. a predicate is represented as a function from pixies to probabilities of truth. this allows the model to account for vagueness. fds does not submit to a fixed interpretation of pixies nor process of obtaining them. rather, pixies are introduced to merely convey information of latent entities. in the work of @xcite , pixies are dimensionality-reduced vectors obtained from a pretrained network. in this work, they are learnt to best represent entities according to our particular formulation by probabilistic graphical models, which are introduced below and in detail in §4. ## enriching the lexical model in this section, we describe our enriched lexical model and explain how it provides a solution to the linguistic and computational challenges mentioned. ## variational autoencoder as mentioned in §2.2, each training instance is a dmrs graph, which can be characterised in terms of n predicates @xmath0, . . . , r n } , and the argument structure @xmath1, j, a) : r i arga ---→ r j . to optimize the parameters θ of the generative model, we use a variational autoencoder (vae) @xcite . the intractable true posterior distributions p θ (z | r, a) over the pixies @xmath2, . . . , z n } are first approximated by tractable distributions chosen a priori (discussed in §4.1). instead of directly performing maximum likelihood estimation on the observed dmrs graphs, the lower bound in (5) is maximized following the β-vae @xcite , using a probabilistic encoder q ϕ (discussed in §4.2) and decoder p θ (discussed in §4.3). §4.4 and §4.5 reformulate the two terms in (5) respectively based on empirical insights for training stability. parameters of the encoder and decoder are thus jointly learnt via gradient descent. ## conclusion we analyzed the linguistic and computational challenges of functional distributional semantics and presented a new formulation where we have improved: applicability to diverse natural language structures, computational efficiency, compatibility with contemporary models, and performances on a range of semantic tasks. we believe this work bridges truth-conditional semantics to practical distributional semantics at scale.