What are the Goals of Distributional Semantics? (2020)

Abstract: Distributional semantic models have become a mainstay in NLP, providing useful features for downstream tasks. However, assessing long-term progress requires explicit long-term goals. In this paper, I take a broad linguistic perspective, looking at how well current models can deal with various semantic challenges. Given stark differences between models proposed in different subfields, a broad perspective is needed to see how we could integrate them. I conclude that, while linguistic insights can guide the design of model architectures, future progress will require balancing the often conflicting demands of linguistic expressiveness and computational tractability.

## introduction in order to assess progress in any field, the goals need to be clear. in assessing progress in semantics, @xcite contrasts "top-down" and "bottomup" approaches: a top-down approach begins with an overarching goal, and tries to build a model to reach it; a bottom-up approach begins with existing models, and tries to extend them towards new goals. 1 like much of nlp, distributional semantics is largely bottom-up: the goals are usually to improve performance on particular tasks, or particular datasets. aiming to improve nlp applications is of course a legitimate decision, but koller points out a problem if there is no top-down goal: "bottom-up theories are intrinsically unfalsifiable... we won't know where distributional semantics is going until it has a top-down element". this is contrasted against truth-conditional semantics, a traditional linguistic approach which is largely topdown: "truth-conditional semantics hasn't reached its goal, but at least we knew what the goal was". in this paper, i take a long-term linguistic perspective, where the top-down goal is to characterise the meanings of all utterances in a language. this is an ambitious goal, and a broad one. to make this goal more precise, in the following sections i will elaborate on several aspects of meaning which could be considered crucial. for each aspect, i identify a plausible goal, lay out out the space of possible models, place existing work in this space, and evaluate which approaches seem most promising. by making the goals explicit, we can assess whether we are heading in the right direction, and we can assess what still needs to be done. if a reader should disagree with my conclusions, they should start by looking at my goals. ## background: distributional semantics the aim of distributional semantics is to learn the meanings of linguistic expressions from a corpus of text. the core idea, known as the distributional hypothesis, is that the contexts in which an expression appears give us information about its meaning. 2 the idea has roots in american structuralism @xcite and british lexicology @xcite foot_2 , and with the advent of modern computing, it began to be used in practice. in a notable early work, @xcite represented word meanings as boolean vectors, based on a thesaurus. distributional semantics has become widespread in nlp, first with the rise of count vectors (for an overview, see @xcite , then of word embeddings @xcite , and most recently, of contextualised embeddings @xcite . 4 what all of these approaches share is that they learn representations in an unsupervised manner on a corpus. while much work takes a bottom-up approach, as koller observes, a notable exception is the typedriven tensorial framework of @xcite and @xcite , which has broad linguistic goals, and will be mentioned in several sections below. this framework represents the meanings of words as tensors, and constructs phrase meanings using tensor contraction based on predicateargument structure. for example, there is one vector space for nouns, and a second vector space for sentences, so intransitive verbs are matrices (mapping noun vectors to sentence vectors). ## meaning and the world language is always about something. in this section, i discuss challenges in connecting a semantic model to things in the world. ## lexical meaning in this section, i discuss challenges in representing the meanings of individual words. ## sentence meaning in the previous section, i discussed meaning at the level of words. i now turn to challenges in representing meaning at the level of sentences. ## conclusion a common thread among all of the above sections is that reaching our semantic goals requires structure beyond representing meaning as a point in space. in particular, it seems desirable to represent the meaning of a word as a region of space or as a classifier, and to work with probability logic. however, there is a trade-off between expressiveness and learnability: the more structure we add, the more difficult it can be to work with our representations. to this end, there are promising neural architectures for working with structured data, such dependency graphs (for example: marcheggiani and titov, 2017) or logical propositions (for example: @xcite . to mitigate computationally expensive calculations in probabilistic models, there are promising new techniques such as amortised variational inference, used in the variational @xcite @xcite . my own recent work in this direction has been to develop the pixie autoencoder @xcite , and i look forward to seeing alternative approaches from other authors, as the field of distributional semantics continues to grow. i hope that this survey paper will help other researchers to develop the field in a way that keeps long-term goals in mind.