Linguists Who Use Probabilistic Models Love Them: Quantification in Functional Distributional Semantics (2020)

Abstract: Functional Distributional Semantics provides a computationally tractable framework for learning truth-conditional semantics from a corpus. Previous work in this framework has provided a probabilistic version of first-order logic, recasting quantification as Bayesian inference. In this paper, I show how the previous formulation gives trivial truth values when a precise quantifier is used with vague predicates. I propose an improved account, avoiding this problem by treating a vague predicate as a distribution over precise predicates. I connect this account to recent work in the Rational Speech Acts framework on modelling generic quantification, and I extend this to modelling donkey sentences. Finally, I explain how the generic quantifier can be both pragmatically complex and yet computationally simpler than precise quantifiers.

## introduction model-theoretic semantics defines meaning in terms of truth, relative to model structures. in the simplest case, a model structure consists of a set of individuals (also called entities). the meaning of a content word is a predicate, formalised as a truth-conditional function which maps individuals to truth values (either truth or falsehood). because of this precisely defined notion of truth, model theory naturally supports logic, and has become a prominent approach to formal semantics. for detailed expositions, see @xcite ; @xcite ; @xcite . mainstream approaches to distributional semantics represent the meaning of a word as a vector (for example: turney and pantel, 2010; @xcite ; for an overview, see: emerson, 2020b). in contrast, functional distributional semantics represents the meaning of a word as a truth-conditional function @xcite . it is therefore a promising framework for automatically learning truth-conditional semantics from large datasets. in previous work @xcite copestake, 2017b, §3.5, henceforth e&c), i sketched how this approach can be extended with a probabilistic version of first-order logic, where quantifiers are interpreted in terms of conditional probabilities. i summarise this approach in §2 and §3. there are four main contributions of this paper. in §4.1, i first point out a problem with my previous approach. quantifiers like every and some are treated as precise, but predicates are vague. this leads to trivial truth values, with every trivially false, and some trivially true. secondly, i show in §4.2-4.4 how this problem can fixed by treating a vague predicate as a distribution over precise predicates. thirdly, in §5 i look at vague quantifiers and generic sentences, which present a challenge for classical (non-probabilistic) theories. i build on tessler and @xcite 's account of generics using rational speech acts, a bayesian approach to pragmatics @xcite . i show how generic quantification is computationally simpler than classical quantification, consistent with evidence that generics are a "default" mode of processing (for example @xcite . finally, i show in §6 how this probabilistic approach can provide an account of donkey sentences, another challenge for classical theories. in particular, i consider generic donkey sentences, which are doubly challenging, and which provide counter-examples to the claim that donkey pronouns are associated with universal quantifiers. taking the above together, in this paper i show how a probabilistic first-order logic can be associated with a neural network model for distributional semantics, in a way that sheds light on longstanding problems in formal semantics. ultimately, model theory requires quantifiers to give truth values to propositions. an example of a logical proposition is given in fig. 1 , with a quantifier for each logical variable. this also assumes a neo-davidsonian approach to event semantics @xcite . equivalently, we can represent a logical proposition as a scope tree, as in fig. 2 . the truth of the scope tree can be calculated by working bottomup through the tree. the leaves of the tree are logical expressions with free variables. they can be assigned truth values if each variable is fixed as an individual in the model structure. to assign a truth value to the whole proposition, we work up through the tree, quantifying the variables one at at time. once we reach the root, all variables have been quantified, and we are left with a truth value. each quantifier is a non-terminal node with two children -its restriction (on the left) and its body (on the right). it quantifies exactly one variable, called its bound variable. each node also has free variables. for each leaf, its free variables are exactly the variables appearing in the logical expression. for each quantifier, its free variables are the union of the free variables of its restriction and body, minus its own bound variable. for a wellformed scope tree, the root has no free variables. each node in the tree defines a truth value, given a fixed value for each free variable. the truth value for a quantifier node is defined based on its restriction and body. given values for the quantifier's free variables, the restriction and body only depend on the quantifier's bound variable. the restriction and body therefore each define a set of individuals in the model structurethe individuals for which the restriction is true, and the individuals for which the body is true. we can write these as r(v) and b(v), respectively, where v denotes the values of all free variables. generalised quantifier theory says that a quantifier's truth value only depends on two quantities: the cardinality of the restriction |r(v)|, and the cardinality of the intersection of the restriction and body |r(v) ∩ b(v)|. each non-terminal node is a quantifier, with its bound variable in brackets. its left child is its restriction, and its right child its body. ## generalised quantifiers in functional distributional semantics functional distributional semantics defines a probabilistic graphical model for distributional semantics. importantly (from the point of view of formal semantics), this graphical model incorporates a probabilistic version of model theory. this is illustrated in fig. 3 . the top row defines a distribution over situations, each situation being an event with two participants. 1 this generalises a model structure comprising a set of situations, as in classical situation semantics @xcite . each individual is represented by a pixie, a point in a high-dimensional space, which represents the features of the individual. two individuals could be represented by the same pixie, and the space of pixies can be seen as a conceptual space in the sense of @xcite . 1 for situations with different structures (multiple events or different numbers of participants), we can define a family of such graphical models. structuring the graphical model in terms of semantic roles makes the simplifying assumption that situation structure is isomorphic to a semantic dependency graph such as dmrs @xcite . in the general case, the assumption fails. for example, the arg3 of sell corresponds to the arg1 of buy. top row: pixie-valued random variables x, y , z together represent a situation composed of three individuals. they are jointly distributed according to the semantic roles arg1 and arg2. their joint distribution can be seen as a probabilistic model structure. bottom row: each predicate r in the vocabulary v has a probabilistic truth-conditional function, which can be applied to each individual. this gives a truth-valued random variable for each individual for each predicate. the bottom row of the graphical model defines a distribution over truth values, so that each predicate has some probability of being true of each individual. each predicate can therefore be seen as a probabilistic truth-conditional function. in this paper, i will not discuss learning such a model (for an up-to-date approach, see: emerson, 2020a). instead, the focus is on how we can manipulate a trained model, to move from single predicates to complex propositions. in previous work (e&c), i sketched an account of quantification. the idea is to follow generalised quantifier theory, but with a truth-valued random variable for each node in the scope tree. similarly to the classical case, the distributions for these nodes are defined bottom-up through the tree. in the classical theory, we only need to know the cardinalities |r(v)| and |r(v) ∩ b(v)|. in fact, all the conditions in more precisely, b and r are truth-valued random variables for the body and restriction, and v is a tuple-of-pixies-valued random variable, with ## quantification with vague predicates truth-conditional functions that give probabilities strictly between 0 and 1 are motivated for both practical and theoretical reasons. practically, such a function can be implemented as a feedforward neural network with a final sigmoid unit (as used by e&c), whose output is never exactly 0 or 1. theoretically, using intermediate probabilities of truth allows a natural account of vagueness @xcite @xcite . however, as we will see in the following subsection, intermediate probabilities pose a problem for e&c's account of quantification. ## vague quantifiers and generics while some, every, no, and most can be given precise truth conditions, other natural language quantifiers are vague. in particular, we can consider the terms few and many. 5 under a classical account (for example: @xcite , many means that r(v) ∩ b(v) is large compared to r(v), but how large is underspecified; similarly, few means this ratio is small. the underspecification of a proportion can naturally be represented as a distribution. so, we can define the meaning of a vague generalised quantifier to be a function from p (b | r, v) to a probability of truth, as illustrated in fig. 6 . a particularly challenging case of natural language quantification involves generic sentences, such as: dogs bark, ducks lay eggs, and mosquitoes carry malaria. generics are ubiquitous in natural language, but they are challenging for classical models, because the truth conditions seem to depend heavily on lexical semantics and on the context of use (for discussion, see @xcite @xcite . while it is tempting to treat generic quantification as underspecification of a precise quantifier (for example @xcite , this is at odds with evidence that generics are easier for children to acquire than precise quantifiers @xcite @xcite , and also easier for adults to process @xcite . in contrast, tessler and @xcite analyse generic sentences as being semantically simple, with the complexity coming down to pragmatic inference. they use rational speech acts (rsa), a bayesian approach to pragmatics @xcite . in this framework, literal truth is separated from pragmatic meaning. communication is viewed as a game where a listener has a prior belief about a situation, and a speaker wants to update the listener's belief. given a truth-conditional function, a literal listener updates their belief by conditioning on truth, ruling out situations for which the function returns false. a pragmatic speaker who observes a situation can choose an utterance which is informative for a literal listener -in particular, the utterance which maximises a literal listener's posterior probability for the observed situation. a pragmatic listener can update their belief by conditioning on a pragmatic speaker's utterance. tessler and goodman's insight is that this inference of pragmatic meanings can account for the behaviour of generic sentences. the literal meaning of a generic can be simple (it is more likely to be true as the proportion increases), but the pragmatic meaning can have a rich dependence on the world knowledge encoded in the prior over situations. for example, mosquitoes carry malaria does not mean that all mosquitoes do (in fact, many do not) but it can be informative for the listener: as most animals never carry malaria, even a small proportion is pragmatically relevant. building on this, we could model the generic quantifier by setting f q as the identity function (the same as many in fig. 6 ). from ( 8 ), the probability of truth is then as shown in (10). however, marginalising out π r and π b is computationally expensive, as it requires summing over all possible functions. we can approximate this by reversing the order of the expectations, and so marginalising out π r and π b before u , as shown in (11), where ψ r and ψ b are vague functions. evaluating a vague function is computationally simple. abbreviating this, similarly to ( 9 ), we can write: for precise quantifiers, using vague functions gives trivial truth values (discussed in §4.1), but for generics, (10) and (11) give similar probabilities of truth. to put it another way, a vague quantifier doesn't need precise functions. modelling generics with (10) was driven by the intuition that generics are vague but semantically simple. the alternative in (11) is even simpler, because we only need to calculate e u|v once in total, rather than once for each possible π r and π b . this would make generics computationally simpler than other quantifiers, consistent with the evidence that they are easier to acquire and to process. in fact, (11) takes us back to e&c's conditional probability, as shown in (13). @xcite 's logical inference, re-analysed as generic quantification. r is the restriction, the logical conjunction of t α,x , t β,y , and t γ,z , while t δ,x is the body. generic quantification gives p(q) = p (t δ,x | t α,x , t β,y , t γ,z ), marginalising out all three bound variables (x, y , and z). this means the logical inference proposed by @xcite can in fact be seen as generic quantification. this is illustrated in fig. 7 , which corresponds to a sentence like rooms that have stoves are kitchens, if α, β, γ, δ are set to room, have, stove, kitchen. 6 not only does this approach to quantification deal with both precise and vague quantifiers in a uniform way, it can also explain why generics are easier to process than precise quantifiers. ## donkey sentences an example of a donkey sentence is shown in (3). they are challenging for classical semantic theories, because naive composition, shown in (4), leaves a variable (y) outside the scope of its quantifier @xcite . the tempting solution in (5) requires a universal quantifier for an indefinite (a donkey), which would be non-compositional. 7 (3) every farmer who owns a donkey feeds it. as discussed in §5, generics are more basic than classical quantifiers, so i first consider generic donkey sentences, as illustrated in ( 6 )-( 8 ). an analysis of (3) is given in appendix a. example (8) shows it is inappropriate to use a universal quantifier: not all mosquitoes carry malaria, and not all bitten birds are infected (even if bitten by a malaria-carrying mosquito). however, this sentence still communicates that malaria is spread between birds by mosquitoes. this relies on pragmatic inference, from prior knowledge that most animals cannot spread malaria. despite the challenge for classical theories, generic donkey sentences can be straightforwardly handled by my proposed probabilistic approach. an example is shown in fig. 8 , which corresponds to (6), if α, β, γ, δ are set to farmer, own, donkey, feed. intuitively, the more likely it is that a farmer owning a donkey implies the farmer feeding the donkey, the more likely it is for the sentence to be true. given world knowledge and a discourse context, this can lead to a sharp threshold for being uttered, using rsa's pragmatic inference. ## related work functional distributional semantics is related to other probabilistic semantic approaches. goodman and lassiter (2015) and bernardy et al. (2018, 2019 ) represent meaning as a probabilistic program. this paper brings functional distributional semantics closer to their work, because a probabilistic scope tree can be seen as a probabilistic program. an important practical difference is that functional distributional semantics represents all predicates in the same way (as functions of pixies), allowing a model to be trained on corpus data. probabilistic ttr @xcite also represents meaning as a probabilistic truth-conditional function. however, in this paper i have provided an alternative compositional semantics, in order to deal with vague quantifiers and generics. in principle, my proposal could be incorporated into a probabilistic ttr appproach. furthermore, although @xcite discuss learning, they assume a richer input than available in distributional semantics. some hybrid distributional-logical systems exist (for example: @xcite @xcite , but these do not discuss challenging cases like generics and donkey sentences. explaining the multiple readings of donkey sentences using pragmatic inference has been proposed using non-probabilistic tools (for example @xcite . i have provided a concrete computational method to calculate such inferences, in the same way that tessler and @xcite have provided a concrete account of generics. ## conclusion in this paper, i have presented a compositional semantics for both precise and vague quantifiers, in the probabilistic framework of functional distributional semantics. i have re-interpreted previous work in this framework as performing generic quantification, building on the approach of tessler and @xcite . i have shown how generic quantification is computationally simpler than classical quantification, consistent with evidence that generics are a "default" mode of processing. finally, i have presented examples of generic donkey sentences, which are doubly challenging for classical theories, but straightforward under my proposed approach.