Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models (2023)

Abstract: Generalized quantifiers (e.g., \textit{few} , \textit{most} ) are used to indicate the proportions predicates satisfy (for example, \textit{some} apples are red). One way to interpret quantifier semantics is to explicitly bind these satisfactions with percentage scopes (e.g., 30%-40% of apples are red). This approach can be helpful for tasks like logic formalization and surface-form quantitative reasoning (Gordon and Schubert, 2010; Roy et al., 2015). However, it remains unclear if recent foundation models (Bommasani et al., 2021) possess this ability due to the absence of direct training signals. To explore this, we introduce QuRe, a crowd-sourced dataset of human-annotated generalized quantifiers in Wikipedia sentences featuring percentage-equipped predicates. We explore quantifier comprehension using PRESQUE, a framework that combines natural language inference and the Rational Speech Acts framework. Experimental results on the HVD dataset (Herbelot and Vecchi, 2015) and QuRe demonstrate PRESQUE’s superiority over a literal listener baseline, showing a 20% relative improvement in F1 in predicting percentage scopes for quantifiers, even with no additional training.

## introduction generalized quantifiers @xcite are used to express relations between subsets of concepts or entities. for instance, the quantifier 'some' in the statement 'some apples are red' indicates that at least one apple is red. quantifiers, being inherently fuzzy, are prevalent in both real-world communication and natural language processing (nlp) benchmarks @xcite . consequently, developing a formal framework for understanding quantifier semantics is essential to enhance the language understanding capabilities of nlp systems, particularly in facilitating natural human-ai language-based interactions, such as in human-robot collaborative tasks @xcite . in this work we present presque (pragmatic reasoning for semantics of quantifiers), a framework to model the semantics of quantifiers for textbased foundation models, such as bert @xcite and roberta @xcite , , through the lens of pragmatic reasoning. while foundation models have shown impressive performance on various text-based tasks @xcite , their ability to infer the semantic meanings of generalized quantifiers remains relatively unexplored. in presque, we represent quantifier semantics in terms of percentage scopes, which indicate the proportion of cases where the associated predicate holds true. for example, in the sentence 'some apples are red', the quantifier 'some' could be associated with a percentage scope of 30-40%, indicating that 30-40% of all apples are red. our framework consists of two components: (1) a natural language inference (nli) component @xcite that models sentence-level semantics between a sentence containing a quantifier word and another sentence containing a percentage value, and (2) a rational speech act (rsa) component (frank and @xcite for pragmatic reasoning. using these components, presque takes a sentence with a quantifier as input and outputs the corresponding percentage scope (further discussed in section 2). ambiguity, as highlighted by @xcite , is beneficial for efficient communication via language. since the percentage values of quantifiers are not universally defined, humans often need to infer the exact percentage value, which is not explicitly conveyed in the utterance @xcite . furthermore, the interpretation of quantifier semantics can be influenced by linguistic and social cues @xcite . the pragmatic theory proposed by @xcite emphasizes the role of communicative goals in interpreting the semantic meaning of natural language expressions, simplifying the required semantic theories @xcite . lastly, nli models are also shown to struggle with ambiguous premises @xcite , quantifier inference @xcite , and quantitative reasoning @xcite , making a direct literal interpretation of generalized quantifiers less reliable. to address these challenges, presque employs rsa, a bayesian framework that follows a gricean approach for modeling communication by reasoning about agent states @xcite . presque incorporates a literal speaker role, based on a foundation model fine-tuned on nli datasets, and a pragmatic listener role, computed using bayesian rules, to reason between the quantifier space and the space of percentage values. existing datasets like hvd @xcite and gqnli @xcite that investigate quantifier semantics either lack gold annotations of the percentage scopes for interpreting quantifier semantics, or are based on artificial setups using a small number of countable objects @xcite . such fictional settings are not generalizable to broader and more complex real-world settings (e.g. describing concepts about a population using quantifiers). for a fair evaluation of the quantifier understanding capabilities acquired by foundation models through their pre-training, we should evaluate these models using text of similar style and content as the pre-training corpora. to address the aforementioned issues with current evaluation corpora for quantifier understanding, we crowd-source a dataset, qure (quantifier reasoning), which contains sentences containing quantifiers paired with annotations for quantifier semantics in terms of percentage scopes. additionally, we characterize the ease of making quantifier predictions for different sentences in qure. using presque to evaluate the quantifier reasoning ability of foundation models on qure, we observe a 20% span-based f1 boost over the literal listener baseline at all specificity levels (section 5.2). our experiments highlight the improved quantifier understanding of foundation models when approached from a pragmatic perspective rather than relying on direct interpretation using textual understanding frameworks like nli. although our framework does not explicitly model mathematical concepts, it is noteworthy that the mean strengths of quantifiers in foundation models, as revealed by presque, echo observations of quantifier hierar-chies from previous works @xcite that involve strong human priors, and findings from @xcite , who associates quantifier usage with the counting behavior of human beings. in summary, our contributions are two-fold: we develop presque based on pragmatic reasoning and nli, and we crowd-source a dataset qure to support quantifier understanding investigation of foundation models. our results on hvd and qure demonstrate that foundation models equipped with pragmatic reasoning (presque) can perform quantifier reasoning similar to humans. ## quantifier semantics understanding through rsa we frame the task of quantifier understanding as the prediction of the percentage scope (e.g., 30%-50%) given a quantified sentence sq (e.g., some apples are red.). specifically, given an interval width β, we divide the percentage spectrum between 0 and 1 into evenly spaced intervals, denoted as w β = {p i } (e.g., w β=0.05 = {0, 5%, 10%, ..., 95%, 100%}). the goal of a quantifier understanding model is to determine the percentage range in w β where the associated predicate holds true (e.g., the proportion of red apples among all apples, 30%-50%). to interpret quantifiers as percentage scopes, we develop presque, a framework that adopts the rational speech act (rsa) framework, with natural language inference (nli) as the backbone for text understanding. the rsa framework consists of a speaker and a listener, where the listener infers the world state from the speaker's utterance by modeling the speaker's state of mind @xcite . in presque, the world state corresponds to percentage values of predicates, while utterances correspond to quantifiers used with those predicates. given a premise p (e.g., some apples are red.) with quantifier q (some) and a hypothesis h (e.g., 30% apples are red.) with a percentage value p (30%), we use the entailment score between the premise and hypothesis, obtained from an nli model, to define the literal listener l 0 : the pragmatic listener l 1 , in the presque framework, interprets the semantics of the quanti-fier word as: here, s 0 represents a literal speaker that maps the semantics of percentage values to quantifier words. practically, we model the speaker by swapping the premise and hypothesis in l 0 : the prior p (p) in eq. 2 can be expanded as: here, p(p|q) is computed similarly to l 0 , and p(q) represents the word frequency of q, obtained from the wordfreq dataset @xcite . ## experimental setup for the experiment in hvd, we compute l(p|q) for presque and l 0 among different foundation models and compare them with human interpretations. in qure, with the target percentage given in section 3, we generate the percentage scope that sq satisfies. all percentage choices are selected from w β , and experiments are run without training. percentage scope generation with specific granularity g and window size w for the operators, a percentage expression in section 3 is converted into a golden scope {p min , p max } ∈ w β (p @xmath0. for example, if β = 0.05, @xmath1.01 and @xmath2, the golden scope of ∼ 0.59 is [0.55, 0.65]. the full generation rules are in appendix e. evaluation metrics for hvd, given a listener l m based on an nli model m. l m (p|q) is computed by averaging the entailment scores over all sq s for all p values in w β and normalize them to be a distribution. we can then compute the cross entropy between the human interpretation of quantifiers p h from section 5.1 and l m (p|q) to measure the similarity of quantifier interpretation between humans and m. ## experiments and results we perform experiments to determine the percentage scope of quantifiers on two datasets: the hvd dataset, which includes predicates annotated with quantifiers but lacks percentage scope annotations, and the qure dataset, which provides annotations for both quantifiers and percentage scopes. as a baseline, we use the literal listener, l 0 . ## conclusion generalized quantifiers are widely used for addressing satisfaction in natural language. however, the quantifier understanding abilities of foundation models are not well-studied or well-supported by the existing benchmarks based on factual context. in this work, we study quantifier understanding by proposing the presque framework that formulates the problem in a pragmatics reasoning perspective and the format of nli. and we collect a dataset qure that includes human annotated quantifiers and percentage mentions for wikipedia sentences. from the experimental results on the hvd dataset and our collected qure dataset, the presque achieves better performance compared to the direct interpretation of quantifier semantics by a literal listener model.