Predicting scalar diversity with context-driven uncertainty over alternatives (2022)

Abstract: Scalar implicature (SI) arises when a speaker uses an expression (e.g., “some”) that is semantically compatible with a logically stronger alternative on the same scale (e.g., “all”), leading the listener to infer that they did not intend to convey the stronger meaning. Prior work has demonstrated that SI rates are highly variable across scales, raising the question of what factors determine the SI strength for a particular scale. Here, we test the hypothesis that SI rates depend on the listener’s confidence in the underlying scale, which we operationalize as uncertainty over the distribution of possible alternatives conditioned on the context. We use a T5 model fine-tuned on a text infilling task to estimate this distribution. We find that scale uncertainty predicts human SI rates, measured as entropy over the sampled alternatives and over latent classes among alternatives in sentence embedding space. Furthermore, we do not find a significant effect of the surprisal of the strong scalemate. Our results suggest that pragmatic inferences depend on listeners’ context-driven uncertainty over alternatives.

## introduction human communication involves not only the transmission of linguistic signals, but also contextguided inference over the beliefs and goals of other conversational agents (e.g., @xcite . one signature pattern of this pragmatic reasoning is scalar implicature (si). the standard view is that sis arise as a result of ordered relationships between linguistic items -when a weaker (less informative) item of a scale is uttered, then a listener can infer that the speaker did not have grounds to utter the stronger (more informative) item on that scale. for example, if alice says "some of the students passed the exam", bob can draw the scalar inference that not all students passed the exam, even though alice's utterance would still be semantically true in that scenario. while this view predicts that sis are contextindependent and generally strong -known as the homogeneity assumption @xcite -empirical studies have demonstrated a remarkable amount of variance in si rates both within @xcite and across lexical scales @xcite @xcite . this raises the question of what factors determine the si strength for a particular scale. in a landmark study, @xcite test two classes of potential predictors of si strength: the availability of the strong scalemate given the weak scalemate, and the degree to which scalemates can be distinguished from each other. they demonstrate that availability is not a reliable predictor of si strengths (but see @xcite , while measures of scalemate distinctness, such as the boundedness of the scale, do robustly predict si. more recent studies (e.g., @xcite @xcite have proposed a variety of other factors such as negative strengthening, polarity, and extremeness. here, we revisit the hypothesis that si rates depend on the availability of the strong scalemate. while prior work has operationalized availability with measures of the strong scalemate such as word frequency or similarity/association with the weak scalemate @xcite @xcite , we re-frame availability as the listener's confidence in the underlying scale. upon hearing a scalar expression, listeners must determine the items on the scale as well as the ordering metric before inference proceeds @xcite . if the listener is less certain about the scale, then they will be less likely to exclude the meaning of a particular strong scalemate. we operationalize scale uncertainty as uncertainty over the alternatives that could serve as a strong scalemate to the observed scalar expression. to estimate the alternatives predicted by humans, we use a t5 model @xcite fine-tuned on a text infilling task. while prior studies have treated alternatives as linguistic forms, we also consider the idea that listeners reason about alternatives at a conceptual level @xcite by treating alternatives as latent classes in a conceptual space. our results support the role of scale uncertainty in determining si rates, and suggest a new way of testing conceptual theories of alternatives for scalar inference. ## human data to obtain human si strengths, we use the data from experiment 2 by @xcite . in our analyses, we only consider the adjectival scales from van tiel et al.'s original materials, resulting in 32 scales. each scale is a pair of adjectives ⟨[weak], [strong]⟩, where the meaning of [strong] entails the meaning of [weak] (e.g., ⟨intelligent, brilliant⟩). the experiment measures whether humans exclude the meaning of @xcite upon observing a speaker use @xcite . on each trial of the experiment, participants read a prompt of the form "john says: [np] is [weak]", where [weak] is an adjective scalar item that may trigger a scalar inference, and [np] is a noun phrase that sets the context for the scalar item. there were 3 such sentences per scale, which differ from each other only in the np. for example, the weak scalar item intelligent is associated with the sentences "this student/that professor/the assistant is intelligent". participants were then asked: "would you conclude from this that, according to john, [np] p is not @xcite ?", where @xcite is the strong scalemate to @xcite , and [np] p is a pronominalized version of the [np] in the speaker's original utterance (e.g., "she is not brilliant"). participants marked their response as yes or no. the si rate for a scale is computed as the proportion of yes responses averaged over participants and sentences. ## predictors we use t5 @xcite to estimate all probabilities in our analyses. t5 is a sequenceto-sequence transformer model @xcite trained to represent language processing tasks as text-to-text problems. our model is based on the pre-trained t5-base model from huggingface transformers @xcite . since the off-the-shelf t5 model is not optimized for text generation, we use a t5 model that has been fine-tuned on a text infilling task @xcite . the model is fine-tuned on a 10-million-token subset of the 2007 portion of the new york times corpus @xcite . the supervision signal is generated by randomly masking some spans of words in a sentence to get the fragmentary context and a plausible completion. at inference time, the model decodes autoregressively via greedy sampling. ## results we computed the four metrics described in section 3 on the data from experiment 2 of @xcite , and evaluated the causal roles of each metric in predicting scalar inference rates across scales. for each of the four metrics, we fit a linear 5 for speed of convergence, we assumed diagonal covariance matrices for each estimated class distribution. regression model to predict mean si rates for each scale (averaged across trials). in all models, we included scale boundedness as an additional predictor, as it is the factor explaining the most variance in @xcite study. our first model tested string-based surprisal as a predictor of si rates. in line with van tiel et al.'s results, boundedness is a highly significant predictor (@xmath0. furthermore, surprisal of the strong scalemate is not a significant predictor (@xmath1.09, @xmath2.928). figure 1a shows the lack of relationship between in-context surprisal of the strong scalemate and si rate. each point represents a scale, with values averaged over the trials and sentence templates (three per scale) presented in van tiel et al.'s experiment 2. this lack of relationship concords with van tiel et al.'s original finding that availability is not predictive of si rate. our second model tested the predictive power of string-based scale uncertainty (i.e., the entropy over completions sampled from t5 in a scalar construction). we found string-based entropy to be a significant predictor of si rate (@xmath3.28, @xmath4.001), suggesting that uncertainty over alternatives (as string forms) may play a role in scalar inference. figure 1b shows the negative relationship between si rates and string-based entropy. next, we turn to the hierarchical metrics, which treat alternatives as latent classes in sentence em- finally, we performed a qualitative evaluation of the classes identified by the gaussian mixture models (gmms). figure 2 shows the alternatives generated by t5 for the template "the salary is adequate, but not .", with each point obtained by projecting the sentence-t5 embedding into 2dimensional space via pca. the bic-minimizing gmm identifies two latent classes, distinguished by color and marker, among the alternatives generated by t5 for the weak scalar item adequate. first, we examine the cluster containing good, the strong scalemate tested in van tiel et al.'s experiments (marked with boldface and outline). this cluster (indicated by blue triangles) contains good as well as semantically similar alternatives such as "great", "sufficient", and "enough". in general, the alternatives in this cluster appear to suggest a scale where high salaries are positive (e.g., from an employee's perspective), with strong scalar items like "generous", "ideal", and "competitive". in contrast, the second cluster (indicated by red circles) contains alternatives such as "extravagant" and "overcharged", capturing the potential of adequate to be on a scale where higher salaries are not always desirable (e.g., from an employer's perspective). while the modelgenerated alternatives and clusters are noisy, we take this to illustrate that a single weak scalar item (like adequate) can plausibly be interpreted as belonging to multiple scales. ## discussion we tested the hypothesis that si rates depend on the listener's confidence in the underlying scale, using two operationalizations of alternatives (surfacelevel string forms and latent classes in a sentence embedding space). using data from a previously conducted experiment @xcite , we found that scale uncertainty was a significant predictor of si rates: on average, when uncertainty over alternatives (i.e., entropy over sampled alternatives, or over classes of alternatives in sentence embedding space) is lower, humans are more likely to draw a scalar inference. on the other hand, the predictability of the strong scalemate (as measured by surprisal of the string form, or of its underlying cluster) was not a significant predictor of si rates. an open question is why scale uncertainty predicts si rates, while strong scalemate surprisal and the availability measures from @xcite are poor predictors. we conjecture that the predictability of the strong scalemate may be shrouded by the paradigm used in experimental investigations of scalar diversity. in these experiments, the participant is explicitly asked to reason about the strong scalemate in the prompt (e.g., "john says: this student is intelligent. would you conclude from this that, according to john, she is not brilliant?"). thus, the effort required to retrieve the strong scalemate (e.g., "brilliant"), which may be captured by its in-context predictability, may no longer be relevant in this setting. we note, however, that our findings likely depend on the chosen clustering algorithm and conceptual representation of the alternatives. we intend to explore this space more broadly in future work. looking forward, our methods can be applied to scales that are ordered by ad-hoc relationships instead of entailment @xcite . beyond predicting scalar diversity, our approach suggests a way to derive quantitative behavioral predictions from non-linguistic alternatives @xcite , and supports the idea that context-driven expectations may give rise to pragmatic behaviors.