Harnessing the linguistic signal to predict scalar inferences (2020)

Abstract: Pragmatic inferences often subtly depend on the presence or absence of linguistic features. For example, the presence of a partitive construction (of the) increases the strength of a so-called scalar inference: listeners perceive the inference that Chris did not eat all of the cookies to be stronger after hearing “Chris ate some of the cookies” than after hearing the same utterance without a partitive, “Chris ate some cookies”. In this work, we explore to what extent neural network sentence encoders can learn to predict the strength of scalar inferences. We first show that an LSTM-based sentence encoder trained on an English dataset of human inference strength ratings is able to predict ratings with high accuracy (r = 0.78). We then probe the model’s behavior using manually constructed minimal sentence pairs and corpus data. We first that the model inferred previously established associations between linguistic features and inference strength, suggesting that the model learns to use linguistic features to predict pragmatic inferences.

## introduction an important property of human communication is that listeners can infer information beyond the literal meaning of an utterance. one well-studied type of inference is scalar inference @xcite , whereby a listener who hears an utterance with a scalar item like some infers the negation of a stronger alternative with all: early accounts of scalar inferences (e.g., @xcite @xcite considered them to arise by default unless explicitly contradicted in context. however, in a recent corpus study, @xcite showed that there is much more variability in scalar inferences from some to not all than previously assumed. @xcite further showed that this variability is not random and that several lexical, syntactic, and semantic/pragmatic features of context explain much of the variance in inference strength. 1 recent bayesian game-theoretic models of pragmatic reasoning @xcite are able to integrate speaker expectations with world knowledge to predict listeners' pragmatic inferences in many cases (e.g., @xcite . however, to compute speaker expectations, these models require manual specification of features as well as specification of a finite set of possible utterances. further, inference becomes intractable when scaling up beyond toy domains to make predictions for arbitrary utterances. 2 neural network (nn) models, on the other hand, do not suffer from these limitations: they are capable of making predictions for arbitrary utterances and do not require manual specification of features. unlike bayesian game-theoretic models, however, nn models have no explicit pragmatic reasoning mechanisms. in this work, we investigate to what extent nn models can learn to predict subtle differences in scalar inferences and to what extent these models infer associations between linguistic features and 1 see section 2 for the operationalization of inference strength that we use throughout this paper and for a description of these features. 2 recent models of generating pragmatic image descriptions @xcite and color descriptions @xcite have overcome this issue by approximating the distributions of utterances given a set of potential referents. however, these models require a finite set of world states (e.g., several referents to choose from) and a corresponding generative model of utterances (e.g., an image captioning model) and are therefore also limited to scenarios with pre-specified world states and a corresponding generative model. ## the dataset we use the annotated dataset collected by @xcite , a dataset of the utterances from the switchboard corpus of english telephone dialogues @xcite with a noun phrase (np) with some. the dataset consists of 1,362 unique utterances. for each example with a some-np, @xcite collected inference strength ratings from at least 10 participants recruited on amazon's mechanical turk. participants saw both the target utterance and ten utterances from the preceding discourse context. they then rated the similarity between the original utterance like (2a) and an utterance in which some was replaced with some, but not all like (2b), on a 7-point likert scale with endpoints labeled "very different meaning" (1) and "same meaning" (7). low similarity ratings thus indicate low inference strength, and high similarity ratings indicate high inference strength. using this corpus, @xcite found that several linguistic and contextual factors influenced inference strength ratings, including the partitive form of, subjecthood, the previous mention of the np referent, determiner strength, and modification of the head noun, which we describe in turn. partitive: (3a-b) are example utterances from the corpus with and without partitive some-nps, respectively. values in parentheses indicate the mean inference strength rating for that item. on average, utterances with partitives yielded stronger inference ratings than ones without. subjecthood: utterances in which the some-np appears in subject position, as in (4a), yielded stronger inference ratings than utterances in which the some-np appears in a different grammatical position, e.g., as a direct object as in ( 4b ). previous mention: discourse properties also have an effect on inference strength. a some-np with a previously mentioned embedded np referent yields stronger inferences than a some-np whose embedded np referent has not been previously mentioned. for example, (5a) contains a some-np in which them refers to previously mentioned mission impossible tape recordings, whereas problems in the some-np in (5b) has not been previously mentioned. determiner strength: finally, it has been argued that there are two types of some: a weak some and a strong some @xcite . this weak/strong distinction has been notoriously hard to pin down @xcite and degen (2015) used empirical strength norms elicited independently for each item. to this end, she exploited the fact that removing weak some from an utterance has little effect on its meaning whereas removing strong some changes the meaning. determiner strength ratings were thus elicited by asking participants to rate the similarity between the original utterance and an utterance without some (of) on a 7-point likert scale from 'different meaning' to 'same meaning'. items with stronger some -e.g., (6a), determiner strength 3.3 -yielded stronger inference ratings than items with weaker some -e.g., (6b), determiner strength 6.7. the quantitative findings from @xcite are summarized in figure 4 , which shows in blue the regression coefficients for all predictors she considered (see the original paper for more detailed descriptions). for our experiments, we randomly split the dataset into a 70% training and 30% test set, resulting in 954 training items and 408 test items. ## model the objective of the model is to predict mean inference strength rating i given an utterance (a sequence of words) @xmath0, w 2 , ..., w n }. we rescale the 1-to-7 likert scale ratings to the interval [0, 1]. figure 1 shows the overall model architecture. the model is a sentence classification model akin to the model proposed by @xcite . it first embeds the utterance tokens using pre-trained embedding models, and then forms a sentence representation by passing the embedded tokens through a 2-layer bidirectional lstm network (bilstm) @xcite with dropout @xcite followed by a self-attention mechanism that provides a weighted average of the hidden states of the topmost bilstm layer. this sentence representation is then passed through a transformation layer with a sigmoid activation function, which outputs the predicted score in the interval [0, 1]. ## context, revisited in the tuning experiments above, we found that including the preceding conversational context in the input to the model did not improve or lowered prediction accuracy. 9 at the same time, we found that the model is capable of making accurate predictions in most cases without taking the preceding context into account. taken together, these results suggest either that the conversational context is not necessary and one can draw inferences from the target utterance alone, or that the model does not make adequate use of the preceding context. @xcite did not systematically investigate whether the preceding conversational context was used by participants judging inference strength. to assess the extent to which the preceding context in this dataset affects inference strength, we re-ran her experiment, but without presenting participants with the preceding conversational context. we recruited 680 participants on mechanical turk who 8 the regression analysis suggests that the model learned to make use of the subjecthood feature and previous work on probing behavior of contextual word representations has found that such models are capable of predicting dependency labels, including subjects (e.g., @xcite . we therefore also hypothesize that the representations of tokens that are part of a subject some-np contain information about the subjecthood status. this in return could be an important feature for the output layer of the model and therefore be providing additional signal for the model to attend to these tokens. 9 as suggested by a reviewer, we conducted post-hoc experiments in which we limited the conversational context to the preceding 2 or 5 utterances, which presumably have a higher signal-to-noise ratio than a larger conversational context of 10 preceding utterances. in these experiments, we again found that including the conversational context did not improve model predictions. the new, no-context ratings correlated with the original ratings (@xmath1.68, see appendix d) but were overall more concentrated towards the center of the scale, suggesting that in many cases, participants who lacked information about the conversational context were unsure about the strength of the scalar inference. since the original dataset exhibited more of a bi-modal distribution with fewer ratings at the center of the scale, this suggests that the broader conversational context contains important cues to scalar inferences. for our model, these results suggest that the representation of the conversational context is inadequate, which highlights the need for more sophisticated representations of linguistic contexts beyond the target utterance. 10 we further find that the model trained on the original dataset is worse at predicting the no-context ratings (@xmath2.66) than the original ratings (@xmath3.78), which is not surprising considering the imperfect correlation between ratings across experiments, but also provides additional evidence that participants indeed behaved differently in the two experiments. ## conclusion and future work we showed that despite lacking specific pragmatic reasoning abilities, neural network-based sentence encoders are capable of harnessing the linguistic signal to learn to predict human inference strength ratings from some to not all with high accuracy. further, several model behavior analyses provided consistent evidence that the model learned associations between previously established linguistic features and the strength of scalar inferences. in an analysis of the contribution of the conversational context, we found that humans make use of the preceding context whereas the models we considered failed to do so adequately. considering the 10 the representation of larger linguistic context is also important for span-based question-answer (qa) systems (e.g., @xcite @xcite and adapting methods from qa to predicting scalar inferences would be a promising extension of the current model. we also only considered the supervised setting in which the model was trained to predict inference strength. it would be interesting to investigate how much supervision is necessary and, for example, to what extent a model trained to perform another task such as predicting natural language inferences is able to predict scalar inferences (see @xcite for such an evaluation of predicting speaker commitment, and @xcite for an evaluation of different nli models for predicting lexically triggered scalar inferences). one further interesting line of research would be to extend this work to other pragmatic inferences. recent experimental work has shown that inference strength is variable across scale and inference type @xcite . we treated some as a case study in this work, but none of our modeling decisions are specific to some. it would be straightforward to train similar models for other types of inferences. lastly, the fact that the attention weights provided insights into the model's decisions suggests possibilities for using neural network models for developing more precise theories of pragmatic language use. our goal here was to investigate whether neural networks can learn associations for already established linguistic features but it would be equally interesting to investigate whether such models could be used to discover new features, which could then be verified in experimental and corpus work, potentially providing a model-driven approach to experimental and formal pragmatics.