Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context (2022)

Abstract: By saying Maria is tall, a human speaker typically implies that Maria is evaluatively tall from the speaker’s perspective. However, by using a different construction Maria is taller than Sophie, we cannot infer from Maria and Sophie’s relative heights that Maria is evaluatively tall because it is possible for Maria to be taller than Sophie in a context in which they both count as short. Can pre-trained language models (LMs) “understand” evaulativity (EVAL) inference? To what extent can they discern the EVAL salience of different constructions in a conversation? Will it help LMs’ implicitness performance if we give LMs a persona such as chill, social, and pragmatically skilled? Our study provides an approach to probing LMs’ interpretation of EVAL inference by incorporating insights from experimental pragmatics and sociolinguistics. We find that with the appropriate prompt, LMs can succeed in some pragmatic level language understanding tasks. Our study suggests that socio-pragmatics methodology can shed light on the challenging questions in NLP.

## introduction this paper concerns pre-trained language models' (lms) interpretation of context-specific implicit elements on the pragmatic level of language understanding. probing lms' competence in implicitness is challenging due to the lack of surface representation. in this paper, we attempt to tease apart exactly what lms "know" about pragmatics through a case study of gradable adjectives such as tall. we draw insights from experimental pragmatics and sociolinguistics, and implement them in probing two types of transformer lms: the traditional auto-regressive gpt-3 @xcite and the encoder-decoder model macaw @xcite . our findings show that the extent to which lms are sensitive to implicitness depends on adjective properties (class, polarity, construction), prompt setting (the speaker is prede-fined as chill or nerdy), and transformers' architecture (decoder-transformer such as gpt-3, encoderdecoder transformer like macaw). by uttering a positive construction (henceforth pos) alex is tall, conversational participants simultaneously extract two kinds of meaning: its descriptive literal meaning about the state of the world -alex's height is above a particular threshold @xcite @xcite ; its socio-indexical meaning which implicitly reveals about the speakers themselves -alex is tall from the speaker's perspective, namely the speaker implies that alex is evaluatively tall @xcite . by contrast, when uttering an equative construction (henceforth eq) like alex is as tall as arthur, or a comparative construction (henceforth comp) such as alex is taller than arthur, there is no such salient evaluative reading because it's likely that alex is as tall as or taller than arthur in a context where (the speaker thinks) they are both short. a construction is evaluative if and only if it contextually entails its pos counterpart @xcite . this is called the bierwisch test: by using alex is tall (pos), the speaker implies that alex is evaluatively tall; while by using alex is as tall as arthur (eq) or alex is taller than arthur (comp), the speaker is not implying that alex is evaluatively tall -hence the linguistic generalization: using pos gives rise to evaluativity (henceforth eval) implicatures, whereas using eq or comp does not. we make all code and test data available for additional testing 1 eval is a central member of the class of contextsensitive phenomena. it arises as a pragmatic inference -a conversational implicature @xcite @xcite . our paper proposes lms examination schemes through a case study on eval implicature. our study is built up on @xcite , which adopts the bierwisch test to test for the the presence of eval inference in different gradable adjectives. their experimental pragmatics findings show a comprehensive picture about eval implicature (human judgment data, @xmath0: humans think that eval implicature is highly dependent on context which is shaped by the speaker's usage of adjective class (relative, e.g. heavy, absolute, e.g. full), adjective polarity (positive: tall, negative: its antonym short), and construction (pos, eq, comp). their experiment result (table2) showed that regarding adjective polarity, there is no clear difference in eval between positive and negative adjectives within either the relative or the absolute class. regarding construction, pos is clearly the most evaluative. regarding adjective class, the relative adjective class is less evaluative than the absolute in pos, but more evaluative than the absolute in eq, and exhibit the same eval as the absolute in comp. our lms investigation implemented @xcite 's dataset to examine the extent to which lms align with humans. throughout @xcite 's dataset, only one template prompt was used. thus, as a sanity check, we varied the prompt by adding two distinct personality illustration to the input. another motivation of taking prompt design to be an independent variable is that an utterance's socio-indexing meaning and its speaker's personality traits are intertwined: it reveals about the speakers' demographic background and ideological orientation @xcite @xcite . we designed the prompt text based on speaker persona: a social construct shown to be central to social meaning across various domains of language @xcite . we argue that the construct of persona is relevant to lms examination because: i) it's wellknown and readily available for perceiving social identity in human interaction; ii) it's been shown to shape human language processing at different levels @xcite @xcite ; iii) personae tend to be indexed by a variety of (non-)linguistic signs, including a mere textual description of the persona at stake (d'onofrio, 2018), making them easy to invoke in lms experiment set up. specifically, our incorporation of persona in the prompt design is inspired by @xcite . they find that to compute the standard of precision required to interpret numeral expressions, human comprehenders reason about the speaker's social identity, particularly about the persona they embody. an utterance produced by a nerdy speaker is associated with higher standard of precision, hence the tendency to interpreting the literal meaning but not necessarily the socioindexing pragmatic implicit meaning, compared to the same utterance in the same context uttered by a chill speaker. our experiments on lms take two opposite sets of characters: a persona interpreting utterance with its literal meaning (nerd), and a persona embodying laid-backness and pragmatic skillfulness (chill). we framed these two persona in the prompt text, and examined if this could help lms "understand" eval implicatures across various adjectives. we found that the answer depends on adjective properties and lms' types. a lot of attention has been paid to increase lms' general transparency @xcite , among which studies on lms' interpretation of implicitness mostly focus on scalar implicature or presupposition @xcite @xcite . to our knowledge, no studies in this line have been done on gradable adjectives' eval implicature, although eval and gradability are classic topics in context sensitivity. this is probably because these phenomena are cognitively too subtle to spot, hence hard to quantify under a lms framework. against this background, our goal is to examine the extent to which pre-trained lms can "understand" implicit eval implicatures. we hypothesized that if pre-trained lms are cognitively plausible, their performance should align with the human data in @xcite and @xcite , namely: i) there should be no eval difference regarding adjective polarity, ii) lms should predict pos constructions to be the most evaluative, iii) whether lms consider the relative adjectives to be more or less evaluative than the absolute adjectives depends on construction type, iv) lms should (at least) show a trend that the chill-persona prompt helps lms' understanding of implicatures, relative to the nerdy-persona prompt. ## experiments we designed our tests in the form of completion tasks, so as to test the pre-trained lms in their most natural setting, without interference from finetuning. we presented all the tasks in a conversa-tion format involving agent(s), meaning lms are expected to interpret the utterance with some conversational level of language understanding. we focus on two distinct types of transformers (table1): macaw @xcite , which is more recent (built on top of @xcite ), and gpt-3 @xcite . we used the 32 gradable antonym pairs (16 relative adjectives and 16 absolute adjectives) in @xcite , because it's already quantitatively justified by human judgments. each antonym pair was syntactically framed in 3 distinct types of constructions: pos, eq and comp. this gave us 32 (adjectives) x 3 (constructions) = 96 strings of sequence. model n params n layers macaw-large (c.f. t5) 770m 24 gpt-3/instructgpt 175b 96 lms completed the prompt with a fixed max-length of sequence (@xmath1. we preset the penalty and the presence coefficients as 0.6, which were reasonable values if the aim is to just reduce repetitive samples @xcite . all the stimuli had the same format, the only strings that changed were the detective's quoted report (underlined), which was replaced by different adjectival constructions. in terms of prompt template, there were 3 variations: in addition to the detective report, we adopted the nerd versus chill persona idea proposed and quantitatively justified by @xcite . their human data showed that arthur, who is overwhelmingly seen as embodying social qualities indicative of nerd, is consistently associated with a geeky stereotype and tend to be insensitive to pragmatic cues, whereas alex is ascribed attributes such as chill and a sociable personality, and he is pragmatically savvy. lms were prompted with (1) nerd persona: arthur is clever, smart, quiet, awkward, nerdy, shy and geeky. what does he mean by saying "maria is tall"? (2) chill persona: alex is chill, laid-back, relaxed, easy, cool, friendly, and outgoing. what doe he imply by saying "maria is tall"?. all the adjectives used in the two persona prompts are from @xcite 's collection of human responses to nerdy/chill stereotypes. all the stimuli had the same format, the only strings that changed were the speaker's (alex or arthur) quoted statement (underlined), which was replaced with various target adjectival constructions. the prompt examples are given in measurement inspired by the bierwisch test @xcite , we hypothesized that a construction is evaluative if and only if it contextually entails its pos counterpart. we therefore used gpt-3 similarity embedding model text-similaritybabbage-001 to embed document as a single vector @xcite . we deployed the model to both lms' responses and the target utterance (the testing adjectival construction's pos counterpart). we then calculated the cosine distance of the two vectors. the similarity score is calculated only between lms' own response and the target utterance (see adopting @xcite 's semantic similarity metrics, where larger amounts of concept overlap between two text segments is interpreted as more similar, we computed the cosine similarity as a proxy to the measurement of the concept overlap between lms' response and the target inference. we took that to be how much implicit meaning lms can pick up in the conversation. for example, in the detective setting with eq in the detective's quoted report "maria is as short as sophie", the target utterance is its pos counterpart maria is short. suppose lms "understand" the eval implicature, lms should draw a pos evaluative inference from eq. this is reflected in the similarity: lms' response is predicted to be similar to the target utterance if lms makes the appropriate pragmatic inference. ## results and discussion with respect to polarity (fig. 1 ), the results align with human data. there is no statistically significant difference in eval between positive and negative adjectives within either the relative adjective class or the absolute adjective class. regarding constructions (fig. 2 ), consistent with humans, the pos construction shows the highest similarity to the target inference: pos is the most evaluative across different lms and adjective types. regarding lms in fig. 2 , gpt-3 is more human-like than macaw regarding construction sensitivity. gpt-3's output shows that using pos implies eval, using eq is less likely to imply eval, and using comp is the least likely to imply eval. by contrast, macaw's output response to different constructions is not as stable: a lot of variance is found especially in macaw -nerdy interpreting eq and comp. relative to gpt-3, macaw is more sensitive to input instructions: with chill personality, macaw's "endorsement" of pos being evaluative gets significantly improved; whereas given nerdy personality (macaw -nerdy) or without any explicit identity, just interpreting detective's report (macaw -detective), macaw's sensitivity to constructions is not as salient. on the other hand, regardless of personality setup, gpt-3 showed similar patterns to different constructions. with respect to adjective class: for pos (fig. 3 left), except for macaw -detective which considers relative adjectives to be slightly more evaluative than the absolute, lms' output shows that the relative adjective class is less evaluative than the absolute adjective class. this effect is statistically significant for macaw -nerdy. surprisingly, for both eq (fig. 3 middle) and comp (fig. 3 right), lms still output representations suggesting that the relative adjectives are less evaluative than the absolute adjectives, especially for macaw in which statistical significance was found. an exception was found in gpt -detective in comp, which judges relative as more evaluative than the absolute. gpt-3 did not seem to outperform macaw, although t-test showed that gpt-3 did not significantly interpret absolute adjectives to be more evaluative than relative adjectives. in almost all of the cases, lms indiscriminately "understood" absolute adjectives to be more evaluative than relative adjectives. overall their interpretation of eval implicatures is not sensitive to construction. introducing socio-pragmatic frameworks in lms evaluation loop, we adopted theory-driven hypothesis and cognitively justified datasets to analyze lms' interpretation of eval implicature across adjective types. we found that lms align with human data in that both suggest that polarity does not influence eval, and both considered pos to be the most evaluative across all adjective types, but deviant from linguistic theory and human cognition, most lms' output suggests that the relative adjectives are less evaluative than the absolute across constructions. the persona setting helped some lms "understand" implicitness. we provide an attempt to tackle challenging nlp questions using validated socio-pragmatic paradigms. ## limitation and future studies in this paper, we investigated the extent to which pre-trained transformer lms (gpt-3 and macaw) capture human inferences regarding the evaluativity of different adjectival constructions (pos, eq, and comp). we acknowledge that there are limitations, which we hope to address in future studies. it might not be fine-grained enough to capture the extent to which lms draw an evaluative inference using the cosine similarity measurement as a proxy. specifically, our methodology design cannot account for the differences of the similarity scores between the target utterance (a) maria is tall and (b) lm's response maria is taller than average, and those between the target utterance (a) and an irrelevant distractor (b′) such as the detective is reporting on the height of maria, given the prompt the detective reported to the police chief: "maria is tall." what can the chief conclude from this?. (b′) is entailed by the prompt, although it's not similar to (a) in a vector space. (b) is contextually entailed by the prompt but it's close to (a) in a vector space. this may distract lms away from the target inference. for future study, we consider using a natural language inference (nli) model to more directly test the contextual entailment relationship between lms' response and the target inference. typically, lms are probed by looking at free form continuation or at the probability assigned to different output continuations under the lms. in this paper, we probed lms using a questionanswering format and measured lms' performance with similarity scores between lms' answer and a target inference. but, even with the non-standard methodology, we still found some evidence that lms do capture human inferences. our study shows that it's worthwhile to adopt existing sets of human behavioral data in evaluating lms' weakness and strength. manipulating the social persona of the "speaker" in a lms setting is novel yet at preliminary stage. for future work, we hope to expand our current dataset and better our motivation of the sociolinguistic persona design. arthur is clever, smart, quiet, awkward, nerdy, shy and geeky. what does he mean by saying "maria is tall"? @xcite maria is taller than arthur. maria is tall because she is taller than most people. alex is chill, laid-back, relaxed, easy, cool, friendly, and outgoing. what does he imply by saying "maria is tall"? @xcite maria is tall implies that maria is taller than most people. maria is taller than alex.