Not Wacky vs. Definitely Wacky: A Study of Scalar Adverbs in Pretrained Language Models (2023)

Abstract: Vector-space models of word meaning all assume that words occurring in similar contexts have similar meanings. Words that are similar in their topical associations but differ in their logical force tend to emerge as semantically close – creating well-known challenges for NLP applications that involve logical reasoning. Pretrained language models such as BERT, RoBERTa, GPT-2, and GPT-3 hold the promise of performing better on logical tasks than classic static word embeddings. However, reports are mixed about their success. Here, we advance this discussion through a systematic study of scalar adverbs, an under-explored class of words with strong logical force. Using three different tasks involving both naturalistic social media data and constructed examples, we investigate the extent to which BERT, RoBERTa, GPT-2 and GPT-3 exhibit knowledge of these common words. We ask: 1) Do the models distinguish amongst the three semantic categories of MODALITY, FREQUENCY and DEGREE? 2) Do they have implicit representations of full scales from maximally negative to maximally positive? 3) How do word frequency and contextual factors impact model performance? We find that despite capturing some aspects of logical meaning, the models still have obvious shortfalls.

## introduction large pretrained language models such as bert @xcite all rely on the assumption that the meanings of words are revealed by the company they keep @xcite ; masked language modelling (mlm) is a useful training objective because words in the context of the masked word provide cues to its identity. this assumption is highly appropriate for nouns and named entities. different topics of discussion involve different entities, and in discussing any given topic, the associated entities will be referred to repeatedly. however, the assumption holds less well for some other classes of words that can be used in discussing virtually any topic. these include quantifiers (e.g. few, many, all), words expressing negation (e.g. not, no, never) and the focus of the present paper, which is scalar adverbs, such as perhaps, certainly, never, often, very, completely. many scalar adverbs tend to occur in similar contexts but have distinct meanings. as @xcite put it, distributional models tend to be 'blind' to logical meanings because the latter are topic independent and thus their meanings tend to not be reflected in their distributional contexts. accurate processing of scalar adverbs is pertinent to a wide variety of nlp applications, including sentiment analysis @xcite , entailment inferences @xcite , detecting contradictions, and indirect question answering @xcite . bert family models succeed without finetuning on a remarkable variety of tasks, and unsupervised embeddings from bert base have been used fairly successfully to rank gradable adjectives according to their scalar position on half-scales from a neutral to an extreme value (e.g., @xmath0. this result suggests that the bert architecture might also be successful in exploiting diffuse or indirect cues for the selection of scalar adverbs. however, the same models have had little success in representing negation @xcite or antonymy @xcite . in fact, even large language models (llms) still struggle substantially with negation @xcite . soler and apidianaki (2020) do not evaluate adjectival antonyms, such as {cold, hot}. however, it is worth noting that scalar adverbs intrinsically include antonym pairs, (e.g.,{never, always}). furthermore, the word not is syntactically possible wherever a scalar adverb can occur, yet would express a contradictory meaning to any positive scalar adverb. these latter observations suggest that scalar adtable verbs might present important challenges for llms, and point to the need for a deep assessment. we look at full scales, and directly compare scalar adverbs to explicit negation. we consider the 24 adverbs in ## background we begin by introducing some concepts from linguistic semantics and pragmatics that motivate our study. ## materials building on the approach of ribeiro et al. (2020) and @xcite , we had the goal of designing a balanced diagnostic dataset for probing how well models capture the meanings of scalar adverbs. our primary dataset consists of 960 items, which are based on posts from the year 2015 in the reddit politosphere dataset introduced in @xcite . this slice represents about 6gb of data from a range of political subreddits (e.g., r/conservative or r/anarchist). it offers naturalness and domain consistency as well as a certain amount of diversity in linguistic usage. to select the posts, we first used spacy (honnibal and montani, 2017) to extract phrases of the form 'adv adj.' where there is a dependency between the adjective and the adverb. we take only phrases in which this construction occurs in final position, so that the phrases are also guaranteed to be well-formed in isolation. we then include the previous context from the same post, up to a maximum of 40 words, cutting at a sentence boundary. in semantic theory, preceding context is known to be important @xcite , but this factor is unfortunately disregarded in most of the related nlp research. aiming for at least 40 different adjectives to occur with each target adverb, we selected 8 distinct adverbs that expressed the speaker's judgment on a scale of likelihood (modality), 8 that express a position on a temporal scale (frequency), and 8 with more general applicability (degree). the adverbs were selected to span the full range of each scale, and hence include adverbs with negative force i.e., hardly and never. to shed light on the contrast between scalar adverbs and outright negation, the word not is reserved as a benchmark and not used as a target. both the adverbs themselves, and the adv adj bigrams, were selected to span the range of available frequencies to the extent possible, using google ngram @xcite frequencies. the target adverbs are listed in target context certainly you are conflating the issue. slavery was not moral but it was [mask] legal. frequently doesn't really matter what republicans say, democrats are going to call them racist. because what republicans say is [mask] racist. very you need verifiable proof. i mean, it's not like saying you're self trained is [mask] reputable. according to @xcite , some of our chosen scalar adverbs are 'maximizers' (e.g., completely) which tend to occur with extreme adjectives (e.g., freezing) or limit adjectives (e.g., dead). others typically combine with stereotypically scalar adjectives (e.g., cold). however, these are tendencies rather than rules @xcite . indeed, phrases such as very dead or completely cold can be perfectly acceptable in some contexts (e.g., i can assure you he was very dead). therefore, we do not restrict our phrases to traditional scalar adjectives and include any occurrences involving the target adverbs. example items and targets can be seen in these items were used as such in an mlm task. the same target adverbs are also used in the entailment task, but constructing items with templates rather than using the natural contexts. for the adverb ranking task, we combined each target scalar adverb with (the same) 40 common adjectives to try and get an average of generic embeddings: able, bad, big, black, clear, different, early, easy, economic, federal, free, full, good, hard, high, human, important, international, large, late, little, local, low, military, national, new, old, only, political, possible, public, real, recent, right, small, social, special, strong, white, young. we describe the construction of the entailment items below in 4.4. ## conclusion the goal of this paper was to examine how well pretrained language models such as bert and roberta represent and process full scales of scalar adverbs in the absence of any specific task fine-tuning. we used naturalistic data from reddit and also constructed sentences in order to explore the language models' ability to predict different types of scalar adverbs in context, and to distinguish them from negation. the models achieved some success when a left context of up to 40 words was available. however, we note many shortcomings: weak differentiation amongst the semantic classes of adverbs, poor ability to discriminate scalar adverbs from negations even in contexts where a negation would create a contradiction, strong effects of adverb frequencies and lack of generalisation across two logically equivalent entailment constructions. acceptability of negation: for some of our natural stimuli, the use of not would be infelicitous or illogical. for others, not would be possible but would express a meaning that contradicts what the speaker chose to express. without a large-scale exploration of alternative contextualizations of the items, it is difficult to separate these cases. for example, for item 498, a substitution of not would appear to be rather infelicitous; however, by imagining further context we can see that this substitution would not be impossible. models: while we tried to explore the predictions from different types of pretrained models (i.e., gpt and bert), we acknowledge that we did not run an extensive study of models from different families. this is in part because these are the most commonly used models in applications, but also because our study is qualitative and we were mostly interested in comparing the models' outputs with and without context and comparing performance between our semantic categories, rather than between different models. we also wished to focus on open-source models for which we could extract embeddings to explore a potential subspace for scalar properties. gold standard: there are few resources for providing gold standard labels of position on the scale for scalar adverbs in general, and especially when including different semantic categories and downsisers as well as maximisers. this limited our available choices for scalar adverbs to investigate. we provided the gold standard labels for the list of adverbs, based on information in wordnet and claims in the research literature, excluding adverbs whose semantics appeared unclear. while these rankings are informed by our best knowledge of semantics as experienced linguists, they were provided by a few researchers rather than by gathering judgments from many crowdsourcers as in other studies. (a) bert large below condition (b) bert large above condition (c) roberta large below condition (d) roberta large above condition c adverb frequencies