RAW - C : Relatedness of Ambiguous Words in Context (A New Lexical Resource for E nglish) (2021)

Abstract: Most words are ambiguous—-i.e., they convey distinct meanings in different contexts—-and even the meanings of unambiguous words are context-dependent. Both phenomena present a challenge for NLP. Recently, the advent of contextualized word embeddings has led to success on tasks involving lexical ambiguity, such as Word Sense Disambiguation. However, there are few tasks that directly evaluate how well these contextualized embeddings accommodate the more continuous, dynamic nature of word meaning—-particularly in a way that matches human intuitions. We introduce RAW-C, a dataset of graded, human relatedness judgments for 112 ambiguous words in context (with 672 sentence pairs total), as well as human estimates of sense dominance. The average inter-annotator agreement (assessed using a leave-one-annotator-out method) was 0.79. We then show that a measure of cosine distance, computed using contextualized embeddings from BERT and ELMo, correlates with human judgments, but that cosine distance also systematically underestimates how similar humans find uses of the same sense of a word to be, and systematically overestimates how similar humans find uses of different-sense homonyms. Finally, we propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics.

## introduction words mean different things in different contexts. sometimes these meanings appear to be distinct, a phenomenon known as lexical ambiguity. in english, approximately 7% of wordforms are homonymous, i.e., they have multiple, unrelated meanings foot_0 (e.g., "tree bark" vs. "dog bark"), and as many as 84% of wordforms are polysemous, i.e., they have multiple, related meanings (e.g., "pet chicken" vs. "roast chicken") @xcite . but even unambiguous words evoke subtly different interpretations depending on the context of use, i.e., their meanings are dynamic and context-dependent @xcite . while the uses of runs in "the boy runs" vs. "the cheetah runs" may not be considered distinct meanings, a human comprehender will likely activate a different mental image when processing each sentence @xcite . these facts present a challenge for computational models of lexical semantics. any downstream task that involves meaning requires models capable of disambiguating among the multiple possible meanings of an ambiguous word in a given context. further, the graded nature of human semantic representations can influence how comprehenders construe events and participants in those events @xcite . in turn, a number of natural language processing (nlp) tasks could benefit from context-sensitive representations that go beyond discrete sense representations and capture the manner in which humans construe events-including sentiment analysis, bias detection, machine translation, and more @xcite . if an eventual goal of nlp is human-like language understanding, models must be equipped with semantic representations that are flexible enough to accommodate the dynamic, context-dependent nature of word meaning-as humans appear to do @xcite . yet a crucial prerequisite to developing better models is evaluating those models along the relevant dimensions of performance. thus, at the minimum, we need metrics that evaluate a model along two critical dimensions: a promising development in recent years is the rise of contextualized word embeddings, produced using neural language models such as bert @xcite , @xcite , xl-net @xcite , and more. advances in these models have yielded improved performance on a number of tasks, including word sense disambiguation (wsd) @xcite . wsd satisfies the disambiguation criterion above, but not the contextual gradation criterion. in fact, there is still a dearth of metrics for assessing the degree to which contextualized representations match human judgments about the way in which context shapes meaning. in section 2, we describe several related datasets that satisfy at least one of these criteria. in section 3, we introduce and describe the dataset construction process for raw-c: relatedness of ambiguous words-in context. 2 in section 4, we describe the procedure we followed for collecting human relatedness norms for each sentence pair. in section 5, we report the results of several analyses that probe how well contextualized embeddings from two neural language models (bert and elmo) predict these norms. finally, in section 6, we explore possible shortcomings in current models, and propose avenues for future work. ## related work most existing datasets fulfill either the disambiguation or the contextual gradation criterion, but few datasets fulfill both (see @xcite for an exception). several datasets contain human relatedness and similarity judgments for distinct words in isolation (see section 2.1). others are used for word sense disambiguation, and contain ambiguous words in different sentence contexts, along with annotated sense labels (see section 2.2); as noted in the introduction, wsd fulfills the disambiguation criterion, but not the contextual gradation criterion. several recent datasets contain graded relatedness judgments for words in different contexts (see section 2.3). however, none focus specifically on graded relatedness judgments for ambiguous words, controlling both the inflection and part of speech of the target word in question. finally, one dataset @xcite ) contains similarity judgments for polysemous words in context, but is more limited in size and does not match the sentence frame across the two uses (see section 2.4). ## raw-c: relatedness of ambiguous words, in context items were adapted from stimuli used in past psycholinguistic studies, which contrasted behavioral responses to homonymous and polysemous words, either in isolated lexical decision tasks @xcite or in a disambiguating context @xcite @xcite . we selected 115 words in total. for each ambiguous word (e.g., "bat"), we created four sentences: two each for two distinct meanings of the word. we attempted to match the sentence frames as closely as possible, in most cases altering only a single word 4 across the four sentences to disambiguate the intended meaning: we also labeled each word according to whether the two distinct meanings were judged by lexicographers to be polysemous or homonymous. distinguishing homonymy from polysemy is notoriously challenging @xcite ; common tests include determining whether the two meanings share an etymology (polysemy) or not (homonymy), or determining whether the two meanings are conceptually related (polysemy) or not (homonymy). both tests can be criticized on multiple grounds @xcite , and do not always point in the same direction (e.g., etymologically related words sometimes drift apart, resulting in apparent homonymy). for our annotation, we consulted both the online merriam-webster dictionary ( https://www. merriam-webster.com/ ) and the oxford english dictionary, or oed ( https://www.oed.com/ ), and identified whether each dictionary listed the two meanings in question in separate lexical entries (homonymy), or as different senses under the same lexical entry (polysemy). 5 for example, both dictionaries list the animal and meat senses of the word "lamb" as different senses under the same lexical entry, whereas they list the animal and artifact senses of the word "bat" under different lexical entries. there was one word ("drill") on which the two dictionaries did not agree; in this case, we labeled the two meanings ("electric drill" vs. "grueling drill") as homonymy (as per the oed). there were also three words for which neither dictionary distinguished the two meanings (either in terms of homonymy or polysemy). for example, "best-selling novel" and "thick novel" refer to cultural and physical artifacts, respectively, but are not listed as distinct senses. again, this highlights the 4 there were 13 words for which at least one of the four sentences used a different article ("a" vs. "an"), in addition to having a different disambiguating word. ## analysis and results the analyses run below were performed on the 112 target words (i.e., excluding the "unsure" items). 7 based on the suggestion of an anonymous reviewer, we also ran a follow-up norming study to collect estimates of sense frequency bias (sometimes called dominance); sense dominance is known to play an important role in the processing of ambiguous words @xcite @xcite . these dominance norms are included in the final dataset. human annotations were assigned to a scale from 0 ("totally unrelated") to 4 ("same meaning"). ## discussion word meanings are dynamic, dependent on the contexts in which those words appear-and some words are even ambiguous, generating distinct, incompatible interpretations in different situations (e.g., "fruit bat" vs. "baseball bat"). raw-c contains graded relatedness judgments (by human annotators) for ambiguous english words in distinct sentential contexts. importantly, the ambiguous wordform (e.g., "bat") is always matched for both part-of-speech and inflection across each sentence pair; 84 of the target words are nouns, and 28 are verbs. each word has relatedness judgments for six different sentences pairs (four unique sentences): two same-sense pairs, and four different-sense pairs. same sense pairs convey the same meaning, according to merriam-webster and the oed (e.g., "fruit bat" and "furry bat"), while different sense pairs correspond to meanings listed in either distinct lexical entries (e.g., "fruit bat" and "wooden bat") or distinct sub-entries (e.g., "marinated lamb" and "baby lamb"). furthermore, different-sense pairs are labeled according to whether they are related via homonymy or polysemy, a relevant distinction for both lexicographers and psycholinguists-recent evidence suggests that polysemous and homonymous meanings are represented differently in the mental lexicon @xcite . finally, the sentential context is always tightly controlled; in most pairs, only one word differs across the two sentences (e.g., "fruit" vs. "furry"). in section 5, we reported several primary findings. first, contextualized representations from both bert and elmo capture the distinction between same-sense and different-sense uses of a word, but their ability to distinguish between homonymy and polysemy is marginal at best. this contrasts with other recent work @xcite , suggesting that bert is able to differentiate between homonymy and polysemy. one possible explanation for this difference in results is that @xcite used naturally-occurring sentences from semcor @xcite , whereas our sentence contexts were more tightly controlled. our results indicate that even the presence of a single disambiguating word can trigger nuanced differences in semantic representation in humans, but not necessarily in current neural language models. second, we found that both bert and elmo explain independent sources of variance in human relatedness judgments, above and beyond same sense and ambiguity type (i.e., homonymy vs. polysemy). this is encouraging, because it demonstrates a direct benefit of graded (rather than categorical) judgments; for example, among the broad category of different-sense polysemous pairs, some are closely related (e.g., "marinated lamb" and "baby lamb"), and others are considerably less closely related (e.g., "hostile atmosphere" and "gaseous atmosphere"). overall, contextualized embeddings from bert were better at predicting human relatedness judgments than those from elmo-this is consistent with past work @xcite suggesting that bert outperforms elmo on tasks involving sense disambiguation. importantly, however, both bert and elmo failed to capture variance in relatedness judgments that is captured by same sense and ambiguity type. as depicted in figure 4 , cosine distance tended to underestimate how related humans find same-sense uses to be, and overestimate how related humans find different-senses to be. this is not entirely surprising, given that neither bert nor elmo are equipped with discrete sense representations-at most, they produce contextualized embeddings that are amenable to supervised classification or unsupervised clustering. yet this also illustrates that-at least on this task-humans do appear to draw on some manner of (likely fuzzy) categorical representation, such that the difference between two contexts of use is compressed for same-sense meanings, and exaggerated for different-sense meanings (particularly for homonyms). this suggests several exciting avenues for future work: can neural language models such as bert be augmented with semantic knowledge or representational schemes that improve their performance on raw-c or similar tasks? both possibilities are explored in section 6.1 below. ## conclusion we have presented a novel dataset for evaluating contextualized language models: raw-c (relatedness of ambiguous words, in context). this resource contains both categorical annotations, derived from expert lexicographers (merriam-webster and the oed), as well as graded relatedness judgments from human participants. we found that contextualized representations from bert and elmo captured some variance (r 2 = .37) in these relatedness judgments, but that the distinction between same-sense and different-sense uses, as well as between homonymy and polysemy, explains considerably more (r 2 = .66). finally, we argued that this gap in performance represents an exciting opportunity for further development, and for crosspollination between experimental psycholinguistics and nlp.