Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality (2021)

Abstract: We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. For writing, lexical substitution systems can assist humans by suggesting words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our Swords benchmark has 3x as many substitutes per target word for the same level of quality, and its substitutes are 1.4x more appropriate (based on human judgement) for the same number of substitutes.

## introduction imagine you are writing the message "i read an amazing paper today" to a colleague, but you want to choose a more descriptive adjective to replace "amazing." at first you might think of substitutes like "awesome" and "great," but feel that these are also unsatisfactory. you turn to a thesaurus for inspiration, but among reasonable alternatives like "incredible" and "fascinating" are words like "prodigious" which do not quite fit in your context. ultimately, you choose to go with "fascinating," but reaching this decision required a non-trivial amount of time and effort. research on lexical substitution @xcite @xcite @xcite @xcite @xcite considers the task of replacing a target word in context with appropriate substitutes. there are two widely-used english benchmarks for this task: semeval @xcite and coinco @xcite . for both benchmarks, data was collected by asking human annotators to think of substitutes from memory. because lexical substitution was originally proposed as a means for evaluating word sense disambiguation systems @xcite , this data collection strategy was designed to avoid a bias towards any particular word sense inventory. in this work, we consider a different use case for lexical substitution: writing assistance. for this use case, we are interested in evaluating a system's ability to produce appropriate substitutes that are likely to be difficult for humans to think of. we show that the data collection strategy used in past benchmarks yields low coverage of such uncommon substitutes-for our previous example, they might contain words like "awesome" and "great," but miss words like "incredible" and "fascinating." furthermore, we observe that these benchmarks have low quality, containing words like "fun," which are easy to think of, but not quite appropriate in context. we present swords-the stanford word substitution benchmark-an english lexical substitution benchmark that raises the bar for both coverage and quality ( coinco candid (3), artless (1), blunt (1), complete (1), direct (1), forthright (1), frank (1), outspoken (1), plainspoken (1), truthful (1), unreserved (1) thesaurus honest, simple, easy, candid, forthright, genuine, sincere, truthful, unequivocal, unambiguous, clear-cut, clear, uncomplicated, aboveboard, barefaced, direct, frank, guileless, honorable, just, laid on the line, ... 33 more swords sincere (80%), genuine (80%), frank (70%), candid (70%), direct (70%), forthright (70%), uncomplicated (60%), up front (60%), clear-cut (60%), clear (60%), plainspoken (60%), complete (50%), straight-arrow (50%), honest (50%), open (50%), blunt (50%), outspoken (50%), truthful (50%), plaindealing (40%), undisguised (40%), unvarnished (40%), unreserved (40%), barefaced (40%), unequivocal (30%), upright (30%), simple (30%), veracious (30%), unconcealed (30%), like it is (30%), squareshooting (20%), upstanding (20%), undissembled (20%), manifest (20%), unambiguous (20%), pretenseless (20%), level (10%), laid on the line (10%), honorable (10%), guileless (10%), ... 20 more with 0% on the other hand, looking up the target word in a thesaurus has higher coverage, but low quality, because it does not consider the context. in swords, we combine the best of both worlds and provide a list of substitutes that has high coverage and high quality, along with fine-grained scores for each substitute (shown in parentheses). substitutes with scores greater than 50% from swords are bolded. determining whether a substitute is appropriate is intrinsically subjective. to address this, we collect binary labels from up to ten annotators for each substitute, inducing a score for each substitute. in coinco, analogous scores are derived from the number of independent annotators who thought of a substitute-hence, as we will show in section 4, these scores tend to correspond more to ease-of-recollection than appropriateness. in contrast, scores from swords correspond to appropriateness, and also allow us to explicitly trade off coverage and quality, permitting more nuanced evaluation. our analysis shows that compared to coinco, swords has 3x more substitutes per target word for the same level of quality, and its substitutes are 1.4x more appropriate based on scores for the same number of substitutes. we demonstrate that swords is a challenging benchmark by evaluating state-of-the-art lexical substitution systems and large-scale, pre-trained language models including systems based on bert @xcite and @xcite . in our evaluation, we find that humans substantially outperform all existing systems, suggesting that lexical substitution can be used as a downstream language understanding task for pre-trained models. we release swords publicly as a benchmark for lexical substitution, coupled with a python library that includes previous benchmarks in a common format, standardized evaluation scripts for prescribed metrics, and reproducible re-implementations of several baselines. foot_1 ## background we describe lexical substitution and briefly introduce two widely-used benchmarks: semeval @xcite , the first benchmark, and coinco @xcite , the largest existing benchmark. for a survey of other benchmarks, we refer readers to @xcite , @xcite @xcite . ## data analysis ## model evaluation in this section, we evaluate several methods on swords. the goals of this evaluation are threefold: (1) to prescribe our recommended evaluation practice for swords, (2) to measure performance of existing large-scale pre-trained models and stateof-the-art lexical substitution systems, and (3) to measure human performance for the purpose of comparing current and future systems. ## related work as we already discussed previous lexical substitution benchmarks in section 2 and models in section 5, we use this section to draw connections to other related literature.