N ibbling at the Hard Core of W ord S ense D isambiguation (2022)

Abstract: With state-of-the-art systems having finally attained estimated human performance, Word Sense Disambiguation (WSD) has now joined the array of Natural Language Processing tasks that have seemingly been solved, thanks to the vast amounts of knowledge encoded into Transformer-based pre-trained language models. And yet, if we look below the surface of raw figures, it is easy to realize that current approaches still make trivial mistakes that a human would never make. In this work, we provide evidence showing why the F1 score metric should not simply be taken at face value and present an exhaustive analysis of the errors that seven of the most representative state-of-the-art systems for English all-words WSD make on traditional evaluation benchmarks. In addition, we produce and release a collection of test sets featuring (a) an amended version of the standard evaluation benchmark that fixes its lexical and semantic inaccuracies, (b) 42D, a challenge set devised to assess the resilience of systems with respect to least frequent word senses and senses not seen at training time, and (c) hardEN, a challenge set made up solely of instances which none of the investigated state-of-the-art systems can solve. We make all of the test sets and model predictions available to the research community at https://github.com/SapienzaNLP/wsd-hard-benchmark .

## introduction in recent years, natural language processing (nlp) has witnessed a quantum leap in benchmark task performance, mainly thanks to the adoption of two major technical innovations: the transformer architecture @xcite and transfer learning from language models pre-trained on massive amounts of textual data @xcite . the impact of these breakthroughs was so strong that, on many benchmarks, the performance of human non-experts was surpassed @xcite , prompting researchers to release new, more challenging benchmarks @xcite . word sense disambiguation (wsd), the task of automatically assigning a meaning to an ambiguous word in context @xcite , is undergoing a similar process: current state-of-the-art systems are now capable of attaining and surpassing the f1 score of 80% foot_0 on standard test datasets @xcite @xcite , a figure often reported as the estimated human performance, because it corresponds to the highest recorded inter-annotator agreement @xcite @xcite . matching and/or surpassing human performance reasonably triggers the assumption that systems are capable of carrying out tasks in real-world scenarios as effectively as their human counterparts @xcite , to the point where non-practitioners would regard such tasks as "solved". and yet, once systems are investigated beyond sheer accuracy figures, their flaws become readily apparent @xcite @xcite @xcite . following this trend of research, our work provides evidence showing why traditional evaluation measures for wsd, such as the f1 score, should not be taken at face value, hence corroborating the thesis that the problem of disambiguation is far from solved @xcite . to provide context, consider the following example, where the sense prediction foot_1 of the currently state-of-the-art escher model @xcite for the word wind is compared with the gold answer from the test set of semeval-2013 task 12 @xcite : context: the banks battling against a strong wind in the usa several years later. investors and regulators (. . . ) escher: air moving (. . . ) from an area of high pressure to an area of low pressure. here, the contextual meaning of the word wind is clear to any english speaker, with no cues in the sentence that would lead a human reader to pick the "air" meaning. this is an illustrative case of why, despite having achieved (on paper) superhuman performance, systems continue to make mistakes that the inter-annotator agreement would not justify. similarly, in the context below, another system which breaks the 80% performance ceiling @xcite makes a trivial mistake on a standard test instance @xcite , and fails to label the word couple properly: context: i was just sitting down to meet with some new therapy clients, a couple, and the building started shaking (. . . ) ## related work wsd has witnessed the creation of many different evaluation benchmarks, most notably as part of the senseval (now semeval) evaluation campaigns @xcite . since the release of the popular unified evaluation framework by @xcite , the experimental setting has become quite standard, with most systems being evaluated on all, i.e., the concatenation of senseval-2 @xcite , senseval-3 task 1 @xcite , @xcite @xcite , and semeval-2015 task 13 @xcite . besides reporting results split by part of speech, which has not been particularly insightful, no specific finer-grained analysis is usually performed. foot_3 this trend runs the risk of promoting a sort of collective hill-climbing behavior, which, in turn, makes it unclear how much the improvement in performance has been due to genuinely stronger generalization power, as opposed to overfitting to increasingly stale test sets. in opposition to this measure-centered style of evaluation, one possible alternative is that of behavioral testing, as proposed by @xcite . in their proposal (which does not address wsd explicitly), the benchmark evaluates separately minimum testable units of behavior, each of which addresses one specific skill required by a usable system. wsd, however, is a tricky problem to address in this way, as it is, in fact, a collection of idiosyncratic, diverse classification problems, which are hard to cluster in a meaningful way. a different kind of analysis, perhaps more specific to wsd, has tackled the problem of the strong imbalance of sense distributions, which makes learning difficult for automatic algorithms, and monitors how this imbalance affects performance @xcite @xcite . we follow this line of research in that we also take sense distribution skewness as the core issue in the development of wsd algorithms. therefore, both in the analysis of current wsd systems and in the creation of our new benchmarks, we check for the excessive influence of the most frequent output classes. ## systems at issue in an effort to make our analysis as thorough and comprehensive as possible, we consider a set of seven representative cutting-edge approaches for wsd. 4 with the exception of syntagrank (scozzafava et al., 2020), all systems are supervised neural architectures exploiting pre-trained language models. below, we describe each of these systems: foot_5 ares @xcite foot_6 is a semisupervised approach to producing contextualized sense embeddings that share the same space as those from bert @xcite . it enables a simple 1 nearest-neighbour algorithm to attain high performance both in the english and multilingual settings despite relying on english training data only. we use the ares english vectors freely available at http://sensembert.org . bem @xcite ) is a biencoder model with high accuracy for the disambiguation of rare word senses. bem maps the target in context and its word senses (as represented by glosses) independently into a shared embedding space, by means of jointly learned context and gloss encoders. disambiguation is then performed simply by predicting the sense whose encoding is most similar to that of the target. we employ the model and code available at https://github.com/ facebookresearch/wsd-biencoders . escher @xcite frames wsd as a span extraction task similar to squad @xcite , in which a system is asked to detect the span matching the gloss of the correct sense for a target word from a pseudodocument constructed by concatenating the con-text of the target word with all the glosses of its possible senses. at the time of writing, es-cher represents the state of the art in wsd. 7 we employ the model and code available at https: //github.com/sapienzanlp/esc . ewiser @xcite is a wsd classifier that exploits relational information included in wordnet by incorporating a sparse adjacency matrix within the architecture. we employ the model and code available at https: //github.com/sapienzanlp/ewiser . generationary @xcite reframes wsd as definition modeling, i.e., the task of generating a gloss from static or contextual embeddings @xcite , therefore recasting disambiguation as a generative problem. we use the gen-uni (mbrr) model reported in the original paper. while the only exposure of the model to wordnet-tagged data was through semcor @xcite , i.e., the most widely employed training set for wsd, the model was also trained on other lexicographic resources, such as the oxford dictionary @xcite . glossbert @xcite formulates wsd as a gloss ranking task, with a crossencoder scoring context-gloss pairs. the model is trained with a simple learning-to-rank @xcite approach, simply predicting whether a gloss is relevant to the context or not. we employ the model and code available at https://github. com/hslcy/glossbert . syntagrank @xcite is a knowledge-based system that jointly exploits the personalized pagerank algorithm and the wealth of syntagmatic information contained in syntagnet @xcite to perform disambiguation in multiple languages. we accessed syntagrank by means of its apis which are freely available at http://api.syntagnet.org/ . ## the hard core to consider wsd as solved, it would be reasonable to expect disambiguation errors to be little more than mismatches between the reference ground truth and another different, but still reasonable interpretation. for example, if we consider the word bold is m-f1 best. ## new benchmarks results from the quantitative and qualitative analysis carried out on the hard core reveal two main reasons why f1 scores can be potentially misleading indicators of the actual capabilities of current systems: (i) scores are actually a long way from estimated human performance when observed in challenging, but nevertheless real-world scenarios, and (ii) errors found in traditional test beds compromise insightful model evaluations. against this background, we put forward a set of evaluation tools to enable a more robust appraisal of system performance in english wsd, namely, (i) 42d, a multi-domain challenge set, (ii) amended versions of all (all n ew ) and semeval-2010 task 17 (s10 n ew ), and (iii) the new harden/soften benchmark. ## conclusion although traditional metrics indicate that wsd systems have attained human-level performances, the actual capabilities of state-of-the-art models are poorly reflected by the current evaluation benchmarks. in this paper, we analyzed the intersection of errors made by a heterogeneous set of seven state-of-the-art systems for english wsd from a quantitative and qualitative perspective, detailing two main reasons why they still falter when compared to their human counterparts, namely, their strong bias towards most frequent word senses and towards senses featured in the training data, as well as the presence of an array of lexical and semantic fallacies in traditional evaluation benchmarks. with the aim of providing a test bench that is more effective in reflecting the actual capabilities of wsd systems, we introduced (i) an amended version of the most popular test bed for wsd, and (ii) the 42d challenge set. as a result of the aforementioned work, we also present the harden/soften benchmark, a unified test bed aimed at moving forward with the disambiguation of so far unresolved instances, while keeping track of the current strong points of wsd systems. we make our test sets and model predictions available at https://github.com/ sapienzanlp/wsd-hard-benchmark .