Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation (2023)

Abstract: Word sense disambiguation (WSD), which aims to determine an appropriate sense for a target word given its context, is crucial for natural language understanding. Existing supervised methods treat WSD as a classification task and have achieved remarkable performance. However, they ignore uncertainty estimation (UE) in the real-world setting, where the data is always noisy and out of distribution. This paper extensively studies UE on the benchmark designed for WSD. Specifically, we first compare four uncertainty scores for a state-of-the-art WSD model and verify that the conventional predictive probabilities obtained at the end of the model are inadequate to quantify uncertainty. Then, we examine the capability of capturing data and model uncertainties by the model with the selected UE score on well-designed test scenarios and discover that the model reflects data uncertainty satisfactorily but underestimates model uncertainty. Furthermore, we explore numerous lexical properties that intrinsically affect data uncertainty and provide a detailed analysis of four critical aspects: the syntactic category, morphology, sense granularity, and semantic relations.

## introduction disambiguating a word in a given context is fundamental to natural language understanding (nlu) tasks, such as machine translation @xcite , question answering @xcite , and coreference resolution @xcite . this task of word sense disambiguation (wsd) targets polysemous or homonymous words and determines the most appropriate sense based on their surrounding contexts. for example, the ambiguous word book refers to two completely distinct meanings in the following sentences: i)"book a hotel, please.", ii) "read the book, please". the phenomenon is universal to all languages and has been paid much attention since the very beginning of artificial intelligence (ai) @xcite . existing supervised methods (blevins and zettlemoyer, 2020; conia and @xcite @xcite cast wsd as a classification task in which a neural networks (nns)-based classifier is trained from wordnet @xcite , a dictionary-like inventory. although they have achieved the state of the art on wsd benchmarks, with some even breaking through the estimated upper bound on human inter-annotator agreement in terms of accuracy @xcite , they do not capture or measure uncertainty. uncertainty estimation (ue) answers a question as follows: to what extent is the model certain that its choices are correct? a model can be unsure due to the noisy or out-of-domain data, especially in a real-world setting. this estimation delivers valuable insights to the wsd practitioners since we could pass the input with high uncertainty to a human for classification. ue is an essential requirement for wsd. interestingly, the word "ambiguous" (in terms of the task of word sense disambiguation) itself is ambiguous: it refers to i) doubtful or uncertain especially from obscurity or indistinctness, and ii) capable of being understood in two or more possible senses or ways, according to the merriam-webster dictionary foot_0 . the conventional treatment only considers its second aspect but disregards the first uncertainty-related sense. in reality, there are many situations where uncertainties arise @xcite . the first situation assumes a true model to which each trained model approximates. uncertainty appears when the structures and parameters of the possible models vary; we refer to it as model uncertainty (figure 1 (a) ) in this paper. model uncertainty can be reduced when collecting enough data, i.e., adequate knowledge to recognize the true model and out-of-distribution (ood) data is always used to test model uncertainty. it has been observed that wsd is prone to domain shift and bias towards the most frequent sense (mfs) @xcite . therefore, it is essential to quantify model uncertainty in the task. another uncertainty is related to the data itself and cannot be explained away, which is referred to as data uncertainty (also called aleatoric uncertainty). data uncertainty happens when the observation is imperfect, noisy, or obscure (figure 1 (b)). even if there is enough data, we cannot obtain results with high confidence. wsd is contextsensitive, and the model output could be divergent due to partial or missing context. even worse, some words have literal and non-literal meanings and can be understood differently. with a fine-grained wordnet @xcite as a reference inventory, the inter-annotator disagreement is up to 20% to 30% @xcite : even human annotators cannot agree on the correct sense of these words. in this paper, we perform extensive experiments to assess the uncertainty of a sota model (conia and navigli, 2021) on wsd benchmarks. first, we compare the probability of the model output with the other three uncertainty scores and conclude that this probability is inadequate to ue, which is consistent with previous research @xcite . then, with the selected score, we evaluate data uncertainty in two designed scenarios: window-controlled and syntax-controlled contexts, which simulate noisy real-world data. further, we estimate model uncertainty on an existing ood dataset @xcite and find that the model underestimates model uncertainty com-pared to the adequate measure of data uncertainty. finally, we design an extensive controlled procedure to determine which lexical properties affect uncertainty estimation. the results demonstrate that morphology (parts of speech and number of morphemes), inventory organization (number of annotated ground-truth senses and polysemy degree) and semantic relations (hyponym) influence the uncertainty scores. ## results and analysis in the first part, we show the quantitative results of different ue scores and the performances of data and model uncertainty. then a qualitative result demonstrates specific instances with a range of uncertainties. this motivates us to analyze which lexical properties mainly affect uncertainty in the last part. ## conclusion we explore the uncertainty estimation for wsd. first, we compare various uncertainty scores. then we choose smp as the uncertainty indicator and examine to what extent a sota model captures data uncertainty and model uncertainty. experiments demonstrate that the model estimates data uncertainty adequately but underestimates model uncertainty. we further explore effects that influence uncertainty estimation in the perspectives of morphology, inventory organization and semantic relations. we will integrate wsd with uncertainty estimation into downstream applications in the future.