Correcting Sense Annotations Using Wordnets and Translations (2023)

Abstract: Acquiring large amounts of high-quality annotated data is an open issue in word sense disambiguation. This problem has become more critical recently with the advent of supervised models based on neural networks, which require large amounts of annotated data. We propose two algorithms for making selective corrections on a sense-annotated parallel corpus, based on cross-lingual synset mappings. We show that, when applied to bilingual parallel corpora, these algorithms can rectify noisy sense annotations, and thereby produce multilingual sense-annotated data of improved quality.

## introduction word sense disambiguation (wsd) is the task of identifying the appropriate meaning of a word in context, from a predefined sense inventory, such as wordnet @xcite and babelnet @xcite . it is one of the central problems in natural language understanding @xcite . the primary approaches to tackle the wsd problem can be divided into supervised and knowledgebased methods. supervised wsd systems have historically achieved the best overall results on standard wsd datasets @xcite . however, these systems rely on large amounts of sense-annotated data for training, which is costly and difficult to produce. in particular, there is a severe lack of high-quality annotated data for languages other than english, which is known as the knowledge acquisition bottleneck problem @xcite . to address this issue, various approaches have been proposed to automate the process of annotating texts in different languages at a large scale. some of the automated annotation approaches operate by leveraging translations from parallel corpora. the idea of using translations for wsd was considered by @xcite , based on the conjecture that different translations of an ambiguous source word in a target language could serve as sense-tagged training examples. this idea was put into practice by @xcite , and then on a large scale by @xcite , as they implemented a semi-automatic approach of disambiguating english nouns using distinct chinese translations, leveraged from an english-chinese parallel corpora. @xcite used a similar semi-automatic approach to create a wsd training set by leveraging the chinese-english part of the multiun corpus @xcite . delli @xcite removed the bottleneck of manual intervention, as they proposed a fully automated approach of producing multilingual sensetagged corpora by jointly disambiguating multiple languages of a parallel corpus. our work is inspired by the central idea of the aforementioned research that translations may provide the necessary information to disambiguate an ambiguous word. however, we focus on leveraging translations to improve the quality of an already sense-tagged parallel corpus, rather than to annotate the corpus from scratch. we propose two algorithms for correcting sense annotations in a parallel corpus. the first algorithm attempts to rectify aligned senses that belong to different multisynsets. the second algorithm considers all alignment links in a bitext to construct a one-to-one mapping between synsets in different languages. both algorithms are based on the theory of synonymy and translational equivalence of @xcite . we empirically show that our algorithms achieve their goal of improving the quality of sense annotations in multiple languages. we extrinsically evaluate the proposed corrections by providing the corrected corpora as training data to a supervised wsd system. an intrinsic evaluation on a random sample of 200 corrected instances in english and spanish confirms the improvement in the overall quality of the annotated corpora. ## multiwordnet (mwn) algorithm algorithm 1 mwn input : set of aligned sense pairs (s, t) lex(s) -word of which s is a sense m (s) -multi-synset that contains sense s m(w) -set of multi-synsets that contain word w 1: for each aligned sense pair (s, t) correct: t ← (lex(t), m (s)) correct: s ← (lex(s), m (t)) the mwn algorithm (algorithm 1) is based on the simplifying assumption that the senses of aligned words are translationally equivalent @xcite . the algorithm consults an existing multilingual wordnet (multi-wordnet) which is composed of multilingual synsets (multi-synsets) that include translationally-equivalent senses of words from both languages. each polysemous word belongs to multiple multi-synsets. if the senses of the aligned words are found to belong to different multi-synsets, this is an indication of a possible annotation error that could be corrected. the algorithm operates on a sense-annotated parallel corpus (bitext). it performs annotation corrections on individual aligned word pairs (line 1) which are annotated with different multi-synsets (line 2). each sense in a multi-wordnet is uniquely defined as a (word, synset) tuple. when applied to a sense, the lex and m operators return the first and second element of the tuple, respectively. we denote as c the set of all multi-synsets that contains both aligned words (line 3). the algorithm is designed to make selective corrections only in those alignment instances where there is little doubt about the appropriate correction. at most one of the two sense annotations in each instance can be corrected. a correction is made if and only if exactly one of the two aligned senses is found in c (lines 4-7). we do not attempt a correction if either both or none of the two senses are in c. if both senses are outside of c, we suspect multiple errors in bitext annotations and/or the multi-wordnet. on the other hand, if both senses are within of c, it is not clear which of the two annotations may be incorrect. ## bipartite (bp) algorithm algorithm 2 bp input : set of aligned sense pairs (s, t) lex(s) -word of which s is a sense s(s) -synset that contains sense s s(w) -set of synsets that contain word w the bp algorithm (algorithm 2) is also based on the assumption that the aligned words should express exactly the same concept. however, it differs from the mwn algorithm in that it globally considers all the alignment links in a given bitext, and makes annotation corrections based on the most frequently observed links. another difference is that bp only corrects the annotations in language l 2 , based on the annotations in the base language l 1 , which are assumed to be always correct. the algorithm is inspired by the concept universality principle of hauer and kondrak (2020) which states that each each monolingual synset corresponds to at most one synset in another language. no access to a multi-wordnet is assumed; instead the algorithm consults two language-specific wordnets, which are composed of monolingual synsets, rather than of multi-synsets. the bp algorithm consists of three stages: (1) construct a bipartite graph g of synsets; (2) identify its subgraph g of degree 1; and (3) correct sense annotations that are not found in subgraph g . in fact, the first two stages constitute a stand-alone algorithm for creating a cross-lingual mapping be-tween synsets. we describe the three stages in more detail below. in the first stage (lines 1-5), we construct a weighted undirected bipartite graph @xmath0, e, weight) in which nodes represent monolingual synsets, and edges represent alignment links that are observed in the bitext. the weight of an edge is equal to the number of the observed alignment links in the bitext between the senses of the corresponding synsets. the weight of a node is simply the sum of the weights of all edges incident with the node, which is equal to the number of times the corresponding synset is used in aligned sense annotations in the bitext. in the second stage (lines 6-10), we construct a graph @xmath1, e ), which is a subgraph of g, such that every node has a degree of at most 1. the goal is to select the edges that represent the most frequent alignments. this is achieved by only retaining the edges with the relative weight above a threshold α (lines 8-9) in both directions. the threshold is constrained to be greater than 0.5, to guarantee that at most one edge per node is selected. in the third stage (lines 11-15), annotation corrections are made based on the edges of the constructed bipartite graph g . unlike the mwn algorithm, the bp algorithm only corrects the annotations of words in language l 2 . if an edge corresponding to a given alignment link is not found in g (line 12), it attempts to correct the annotation in l 2 by following the edge in g between the node s(s), which represents the synset used to annotate the word in l 1 , and the node s , which represents the synset in l 2 that expresses the same concept as s(s). ## extrinsic wsd evaluation to extrinsically evaluate the algorithms, we apply them to @xcite , an automatically constructed sense-annotated resource based on the europarl parallel corpus @xcite . in eurosense, words (which include noncompositional mwes) are tagged with multilingual synsets from babelnet 4.0 @xcite , and accompanied by their respective lemmatized forms. we extract four sentence-aligned bitexts from eurosense, by considering four different language pairs: english-italian (en-it), english-german (en-de), english-french (en-fr) and englishspanish (en-es). we employ babalign @xcite to align the bitexts at the word level; the aligned word or phrase of each annotated token is taken as its translation. the annotated translation pairs in eurosense are filtered to remove non-existent senses, non-literal translations, and hypernym translations. a sense of a word is considered non-existent if it is not found in the respective babelnet synset. if the aligned words have no synsets in common, they are treated as non-literal translations. finally, we detect non-synonymous translations pairs by traversing hypernymy and hyponymy links in babelnet @xcite . in our development experiments, we found that approximately 3% of the pairs contain invalid senses, 13% are cases of non-literal translations. and 5% involve word entailment. following this filtering procedure, the remaining translation pairs are used as inputs to both algorithms to perform annotation corrections for each language separately. the bp threshold α is set to 0.8 on the basis of the development experiments. for it, de, fr and es corrections, we use english as the base language. to perform en corrections, we use italian as the base language as it is reported to have good babelnet coverage @xcite . 75.3% of the english-italian synset mappings returned by the bp algorithm match ba-belnet concepts. we extrinsically evaluate the corrections by providing the corrected corpora as training data for a supervised wsd system, which is then evaluated on standard benchmark datasets. to this end, we employ ims @xcite , a supervised wsd system based on lexical features. to keep the corpus at a reasonable size, we consider a maximum of 10,000 randomly sampled training examples per sense. for english, in cases where the system fails to make a prediction, we back off to the most frequent sense. for all languages, any monosemous words are automatically tagged with their single possible sense. ## intrinsic evaluation to intrinsically evaluate the quality of the sense annotation corrections made by the algorithms, a random sample of 200 english and spanish instances were annotated manually. for each instance, an annotator was shown the corresponding sentence from eurosense, and asked to decide whether the focus word is used in the original or the corrected sense (or neither). the senses were defined using babelnet glosses and synonyms. and provided in a random order. the results in the wrong corrections may be grouped into three types: of the concept that it represents. for example, the synset bn:00109131a, which is glossed in english as "related to the future", contains the spanish adjective futuro but not its english translation future. such omissions, which are frequent in babelnet because of its semi-automatic construction method, prevent the mwn algorithm from making a correction. ## conclusion our extrinsic and intrinsic evaluation results constitute a strong proof-of-concept that translations and wordnets can be leveraged to make effective annotation corrections in a sense-annotated bitext. manual analysis indicates that most of the invalid corrections can be traced to errors and omissions in existing lexical resources. in the future, we plan to investigate the use of machine translation instead of bitexts for the purpose of automatically annotating raw monolingual text corpora.