Testing agreement between lexicographers: A case of homonymy and polysemy (2021)

Abstract: In this paper we compare Oxford Lexico and Merriam Webster dictionaries with Princeton WordNet with respect to the description of semantic (dis)similarity between polysemous and homonymous senses that could be inferred from them. WordNet lacks any explicit description of polysemy or homonymy, but as a network of linked senses it may be used to compute semantic distances between word senses. To compare WordNet with the dictionaries, we transformed sample entry microstructures of the latter into graphs and cross-linked them with the equivalent senses of the former. We found that dictionaries are in high agreement with each other, if one considers polysemy and homonymy altogether, and in moderate concordance, if one focuses merely on polysemy descriptions. Measuring the shortest path lengths on WordNet gave results comparable to those on the dictionaries in predicting semantic dissimilarity between polysemous senses, but was less felicitous while recognising homonymy.

## introduction we talk about polysemy when different word senses are semantically related. homonymy is the opposite phenomenon in which etymologically unrelated senses are signified by the same word-form @xcite . 1 the main source of 1 for the needs of this paper, we define homonyms (homographs) as a pair of senses which are characterised by the same part of speech, share the same lemma, but are not related semantically and etymologically @xcite . a pair of polysemous senses (polysemes) -on the contrary -is constituted by the two senses of the same pos category, sharing the same lemma, semantically related and of the same etymology. the goals of this paper are two-fold: (i) we check the degree of agreement between polysemy descriptions in two general english dictionaries, namely oxford lexico and american english merriam-webster dictionary, and in word-net, (ii) we test the applicability of wordnet in measuring semantic similarity between senses (that is assessing polysemy vs. homonymy distinctions). for these purposes, we have created a data set of 57 nouns, noted by the three lexicons (sec. 3.1). we represented dictionary microstructures as graphs with the equivalent word-net synsets attached to them (sec. 3.2). the approach resulted in 889 sense pairs in total. the set of the mapped synsets served as a common denominator for the subsequent comparisons between the three lexical resources. measuring distances between particular sense pairs allowed us to compare the polysemy/homonymy description in the two dictionaries with the structural description in wordnet (via lexico-semantic relations, sec. 3.3 and 4). 2 it turned out that the dictionaries are in high concordance with each other, if we consider the homonymy-polysemy distinction, and in moderate agreement, if we look at polysemy descriptions (in terms of spearman's correlation coefficient). wordnet did not differ much from lex-ico and merriam-webster in its capability to describe similarity between polysemous senses. it was homonymy that made the difference (sec. 5). ## related work in natural language processing accessing word or sense dissimilarity via measuring distances in lexical networks is a well-known procedure @xcite @xcite . among many measures, some are of special interest for semantic relatedness assessment, that is path-based indices (the shortest-path, wu-palmer's, leakcock-chodorow's or li's measures) and information content-based measures (resnik's, lin's or jiang's methods, see @xcite ). in the context of recognising polysemous sense proximity, wu-palmer's measure was used to calculate concept similarity within the taxonomy of the historical thesaurus of english @xcite . each sense was compared with all other word senses, then the obtained matrices of similarity were used to arrange polysemous word meanings into a chain of extended senses. in @xcite polysemy networks for many world languages were compared with the use of path distances between swadesh' concepts mapped to them. the authors found the distance distribution of polysemy structures universal across languages, despite clearly different geographical and cultural conditions. out of various measures of semantic relatedness, we made use of one of the simplest -the shortest path length. since our graphs were weighted, we utilised dijkstra's distance algorithm @xcite which finds the geodesics for weighted networks. we applied it to measuring semantic distances in both english dictionaries and in wordnet. many traditional dictionaries depict word senses in the form of nested clusters of definitions. starting from the basic sense @xcite , @xcite , they unfold a network of inter-dependencies in the form of a sense hierarchy. in such structured polysemy nets main senses are linked into meaning chains with groups of subsenses attached to them @xcite . hierarchical sense differentiation is more intuitive for dictionary users than a flat arrangement. in such a set-up senses are ordered according to their semantic "closeness" @xcite . lexico and merriam-webster both represent this type of polysemy structuring. the problem of consistency of lexicographic entries is widely acknowledged @xcite . the same word may be differently treated in different dictionaries @xcite . splitting or merging senses is not an easy task even for a specialist. the issue seems highly intuitive and decisions are supposed to be highly arbitrary. this is not entirely true. in distinguishing senses lexicographers rely on specific rules, like observing usage restrictions (e.g., for specialised vocabulary), differences in syntactic frames (cf. transitive -intransitive frame) or other grammatical properties, like grammatical number (cf. pluralia and singularia tantum) @xcite . yet another way to tame lexicographers' intuitions is to rely on taxonomic and other sense relationships, in such a way genus proximum (a hypernym) and differentia specifica (a meronym/holonym, antonym etc.) might be captured @xcite . the lexicographic process of splitting and clustering senses was widely studied in the context of word sense disambiguation (e.g. @xcite ). we relate to several research studies which are most relevant to our approach. @xcite proposed a method of measuring sense distances on hector -a hierarchical dictionary @xcite ). they postulated that the penalty applied to a homonymous pair should be much higher than the cost of a polysemy step. in some aspects our methodology resembles this approach to the construction of an adjacency matrix. 3 @xcite counterargued against the possibility of an honest measure based on hierarchical dictionaries. the argument is as follows: since in metaphorical shifts extended senses completely change their semantic domain, dictionary provided sense relations do not mirror mental lexicon sense proximities. to this plea we answer that polysemy topologies are often multicentred @xcite and are governed by their own rules (naming ̸ = knowing, see @xcite ). @xcite executed experiments in the assessment of the number of word senses obtained from a tagged corpus which was collated with dictionary data (petit larousse). the spearman's rank correlation equal to 0.5 was reported for nouns. in various senseval editions research teams also reported rather mediocre agreement values between annotators @xcite , e.g., @xcite in senseval-3 observed ca. 70% ita (percentage agreement) and κ = 0.58; similar results were obtained by @xcite . according to @xcite , "[w]ord sense tagging is one of the hardest annotation tasks." ## comparison with wordnet dictionaries are in high agreement when we consider both homonymy and polysemy, and in moderate concordance when we look solely at polysemy. the moderate correlations in polysemy depiction are not surprising. if one took into account the fact that our dictionaries might have differently clustered meanings, subsenses and meaning shades; might have distinguished more or less 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 lexico merriam-webster 50 100 150 200 count figure 3: 2d histogram of dijkstra's distances between pwn senses (in steps). this time the overlooked senses landed in the top-most and rightmost sides of the square. ρ mw hp p lex .71 .43 ci (0.65, 0.76) (0.38, 0.53) when one merges the information from both dictionaries (see ## conclusions the performed experiments aimed at comparing how similarly two dictionaries described semantic distances in polysemy and homonymy. we found out that traditional english dictionaries showed traces of positive correlation between dijkstra's path lengths on corresponding polysemy nets (0.7 for polysemy and homonymy, and ρ = 0.4 for sole polysemy). with regard to the homonymy/polysemy binary distinction, we obtained cohen's κ = 0.67 and 85% percentage agreement. the agreement with wordnet was moderate in the case of homonymy and polysemy (ρ ∼ 0.46). when the oracle was considered (hints on the status of homonymous pairs), the correlation rose to the level of ρ = 0.7 which value was comparable to the confidence interval calculated for dictionaries. the values calculated for the sole polysemy (i.e. excluding homonymy) were slightly smaller than those obtained from the lexico and merriam-webster comparison. the achieved results resembled agreement measurements reported in the literature (see sec. 2 above). the performed experiments gave an insight into the debate on the quality of dictionary descriptions. it turned out that lexicographers from different publishing companies provided very similar semantic description of homonymy -senses were similarly grouped according to their shared etymology. dictionaries comparably described also semantic distance between related senses, when measured shortest paths on entry microstructures (micro-hierarchies). wordnet proved its usefulness in capturing the strength of polysemy links, but failed in homonymy recognition.