Homonym normalisation by word sense clustering: a case in J apanese (2020)

Abstract: This work presents a method of word sense clustering that differentiates homonyms and merge homophones, taking Japanese as an example, where orthographical variation causes problem for language processing. It uses contextualised embeddings (BERT) to cluster tokens into distinct sense groups, and we use these groups to normalise synonymous instances to a single representative form. We see the benefit of this normalisation in language model, as well as in transliteration.

## introduction this work presents a method of clustering homonyms that uses contextualised word embeddings, assesses its performance for polysemy detection and shows two use cases in japanese, normalisation and transliteration. japanese has multi-typed orthography, using three different alphabets, which creates challenges as homonyms/homophones are extensively observed. our main proposal is to detect semantically equivalent groups in them, thereby cerating normalised, semantically less noisy data in pre-processing, to the benefit of language modelling. we also apply the same technique to a long-standing 'annoyance' in japanese input, i.e. transliteration process. homonymy is a universal phenomenon, and distinguishing their distinct meanings is a major problem in language processing. the advent of contextualised word embeddings @xcite has made it possible to differentiate token level occurrences, and they have indeed been utilised for supervised word sense disambiguation @xcite , but whether it can also capture latent meaning distinctions without labels is unclear, especially when the meaning boundaries are vague @xcite . our strategy is to detect relatively clear-cut cases by using a halting criterion on the number of clusters. japanese offers a unique platform to conduct such clustering on, since along with homonyms, their opposite counterpart, heterographs -words semantically and phonemically identical but written differently-systematically coexist. we will show first that accurate enough clustering is possible via contextualised embeddings and then that the clustering improves the two aspects mentioned above, normalisation and transliteration. ## terminology terminology in which to describe homonymy and heterography and related phenomena is rather confusing and often confused, and some tidying-up is in order for clarity. we have at least six related terms, homo-/hetero-prefixes with -phone/-nym/-graph suffixes that describe a relation that holds in a set (mostly pair) of word types. in accordance with how the terms are conventionally used, we could understand these roughly to correspond to the combinations of the identity/difference in three parameters, sound, writing and meaning: the combination of same sound, same spelling but different meaning is for homonyms and so on. leaving out the uninteresting combinations of the same and different in all these respects, we are left with six combinations. the following table shows the correspondence. while the conventional usage unfortunately does not lend itself to a clear one-toone mapping, our targets are fortunately amongst the teminologically clear ones, namely homonyms/homophones on one hand, and heterographs on the other. that is, words pronounced identically but not synonymous, and ones that are identical in meaning but are written differently. homonyms and homophones are differentiated based upon whether the writing is also identical (homonyms) or not (homophones) in addition to sound, but this distinction is sometimes not essential, and we take liberty to use the term homonym to represent both, unless the distinction is crucial. in short, we are dealing with one-to-many mappings in both directions, from sound to meaning, and from meaning to sound. it is important to note, in this relation, an inter-language contrast, mainly caused by the orthographical charateristics, i.e. the phonemic/ideographic contrast. heterography is not a prominent phenomenon in a single-alphabet orthography, but is pervasive in japanese. with both heterography and homonymy pervasive, japanese poses the issue of multiplicity in both directions, calling for resolution on both counts. clustering of word types provide a uniform solution for these problems. ## motivation with our terminology definition in place we can now succinctly state what our objectives are. our ovearching goal, homonym normalisation, is to render the data more compact (less typesparse) and less ambiguous by grouping together heterographs -different-looking synonymous words-and separating out homonyms -same-looking heteronymous words. we do this by clustering, attempting to achieve the following: heterographs be merged into a single group and homonyms be grouped into different meanings. aside from being free from labelling, clustering has this advantage of achieving these two tasks in one go, which would have to be done in two separate tasks in the classification approach. there is a further advantage in the clustering approach for a set of sutble cases: avoiding overfitting on words with very small semantic differences, i.e. 'very close' synonyms such as 'midday' and 'noon'. furthermore, japanese has a sometimes quite extensive set of 'almost synonymous' homophones, due to the 'borrowing' of chinese characters with similar meanings that are riginally pronounced differerntly in chinese but end up as homophonic, in the absence of differentiable equivalents in japanese. such cases are observed particularly for verbs and adjectives. for example, homophonic '停める', '止める', '留める', '駐める', all roughly meaning to stop/stall are so close that the choice could be simply up to the stylistic preference of the writer. we would like to allow these close synonyms to belong to the same group. another advantage is a robust, gradient approach to vagueness. a homonym can show 'shades' of meanings. 'bank' in bank account and food bank may be considered a borderline case, as opposed to river bank vs. bank account. clustering approach provides the flexibility to handle all these cases. generally speaking, contexts does the job of disambiguation, even a very short one at times. for example, for japanese, くま (kuma), homonymous between bear (animal) and black eye, can be disambiguated by a single-verb context: く ま が 出 た/く ま が で き た ('a kuma showed up/developed'). so our hope is that with the use of contextual embeddings and an appropriate halting criterion, we will end up with two clear cut clusters. for those borderline cases like the above, on the other hand, we let the data decide if the purported synonyms are contextually similar enough. in the context of disambiguation, the multiplicity of alphabets in japanese is something of a mixed blessing. the co-existence of phonemic alphabets (two sets of kana) and ideographic characters (kanji) does make the task of clustering more complicated, but the latter provides a natural reference for homonymy. sticking with our kuma example, of which we earlier had the hiragana version, it can be written in katakana (クマ) too, where we have exactly a parallel situation (homonymic and two-way ambiguomus), and then we have ideographic versions, 熊 and 隈, which do correspond to distinct meanings. our desired end results would therefore be to end up with two clusters for the whole set, 'bear's and 'black-eye's, the first containing 熊 and bear-meaning kana occurrences, the second 隈 and black-eye meaning kana occurrences. another important motivation comes from a practical application essential for the input of ideographic orthography on computer: input method, or transliteration. in japanese or chinese input, the user copes with the numerous characters by inputting first phonemic form (kana for japanese, pinyin for chinese) and 'convert' it into ideographs. this conversion process itself is cumbersome enough, but in the presence of homophones, the user is further required to choose amongst multiple ideographs shown on screen, even when the the choice is obvious from the context. this issue can also be handled by our clustering model, by additionally turning the model into a predictor. we will see this application later in section 6.2 too. ## related work clustering on word sense has attracted interest in the community of computational semantics since early days, with the unsupervised word sense induction tasks often featuring in semeval workshops @xcite . theoretical motivations for word sense clustering are well described in @xcite , as well as the comparison with the traditional classification method. since word embeddings emerged as a standard tool in nlp, and particularly after the advent of contextualised embeddings, embeddings-oriented work also started to appear @xcite for polysemy detection, while others have started to use them for diverse applications such as transfer learning @xcite , paraphrasing @xcite and search engine improvement @xcite . normalisation has been conducted primarily on heterographs (such as abberation from the standard form) with a classification method @xcite , including japanese @xcite , with noisy text as the target. contextualised embeddings have started being used in this context @xcite , again usually under the supervised framework. work combining normalisation with clustering is not so common though exceptions exist @xcite . as for transliteration systems into ideographs, several statistical methods have been proposed and implemented both in chinese and japanese. for chinese, neural-net based systems are actively developed, such as @xcite and @xcite . for japanese, google's mozc is probably the most popular statistical input method commercially available @xcite . being essentially a bigram model, however, it is not sensitive to meaning distinctions of homophones when the relevant context is located farther than the immediate vicinity, or the right choice is infrequent. for example, consider the plausible rendering of kōsō in this context: 仏教史に名を残した中国の ('a chinese who has gone down in the buddhist history'). for the human mind, the clear choice, though its relative infrequency, is 高 僧 ('prominent monk') rather than other homophones, such as 高層 ('high-rise'), 構想 ('design') and 抗争 ('infighting'), which are nevertheless its top candidates. more advanced engines have been proposed, e.g. a recurrent neural-network (rnn) based model by @xcite , but it is stated that the latency issue prevents such a model from being deployed. while we do not mean to construct a full input method, we will show that our 'module' can improve the choice in many cases without much overhead. ## applications to gauge the benefit of homonym normalisation by clustering, we applied the results to two applications, language model and transliteration. we will evaluate our clustering performance on the normalisation results themselves, but also on these two applications. ## concluding remarks we described a method of word sense clustering which uses contextualised word embeddings, evaluate its meaning differentiation capacity and use the model for meaning-based normalisation of orthography and context-sensitive transliteration, with promising improvement. we might emphasise that the proposed clustering procedure being unsupervised, it can be applied to any language. though we focused on japanese, where the homophone/homonym issue is most serious, the same procedure can be used in other languages for various purposes. the immediate application can be made to chinese, which have similar challenges concerning orthography. we can apply normalisation perfectly well also to non-ideographic languages, with the proviso of making reference data available for evaluation. the most serious outstanding problem concerns the issue of halting criterion for the optimal number of clusters. the clustering result is adequate enough except in homogeneity, which indicates oversplitting. we tried to find the optimum experimentally with various criteria but a better alternative is to use dirichlet process @xcite , which can find it in a more principled manner. unfortunately, its computational cost has prevented us from finishing the experiment and constitutes a major future task.