Development and evaluation of phonological models for cognate identication (2018)

Abstract: The paper presents a methodology for the development and task-based evaluation of phonological models, which improve the accuracy of cognate terminology identification, but may potentially be used for other applications, such as transliteration or improving character-based NMT. Terminology translation remains a bottleneck for MT, especially for under-resourced languages and domains, and automated identification of cognate terms addresses this problem. The proposed phonological models explicitly represent distinctive phonological features for each character, such as acoustic types (e.g., vowel/ consonant, voiced/ unvoiced/ sonant), place and manner of articulation (closed/open, front/back vowel; plosive, fricative, or labial, dental, glottal consonant). The advantage of such representations is that they explicate information about characters’ internal structure rather than treat them as elementary atomic units of comparison, placing graphemes into a feature space that provides additional information about their articulatory (pronunciation-based) or acoustic (soundbased) distances and similarity. The article presents experimental results of using the proposed phonological models for extracting cognate terminology with the phonologically aware Levenshtein edit distance, which for Top-1 cognate ranking metric outperforms the baseline character-based Levenshtein by 16.5%. Project resources are released on: https://github.com/bogdanbabych/cognates-phonology

## introduction: development of phonological models for cognate terminology identification this paper presents a methodology for the development and automated evaluation of linguistic phonological features sets that can extend traditional methods of cognate terminology identification, such as levenshtein edit distance. cognate identification is important for a range of applications. this paper evaluates its use for assisting mt developers in creating cognate term banks used in rule-based and hybrid mt, as well as in computer-assisted translation, development of dictionaries and between closely related languages (e.g., ukrainian (uk) and russian (ru), portuguese (pt) and spanish (es), dutch (nl) and german (de)). for many of such language pairs one of the languages can be under-resourced, therefore no electronic dictionaries are available, and only small parallel corpora with limited lexical coverage can be collected. typically these parallel corpora can provide translations for frequently used general words, but miss the 'long tail' of less frequent, often topic-specific or terminological words. however, in closely related languages these words are often cognates, which creates a possibility to rapidly extend bilingual lexicons in semi-automated way using nonparallel, comparable corpora and automated cognate identification techniques. in this task, cognate candidates are generated from word lists created from large monolingual comparable corpora in both languages the assumption is that the developers have good linguistic intuition of both languages and work through lists of cognate candidates, checking which pairs can be added to the bilingual dictionary. their productivity depends on whether cognates are presented high up in the list of candidates, ideally at the top of the list, or at least in the top n items, where n should be relatively small, e.g., the number of lines which fit on a single screen. other uses of cognates for terminology identification include term extraction from parallel corpora. if multiword source terms are known, the task is to identify the boundaries of the corresponding multiword target terms in the aligned target sentences, where component words or stems of compound words within the target terms may not be necessarily cognate with the corresponding source, so correctly identified cognates can facilitate adding adjacent non-cognate words according to part-of-speech and word order patterns, e.g., en: 'information requirements' ~ uk: 'інформаційні потреби' ('informatsijni potreby'); or splitting and extending compounds which have cognate parts, e.g., en: 'multinational' ~ uk: 'багатонаціональний'-('bahatonatsionalnyj'). yet another application of cognate identification is sentence alignment of parallel corpora, where statistical alignment methods are more accurate if cognates are used as an additional data source @xcite . inaccuracies in cognate identification, which are due to orthographic differences, often create unnecessary bottlenecks for this task @xcite . in this scenario identified cognates are not necessarily terms, but they contribute to a more accurate alignment and extraction of non-cognate terminology, produced from word alignment and monolingual terminology detection. an additional complication for the multilingual terminology extraction scenarios that rely on cognate identification is the use of different writing systems in the source and target (e.g., cyrillic or georgian vs. latin script), which requires transliteration between those languages. transliteration is often non-trivial, because of differences in pronounciation of the same letters, the lack of direct graphemic equivalents across languages, contextual dependencies in transliteration rules, different historical conventions for different words (e.g., en/de "h" à ru "х" (hockey ~ хоккей, since borrowed directly from en), or "г" (hermeneutics ~ герменевтика, since borrowed via ukrainian, where en: h à uk: г [ɣ] à ru: г [g]). also, even if languages use the same alphabet, pronounciation of letters and corresponding transliteration rules may differ (e.g., cyrillic letter "и" = [i] in @xcite in uk, latin letter "g" = [g] in en/de, and [ɣ] in nl), so new transliteration mappings need to be created for each translation direction, each with their potential language-specific problems. as a result, the complexity of transliteration in some cases is comparable to the complexity of mt, and it is often addressed not via simple character mappings, but via fully developed character-based mt models that require an aligned training corpus for each translation direction, and which are used in mt applications to cover out-of-vocabulary words, such as compounds, morphologically complex words, named entities and cognate terminology @xcite transliteration problem resembles a traditional "direct translation" bottleneck in mt: this approach cannot reuse any of the previously created mappings between languages if a new language pair or translation direction need to be covered. a more principled approach to the transliteration problem in the context of automated cognate identification, developed in this paper, is mapping characters for each language into a language-independent ("interlingual") phonological feature space. ## related work the use of phonological features for cognate identification has been initially proposed in the context of dialectological studies @xcite and diachronic phonology @xcite , @xcite . some limitations of these approaches for mt-related tasks have been discussed in @xcite , such as the need for phonological transcription of orthographic words and the absence of reliable evaluation for different ways of organising the complex phonological feature space and computing similarity between phonological segments. for instance @xcite acknowledges that different phonological features make unequal contribution in computing similarity between segments. to address this problem, in the aline phonetic aligner an introspective set of weights for each of the features is adopted from @xcite . machine-learning algorithms based on learning phonetic mappings from bilingual texts @xcite outperform the introspective linguistic model based on weighted phonological features. however, the most important difference between identification of cognates for dialectological or historical studies of language vs. for mt-oriented tasks of cognate term identification is the range of the compared candidate cognates and therefore the need of the metric to be optimised for both recall and precision on the large dictionary data sets. addition of phonological features on such tasks often results in overgeneration, so additional features have to be used, such as semantic similarity of terms, wordnet-based and semantic features, clustering @xcite . on the large scale for cognate identification for mt, where datasets are not limited only to candidate cognate pairs, a character-based levenshtein edit distance @xcite is typically used, without additional linguistic features. levenshtein metric calculates the number of insertions, deletions and substitutions between compared word pairs from different languages and determines if they pass a threshold to be considered cognate candidates. for example, if cognate candidates are extracted from a non-aligned or non-parallel corpus, the levenshtein distance is computed for every pair of words in the two word lists created for each language (the cartesian product of the lists), the search space may be restricted to comparing words with the same part-of-speech (pos) codes, if pos annotation is available for the corpus. however the problem with the character-based levenshtein metric is that all characters in comparison are treated as atomic units that do not have any internal structure and therefore, can be substituted only as a whole character. because of this the levenshtein metric does not distinguish between the substitutions of characters that correspond to acoustically/articulatory similar sounds vs. the substitution of phonologically distant letters. as a result, words that are intuitively close may receive a large distance score, e.g., uk "жовтий" (zhovtyj)='yellow' ru "жёлтый" (zheltyj) = 'yellow' (lev @xmath0, where, for historical reasons, articulatory similar sounds are represented by different characters: the sound [o] -by 'о' in uk and 'ё' in ru, the sound [y] -by 'и' in uk and 'ы' in ru. on the other hand, words that are not cognates and are phonologically and intuitively far apart, still receive the same distance scores, such as: uk "жовтий" (zhovtyj) = 'yellow' and ru "жуткий" (zhutkij) = 'dismal' (@xmath1. for example, here no distinction is made between, on the one hand, the substitution "о" (o) à "ё" ( i o) of phonologically similar sounds (which differ only in a peripheral feature -trig-gering palatalization of the preceding consonant (uk: --ru: +; in addition, this feature is neutralised after the sibilant "ж" (zh)), and on the other hand -the substitution "о" (o) à "у" (u), where sounds differ in core articulatory features of the place of vowel articulation (uk: middle; ru: close/high). some existing modifications and extensions of the levenshtein metric introduce weightings for different character mapping, but these weights need to be set or empirically determined for each specific mapping: compared characters still do not have internal structure and there is no way to predict the weights in advance for any possible pair in a principled way. this paper presents an automated task-based evaluation framework for an extension to the levenshtein edit distance metric, which explicitly represents linguistic phonological features of compared characters, so the metric can use information about characters' internal feature structure rather than treat them as elementary atomic units of comparison. similar sets of distinctive features have been used for comparing transcriptions of spoken words in modeling dialectological variation and historical changes in languages @xcite . in the proposed approach, phonological feature representations are applied to cognate identification and terminology extraction tasks, transliteration, and as well as modeling morphological variation. previously it has been shown that there are multiple ways of identifying, representing, structurally arranging and comparing these features in a phonological feature space @xcite , so there is a need for a methodology for evaluating alternative feature configurations. the results of the previously reported pilot experiment, using a small-scale manual evaluation, indicated the need to use hierarchical phonological feature structures for consonants rather than flat feature vectors previously used in dialectological research. manual evaluation methods in previous pilot experiment cannot be used for systematically testing and optimising weights or alternative phonological feature representations used in the levenshtein phonological metric. for instance, a serious problem for the proposed phonologically aware metric has been overestimation of its insertion and deletion costs, which is mainly due to the relatively smaller average substitution cost, and no corresponding reduction in the average insertion or deletion costs. e.g., for non-cognates a replacement of a consonant with another phonologically unrelated consonant produces a substitution distance of 0.8, because one feature -"type:consonant" does not have to be rewritten (phonological structure of consonants in the proposed models has 5 features). if insertion and deletion costs @xmath2, this leads to disproportional under-generation of cognates that contain inserted or deleted characters. even though the need of adjusting insertion/deletion distances has been highlighted in the pilot stage, manual evaluation methods used then did not allow us to test and optimise multiple parameter settings for the phonological metric, such as a range of different insertion and deletion costs. their values have to be determined experimentally using an automated evaluation methodology. this paper develops an automated framework for evaluating different arrangements of phonological features and parameters using the task of cognate identification, which enables us to experimentally find optimal setup of a metric for a given task. apart from practical applications mentioned above, this methodology creates a framework for feature engineering for phonologically aware character-based models for a wider range of machine translation and machine learning methods and tools, to design and calibrate phonological feature structures in a systematic way tuned for optimal the performance on specific tasks. the proposed automated evaluation framework uses standard automatically computed evaluation metrics, such as number of cognates in top-n candidates and an average rank of a correct cognate in an ordered candidate list. evaluation is performed on a larger data set of candidate cognate lists generated from large ukrainian and russian corpora on a high-performance computing cluster. the evaluation results show the settings where phonological levenshtein metrics achieves best performance on the cognate identification task and allow us to rule out some unproductive modifications. ## set-up and results of the evaluation experiment this section presents a methodology for automated performance-based evaluation that is used in testing different settings of phonological categories and values for the extended levenshtein metric. the experiment is set up in the following way: in future, for the task of a large-scale induction of cognates between languages phonological feature representations will be optimised for speed and other techniques such as hashing of phonological features for the searched target entries will be implemented, which is expected to make the developed metric more usable for generation of wide-coverage translation resources. for each ukrainian word from the evaluation set, its russian translation equivalents from the gold standard dictionary translations were automatically searched in the ranked cognate lists generated for that word by different variants of the levenshtein metric. the position of the top dictionary translation equivalent was recorded in each of the ranked cognate lists. basel @xmath3.@xmath4: i0.2 = the cost of insertion deletion is set to 0.2, i0.8 = is set to 0.8 (it is set to 1 in the phon lev metrics. ## discussion of the results, conclusion it can be seen from the results show that phonological extension to the levenshtein edit distance metric on the task of cognate identification outperforms the character-based baseline. the proposed framework also allows accurate calibration of the feature arrangement and other parameter settings of the metric. the modified levenshtein metrics, phonological features sets for several alphabets and sample input files are released as an open-source software on the github repository @xcite . future work will include systematic evaluation of different possible feature hierarchies and costs, and metrics application to other tasks, such as transliteration.