Transliteration for Cross-Lingual Morphological Inflection (2020)

Abstract: Cross-lingual transfer between typologically related languages has been proven successful for the task of morphological inflection. However, if the languages do not share the same script, current methods yield more modest improvements. We explore the use of transliteration between related languages, as well as grapheme-to-phoneme conversion, as data preprocessing methods in order to alleviate this issue. We experimented with several diverse language pairs, finding that in most cases transliterating the transfer language data into the target one leads to accuracy improvements, even up to 9 percentage points. Converting both languages into a shared space like the International Phonetic Alphabet or the Latin alphabet is also beneficial, leading to improvements of up to 16 percentage points.

## introduction the majority of the world's languages are synthetic, meaning they have rich morphology. as a result, modeling morphological inflection computationally can have a significant impact on downstream quality, not only in analysis tasks such as named entity recognition and morphological analysis @xcite , but also for language generation systems for morphologically-rich languages. in recent years, morphological inflection has been extensively studied in monolingual high resource settings, especially through the recent sig-morphon challenges @xcite @xcite . the latest sigmoprhon 2019 challenge @xcite focused on lowresource settings and encouraged cross-lingual training, an approach that has been successfully applied in other low-resource tasks such as machine transfer lang. test lang. acc. (l 1 ) (script) (l 2 ) urdu (arabic) 42 sanskrit (devanagari) bengali 44 hindi (devanagari) (bengali) 49 greek (greek) 42 arabic (arabic) maltese 18 hebrew (hebrew) (roman) 22 italian (roman) 34 translation (mt) or parsing. cross-lingual learning is a particularly promising direction, due to its potential to utilize similarities across languages (often languages from the same linguistic family, which we will refer to as "related") in order to overcome the lack of training data. in fact, leveraging data from several related languages was crucial for the current state-of-the-art system over the sigmorphon 2019 dataset @xcite . however, as anastasopoulos and neubig (2019) point out, cross-lingual learning even between closely related languages can be impeded if the languages do not use the same script. we present a few examples taken from @xcite in the aim of this work is to investigate this potential issue further. we first quantify the effect of script differences on the accuracy of morphological inflection systems through a series of controlled experiments ( §2). then, we attempt to remedy this problem by bringing the representations of the transfer and the test languages in the same, shared space before training the morphological inflection system. in one setting, we achieve this through transliteration of the transfer language into the test language's script as a preprocessing step. in another setting, we convert both languages into a shared space, using grapheme-to-phoneme (g2p) conversion into the international phonetic alphabet (ipa) as well as romanization. we discuss both settings and their effects on morphological inflection in low-resource settings ( §3). our approach bears similarities to pseudo-corpus approaches that have been used in machine translation (mt), where low-resource language data are augmented with data generated from a related highresource language. among many, for instance, de @xcite built a catalan-english mt by bridging through spanish, while @xcite show that word-level substitutions can convert a high-resource (related) language corpus into a pseudo low-resource one leading to large improvements in mt quality. such approaches typically operate at the word level, hence they do not need to handle script differences explicitly. nlp models that handle script differences do exist, but focus mostly on analysis tasks such as named entity recognition @xcite @xcite or entity linking @xcite , whereas we focus in a generation task. character-level transliteration was typically incorporated in phrase-based statistical mt systems @xcite , but was only used to handle named entity translation. notably, there exist nlp approaches such as the document classification approach of @xcite showing that indeed shared character-level information can facilitate cross-lingual transfer, but limit their analysis to same-script languages only. specific to the the morphological inflection task, @xcite use cognate projection to augment low-resource data, while @xcite explore the inflection task using inputs in phonological space as well as bundles of phonological features from panphon @xcite , showing improvements for both settings. our work, in contrast, focuses on better cross-lingual transfer, attempting to combine the phonological and the orthographic space. ## quantifying the issue in the intuition behind our analysis is that a model trained cross-lingually can only claim to indeed learn cross-lingually if it ends up sharing the representations of the different inputs, at least to some extent. this observation of a learned shared space has also been noted in massively multilingual models like the multilingual bert @xcite , or for cross-lingual learning of word-level representations @xcite . for a character-level model, such as the ones typically used for neural morphological inflection, this implies a learned mapping between the characters of the two inputs. our hypothesis is that such a learned character mapping, and in particular between related languages, should resemble a transliteration mapping, assuming that both languages use a phonographic writing system (such as the latin or the cyrillic alphabet and their variations), to use the notation of faber (1992). 2 to verify whether this intuition holds, we trained models on armenian-kabardian and bashkir-tatar (see details in section §3). in the first setting, the transfer language (armenian) uses the armenian alphabet, while the test language (kabardian) uses the cyrillic one. in the second, we are transferring from bashkir, which currently uses the cyrillic alphabet, to tatar, which is written with the latin alphabet. we obtain the character representations from the final trained models, and we perform a simple search over the embedding space, returning for each of the transfer language characters the nearest neighbor from the test language alphabet. our findings are that this type of mapping does not resemble a transliteration one, at all. for example, one would expect that the bashkir characters е, ә, or э would map to the tatar e character, or at least to another vowel. bashkir е indeed maps to tatar e, but ә maps to tatar i (which might be somewhat fine since they are both vowels), while bashkir э maps to tatar r. after a manual annotation of the mappings in both language pairs, we find that the absolute accuracy is less than 5% in both settings (2 of 54 are correct in bashkir-tatar, and 1 of 47 in armenian-kabardian). we also present a visualization (obtained through pca @xcite ) of the character embeddings in figure 1 for these two settings, which shows that the two languages are still, to an extent, separable. in an attempt to also take into account potential slight differences in pronunciation, which are common across related languages, we also count mappings that agree in coarse phonetic categories as correct. we obtain rough grapheme-to-phoneme mappings from omniglot 3 @xcite which allows us to classify each character as mapping to a vowel, or a consonant category (we devise categories across both manner and place). for instance, the bashkir characters с,ҫ,һ,ҙ,ш map to sibilant 3 https://omniglot.com/ fricatives, so we count any mapping to tatar characters that also map to sibilant fricatives (ç,z,s,ş) as correct. overall, however, even this more flexible evaluation only leads to an accuracy of less than 30% (16 out of 54 characters for bashkir-tatar, 12 of 47 in armenian-kabardian). ## methodology the previous section ( §2) showcases that different scripts can inhibit the model's ability to represent both languages in a shared space, which can be damaging for downstream performance in crosslingual learning scenarios. in order to bring the transfer and test languages into a shared space we explore two straightforward approaches: the following sections provide details on transliteration, grapheme-to-phoneme conversion, the inflection model, and the data that we use for training and evaluation. transliteration in the absence of some sort of a universal transliteration approach, we rely on various libraries for our experiments. for transliterating between the indic scripts (devanagari, bengali, kannada, and telugu in our experiments) we rely on the indicnlp library. 4 we also use the uroman foot_3 library @xcite to transliterate into the roman alphabet for the arabic, hebrew, armenian, and cyrillic scripts. the lack of resources and transliteration tools for some directions severely limited the extent of the experiments that we could conduct. notably, even though romanization is fairly well-studied and are easily attainable through tools like uroman, the opposite direction is fairly understudied. most of the related work has focused on either to-english transliteration specifically @xcite or on named entity transliteration @xcite . even then, the state-of-the-art results on the recent news named entity transliteration task @xcite ranged from 10% to 80% in terms of accuracy across several scripts. the high variance in expected quality depending on the transliteration direction showcases the need for further work towards tackling hard transliteration problems. ## conclusion with this work we study whether using transliteration as a preprocessing step can improve the accuracy of morphological inflection models under cross-lingual learning regimes. with a few exceptions, most cases indeed show accuracy improvements, some of them statistically significant. we also note that the improvements are orthogonal to those obtained by data augmentation through hallucination, even in typologically distant languages. while this work represents a first step in the direction of understanding the effect of script differences in morphological inflection, it is still limited in scope, as the experiments were restricted by the lack of reliable transliteration tools for most scripts. the sigmorphon 2020 shared task on morphological inflection also provides more languages and better systems are being developed, so we plan to expand our analysis to the latest stateof-the-art models @xcite . additionally, some of the transliteration models do not account for phenomena that could have an impact in downstream performance, such as vowelization for abjad scripts like arabic. as we aim to expand the scale of this study, a future direction will involve training transliteration models between most scripts of the world. this will allow more extensive experimentation, both by incorporating more language pairs and by allowing more control experiments across various scripts. we will also further explore the usage of more advanced g2p systems, such as those developed for the sigmorphon 2020 shared task on grapheme-to-phoneme conversion, or the models of @xcite .