How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology? (2021)

Abstract: Data-driven subword segmentation has become the default strategy for open-vocabulary machine translation and other NLP tasks, but may not be sufficiently generic for optimal learning of non-concatenative morphology. We design a test suite to evaluate segmentation strategies on different types of morphological phenomena in a controlled, semi-synthetic setting. In our experiments, we compare how well machine translation models trained on subword- and character-level can translate these morphological phenomena. We find that learning to analyse and generate morphologically complex surface representations is still challenging, especially for non-concatenative morphological phenomena like reduplication or vowel harmony and for rare word stems. Based on our results, we recommend that novel text representation strategies be tested on a range of typologically diverse languages to minimise the risk of adopting a strategy that inadvertently disadvantages certain languages.

## introduction data-driven subword-level segmentation of text @xcite is a well-known and widely used text representation strategy in the natural language processing (nlp) community. while subword segmentation largely solves the open vocabulary problem, previous research has shown that models often break down in outof-domain contexts @xcite , when encountering spelling errors @xcite , when translating morphologically-rich languages @xcite and in multilingual scenarios @xcite . the reason for this is that even slight deviations from the text seen when learning a segmentation model can result in entirely different segmentations and often aggressively over-segmented text. given the rich morphological diversity across natural languages, it is especially interesting to investigate the suitability of subword segmentation to represent different morphological phenomena. for example, reduplication is a non-concatenative morphological phenomenon foot_1 that is common across the world's languages, but is marginal in higherresource european languages, foot_2 which raises the question if the dominant text representation strategies inadvertently disadvantage nlp systems for languages that feature it. many types of morphological phenomena (see examples in to address this issue, we design a test suite that can be used to evaluate how well a range of morphological phenomena can be learned with sentencelevel sequence-to-sequence models. we focus on compounding circumfixation infixation vowel harmony reduplication (german) (chickasaw) (bontoc) (turkish) (itza') schild / kröte lakna fikas üzüldünüz tz'eek 'shield' / 'toad' 'it is yellow' 'strong' üz-ül-dü-nüz 'few' schildkröte iklakno fumikas sadden-pass-past-2pl tz'eek-tz'eek 'turtle' the task of neural machine translation (nmt) and evaluate in a semi-synthetic @xmath0, allowing for an automatic evaluation that controls for various confounding factors. in our experiments, we test how well current segmentation strategies on subword-and character-level can learn to translate compounds, circumfixed words, infixed words, vowel harmony and reduplicated words. our contributions are the following: ## related work isolated morphological analysis and reinflection have long been of interest to the nlp community, with yearly shared tasks @xcite ) that result in dedicated architectures that perform well for many languages @xcite @xcite . however, despite the large morphological diversity of natural languages, many approaches for sentence-level sequence-to-sequence tasks are often only tested on a subset of (morphologically similar) languages and then adopted without much questioning @xcite . one such example is subword-level representation of text @xcite which has contributed greatly to the success of deep learning in various nlp tasks and has become a necessary preprocessing step to train state-of-theart models @xcite . due to its data-dependent nature, subword segmentation algorithms often produce subword splits that do not adhere to morpheme boundaries which can limit the generalisation to rare or unseen words and can lead to performance loss. furthermore, it is unclear if models trained with subword segmentation can learn to generalise to non-concatenative morphological phenomena such as reduplication or vowel harmony, even in high-resource settings. previous work that evaluated how well morphology or morpho-syntax is captured by sequence-tosequence models either used contrastive test sets to evaluate whether models assign a higher probability to sentences e.g. with correct subject-verb agreement @xcite @xcite or probing classifiers to evaluate how well morphological features such as case, number or gender can be predicted from the models' hidden representations @xcite @xcite @xcite . our work is similar to @xcite who also evaluate morphological competence based on the output of machine translation models rather than probabilities or hidden states. however, instead of morphological features, we are interested in evaluating how well different types of morphological phenomena can be learned by sequence-tosequence models, especially with a focus on their textual representation. closely related is work by @xcite who compare language model perplexities for different segmentation strategies on morphologically diverse languages and by @xcite who show that multilingual bert @xcite subwords do not reflect the morphological structure of a nonconcatenative language like hebrew well. our setup with synthetic morphological phenomena is similar to work by @xcite who generate synthetic treebanks by reordering nodes in existing treebanks for various natural languages. instead of simply reordering components, we need to apply a more complex preprocessing to generate synthetic morphological phenomena that fit the natural context. we discuss this preprocessing in more detail in section 5.2. ## morphological phenomena we choose five morphological phenomena which we believe may be hard to learn with subword segmentation strategies. we show a natural language example for each morphological phenomenon in compounding: a compound is a word composed of more than one free morpheme. compounding can affect the subword segmentation of the individual components which can make it harder to translate compounds even if the individual parts are seen regularly in the training data. circumfixation: a circumfix is an affix that consists of two parts, one added at the start of a word stem, the other at the end. with circumfixation, it is not guaranteed that the subword segmentation adheres to the morpheme boundaries and that the base is segmented in the same way as without any affixation. it may be difficult to learn the correct form for rare or unseen circumfixed words. infixation: an infix is an affix inserted inside a word stem. a word with an infix cannot be segmented in the same way as without infixation and it is not guaranteed that the segmentation splits the infix into a separate token. infixation may also be hard to learn for rare or unseen cases. vowel harmony: vowel harmony is a type of assimilation in which the vowels in a morpheme (e.g. an affix) are assimilated to vowels in another morpheme (e.g. the word stem). vowel harmony is a non-concatenative morphological process and it is unclear whether an nmt model trained with subword segmentation can learn to generate the correct vowels for rare or unseen words. reduplication: reduplication is another nonconcatenative morphological process in which the whole word (full reduplication) or a part of a word (partial reduplication) is repeated exactly or with a slight change. in some cases, the repetition can also occur twice (triplication). reduplication often marks features such as plurality, intensity or size, depending on the language and raises the same generalisation question as vowel harmony. ## segmentation test suite we identify four key requirements for our test suite and address them as follows: the morphological phenomena should also be evaluated in isolation, i.e. not on the level of bleu. to achieve this, we do not use naturally occurring morphemes to create our synthetic morphological phenomena. rather, we generate artificial morphemes that do not occur in our training data otherwise and are distinct between source and target. 4 in this way, there are also no cognates among the artificial morphemes we evaluate. ## conclusion we develop a test suite to evaluate how well various types of morphological phenomena can be translated in nmt. we show that the choice of segmentation strategy can have a considerable influence on the performance, especially for nonconcatenative phenomena such as reduplication and vowel harmony. our results with current segmentation strategies show a) that there is potential for more work on text representation strategies, b) that abstract representations may be a helpful source of information, especially for languages with non-concatenative morphology (if reliable tools for morphological analysis and generation are available) and c) that bpe-dropout should be adopted in state-of-the-art models since it improves learning non-concatenative morphology. based on our results, we recommend that novel approaches in nlp always be tested on a range of typologically diverse languages that cover different types of morphological phenomena. in the future, we are interested in evaluating a wider variety of text representation strategies, including tokenisation-free input such as canine @xcite or visual text representations @xcite , although these are limited to the source side. we would also like to investigate the effects of out-of-domain contexts where we expect more rare word stems. elizabeth salesky, david etter, and matt post. 2021. robust open-vocabulary translation from visual text representations. rico sennrich. 2017. how grammatical is characterlevel neural machine translation? assessing mt quality with contrastive translation pairs. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pages 376-382, valencia, spain. association for computational linguistics. rico sennrich, orhan firat, kyunghyun cho, alexandra birch, barry haddow, julian hitschler, marcin junczys-dowmunt, samuel läubli, antonio valerio miceli barone, jozef mokry, and maria nȃdejde. 2017. nematus: a toolkit for neural machine translation. in proceedings of the software demonstrations of the 15th conference of the european chapter of the association for computational linguistics, pages 65-68, valencia, spain. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. nitish srivastava, geoffrey hinton, alex krizhevsky, ilya sutskever, and ruslan salakhutdinov. 2014. dropout: a simple way to prevent neural networks from overfitting. journal of machine learning research, 15(56):1929-1958. aleš tamchyna, marion weller-di marco, and alexander fraser. 2017. modeling target-side inflection in neural machine translation. in proceedings of the second conference on machine translation, pages 32-42, copenhagen, denmark. association for computational linguistics. clara vania and adam lopez. 2017. from characters to words to in between: do we capture morphology? in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2016-2027, vancouver, canada. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. ekaterina vylomova, trevor cohn, xuanli he, and gholamreza haffari. 2017. word representation models for morphologically rich languages in neural machine translation. in proceedings of the first workshop on subword and character level models in nlp, pages 103-108, copenhagen, denmark. association for computational linguistics.