Modeling Word Formation in E nglish– G erman Neural Machine Translation (2020)

Abstract: This paper studies strategies to model word formation in NMT using rich linguistic information, namely a word segmentation approach that goes beyond splitting into substrings by considering fusional morphology. Our linguistically sound segmentation is combined with a method for target-side inflection to accommodate modeling word formation. The best system variants employ source-side morphological analysis and model complex target-side words, improving over a standard system.

## introduction a major problem in word-level approaches to mt is a lack of morphological generalization. both inflectional variants of the same lemma and derivations of a shared word stem are treated as unrelated. for morphologically complex languages with a large vocabulary, this is problematic, especially in lowresource or domain-adaption scenarios. a simple and widely used approach to reduce a large vocabulary in nmt is byte pair encoding (bpe) @xcite , which iteratively merges the top-frequent bigrams from initially character-level split words until a set vocabulary size is reached. this strategy is effective, but linguistically uninformed and often leads to sub-optimal segmentation. also, by only segmenting words into substrings, bpe cannot handle non-concatenative operations, for example: in this paper, we apply word segmentation on both the source and target sides that goes beyond merely splitting into exact substrings. this overcomes the issues caused by fusional morphology by accomodating modeling word formation across languages. productive word formation can lead to a high number of infrequent word forms even though the morphemes in these words are frequent. a linguistically motivated segmentation method to handle processes such as compounding and derivation allows for better coverage and generalization, both on the word level and on the morpheme level, and also enables the generation of new words. sound morphological processing on the source and target side aims at learning productive word formation processes during translation, such as the english-german translation pair ungovernability↔unregierbarkeit: un|pref govern|v able|suff-adj ity|suff-noun ↔ un|pref regieren|v bar|suff-adj keit|suff-noun morphological information should not only handle isomorphic translation equivalents as above, but also help to uncover relations between source and target side for structurally different translations. ## related work there is a growing interest in the integration of linguistic information in nmt. for example, @xcite and @xcite demonstrate the positive impact of source-side syntactic information; @xcite report improved translation quality when using syntactic information in form of ccg tags on source and target side. to address data sparsity, compound modeling has already been proved useful for phrase-based mt, e.g., @xcite who model source-side compounds, and @xcite who generate compounds on the target side. for nmt, @xcite apply compound and suffix segmentation using a stemmer. @xcite reduce complex source-side vocabulary by means of an unsupervised morphology learning algorithm. ataman and federico (2018) forego a traditional morphological analysis of the source language, and instead compose word representations from character trigrams. however, these three papers only apply segmentation on the string level and cannot properly handle fusional morphology. addressing morphology in nmt, @xcite combine bpe with a morphological analyzer to "guide" the segmentation of surface forms into substrings. their approach does not result in morphemes, for example @xmath0, which does not match with google, while in our work we match such morphemes. @xcite present an nmt system to generate inflected forms on the target side, with a focus on overcoming data-sparsity caused by inflection. their work contains a simple experiment on compound splitting with promising initial results that encouraged us to systematically explore word formation, including compounding, in nmt. our contribution is a segmentation strategy that includes modeling non-concatenative processes, by implementing an english morphological analyzer suitable for this task, and by exploiting an existing tool for german, in order to obtain a consistent morphological sub-word representation. ## modeling target-side morphology our strategy to model word formation operates on lemma level as this allows for a better generalization than using surface forms. to model target-side inflection, we follow the simple lemma-tag generation approach by @xcite , but we improved the lemma representation to better support modeling word formation, and also implement a novel source-side morphological representation. lemma-tag generation (existing strategy): in a pre-processing step, all inflected forms of the target-side training data are replaced by pairs of the lemma and its rich morphological tag. in a postprocessing step, the system's output is re-inflected by generating inflected forms using the morphological tool smor @xcite . old @xmath1 ## simple english morphological analysis we implemented a simple morphological analyzer that is generally based on @xcite , in that a word is segmented into strings that are already observed in the training data. our method additionally relies on tag information (similar to the compound splitter of weller-di marco (2017)), and on a hand-crafted set of prefixes and suffixes in combination with rules such as @xmath3. the segmentation is based on statistics derived from tagged and lemmatized data. this has several advantages: (i) the lemma and tag information restricts the possible operations (e.g., -ion as suffix is only applicable to nouns); (ii) there is no need to handle inflection; (iii) the tag provides a flat morpho-syntactic structure of the segmented word. the analysis first identifies a potential prefix by finding a combination with a prefix in the training data, for example deactivation|@xmath4. the tag restriction at this step is important to maintain the word class of the original word, and to avoid analyses such as decent|@xmath5. the remaining part undergoes splitting into either word+suffix (e.g., activation|@xmath6.g., evildoer|@xmath7. in case of several possibilities, the analysis whose components lead to the highest geometric mean is selected. the splitter in its present form is rather aggressive and tends to oversplit. while it is often assumed that this is not harmful in mt (e.g., koehn and knight ( 2003 )), we have not investigated the impact of oversplitting vs. undersplitting. ## data representation in nmt the morphological analyses provide a straightforward basis for the segmentation experiments. german: the lemma-tag approach (oldlemtag) is contrasted to the system variant with new lemma selection (newlemtag). for the segmentation experiments (newlemtagsplit), we apply compound splitting, such as @xmath8. also, nominalization, e.g., @xmath9, is segmented, but different adjective suffixes (such as -lich) are kept attached. generally, we found that variation of the splitting granularity of adjective suffixes does not have a large impact. english: we first look at a representation where lemma-tag pairs replace surface forms (lemtag). to evaluate the effect of morphological information, we compare the three settings in in morph-markup-split, words are split following the analysis, with tags indicating word-internal structure. morph-nomarkup-split is the same, but without word-internal tags. the annotation of prefixes/suffixes (-@xmath10. in addition to explicit splitting, we consider a variant where lemmas are replaced by the unsplit morphological analysis (morph-nomarkup-nosplit), and all segmentation is done by bpe, which can now access actual words (enthusiasm instead of *enthusias) that already occur in the training data. this representation is conceptionally similar to the german lemma-tag representation. ## experiments we compare four training settings: small (248,730 sentences: news-commentary), large2m (1,956,444 sentences: europarl + news-commentary), large4m (4,116,215 sentences: europarl + news-commentary + commoncrawl) and medium (1m sentences) where the medium corpus consists of the newscommentary corpus and the first ∼750k sentences of europarl. we use wmt'15 as dev-set (2169 sentences) and wmt'16 as test-set (2999 sentences). the lemma-tag approach doubles the sentence length by inserting tags. to avoid overly long sentences, the training data was first filtered to sentences of length 50, and after that, sentences more than 60 words long after bpe splitting were removed (e.g., sentences containing mostly foreign language words split nearly at character level). ## conclusion we showed that morphologically sound segmentation that considers non-concatenative processes in order to obtain a consistent representation of subwords improves translation. the findings of our experiments provide important insights for translating morphologically rich languages, and are particularly important for low-resource settings.