Automatic Construction of Morphologically Motivated Translation Models for Highly Inflected, Low-Resource Languages (2016)

Abstract: Statistical Machine Translation (SMT) of highly inflected, low-resource languages suffers from the problem of low bitext availability, which is exacerbated by large inflectional paradigms. When translating into English, rich source inflections have a high chance of being poorly estimated or out-of-vocabulary (OOV). We present a source language-agnostic system for automatically constructing phrase pairs from foreign-language inflections and their morphological analyses using manually constructed datasets, including Wiktionary. We then demonstrate the utility of these phrase tables in improving translation into English from Finnish, Czech, and Turkish in simulated low-resource settings, finding substantial gains in translation quality. We report up to +2.58 BLEU in a simulated low-resource setting and +1.65 BLEU in a moderateresource setting. We release our morphologically-motivated translation models, with tens of thousands of inflections in each of 8 languages.

## introduction statistical machine translation systems are typically trained on large bilingual parallel corpora (bitext). low-resource machine translation focuses on translation of languages for which there exists little bitext, and where translation quality is subsequently often poor. highly inflected languages-those that exhibit large inflectional paradigms of words with a common dictionary entry-excacerbate the problems of a low-resource setting. many inflections of words in an inflectional paradigm are complex and rare, and their translations are unlikely to be wellestimated even in a moderately large parallel corpus. for example, @xcite point to the highly inflected nature of finnish as a reason for poor translation performance into english even in high-resource settings. however, even where bitext may be lacking or scarce, there are often many other resources available. one source of rich morphological information is wiktionary. 1 this paper describes a method for using resources extracted from wiktionary to automatically map inflections in paradigms of morphologically rich languages to ranked sets of english phrasal translations. this is done by the following procedure: 1. we begin with resources from the unimorph project @xcite , which produced millions of tuples pairing inflected words forms with their lemmas and a rich morphological tag (which we refer to as a unimorph tag or vector) that was designed to be a universal representation of morphological features ( §3). a depiction of the full pipeline is in figure 1 . ## prior work maximizing the utility of a baseline phrase table has been the focus of a large body of prior work in translating from morphologically rich languages. @xcite work on the oov problem in arabic, mapping oov types to in-vocabulary (inv) types by orthographic and morphological smoothing methods. @xcite take inspiration from the textual entailment (te) problem, using wordnet to determine a set of entailed alternatives for english oov tokens. however, since this oov-resolution scheme is dependent on the existence of a semantic resource like wordnet in the source language, it is unsuitable in general low-resource settings. @xcite implement a backoff model for finnish and german, stemming and splitting oov tokens at test time and searching a baseline phrase table for the resulting simplified forms. many systems attempt to address the incorrect independence assumptions traditional phrase-based mt systems impose on inflections in the same paradigm. @xcite train a baseline phrase-based translation model, and back off to a factored model that decomposes oov tokens into lemma+morphology. @xcite address the intuition that even inv tokens may have poorly estimated translations. they decode on a confusion network of tranformations of source sentences, imposing penalties for backing off from surface tokens. each of these approaches attempts to use pre-or post-processing to make up for poorly estimated, low-coverage phrase tables. we take advantage of an entirely new resource, a very large, massively multilingual, morphologically-annotated dictionary, to directly improve phrase table coverage and provide improved estimations for inv types. thus, the practical translation gains we see through our methods should be orthogonal to those of prior work. this paper presents a method of constructing english phrases that express inflectional features, and a novel system of mapping these phrases to foreign inflections, with the following qualities: ## unimorph inflectional paradigms the universal morphological feature schema (unimorph) represents fine-grained distinctions in meaning expressed through inflectional morphology cross-lingually @xcite . the schema defines 23 distinct dimensions across which inflections can vary independently. though english does not express many of these features through morphology, their purposes can still be intuitive: 1 , is a list of inflections with corresponding lemmata, and inflectional values as a unimorph vector. each language has a corresponding table of inflection entries with their corresponding unimorph vector. the size of the wiktionary dataset varies from language to language, including 2,990,482 finnish inflections and 15,132 swahili inflections. to a large extent, combining a lemma with the information in the unimorph vector reconstructs the meaning of the inflection. we do not claim perfect reconstruction, as no inflectional morphological schema can perfectly encode language-independent meaning. however, practical gains in translation quality do not require a perfect schema. we instead focus on the substantial signal provided by unimorph annotations, and show that they are highly effective at providing cross-linguistic information. ## english inflectional elicitation wiktionary provides us with unimorph vectors for inflections, but provides no information about how to express these vectors in english. these expressions are difficult to generate, as english expresses inflectional information phrasally. for this, we use a corpus of 8,000 unimorph-annotated english sentences, which we call the elicitation corpus. the corpus was developed in an attempt to document the full inflectional variation in 49 languages of morphological interest. the process worked as follows. for each language, we first collected all its important morphological tags. we then hand-built the cartesian product of all inflectional variation expressed by the language. thus, if a language inflected nouns exactly for 4 values of number and 3 values of case, we constructed 12 vectors of (number × case). then, for each unimorph tag, keeping the specific language and part of speech in mind, we manually wrote an english sentence with a head word on which the same inflectional features are expressed. for example, for the tag vb, 3, prs, prf, pass (verb, 3rd person, present, perfect, passive) we might generate the sentence [the apple] has been eaten. these sentences were given to a bilingual native speaker. the goal of each sentence was to elicit, in the foreign language, the inflection encoded by the lemma we used and the unimorph vector expressed in english. we wanted to avoid sentential translations, aiming instead for individual inflections. to this end, portions of each sentence necessary for syntactic coherence but deemed unnecessary to express inflectional values were enclosed in brackets. each line in the corpus is a tuple of (english sentence, unimorph vector). by construction, the corpus spans a large amount of the variance of inflectional expression of most languages. further, equivalent unimorph vectors in multiple languages were not english sentence unimorph vector [they] were eating [when the door broke.] ind, pst, prog, 3, pl [the view was blocked] by the houses ins, pl was [he] not speaking? pst, prog, neg, int, 3, sg [he is] sleeping prs, prog, pos, 3, sg in this section, we detail the pipeline by which we automatically construct a translation model or phrase table for each language represented in the wiktionary dataset. ## experimental design we evaluate our system by examining its effectiveness in improving the quality of end-to-end mt from czech, finnish, russian and turkish into english. we use the joshua decoder @xcite with hiero grammars @xcite . for many of our target languages, the only bitext available is the bible. we thus simulate a low-resource setting by training on the bible. we simulate moderately higher-resource settings by appending differing numbers of lines of modern bitext (described below) to the bible. we test on newswire from the workshop on statistical machine translation @xcite . for all translation models, we use a gigaword-trained 5-gram language model (lm). we anticipated that the english phrases in our tables might be discounted by the language model due to higher-order (3-and 4-gram) misses in the gigaword corpus. in preliminary experiments, we trained language models with lmplz @xcite on the training data with our phrases appended. however, we saw best performance when using the gigaword lm, and by using it across all tms, the bleu scores are kept comparable. ## results and analysis figure 3 illustrates that our system is able to correctly generate the english expression of complex inflectional information, both intelligently filling in oovs and providing better translations for poorly-estimated inflections. wiktionary dataset. we also note that the potential benefit of our system is constrained by the quality and size of the wiktionary dataset for a given language. ## summary and future work translation of highly inflected languages presents compounding data scarcity problems for smt systems with little bitext. the information encoded in the inflections of highly inflected languages is formalized in unimorph, and a large, multilingual, freely available repository of unimorph-annotated inflections exists in the wiktionary dataset. using a small unimorphannotated english corpus, we generalize english inflectional phrase templates to express a wide range of unimorph vectors. we then use the inflectional information to assign english phrase templates as translation candidates to inflections from wiktionary. building translation models from these pairs, we substantially improve the quality of mt in a range of low-resource settings. analyzing a range of resource settings and levels of morphological information, we find that a full morphological system outperforms inflection-lemma mappings and lemma-lemma mappings. we also find that morphological information is less useful in higher-resource settings, but can still provide substantial gains. we believe this approach holds promise for constructing translation systems for language pairs that do not have much in the way of bitext. in service of this goal, we release translation models constructed for finnish, czech, russian, korean, georgian, swahili, turkish, and urdu. for future work, we believe it would be useful to investigate how well these phrase-table augmentation techniques work when combined with other approaches to low-resource machine translation into english, such as lemmatized backoff models.