Low-resource neural machine translation with morphological modeling (2024)

Abstract: Morphological modeling in neural machine translation (NMT) is a promising approach to achieving open-vocabulary machine translation for morphologically-rich languages. However, existing methods such as sub-word tokenization and character-based models are limited to the surface forms of the words. In this work, we propose a framework-solution for modeling complex morphology in low-resource settings. A two-tier transformer architecture is chosen to encode morphological information at the inputs. At the target-side output, a multi-task multi-label training scheme coupled with a beam search-based decoder are found to improve machine translation performance. An attention augmentation scheme to the transformer model is proposed in a generic form to allow integration of pre-trained language models and also facilitate modeling of word order relationships between the source and target languages. Several data augmentation techniques are evaluated and shown to increase translation performance in low-resource settings. We evaluate our proposed solution on Kinyarwanda \leftrightarrow English translation using public-domain parallel text. Our final models achieve competitive performance in relation to large multi-lingual models. We hope that our results will motivate more use of explicit morphological information and the proposed model and data augmentations in low-resource NMT.

## introduction neural machine translation (nmt) has become a predominant approach in developing machine translation systems. two important innovations in recent state-of-the-art nmt systems are the use of the transformer architecture @xcite and sub-word tokenization methods such as byte-pair encoding (bpe) @xcite . however, for morphologically-rich languages(mrls), bpe-based tokenization is only limited to the surface forms of the words and less grounded on exact lexical units (i.e. morphemes), especially in the presence morphographemic alternations @xcite and nonconcatenative morphology @xcite . in this work, we tackle the challenge of modeling complex morphology in low-resource nmt and evaluate on kinyarwanda, a low-resource and morphologically-rich language spoken by more than 15 million people in eastern and central africa foot_0 . to model the complex morphology of mrls in machine translation, one has to consider both source-side modeling (i.e. morphological encoding) and target-side generation of inflected forms (i.e. morphological prediction). we explicitly use the morphological structure of the words and the associated morphemes, which form the basic lexical units. for source-side encoding, morphemes are first produced by a morphological analyzer before being passed to the source encoder through an embedding mechanism. on the target side, the morphological structure must be predicted along with morphemes, which are then consumed by an inflected form synthesizer to produce surface forms. therefore, this approach enables open vocabulary machine translation since morphemes can be meaningfully combined to form new inflected forms not seen during training. previous research has shown that certain adaptations to nmt models, such as the integration of pre-trained language models @xcite , can improve machine translation performance. we explore this idea to improve low-resource machine translation between kinyarwanda and english. our model augmentation focuses on biasing the attention computation in the transformer model. beside augmentation from pre-trained language model integration, we also devise an augmentation based solely on the word order relationship between source and target languages. these model augmentations bring substantial improvement in translation performance when parallel text is scarce. one of the main challenges facing machine translation for low-resource languages obviously is parallel data scarcity. when the training data has limited lexical coverage, the nmt model may tend to hallucinate @xcite . additionally, for a morphology-aware translation model, there is a problem of misaligned vocabularies between source and target languages. this makes it harder for the model to learn to copy unknown words and other tokens that need to be copied without translation such as proper names. to address these challenges, we take a data-centric approach by developing tools to extract more parallel data from public-domain documents and websites. we also use various data augmentation techniques to increase lexical coverage and improve token copying ability where necessary. by combining these data-centric approaches with our morphology-aware nmt model, we achieve competitive translation performance in relation to larger multi-lingual nmt models. to have a comprehensive evaluation, we evaluate our models on three different benchmarks covering different domains, namely wikipedia, news and covid-19. in short, our contribution in this work can be summarized as follow: ## methods machine translation (mt) can be considered as the task of accurately mapping a sequence of tokens (e.g. phrase, sentence, paragraph) in the source language @xmath0, s 2 , ...s n ) to a sequence of tokens in the target language @xmath1, t 2 , ...t m ) with the same meaning. the learning problem is then to estimate a conditional probability model that produces the optimal translation t * , that is: in this section, we describe our model architecture as an extension of the basic transformer architecture @xcite to enable morphological modeling and attention augmentation. we also describe our data-centric approaches to dataset development and augmentation in the context of the low-resource kinyarwanda↔english machine translation. ## related work morphological modeling in nmt is an actively researched subject often leading to improvements in translation. however most of this research has focused on european languages. @xcite shows that an rnn-based word compositional model improves nmt on several lan-guages. weller-di marco and fraser (2020) evaluates both source-side and target-side morphology modeling between english and german using a lemma+tag representation. @xcite proposes using multi-task learning of target-side morphology with a weighted average loss function. however, @xcite does not find improvement when using unsupervised morphological analysers. our studies differs in that it uses a different morphological representation, that is the two-tier architecture, and we also evaluate on a relatively lower resourced language. the idea of model augmentation with pre-trained language models (plm) have been previously explored by @xcite , and @xcite who use a drop-net scheme to integrate bert embeddings. also, there have been attempts to model word order relationships between source and target languages @xcite . our model architecture provides a more generic approach through the attention augmentation. ## conclusion this work combines three techniques of morphological modeling, attention augmentation and data augmentation to improve machine translation performance for a low-resource morphologically-rich language. our ablation results indicate improvement from each individual contribution. baseline improvements from morphological modeling are more pronounced at the target side than at the source side. this work expands the landscape of modeling complex morphology in nmt and provides a potential framework-solution for machine translation of low-resource morphologically rich languages.