Multi-Task Neural Model for Agglutinative Language Translation (2020)

Abstract: Neural machine translation (NMT) has achieved impressive performance recently by using large-scale parallel corpora. However, it struggles in the low-resource and morphologically-rich scenarios of agglutinative language translation task. Inspired by the finding that monolingual data can greatly improve the NMT performance, we propose a multi-task neural model that jointly learns to perform bi-directional translation and agglutinative language stemming. Our approach employs the shared encoder and decoder to train a single model without changing the standard NMT architecture but instead adding a token before each source-side sentence to specify the desired target outputs of the two different tasks. Experimental results on Turkish-English and Uyghur-Chinese show that our proposed approach can significantly improve the translation performance on agglutinative languages by using a small amount of monolingual data.

## introduction neural machine translation (nmt) has achieved impressive performance on many high-resource machine translation tasks @xcite @xcite . the standard nmt model uses the encoder to map the source sentence to a continuous representation vector, and then it feeds the resulting vector to the decoder to produce the target sentence. however, the nmt model still suffers from the low-resource and morphologically-rich scenarios of agglutinative language translation tasks, such as turkish-english and uyghur-chinese. both turkish and uyghur are agglutinative languages with complex morphology. the morpheme structure of the word can be denoted as: prefix1 + … + prefixn + stem + suffix1 + … + suffixn @xcite . since the suffixes have many inflected and morphological variants, the vocabulary size of an agglutinative language is considerable even in small-scale training data. moreover, many words have different morphemes and meanings in different context, which leads to inaccurate translation results. recently, researchers show their great interest in utilizing monolingual data to further improve the nmt model performance @xcite @xcite . @xcite pair the target-side monolingual data with automatic back-translation as additional training data to train the nmt model. @xcite use the source-side monolingual data and employ the multi-task learning framework for translation and source sentence reordering. @xcite modify the decoder to enable multi-task learning for translation and language modeling. however, the above works mainly focus on boosting the translation fluency, and lack the consideration of morphological and linguistic knowledge. stemming is a morphological analysis method, which is widely used for information retrieval tasks @xcite . by removing the suffixes in the word, stemming allows the variants of the same word to share representations and reduces data sparseness. we consider that stemming can lead to better generalization on agglutinative languages, which helps nmt to capture the in-depth semantic information. thus we use stemming as an auxiliary task for agglutinative language translation. in this paper, we investigate a method to exploit the monolingual data of the agglutinative language to enhance the representation ability of the encoder. this is achieved by training a multi-task neural model to jointly perform bi-directional translation and agglutinative language stemming, which utilizes the shared encoder and decoder. we treat stemming as a sequence generation task. ## related work multi-task learning (mtl) aims to improve the generalization performance of a main task by using the other related tasks, which has been successfully applied to various research fields ranging from language @xcite , vision @xcite , and speech @xcite . many natural language processing (nlp) tasks have been chosen as auxiliary task to deal with the increasingly complex tasks. @xcite employ a small amount of data of syntactic parsing and image caption for english-german translation. @xcite present a joint mtl model to handle the tasks of part-of-speech (pos) tagging, dependency parsing, semantic relatedness, and textual entailment for english. @xcite utilize the pos tagging and dependency parsing for english-german machine translation. to the best of our knowledge, we are the first to incorporate stemming task into mtl framework to further improve the translation performance on agglutinative languages. recently, several works have combined the mtl method with sequence-to-sequence nmt model for machine translation tasks. @xcite follow a one-to-many setting that utilizes a shared encoder for all the source languages with respective attention mechanisms and multiple decoders for the different target languages. @xcite follow a many-to-many setting that uses multiple encoders and decoders with two separate unsupervised objective functions. @xcite follow a many-to-one setting that employs multiple encoders for all the source languages and one decoder for the desired target language. @xcite propose a more simple method in one-to-one setting, which trains a single nmt model with the shared encoder and decoder in order to enable multilingual translation. the method requires no changes to the standard nmt architecture but instead requires adding a token at the beginning of each source sentence to specify the desired target sentence. inspired by their work, we employ the standard nmt model with one encoder and one decoder for parameter sharing and model generalization. in addition, we build a joint vocabulary on the concatenation of the source-side and target-side words. several works on morphologically-rich nmt have focused on using morphological analysis to pre-process the training data @xcite @xcite . @xcite segment each turkish sentence into a sequence of morpheme units and remove any nonsurface morphemes for turkish-english translation. @xcite propose a vocabulary reduction method that considers the morphological properties of the agglutinative language, which is based on the unsupervised morphology learning. this work takes inspiration from our previously proposed segmentation method @xcite that segments the word into a sequence of subword units with morpheme structure, which can effectively reduce language complexity. ## results and discussion ## conclusions in this paper, we propose a multi-task neural model for translation task from and into a low-resource and morphologically-rich agglutinative language. the model jointly learns to perform bi-directional translation and agglutinative language stemming by utilizing the shared encoder and decoder under standard nmt framework. extensive experimental results show that the proposed model is beneficial for the agglutinative language machine translation, and only a small amount of the agglutinative data can improve the translation performance in both directions. moreover, the proposed approach with external monolingual data is more useful for translating into the agglutinative language, which achieves an improvement of +1.42 bleu points for translation from english into turkish and +1.45 bleu points from chinese into uyghur. in future work, we plan to utilize other word segmentation methods for model training. we also plan to combine the proposed multi-task neural model with back-translation method to enhance the ability of the nmt model on target-side language modeling.