Improving Low-Resource NMT through Relevance Based Linguistic Features Incorporation (2020)

Abstract: In this study, linguistic knowledge at different levels are incorporated into the neural machine translation (NMT) framework to improve translation quality for language pairs with extremely limited data. Integrating manually designed or automatically extracted features into the NMT framework is known to be beneficial. However, this study emphasizes that the relevance of the features is crucial to the performance. Specifically, we propose two methods, 1) self relevance and 2) word-based relevance, to improve the representation of features for NMT. Experiments are conducted on translation tasks from English to eight Asian languages, with no more than twenty thousand sentences for training. The proposed methods improve translation quality for all tasks by up to 3.09 BLEU points. Discussions with visualization provide the explainability of the proposed methods where we show that the relevance methods provide weights to features thereby enhancing their impact on low-resource machine translation.

## introduction neural machine translation (nmt) @xcite @xcite is known to give state-of-the-art translation quality for language pairs having abundance of parallel corpora. in case of resource poor scenarios, additional translation knowledge is acquired either through transfer learning in the form of pre-trained model parameters or by supplying external monolingual corpora. however, exploiting linguistic information effectively in low-resource conditions is still an under-researched field. annotating the source side with various syntactic features e.g. part-of-speech (pos), lemma, dependency labels etc. can help in accurate translation when either a surface word is polysemous i.e. having multiple senses on varying context or the same word form is shared by different root words due to the inflectional nature of the language. hence, for morphologically rich languages, ideally, the use of language specific knowledge should improve the translation quality. to the best of our knowledge, there are not an adequate amount of research works on effectively incorporating arbitrary syntactic information into nmt. one possible reason could be that in high resource scenario the network learns from the large amount of training data to handle the problem caused by polysemy and morphological variants. in this direction, the notable works are done by @xcite @xcite . @xcite incorporated several features at the source side by employing a separate embedding matrix for each component of a source token including the word and its associated features. finally, all embeddings are concatenated to enrich the representation. inspired by this work, @xcite developed a method to process feature sequences of the source sentence by separate recurrent neural networks (rnns) and combined the output of all rnns using a hybrid global-local attention strategy. @xcite proposed a complex rnn architecture to model source-side linguistic knowledge. their approach, at the first level, passes the morphological properties of each word sequentially through an rnn to build a composite representation of the features. these representations are further encoded by an rnn at the sentence level. very recently, @xcite came up with a dual-source transformer model to process words and features in isolation. the output of the word transformer and the feature transformer are fed to the decoder by two encoder-decoder attention sub-layers put in a series. all the models discussed so far ignore to consider whether a particular feature of a given word is really relevant for the translation task. the above point motivates us to conduct the present research. we hypothesize that only including the features alongside the word by using a generalized embedding layer or forming a composite representation by taking all features together does not exploit the features completely. there should be some mechanism which can justify the relationship between a word and its supporting features as well. driven by this idea, we come up with two simple strategies which measure the relevance of the word and the feature embeddings obtained from the output of the embedding layers. the first one is self relevance which considers the relevance of a feature with respect to itself. we apply an attention function to the feature embedding, which in turn generates a mask determining the importance of that feature. finally, the mask is applied on the feature to effectuate the attention. our second approach considers the feature relevance with respect to the corresponding word which is the most vital component of a source token. in this case, the attention function operates on a word-feature pair and returns the mask determining the word-based relevance of the input feature. for experimentation, we choose the transformer network of @xcite and assess our proposed techniques on eight low-resource language pairs having diverse morphological variations taken from the asian language treebank @xcite . our hypothesis is empirically validated showing the fruitfulness of the relevance checking mechanisms in low-resource scenario. we achieve up to 3.09 bleu points gain over the standard baseline models of @xcite and @xcite . in the next section, the related works are briefly described. ## related works incorporating morphological information for nmt is a challenging area of research. a significant number of works involve dependency structure at the source side @xcite @xcite @xcite @xcite . @xcite proposed a syntax-aware encoding mechanism that encodes the source sentence maintaining the hierarchy of its dependency tree. a long-short-term memory (lstm) network @xcite is used to encode the constituent phrases recursively in bottom-up direction. on the contrary, @xcite claimed that an rnn encoder can capture the inherent syntactic properties automatically from a source sentence as a by-product of training. they used a multi-layer lstm and found that its different layers represent different types of syntax. this work gives the intuition that using linguistic prior at the source side may be redundant for translation. following @xcite , @xcite presented a bidirectional tree encoder which builds the sentence representation considering both top-down and bottom-up directions of the dependency tree. a different approach was taken by @xcite to propose a syntactic graph-convolutional network for encoding source sentences. @xcite came up with a model that learns the latent graph structure of a source sentence optimized by the translation objective. in the study by @xcite , the parse tree of the source sentence are linearized to label sequence. next, three different encoding strategies using rnn namely parallel, hierarchical and mixed, are tried to integrate the dependency information. @xcite used dependency information of both the source and the target languages. as their model needs multiple encoders and decoders, so it is not worthy for use under low-resource condition. in the work of @xcite , the authors employed a supervised encoder-decoder dependency parser and used the outputs from the encoder as a syntax-aware representations of words, which in turn, are concatenated to the input embeddings of the translation model. most recently, @xcite proposed dependency-aware self-attention in the transformer that needs no extra parameter. for a pivot word, its self-attention scores with other words are weighted considering their distances from the dependency parent of the pivot word. apart from the above works, there are some studies which use the factors in the target side @xcite @xcite . in general, their approach is to predict the roots and other morphological tags of the target words instead of producing the surface forms. additionally, a morphological analyzer is applied for the reinflection task. ## nmt architecture we employ the transformer architecture for execution of our experiments. it is a specific type of encoderdecoder neural model that can be applied for sequence modelling tasks. unlike rnn, the working policy of the transformer does not rely on recurrence and hence, is much more parallelizable. as in general encoding-decoding framework, the learned embeddings of the source sentence @xmath0, ..., s m ) are mapped by the encoder to continuous representations @xmath1, ..., z m ). each representation z i contains the contextual information about its surrounding tokens. to keep track of the order of the sequence, positional encodings are added to the input embeddings. the encoder comprises a stack of identical layers each having two sub-layers -multi-head selfattention and position-wise fully connected feed-forward network. for each position in the source sentence, the self-attention block calculates a probability distribution over all positions. this distribution is then used to make a new representation of the reference position. multi-head self-attention repeats the process a number of times resulting multiple representations in different subspaces. finally, all of them are concatenated and the output is passed to the fully connected feed-forward network which contains two linear layers with relu activation function in between. as no recurrence is involved, so the encoding process can be parallelized during both training and inference phases. the decoder is also made of a stack of identical layers having multi-head self-attention and feedforward networks with the addition of an extra computation called multi-head encoder-decoder attention over the output of the encoder stack. here, the self-attention sub-layer is masked so that at a particular position it would not be able to consider the subsequent positions. this must be enforced for preserving the auto-regressive property. all sub-layers in the encoder and the decoder stacks have residual connections followed by layer normalization. ## conclusion in this article we revisit the ways to incorporate linguistic features in nmt. we argue that it is important to check the relevance of the features instead of just plugging them into the model. to establish our claim two novel methods are proposed and evaluated under extremely low-resource condition on eight language pairs. we design word-dependent as well as word-agnostic relevance checking mechanisms and show that by controlling the effects of features we can obtain substantial improvement in translation quality. our models yield significantly higher bleu scores compared to the baselines with a modest increase in the number of model parameters. additionally we observe lower validation perplexity that shows applying feature relevance helps to reduce prediction uncertainty. the methods are further analyzed by visualization of the relevance weights. in case of the word-based relevance, the feature embeddings are tuned similarly based on the semantics of the corresponding word. it indicates that the proposed models actually pay attention to morphological features leading to enriched word representation. moreover, we assess the proposed relevance methods on rnn model with attention. the results are not as satisfactory as obtained for the transformer model, which requires further investigation. one notable issue in the present work is that the source language has been annotated by a robust and highly accurate parser which is unlikely to exist for a low-resource language. checking the effectiveness of relevance methods is necessary when the annotation is noisy. the future extension of the present work will also focus to exploit the features under high resource scenario. in a resource rich setting usually features are redundant as the model learns from a large variety of context. hence, using them effectively in that case will be beneficial for nmt research.