F eature BART : Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT (2022)

Abstract: In this paper we present FeatureBART, a linguistically motivated sequence-to-sequence monolingual pre-training strategy in which syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the span prediction based pre-training framework (BART). These automatically extracted features are incorporated via approaches such as concatenation and relevance mechanisms, among which the latter is known to be better than the former. When used for low-resource NMT as a downstream task, we show that these feature based models give large improvements in bilingual settings and modest ones in multilingual settings over their counterparts that do not use features.

## introduction sequence-to-sequence (s2s) pre-training done via denoising objectives on monolingual corpora is known to improve generation quality in lowresource settings @xcite . this has been extensively explored for neural machine translation, however most works show that the improvements for translation into english are more pronounced than those for translation into a non-english language @xcite . one reason for this is that pre-training leads to a strong decoder that learns to de-noise masked inputs, whereas the knowledge retained in the encoder is rather limited. thus far, there has been no explicit effort towards improving the contribution of the encoder during pre-training. most pre-training methods rely on the power of large models and large corpora, but ignore the possibility of incorporating linguistic knowledge into the pre-trained model. on the other hand, there are several works which show that incorporating linguistic knowledge in the form of lemma, part-of-speech tags and dependency labels, lead to a significant improvement in translation quality @xcite @xcite @xcite in both low-and high-resource settings. most recently, in an extremely low-resource setting, @xcite show the effectiveness of using the aforementioned linguistic features, especially when their influence on the model is controlled via relevance mechanisms that appropriately scale feature embeddings before they augment word embeddings. we hypothesize that incorporating linguistic features, into the denoising based pre-training framework, should improve the quality of pre-training, which should then have a positive impact on the translation quality via finetuning. to this end, we propose featurebart, a feature based sequence-to-sequence pre-training method. in featurebart, linguistic features are obtained via automatic annotators and then converted into embeddings, which are used to augment the word embeddings of the encoder. feature embeddings, are incorporated by either naive concatenation with the word embeddings or by first weighing them with a relevance mechanism and then adding them to word embeddings. the model itself is trained using a monolingual corpus via either the textinfilling or the mask prediction approaches, the former used in bart and the latter used in bert. this featurebart model is then fine-tuned for low-resource language pairs, where linguistic features are also used during fine-tuning. experiments on english to 8 asian languages from the asian language treebank (alt) dataset @xcite , using bilingual as well as multilingual finetuning, show that our feature based pre-training and fine-tuning leads to significant improvements in translation quality indicating the complementary nature of denoising pre-training and features. analyses of training curves show that when compared to non-feature based pre-training, feature based pre-training leads to significantly lower perplexities during the initial stages of fine-tuning. ## methodology we first give some background knowledge about feature based nmt modeling, followed by an explanation of featurebart. ## experiments we describe the datasets for pre-training and finetuning, features used and model training details. datasets: for pre-training of our bart and fea-turebart models, we choose the english monolingual news crawl articles foot_0 of 2007, 2010, and 2013 from wmt-2017 with varying sizes that contain 3.8, 6.8, and 21.7 million sentences respectively. for fine-tuning, we experiment with the multilingual, multi-parallel asian language treebank (alt) @xcite foot_1 for english to asian language translation. following @xcite , eight asian languages -bengali (bg), filipino (fi), hindi (hi), indonesian (id), khmer (khm), malay (ms), myanmar (my) and vietnamese (vi) are set as the targets. so, bilingual experiments cover eight language pairs from en-bg to en-vi. for our multilingual experiments, we explore one-tomany multilingual translation setup keeping the source side fixed to english (en) and the target side to the eight asian languages mentioned above. the source side is fixed to english throughout because as the initial attempt on feature-based pretraining, we can rely on high quality automatic morphological analyzers available for english. given the potential of this work established by empirical results, other languages with moderate quality morphological annotation can also be tried in the future. we use the official train/dev/test splits containing 18088, 1000, and 1018 sentences, respectively. pre-processing: the monolingual corpora for pretraining and the source side of fine-tuning corpora are tokenized and true-cased by moses tokenizer @xcite and the target languages are tokenized to separate the delimiters and the punctuation symbols. following @xcite , each source language sentence in multilingual translation setup is appended with a token @xmath1. byte-pair-encoding (bpe) @xcite is performed to obtain subword vocabularies. we train a single bpe model of vocabulary size 32k on the combined training corpora of all 9 languages and use it during pre-training as well as fine-tuning. features used: morphological annotation of the english datasets is done using stanford corenlp toolkit @xcite . there are three word-level linguistic features -lemma, part-ofspeech (pos), and dependency labels. all subwords of a word take the features of that word. we use subword tags @xcite to denote beginning, middle and ending of a subword unit. hyperparameters and training details: we use variations of the transformer-base model @xcite for our experiments available from opennmt pytorch @xcite , which we modify for feature based experiments. wherever possible, we perform hyperparameter tuning of layers, hidden sizes, number of attention heads, dropouts, number of training epochs etc. (see a for details). all training is done on a single 32 gb v-100 gpu. pre-training is done for 3 epochs for each monolingual corpora. during fine-tuning, validation is done after every 10000 steps and training stops if validation accuracy does not improve for consecutive 5 evaluations. test set decoding is done using beam search with a beam size of 5 and length penalty of 1.0. translation performance is measured by bleu score calculated using multibleu.perl. ## conclusion we have presented featurebart, an encoderdecoder pre-trained model that augments the encoder's embeddings with linguistic feature embeddings. our experiments on english to asian language translation in an extremely low-resource setting show that featurebart leads to better translation quality compares to its counterpart that does not use features. analyses of training curves reveal that compared to pre-training without features, feature based pre-training leads to significantly lower perplexities during the initial stages of fine-tuning, which we think is responsible for improvement in translation quality. future work will focus on: (1) exploring the impact of individual feature category on feature based pre-training and (2) multilingual version of featurebart which uses features in the encoder for languages other than english.