Enriching the Transformer with Linguistic Factors for Low-Resource Machine Translation (2021)

Abstract: Introducing factors, that is to say, word features such as linguistic information referring to the source tokens, is known to improve the results of neural machine translation systems in certain settings, typically in recurrent architectures. This study proposes enhancing the current state-of-the-art neural machine translation architecture, the Transformer, so that it allows to introduce external knowledge. In particular, our proposed modification, the Factored Transformer, uses linguistic factors that insert additional knowledge into the machine translation system. Apart from using different kinds of features, we study the effect of different architectural configurations. Specifically, we analyze the performance of combining words and features at the embedding level or at the encoder level, and we experiment with two different combination strategies. With the best-found configuration, we show improvements of 0.8 BLEU over the baseline Transformer in the IWSLT German-to-English task. Moreover, we experiment with the more challenging FLoRes English-to-Nepali benchmark, which includes both extremely low-resourced and very distant languages, and obtain an improvement of 1.2 BLEU

## introduction many classical natural language processing (nlp) pipelines used linguistic features @xcite . in recent years, the rise of neural architectures has diminished the importance of the aforementioned features. nevertheless, some works have still shown the effectiveness of introducing linguistic information into neural machine translation systems, typically in recurrent sequence-to-sequence (seq2seq) architectures @xcite @xcite . by factored neural machine translation (nmt), we refer to the use of word features alongside the words themselves to improve translation quality. both the encoder and the decoder of a seq2seq architecture can be modified to obtain better translations @xcite . the most prominent approach consists of modifying the encoder such that instead of only one embedding layer, the encoder has as many embedding layers as factors, one for words themselves and one for each feature, and then the embedding vectors are concatenated and input to the rest of the model, which remains unchanged @xcite . the embedding sizes are set according to the respective vocabularies of the features. note that they used byte pair encoding (bpe) @xcite , an unsupervised preprocessing step for automatically splitting words into subwords with the goal of improving the translation of rare or unseen words. thus, the features had to be repeated for each subword. in @xcite , the exact same architecture was used, except that this new proposal used concepts extracted from linked data database, babelnet @xcite . these semantic features, synsets, were shown to improve zero-shot translations. all the cited works obtained moderate improvements with respect to the bleu scores of the corresponding baselines. some works have previously proposed additional ways to combine sources and introduce hierarchical linguistic information @xcite @xcite . the main goal of this work, and differently from previous works using nmt architectures based on recurrent neural networks, is to modify the transformer to make it compatible with factored nmt with an architecture that we call factored transformer and inject linguistic knowledge and concepts extracted from linked data, babelnet @xcite . we focus on low-resource datasets. ## factored transformer unlike the vanilla transformer @xcite , the factored transformer can work with factors; that is, instead of just being input the original source sequence, it can work with an arbitrary number of feature sequences. those features can be injected at embedding-level, as in the previous works we described above (but in a transformer instead of a recurrent-based seq2seq architecture), or at the encoder level. n-encoders model (depicted in figure 1 , bottom): we intuited that features with large vocabulary sizes could benefit from having a specific encoder. in this variant, each factor has its own full encoder (instead of just its own embedding layer). the outputs from the encoder are combined and input to the following layer. the rest of the model remains unchanged. once we have the outputs of the multiple embedding layers (the 1-encoder) or the n-encoders, they must be aggregated before being input to the next layer. we have considered two combination strategies: concatenation: the outputs of the different embedding layers or encoders are concatenated. summation: the outputs of the different embedding layers or encoders are summed. in both cases, the dimensions must agree. the decoder embedding size must be equal to the encoder embedding size. if the outputs from the different encoders or embedding layers are concatenated, they do not need to have the same embedding size, but the resulting embedding size is increased. instead, if they are summed, they must share the same dimensionality, but the resulting vector size is not increased. ## linguistic features an arbitrary number of features can be injected into the factored transformer, provided they are aligned with words. as follows we describe how linguistic features were extracted and how they were aligned at the subword level. linguistic tagging with stanfordnlp: the corpus was tagged with linguistic information, namely lemmas, part-of-speech (pos), word dependencies and morphological features, using stan-fordnlp @xcite , and aligned with respect to the original tokenization. synsets extraction: babelnet's api retrieves all possible synsets (semantic identifiers) that a given token may have. babelfy @xcite is a word sense disambiguation service based on ba-belnet that retrieves the disambiguated synset for each token depending on the sentence-level context. we split the corpus into chunks such that the daily usage limits of the api were not exceeded and no sentence was split in half (because otherwise babelfy would have missed the context). babelfy returns a list of all the detected synsets with their character offsets, and they must be assigned and aligned to the original tokenization of the corpus. the following step was performed to resolve multiword synset conflicts since in the case of synsets composed of more than one token, babelfy may retrieve one individual synset for each token and a collective one. we decided to prioritize the synset with the largest number of tokens since it seemed to give the most disambiguated information (e.g. the synset semantic network gives more specific information than the individual synsets semantic and network). for the tokens in the corpus that do not have an assigned synset (e.g. articles or punctuation marks), we assign a backup syntactic feature, namely, part-of-speech. ## experimental framework and results data: experiments were conducted with a pair composed of similar languages, the german-to-english translation direction of the iwslt14 @xcite , which is a low-resource dataset (the training set contains about 160,000 sentences). for cleaning and tokenizing, we use the data preparation script proposed by the authors of fairseq @xcite . we took the test sets from the corpus released for iwslt14 and iwslt16. the former was used to test the best configuration, and the latter was used to see the improvement of this configuration in another set. a joint bpe (ie. german and english share subwords) of 32,000 operations is learned from the training data, with a threshold of 50 occurrences for the vocabulary. other experiments were conducted with the english-to-nepali translation direction of the flo-res low resource mt benchmark @xcite . although this pair has more sentences than the previous one (564,000 parallel sentences), it is considered to be extremely low-resource and far more challenging because of the lack of similarity between the involved languages. in this case, we learn a joint bpe of 5000 operations (both with an algorithm based on bpe, sentencepiece @xcite , as proposed by the flores authors, and with the original bpe algorithm). ## conclusions we have shown that the transformer can take advantage of linguistic features but not synsets. we conclude the best configuration for the factored transformer is the 1-encoder model (with multiple embedding layers) with summation instead of concatenation. for the german-to-english iwslt task, the best configuration for the factored transformer shows an improvement of 0.8 bleu, and for the extremely low-resourced english-to-nepali task, the improvement is 1.2 bleu. in future work, we suggest adapting the alignment algorithm to sentencepiece by combining features coming from different words into a single feature, provided their respective subwords have been merged into a single token. we suggest investigating whether linguistic features are still useful with backtranslation too.