Better Neural Machine Translation by Extracting Linguistic Information from BERT (2021)

Abstract: Adding linguistic information (syntax or semantics) to neural machine translation (NMT) have mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT(Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT.

## introduction probing studies into large contextual word embeddings such as bert @xcite have shown that these deep multi-layer models essentially reconstruct the traditional nlp pipeline capturing syntax and semantics @xcite ; information such as part-of-speech tags, constituents, dependencies, semantic roles, coreference resolution information @xcite and subject-verb agreement information can be reconstructed from bert embeddings @xcite . in this work, we wish to extract the relevant pieces of linguistic information related to various levels of syntax from bert in the form of dense vectors and then use these vectors as linguistic "experts" that neural machine translation (nmt) models can consult during translation. but can syntax help improving nmt? @xcite ; @xcite ; @xcite have reported that learning grammatical structure of sentences can lead to higher levels of performance in nlp models. in particular, @xcite show that augmenting nmt models with explicit linguistic annotations improves translation quality. bert embeddings have been previously considered for improving nmt models. @xcite replace the encoder token embedding layer in a transformer nmt model with bert contextual embeddings. they also experiment with initializing all the encoder layers of the translation model with bert parameters, in which case they report results on both freezing and fine-tuning the encoder parameters during training. in their experiments bert embeddings can help with noisy inputs to the nmt model, but otherwise do not help improving nmt performance. @xcite suggest that replacing the encoder layer with bert embeddings and fine-tuning bert while training the decoder leads to a catastrophic forgetting phenomenon where useful information in bert is lost due to the magnitude and number of updates necessary for training the translation decoder and fine-tuning bert. they present a two-step optimization regime in which the first step freezes the bert parameters and trains only the decoder while the next step finetunes the encoder (bert) and the decoder at the same time. @xcite also try to address the catastrophic forgetting phenomenon by thinking of bert as a teacher for the encoder of the neural translation model (student network) @xcite . they propose a dynamic switching gate implemented as a linear combination of the encoded embeddings from bert and the encoder of nmt. however these papers do not really focus on the linguistic information in bert, but rather try to combine pre-trained bert and nmt encoder representations. @xcite identify part-ofspeech, case, and sub-word position as essential lin-guistic information to improve the quality of both bert and the neural translation model. they extract each linguistic feature using the viterbi output of separate models, embed the extracted linguistic information (similar to trained word embeddings) and append these vectors to the token embeddings. however, their model uses point estimates of the syntactic models and they do not use the linguistic information in bert embeddings. ## nmt and bert machine translation is the problem of transforming an input utterance sequence x in source language l f into another utterance sequence y (possibly with varying length) in target language l e . machine translation models search among all possible sequences in target language to find the most probable sequence based on the probability distribution of equation 1 . bert @xcite adopts the encoder part of the transformer model and requires training it on large amounts of text data using a masked language model objective over sub-words p(y i |x, y 0 , ..., y i-1 , y i+1 , ..., y max len ) instead of guessing the next sub-word p(y i |x, y 0 , ..., y i-1 ). this bidirectional context turns bert into a provider of strong contextual sub-word embeddings in many languages. these massively overparameterized neural networks have revolutionized many different nlp tasks. effective application of bert in nmt has been studied in a number of contemporary research projects; language modeling, named entity recognition, question answering, natural language inference, text classification @xcite , and question generation @xcite . we approach this problem from the novel perspective of extracting linguistic information encoded in bert and applying such information in nmt. ## linguistic aspect extraction from bert since bert contextual embeddings contain a variety of information (linguistic and non-linguistic), extraction of relevant information plays an important role in further improvement of the downstream tasks. in the rest of this section, we define aspect vectors as single-purpose dense vectors of extracted linguistic information from bert, discuss how aspect vectors can be extracted, and explain how to integrate aspect vectors into nmt. ## experiments in this section, we initially examine our designed aspect extractor and report its classification accuracy scores. next, we integrate the extracted aspect vectors into the neural machine translation framework as explained in section 3.3 and study the effects of integrated vectors on the performance of the models. ## conclusion and future work in this paper, we proposed a simple method of extracting linguistic information from bert contextual embeddings and integrating them into neural machine translation framework. we showed that the linguistic aspect vectors provide the translation models with out-of-domain knowledge which not only improves the translation quality but also helps the model to better deal with out-of-vocabulary words. in the future, we would like to reconsider the integration module as a multi-head attention module, except that it will attend to different linguistic aspects of the current sub-word or sub-word tokens of a single word. increasing the number of linguistic aspects (especially the use of syntactic dependencies and morphology) and studying the effects of the aspect vector size on the quality of generated translations are other directions of future research. we would also like to examine the effectiveness of aspect vectors trained on large out-of-domain data in low-resource settings and explore the effects of using linguistic aspect vectors in tasks other than machine translation. source ihm werde weiterhin vorgeworfen, unerlaubt geheime informationen weitergegeben zu haben. reference he is still accused of passing on secret information without authorisation. @xcite he has also been accused of having illegally passed on secret information. @xcite he continues to be accused of fraudulently passing on secret information. @xcite he is also accused of having pass unauthorised secret information on. aspect augmented nmt he is still accused of passing on illegal secret information. source auto und traktor krachen zusammen: frau stirbt bei schrecklichem unfall reference car and tractor crash together: woman dies in terrible accident @xcite car and traktor cranes together: women die in the event of a terrible accident. @xcite cars and tractors are killing women in the event of a terrible accident. @xcite auto and tractor are blowing together: woman dies when the terrible accident occurs. aspect augmented nmt car and tractor crash together: woman dies in terrible accidents. source die deutsche bahn will im kommenden jahr die kinzigtal-bahnstrecke verbessern. reference the deutsche bahn hopes to improve the kinzigtal railway line in the coming year. @xcite the german railway wants to improve the kinzig valley railway line next year. @xcite christian deutsche bahn intends to improve the kinzig valley railway next year. @xcite the german railway wants to improve the kinziggia railway line next year. aspect augmented nmt deutsche bahn wants to improve the kinzig valley railway in the coming year.