A Simple and Fast Strategy for Handling Rare Words in Neural Machine Translation (2022)

Abstract: Neural Machine Translation (NMT) has currently obtained state-of-the-art in machine translation systems. However, dealing with rare words is still a big challenge in translation systems. The rare words are often translated using a manual dictionary or copied from the source to the target with original words. In this paper, we propose a simple and fast strategy for integrating constraints during the training and decoding process to improve the translation of rare words. The effectiveness of our proposal is demonstrated in both high and low-resource translation tasks, including the language pairs: English → Vietnamese, Chinese → Vietnamese, Khmer → Vietnamese, and Lao → Vietnamese. We show the improvements of up to +1.8 BLEU scores over the baseline systems.

## introduction neural machine translation (nmt) @xcite @xcite has recently shown impressive results compared to statistical machine translation (smt) @xcite . however, nmt systems still have great challenges @xcite , in that, addressing rare words is one of them. due to nmt that has tended to bias in highfrequency words, low-frequency words have little chance of being considered in the inference process. to tackle this problem, some previous works have proposed various strategies to augment translation of low-frequency words. typically, @xcite demonstrate the effectiveness of nmt systems by replacing rare words by special symbols such as unk 1 , unk 2 , ..., unk i in the sentence. they use an aligned dictionary to map between an unk i in the source sentence and an other unk j in the target sentence. this approach tends to raise ambiguity in context of sentence as shown in @xcite . in addition, @xcite suggest for applying byte pair encoding (bpe) @xcite to nmt systems. this technique significantly reduces the vocabulary size and shows substantial improvements on performance translation, and it is widely applied for almost all translation systems nowadays. following this approach, a rare word will be split into sub-words and the sentence context is still preserved, nevertheless, other new rare words can be also generated. moreover, this segmentation could make it more difficult to discover original rare words from their sub-words. to overcome this issue, @xcite propose pointer networks which automatically copy rare-words from the source sentence into the target sentence. to achieve this aim, they integrate a copy probability to the output distribution with a copy coefficient learnt during the training process. there are also some other variances of these networks such as @xcite @xcite . however, these techniques often copy only a part of rare words when they are separated into sub-words, and we find that they do not have benefit in the data sparsity situation. inspired by above pointer networks, we propose a simple and fast idea for representing the output probability distribution, though our strategy does not require learning any additional weights. besides, we leverage neighboring words to identify the suitable position of translation in the source side that corresponds to a rare word in the target side when it has the wrong attention. the proposal is only performed during the inference process, therefore, it does not affect the training. our experiments show the improvements overcoming the baseline systems. the background of nmt is shown in 2. the detail of our method is described in section 3, experiments and results are shown in section 4. finally, the related work is presented in section 5. ## neural machine translation the goal of nmt systems is to translate a source sentence @xmath0, ..., x |x| to a target sentence @xmath1, ..., y |y| . nmt has suggested in @xcite with recurrent neural networks (rnns) which use gru (gated recurrent unit) or lstm (long short term memory) for handling the memory context of long sentences. however, these networks face the difficulty of parallel computations during the training process. @xcite presents the transformer model to overcome this issue, which has shown the state-of-the-art in current nmt systems. the probability p (y|x, θ) indicate a nmt model @xcite parameterized by θ. during the training process, parameters are optimized by minimizing the maximum likelihood of the sentence pairs: self-attention in the transformer architecture, both encoder and decoder are stacks of i identical layers, each layer contains number heads of selfattention to learn context representations. attention mechanism in transformer between the source sentence and the target sentence is a variance of the attention proposed by @xcite . its presentation is the same as in 2, however, keys and values are representations of the source sentence while queries are those of the target sentence. ## rare words translation this section will present details of steps in our method to enhance the translation system. tagging data in the first place, we use giza++ foot_0 to align between source and target sentences. secondly, a dictionary will be generated from the alignment table. finally, we tag a special token for rare words in both source and target sentences as in the figure 1 . in our experiments, words that have a frequency below 4 are considered as rare words. each rare words is inside a pair of "#". we hope that these tokens will help translation systems detect rare words during the training and decoding process. to address this issue, we fix the copy factor α to constants during the inference process. base on tagged labels as mentioned above to the detection rare words, we set α to 1 after detecting the token "#" that mark the start position of each rare word in the step i th , otherwise, it is set to 0 when the other stop token "#" is discovered. heuristic due to the systems that could attend to the translation of a rare word in the target side to the wrong translation in the source side may contain many rare words. in this issue, around words are also used to support the inference process discovering the best suitable position of the corresponding translation in the source sentence. to implement this idea, we define a heuristic function score ij to estimate alignment weight between each rare word i th in the target side and the rare word j th in the source side during the inference process as in the formula 4. score @xmath2. similarly, p # mj is the attention weight of the token "#" m th figure 1 : a illustration of our method: the rare word "bệnh đậu mùa khỉ (猴痘)" is put on a pair of "#", the non rare word "triệu trứng" is used to support the inference process detecting the suitable position of translation in the source side. the α is a copy factor which is described in the formula 3 . that starts or ends the rare word i th , and p a lj is the attention weight of the around word l th of the rare word i th . in the experiment, we only consider two content words that are nearest to the rare word i th , in which, one word before and another one after the the rare word i. p sw kj , p # mj , and p a lj will assigned to 1 if they attend to the rare word j th in the source side, otherwise, they are 0. besides, β, γ and ϵ are constants, in our experiments, both β and γ are the same value and they are assigned to 0.4 while ϵ is 0.2. the rare word j th in the source side that has the highest score corresponding to the rare word i th in the target side will be chosen as the its best translation. our algorithm is detailed in the algorithm 1. thus, at each step n th , the cross attention weights will be computed for each target token. we utilize head 0 with layer 0 in order to evaluate these attentions .to save time, we employ "cache maintenance" strategy inspired by @xcite to archive q t , k t , v t , and attn t in beam search. ## experiments this section presents our implementation of the translation systems. our method show the efficiency of both bilingual and multilingual translation systems. the sacrebleu score @xcite 2 is employed to evaluate the quality translation. for all experiments, the development and test sets from the asian language treebank (alt) corpus are utilized for early stop and evaluate the efficiency of our strategy. the development set includes 1000 sentence pairs while the test set contains 1018 other ones. to generate alignment dictionaries for tagging rare words, we use various segmentation tool for each language: pkuseg foot_4 for chinese texts, laonlp foot_5 for laos texts, khmernltk foot_6 for khmer texts, and moses foot_7 tokenizer for english and vietnamese texts. algorithm 1: finding the suitable position in the source sentence for a target rare word input : attn_cache contains cross attentions, marked_src_sent: marks the position of the rare word, word_query: the translation of the rare word in target sentence including special token "#". output :the best candidate correspond to rare word p sw kj , p # mj , p a @xmath3. ## related work dealing with rare words has investigated by many previous works in machine translation. @xcite employed a model of lexical borrowing to enhance smt systems. however, this approach claim extraction of complex features such as phonetic and semantic features, or pre-trained smt systems, and it is difficult to apply to nmt systems. @xcite used a large vocabulary to solve the rare words but this increases parameters and leads to augmentation the size of models and rare words still exist. some other works require additional resources to tackle rare words such as trieu (2016) exploited word similarity, or @xcite utilize synonyms to advance translation systems. @xcite employed special symbols to present rare words or unknown words but this tend to increase the ambiguous of sentence context. @xcite applied bpe algorithm to separate rare words into sub-words, however, new sub-words are again generated. furthermore, @xcite suggested point networks allow to copy automatically rare words from source side to the target side with learning supplemental parameters, and in some case, it has only a part of rare words (sub-words) are copied. this approach is also considered in recent studies @xcite ; @xcite ; @xcite . our proposal also relies on the idea of point networks, nevertheless, we fix the copy factor to a constant. besides, we leverage neighbouring words to specify the best position of the rare word in the source side that corresponds to the one in the target side. ## conclusion in this work, we propose a simple and fast method to improve the translation quality for rare words. our technique does not require training supplemental parameters, and this strategy is only performed in the inference process, therefore, the training time does not change. in the future, we would like to consider more neighbouring words around rare words for selecting position to improve the quality of translation tasks.