Improving Neural Machine Translation by Multi-Knowledge Integration with Prompting (2023)

Abstract: Improving neural machine translation (NMT) systems with prompting has achieved significant progress in recent years. In this work, we focus on how to integrate multi-knowledge, multiple types of knowledge, into NMT models to enhance the performance with prompting. We propose a unified framework, which can integrate effectively multiple types of knowledge including sentences, terminologies/phrases and translation templates into NMT models. We utilize multiple types of knowledge as prefix-prompts of input for the encoder and decoder of NMT models to guide the translation process. The approach requires no changes to the model architecture and effectively adapts to domain-specific translation without retraining. The experiments on English-Chinese and English-German translation demonstrate that our approach significantly outperform strong baselines, achieving high translation quality and terminology match accuracy.

## introduction in the workflow of translation, human translators generally utilize different types of external knowledge to simplify the process and improve translation quality and speed, such as matching terminologies and similar example sentences. the knowledge used in machine translation mainly includes high-quality bilingual sentences, a bilingual terminology dictionary and translation templates. intuitively, it is reasonable to believe that it is beneficial for improving translation quality to integrate multiple types of knowledge into nmt models in a flexible and efficient way. however, most existing methods focus on only how to integrate a single type of knowledge into nmt models, either a terminology dictionary @xcite , bilingual sentences @xcite or translation templates @xcite . as a primary technique to utilize a terminology dictionary, lexically constrained translation allows for explicit phrase-based constraints to be placed on target output strings @xcite . several research works @xcite impose lexical constraints by modifying the beam search decoding algorithm. another line of approach trains the model to copy the target constraints by data augmentation @xcite @xcite . some researchers @xcite introduce attention modules in the architecture of nmt models to integrate constraints. these methods using terminologies or phrases as the knowledge suffer from either high computational overheads or low terminology translation success rates. in the majority of methods that utilize sentence pairs, the most similar source-target sentence pairs are retrieved from a translation memory (tm) for the input source sentence @xcite @xcite . several approaches focus on integrating a tm into statistical machine translation (smt) @xcite @xcite . some researchers use a tm to augment an nmt model, including using ngrams from a tm to reward translation @xcite , employing an auxiliary network to integrate similar sentences into the nmt @xcite and data augmentation based on tm @xcite . these methods consume considerable computational overheads in training or testing. although these approaches have demonstrated the benefits of combining an nmt model with a single type of knowledge, how to integrate multiple types of knowledge into nmt models remains a challenge. in this work, we propose a promptbased neural machine translation that can integrate multiple types of knowledge including both sen-tences, terminologies/phrases and translation templates into nmt models in a unified framework. inspired by @xcite , which has redefined different nlp tasks as fill in the blanks problems by different prompts, we concatenate the source and target side of the knowledge as prefixprompts of input for the encoder and decoder of nmt models, respectively. during training, this model learns dynamically to incorporate helpful information from the prefixes into generating translations. at inference time, new knowledge from multiple sources can be applied in real time. the model has automatic domain adaption capability and can be extended to new domains without updating parameters. we evaluate the approach in two tasks domain adaptation and soft lexical (terminology) constraint. the metric of 'exact match' for terminology match accuracy has significantly improved compared to strong baselines both in english to german and english to chinese translation. this approach has shown its robustness in domain adaptation and performs better than finetuning when there are domain mismatch or noise data. the contributions of this work include: ## related work nmt is increasingly improving translation quality. however, the interpolation of the reasoning process has been less clear due to the deep neural architectures with hundreds of millions of parameters. how to guide an nmt system with user-specified different types of knowledge is an important issue of nmt applications in real world. the first and most studied knowledge is simple constraints such as lexical constraints or in-domain dictionaries. @xcite proposes grid beam search (gbs) by modifying the decoding algorithm to add lexical constraints. @xcite introduces dynamic beam allocation to reduce the runtime complexity of gbs by dividing a fixed size of beam for candidates. @xcite proposes vectorized dynamic beam allocation (vdba) to improve the efficiency of the decoding algorithm further. the beam search decoding algorithm by adding lexical constraints is still significantly slower than the beam search algorithm. some data augmentation works propose to replace the corresponding source phrases with the target constraints @xcite , to integrate constraints as inline annotations in the source sentence @xcite , to insert target constraints using an alignment model @xcite and to append constraints after the source sentence with a separation symbol @xcite . these data augmentation methods can not guarantee the presence of the target constraints in the output. some works concentrate on adapting the architecture of nmt models to add lexical constraints. @xcite invokes lexical constraints using a non-autoregressive decoding approach. @xcite introduces explicit phrase alignment into the translation process of nmt models by building a search space similar to phrase-based smt. @xcite proposes to use external continuous memory to store constraints and integrate the constraint memories into nmt models through the decoder network. @xcite proposes a template-based method for constrained translation while maintaining the inference speed. @xcite proposes to integrate vectorized lexical source and target constraints into attention modules of the nmt model to model constraint pairs. these methods may still suffer from low match accuracy of terminology when decoding without the vdba algorithm. the use of tm is very necessary for computeraided translation @xcite and computational approaches for machine translation @xcite . similar sentence pairs retrieved from a tm are also utilized as a type of knowledge to enhance the translation @xcite @xcite . @xcite exploits the retrieved sentence pairs from a tm to update the generic nmt models on-the-fly. @xcite utilizes translation pieces based on n-grams extracted from a tm during beam search by adding rewards for matched translation pieces into the nmt model output layer. @xcite proposes to add the word position information from a tm as additional rewards to guide the decoding of nmt models. figure 1 : knowledge integration framework with prompting and an example representation. @xcite uses an auxiliary network to fuse information from the source sentence and the retrieved value from a tm and then integrate it into the nmt model architecture. @xcite proposes to pack a tm into a compact graph corresponding to multiple words for different sentences in a tm, and then encode the packed graph into a deep representation during the decoding phase. @xcite utilizes data augmentation to train an nmt model whose training instances are bilingual sentences augmented with the translation retrieved from the tm. for input sentences that are not very similar to their tms, the translation performance of these methods suffers significantly. @xcite introduces example layer consisting of multi-head attention and cross attention to translate any input sentences whether they are similar to their tm or no. @xcite extends the tm from the bilingual setting to the monolingual setting through learnable memory retrieval in a cross-lingual manner. the key idea of tm is to integrate the retrieved sentence pairs from a tm into the nmt architecture for accurate translations. most of the works integrate the tm knowledge via model modification and the models need to be retrained when loading another tm in new domains. the general knowledge integration to nmt is an ongoing work @xcite @xcite . @xcite proposes to use extracted templates from tree struc-tures as soft target templates to incorporate the template information into the encoder-decoder framework. @xcite introduce to a templatebased machine translation (tbmt) model to integrate the syntactic knowledge of the retrieved target template in the nmt decoder. @xcite represents prior knowledge sources as features in a log-linear model to guide the learning process of nmt models. these approaches have demonstrated the clear benefits by incorporating different single types of knowledge into nmt models. our approach is designed to integrate multiple types of knowledge into nmt models through an unified framework. ## approach as shown in figure 1 , we use source-side knowledge sequence and target-side knowledge sequence to prepend a source sentence and a target sentence as a prefix separately. the model uses multiple types of knowledge as prefix-prompts of the input to guide the process of translating target sentence. the form is intended to lead the nmt model how to utilize relevant information from the redundant prefixes to guide the translation process for improving translation quality. we use three types of special tokens to separate different types of knowledge sequence, source sentence and target sentence. for each sentence pair ⟨x, y⟩, corresponding similar sentence pairs, matching terminologies and translation templates are concatenated into source knowledge sequence x k and target knowledge sequence y k with the corresponding special token [sentence], [term] and [template] on the source and target side, respectively. then the sentence pair ⟨x, y⟩ are preprocessed as follows: the source sentence x and target sentence y are concatenated with source knowledge sequence x k and target knowledge sequence y k , respectively. an example of the format of the input and output sequences is given in figure 1 . when similar sentences and matching terminologies retrieved are empty, the input sequence and output sequence contain only translation template knowledge sequence. although in this paper we integrate similar sentences, terminologies and translation templates into the nmt models, our approach can utilize more types of knowledge to improve translation performance by using this labeling strategy. ## experiments in this section, we validate the effectiveness of the proposed approach on translation quality and terminology match accuracy by comparing with the previous methods used only a single type of knowledge. we evaluate translation quality with the caseinsensitive detokenized sacrebleu score @xcite and terminology match accuracy with exact match accuracy @xcite which is defined as the ratio between the number of matched source term translations in the output and the total number of source terms. ## conclusions in this paper, we propose a unified framework to integrate multi-knowledge into nmt models. we utilize multiple types of knowledge as prefixes of the encoder and decoder of nmt models, which guides the nmt model's translation process. especially, our approaches do not actually require the model to see the domain-specific data in training. the model has automatic domain adaption capability and can be extended to new domains without updating parameters. the experimental results on multi-domain translation tasks demonstrated that incorporating multiple types of knowledge into nmt models leads to significant improvements in both translation quality and exact match accuracy. monolingual data is valuable to improve the translation quality of nmt models. in the future, we would like to integrate monolingual knowledge into the nmt model. furthermore, our approach can be applied for tasks where there are multiple types of knowledge, such as question answering and image to text.