An Energy-based Model for Word-level A uto C ompletion in Computer-aided Translation (2024)

Abstract: Word-level AutoCompletion (WLAC) is a rewarding yet challenging task in Computer-aided Translation. Existing work addresses this task through a classification model based on a neural network that maps the hidden vector of the input context into its corresponding label (i.e., the candidate target word is treated as a label). Since the context hidden vector itself does not take the label into account and it is projected to the label through a linear classifier, the model cannot sufficiently leverage valuable information from the source sentence as verified in our experiments, which eventually hinders its overall performance. To alleviate this issue, this work proposes an energy-based model for WLAC, which enables the context hidden vector to capture crucial information from the source sentence. Unfortunately, training and inference suffer from efficiency and effectiveness challenges, therefore we employ three simple yet effective strategies to put our model into practice. Experiments on four standard benchmarks demonstrate that our reranking-based approach achieves substantial improvements (about 6.07%) over the previous state-of-the-art model. Further analyses show that each strategy of our approach contributes to the final performance.1

## introduction computer-aided translation (cat) @xcite @xcite , * work done during internship at tencent ai lab. † corresponding authors. 1 our codes are available at https://github.com /yc1999/energy_wlac . previous research includes statistical methods @xcite and neural methods @xcite . with the help of word alignment toolkits @xcite , statistical approaches build a translation table and use it to predict the target word. more recently, @xcite use a transformer-based classification model, which firstly encodes the input context to a hidden vector and then maps the hidden vector into the candidate target word through a linear classifier. this strong baseline method achieves the state-of-the-art (sota) performance. in the aforementioned classification paradigm, the hidden vector of the input context inherently does not take the candidate target word into consideration. as a result, it may not effectively leverage valuable information carried by the candidate target word when occurring in the input context, as shown in figure 1 @xcite . specifically, given the input context and human typed characters ''d'', the user may tend to type ''disease'' @xcite . however, through visualizing attention weights, it shows that the baseline method captures more information from ''gemeinsame @xcite than that from the most informative word ''krankheit'' in the source side, which may underestimate the suppose that a user has input a source sentence x, partial translations (c l , c r ) and is now typing some characters (s). a well-trained wlac model is expected to suggest ''disease'' to complete s. the expected translation for x is ''and disease is the common enemy of these desperate people.'' (b) attention weights from '' @xcite '' to words in x of the baseline method. (c) attention weights from ''disease'' to words in x of our energy-based model. (color intensity reflects the strength of attention weights.) model score of the ground-truth word ''disease'' and thereby leads to incorrect prediction. to alleviate the above issue, we formalize the wlac task with an energy-based model @xcite based on transformer, where the hidden vector is defined on top of both the candidate target word and the input context through a deep energy function. furthermore, with the help of deep neural networks, the energy-based function is expected to capture sufficient information for each candidate target word through the attention mechanism. in this way, the energy function is able to capture informative context (i.e., ''krankheit'') to evaluate the target word (i.e., ''disease''), and thereby the score from the energy-based model is more reliable, as shown in figure 1(c ). unfortunately, training and inference with the energy-based model suffer from efficiency and effectiveness challenges due to the normalization term in the model. to alleviate the effect of these barriers, we systematically incorporate three sim-ple yet effective strategies inspired by previous studies: (1) a negative sampling method for efficient training @xcite @xcite , (2) a reranking paradigm as an approximate proxy for efficient inference @xcite @xcite , and (3) a pre-training method for effective training @xcite . experiments on four standard benchmarks demonstrate that the energy-based model is indeed better at capturing informative signals for the prediction of a candidate target word and thereby yields substantial improvements over strong baselines. to sum up, our contribution is three-fold: ## preliminary in this section, we review the setting of the wlac task and introduce the state-of-the-art baseline method, which will be reused in section 3. task definition given the input tuple (x, c, s), the wlac task aims at predicting the target word w, which starts with s and is the most appropriate to be placed between c l and c r @xcite . in partial translation consisting of c l , w, and c r , w is not necessary to be consecutive to c l,m and c r,1 . figure 1 (a) gives an illustrative example. to be more general in real-world scenarios, the wlac in the baseline model, h [mask] does not capture the information from ''disease'' whereas h [disease] does in the energy-based model. note that ''target encoder'' is a variant of the transformer decoder which can capture bidirectional information on the target side. it is noteworthy that context types described above are general and encompass context of several conventional translation scenarios, such as prefix-decoding for left-to-right interactive machine translation (imt) @xcite and post-editing @xcite . to elaborate, in prefix-decoding, the context falls into the special case of prefix, where c r is empty and c l is consecutive to w. in post-editing, the context corresponds to the special case of bi-context, where both c l and c r are consecutive to w. a baseline wpm is defined by transformer architecture @xcite for nmt. specifically, it first uses a placeholder @xcite to represent the position of the target word w and put it between c l and c r . ultimately, it uses the representation of [mask] defined through transformer to predict the target word. figure 2 (a) shows the model architecture of the baseline wpm. formally, the conditional probability distribution of the target word w is: ## training and inference to relieve the aforementioned challenges, we systematically employ three simple yet effective methods inspired by previous studies. first, we employ negative sampling to address the normalization computation during the training @xcite @xcite ; similarly, during the inference, we adopt a reranking paradigm, where the energy-based model is used as a reranker over a small subset of candidates @xcite @xcite . moreover, we harness a conditional mask bilingual language modeling pre-training strategy for parameter initialization @xcite . ## experiments in this section, we first describe the experimental setup. then we report the main results and analyze the proposed approach. ## conclusion word-level autocompletion is a critical yet challenging task in computer-aided translation. existing work casts this task as a classification problem. however, it cannot make full use of the contextual information from the input context for its prediction. to alleviate such issue, we introduce a reranking perspective by an energy-based model, which directly defines the energy function on top of the input context and the candidate target word. extensive experiments and analyses demonstrate the effectiveness of our proposed approach on four standard benchmarks: it achieves about 6.07% improvements over the strongest baseline.