Translation Word-Level Auto-Completion: What Can We Achieve Out of the Box? (2022)

Abstract: Research on Machine Translation (MT) has achieved important breakthroughs in several areas. While there is much more to be done in order to build on this success, we believe that the language industry needs better ways to take full advantage of current achievements. Due to a combination of factors, including time, resources, and skills, businesses tend to apply pragmatism into their AI workflows. Hence, they concentrate more on outcomes, e.g. delivery, shipping, releases, and features, and adopt high-level working production solutions, where possible. Among the features thought to be helpful for translators are sentence-level and word-level translation auto-suggestion and auto-completion. Suggesting alternatives can inspire translators and limit their need to refer to external resources, which hopefully boosts their productivity. This work describes our submissions to WMT’s shared task on word-level auto-completion, for the Chinese-to-English, English-to-Chinese, German-to-English, and English-to-German language directions. We investigate the possibility of using pre-trained models and out-of-the-box features from available libraries. We employ random sampling to generate diverse alternatives, which reveals good results. Furthermore, we introduce our open-source API, based on CTranslate2, to serve translations, auto-suggestions, and auto-completions.

## introduction translation auto-suggestion and auto-completion are among the important features that can help translators better utilize machine translation (mt) systems. in a computer-aided translation (cat) environment, a translator can make use of the mt word auto-suggestion feature as follows: the wmt's word-level autocompletion (wlac) shared task addresses a more specific scenario, where the user types a few characters, and the system predicts and auto-completes the correct word, given the current context. the wlac task even suggests that the context might be partial, and it can consist of preceding and/or following words. given a source sequence x, typed character sequence s and a context c, wlac aims to predict a target word w which is to be placed in the middle between the left context c l and right context c r to constitute a partial translation. note that the last word of c l , the auto-completed word w, and the first word of c r are not necessary consecutive. previous work proposed diverse approaches, mostly to translation sentence-level autosuggestion and auto-completion. in their work, @xcite proposed an approach to tackle the word-level auto-completion task. given a tuple (x, c, s), the system decomposes the word autocompletion process into two parts: 1) model the distribution of the target word w based on the source sequence x and the translation context c; and 2) find the most possible word w based on the distribution and human typed sequence s. hence, they first use a single placeholder @xcite to represent the unknown target word w, and use the representation of [mask] learned from the word prediction model, based on bert @xcite , to predict it. then, the predicted distribution of the masked token is used over the vocabulary to filter out invalid words, namely those that do not start with the human typed sequence s. finally, they return the token with the highest probability over the new distribution. researchers in other natural language processing areas such as language modelling offered approaches to improve predictions of decoder-only autoregressive models, trained to predict the next word given the previous context. among these approaches are top-k sampling and top-p (nucleus) sampling @xcite @xcite . since neural machine translation inference depends on a decoder model, such approaches from language modelling can be employed. in particular, we investigate the use of top-k sampling during decoding to generate better word-level auto-completions. ## user survey previous work reported that a user can save over 60% of the keystrokes needed to produce a translation in a word completion scenario @xcite . other researchers noted that post-editing is faster than mt auto-completion @xcite while mt auto-completion can yield higher quality translation when the baseline mt quality is high @xcite . in a user survey we designed and distributed via social media networks, we asked participants whether they thought an mt word-level autosuggestions feature could be helpful, and provided a simple definition and an illustrative image. if their answer was "yes", the respondent was asked to specify a reason. by the time of writing this paper, we received 41 responses to our survey. while we do not believe this survey is enough to justify introducing an auto-suggestions feature into every mt system, it can be an indicator as to why some users think such a feature could be helpful. to answer the question, "which of the following best describes you?" 46.3% (19) of the respondents chose "translator/linguist", 31.7% (13) selected "nlp engineer/researcher", and the rest 22% (9) were other "mt users", not included in the two aforementioned categories. among the respondents to the survey, 90.2% (37) answered "yes" to the question "in general, do you believe that a word-level auto-suggestions feature is helpful?" figure 3 shows the breakdown of answers to the question, "why do you believe that a word-level auto-suggestions feature can be helpful?" taking into consideration those who answered "no" to the previous question. out of the 37 persons who believed a word-level auto-suggestions feature can be helpful, 40.5% (15) of the respondents specified that it can give them some inspiration. this answer is specifically interesting as it is not constrained by time-saving benefits; hence, it focusses more on effectiveness rather than efficiency. the respondent that answered with "other" mentioned that it allows them to look for alternative senses or phrasings, especially when they suspect the initial translation is bad, and referred to this as "human in the loop". respondents were allowed to give extra comments; among the notable comments were: • i think word-level suggestions can be a useful feature, among the respondents who chose "for me, it is easier or faster than typing", comments included: the last comment above might be referring to the fact that in some cat tools, auto-suggestions can also include glossary terms, and translation memory sub-segments, which encourages further research efforts to investigate methods to enhance leveraging and interaction between various translation resources in human-in-the-loop environments. we hope this survey will inspire future user studies to look deeper into how diverse users of mt and cat tools prefer to utilize certain features, such as auto-suggestions, and the value they seek. more aspects should be taken into consideration such as language pairs, translation workflows, and user interfaces. this can help improve these features to better support linguists and other mt users and boost their productivity as well as translation quality. ## experimental setup models we use opus pre-trained models foot_1 based on the transformer architecture @xcite for the chinese-to-english, englishto-chinese, german-to-english, and english-to-german language directions. tokenizers opus models depend on sentence-piece foot_2 @xcite for tokenization. hence, we use their provided subword models during our pre-processing and post-processing processes. as opus's english-to-chinese model requires defining the target dialect using a prespecified token, we prepend [">>@xmath0. for word-level tokenization, we use nltk for english and german, and jieba foot_3 for chinese. this list of words can be used later to find the word that starts with the typed sequence. inference engine we employ ctranslate2 @xcite for sentence-level mt, as well as for translation auto-suggestions. to this end, we first convert opus models into the ctranslate2 format. after that, we utilize a number of ctranslate2 decoding features, including "alternatives at a position" and "auto-completion". 5 the translation options return_alternatives and num_hypotheses are essential for all our experiments; the former should be set to t rue while the latter determines the number of returned alternatives. these decoding options can be used with regular beam search, prefix-constrained decoding, and/or random sampling. if the decoding option return_alternatives is used along with target_pref ix, the provided target left context is fed into the decoder in the teacher forcing mode, foot_5 then the engine expands the next n most likely words, and continues (auto-completes) the decoding for these n hypotheses independently. the shared task investigates four context cases: pinyin the official romanization system for standard mandarin chinese is called pinyin. since the task organizers used the pypinyin library 7 to prepare the test files, we did too. opus english-to-chinese models accept chinese input, so we had to use the library to convert between the two writing systems. since the conversion from chinese characters to pinyin is a lossy process and cannot be perfectly converted back, we keep a list of chinese words resulted from tokenization with jieba to be able to map pinyin tokens to chinese tokens later. ## method we experimented with both beam search alternatives and random sampling, and found that the latter achieves better results. this could be due to the fact that alternatives generated from each beam are usually very similar, and lower beam values tend to generate translations of lower quality. this section elaborates on the actual methods we used for our submissions, while more details about initial experiments that led us to these decisions are explained in section 5. random sampling is a decoding mode that randomly samples tokens from the model output distribution. in our experiments, we restrict the sampling to the top-10 candidates at each time-step. to obtain diverse generations from the mt model, we rely on randomness in the decoding method, in particular through top-k sampling that samples the next word from the top-k most probable choices 7 https://github.com/mozillazg/ python-pinyin @xcite @xcite , instead of aiming to decode text that maximizes likelihood. for each translation, we use the ctranslate2 option return_alternatives to return 10 sequences, with 10 top-k sampling. if the entry has a left context starting with a capital letter, we use the prefix to constrain the decoding. in ctranslate2, combining target_prefix with the return_alternatives flag returns alternative sequences just after the prefix. we compose a list of alternatives with and without the prefix, and try to find the word starting with the typed sequence. 8 if the word is not found, we repeat the same process for up to five runs. in each new run, random sampling can generate a new set of alternatives. our experiments show that returning 20 sequences with 20 top-k sampling could lead to more correctly predicted words (cf. ## other experiments this section elaborates on some initial experiments we conducted to decide what approach to use. the final approach we actually used in our submissions is explained in section 4. we used 10,000 entries of a chinese-to-english golden sample provided by the organizers to evaluate various experiments. for sentence translation, when there is no left context, we experimented with the following values: ## api our api project foot_8 offers an easy way to integrate translation, auto-suggestion, and auto-completion features into translation environments. we chose fastapi foot_9 for its high performance that beats many other python web frameworks foot_10 in addition to its easy integration with openapi (swagger) documentation.