IMTL ab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems (2023)

Abstract: We present IMTLab, an open-source end-to-end interactive machine translation (IMT) system platform that enables researchers to quickly build IMT systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. IMTLab treats the whole interactive translation process as a task-oriented dialogue with a human-in-the-loop setting, in which human interventions can be explicitly incorporated to produce high-quality, error-free translations. To this end, a general communication interface is designed to support the flexible IMT architectures and user policies. Based on the proposed design, we construct a simulated and real interactive environment to achieve end-to-end evaluation and leverage the framework to systematically evaluate previous IMT systems. Our simulated and manual experiments show that the prefix-constrained decoding approach still gains the lowest editing cost in the end-to-end evaluation, while BiTIIMT achieves comparable editing cost with a better interactive experience.

## introduction in recent years, there has been significant development in neural machine translation (nmt) @xcite @xcite . however, the quality of machinetranslated texts still cannot meet the rigorous demands of industrial applications, necessitating costly and inefficient human intervention. interactive machine translation (imt) @xcite @xcite @xcite is a promising solution that can guarantee high-quality, error-free translations. it involves an iterative collaboration process between humans and * work was done during internship at tencent ai lab. † corresponding author machines, with multiple interactive steps to obtain a satisfactory translation. traditional imt systems use a left-to-right completion paradigm @xcite @xcite where human translators revise words in the translation prefix. this paradigm can be easily implemented using a prefix-constrained decoding strategy. however, this strict left-to-right manner limits its flexibility, as some translators may prefer to revise words in a different order. recently, an alternative imt paradigm has been proposed, allowing human translators to revise words at arbitrary positions in the translation. the essential technique for this paradigm is lexical-constrained translation @xcite @xcite @xcite , which leverages modified words as constraints to generate a satisfactory translation. although various imt techniques have been proposed, there is currently a lack of a unified platform to fairly and systematically evaluate these methods, especially in an end-to-end manner. one of the difficulties is that previous imt methods differ in their interactive policies, evaluation environments and metrics, making it challenging to fairly compare their performance and efficiency. moreover, the evaluation methods previously employed, such as randomly deleting words or phrases and then comparing bleu scores after one or several interactions @xcite @xcite , are far from the real-world interactive experience. the final hypothesis may be not the golden translation and post-editing is still indispensable. instead, the end-to-end paradigm, which evaluates the performance of imt systems when the human translator finishes editing the translation, is a more accurate measure of the total cost of the whole iterative collaboration process. in this paper, we introduce an open-source end-to-end imt system platform, namely imtlab, to fill this gap. this platform enables both academia and industry to quickly build imt systems using state-of-the-art models, perform end-to-end evaluations, and diagnose weaknesses in imt systems. imtlab treats the entire interactive translation process as a task-oriented dialogue, which includes a human-in-the-loop setting and allows imt systems to leverage explicit human interventions to produce high-quality, error-free translations. specifically, the user's goal is to obtain a correct translation, while the imt system constantly generates translation based on the user's behaviors. during this process, the user iteratively performs editing operations to request the response of imt systems until the pre-defined goal is achieved or the user loses patience with this interactive process. to support the flexible architectures of various imt systems, we propose a general communication interface between imt systems and users, where users are limited to five common types of editing operations: keep, insert, replace, delete and blank-filling. then imtlab leverages the revised translation and the corresponding character-level editing operations from users to form lexical constraints for imt systems. moreover, we build a simulated or real interactive environment for imt systems and introduce new evaluation metrics to better verify the effectiveness of imt systems in an end-to-end manner. we conduct simulated and manual experiments to systematically evaluate several popular imt systems with this platform. experimental results indicate that the prefix-constrained decoding approach still obtains the lowest editing cost in the end-toend evaluation, while bitiimt @xcite achieves comparable editing cost with a better interactive experience, i.e., better success rate, lower average turns and response time. imtlab is also compatible with large language models, such as chatgpt. additional experiments show that chat-gpt yields a promising performance of editing cost in end-to-end evaluation, but it is not very robust to flexible lexical constraints. in summary, our contributions are as follows: ## related work imt has been developed to assist professional translators since the era of statistical machine translation (smt) @xcite @xcite @xcite . recently, with the development of nmt @xcite @xcite , the field of imt has undergone a major shift towards powerful deep neural models @xcite . early imt systems typically adopt a left-to-right sentence completing framework @xcite @xcite , where users process the translation from the beginning and revise the leftmost error. however, this left-to-right approach is inflexible and inefficient for modifying critical translation errors near the end of the sentence. as a result, many researchers have explored alternative imt paradigms that allow human translators to revise critical translation errors at any position. one such paradigm is the pick-revise framework proposed by @xcite to improve efficiency. with respect to neural translation models, lexical-constrained decoding (lcd) methods are employed to satisfy arbitrary constraints, such as grid beam search (gbs) @xcite and dynamic beam allocation (dba) @xcite . @xcite design a bidirectional imt framework on top of lcd method. while lcd methods suffer from slow inference speed, some methods treat the lexical constraints as part of the input and let vanilla nmt directly learn the decoding strategy. for example, leca @xcite and bitiimt @xcite are two such methods that employ soft and hard lexical constraints, respectively, to improve translation quality and inference speed. despite the numerous imt techniques, there is currently no unified platform to evaluate these methods in a fair and systematic end-to-end manner. to fill this gap, we develop the first open-source platform for building, the greatest advantage of vr works is that the audience has a unique sense of "immersion" in the film. the biggest advantage for vr works is __ the audience __ hasve a unique __ "immersion" in the film. my goal: the biggest advantage for vr works is to enable the audience to have a unique movie watching "immersion". the biggest advantage for vr works is that the audience will have a unique sense of "immersion". the biggest advantage for vr works is to __ the audience to have a unique movie __ "immersion". the biggest advantage for vr works is to enable the audience to have a unique movie watching "immersion". keep insert delete blank: __ replace vr作品的最大优势是令观众具有独一无二的观影 "沉浸"感。 figure 1 : overview architecture of imtlab, involving the iterative collaboration process between a user and an imt system. during this process, the user provides editing operations (i.e., keep, insert, replace, delete and blank-filling) interactively to query imt systems until the user's goal is achieved or the user loses patience with this interactive process. another research line related to this topic is online learning from translation memory or human feedback, which is particularly effective when translating documents in an unseen domain. this approach typically enhances the performance of nmt systems by fine-tuning through humancorrected translations @xcite @xcite , or by utilizing tm-nmt frameworks @xcite @xcite or knn-mt methods @xcite @xcite @xcite to incorporate humancorrected translation pairs. in this paper, we focus on the interactive process within a single sentence, rather than the knowledge transfer across sentences. we leave the incorporation of online learning in our platform for achieving a stronger imt system as a future direction. ## imtlab this section details the design of imtlab and its flexibility to support a wide range of experiments. ## conclusion in this paper, we introduce imtlab, an opensource platform for building, evaluating and diagnosing imt systems. imtlab treats the whole interactive translation process as a task-oriented dialogue. to this end, we design a general communication interface to support the flexible architectures of imt systems and a simulated or real interactive environment is further constructed for the end-toend evaluation. experiments demonstrate that the prefix-constrained decoding approach still achieves the lowest editing cost in the end-to-end evaluation, while bitiimt achieves comparable editing cost with a better interactive experience. imtlab is also compatible with llms, such as chatgpt.