How STAR Transit NXT can help translators measure and increase their MT post-editing efficiency (2023)

Abstract: As machine translation (MT) is being more tightly integrated into modern CAT-based translation workflows, measuring and increasing MT efficiency has become one of the main concerns of LSPs and companies trying to optimise their processes in terms of quality and performance. When it comes to measur-ing MT efficiency, STAR’s CAT tool Transit NXT offers post-editing distance (PED) and MT error categorisation as two core features of Transit’s compre-hensive QA module. With DeepL glossa-ry integration and MT confidence scores, translators will also have access to two new features which can help them in-crease their MT post-editing efficiency.

## star transit nxt in the context of today's technology-shaped localisation business, transit nxt keeps evolving as a sophisticated cat tool to respond to the needs of language professionals. it does so by supporting a variety of mt providers ranging from star's proprietary mt system, commercially available third-party mt providers, through to customisable mt solutions, while also providing the appropriate tools to evaluate mt-based projects and enhance the overall mt post-editing experience. @xcite the ped uses a slightly modified levensthein distance formula. through string comparison, the ped metric returns a value ranging from 0% (worst) to 100% (best) based on the number of manipulations (additions, deletions, substitutions) in comparison to the total number of characters of a text string. ## quality rating with the proofreading mode enabled, translators can mark the mt output in the target language, select an error category and choose a weighting for it. the error categories are based on a slightly condensed interpretation of the sae j2450 foot_0 metric. the weighting is divided into "minor" and "serious", depending on the impact the translation error might have on the understanding of or action related to the translation. in the transit nxt quality report, users can set filters to get an overview of the error category distribution within an entire project or individual files and the possible impact these errors may have on the post-editing process. despite the fact that newer qa metrics such as mqm have been around for quite a while now, our experience has shown that the sae j2450 metric does provide sufficient coverage of all error categories needed to evaluate the quality of the raw mt output. ## post-editing distance the revision mode allows translators to keep track of all changes made during the editing process. changes are saved on a per-segment basis and can be conveniently displayed in the segment information window in the transit nxt editor. in the quality report, users are then provided with an overview of all mt segments. for each mt segment, the report shows the source text, the mt output as well as the final translation, complemented by a per-segment ped value. changes made during the editing process are highlighted in both the mt output and the final translation column. this new reporting feature is currently being reviewed and will be released for transit nxt users in the near future within the service pack 15 update cycle foot_1 . ## quality report the transit nxt quality report is a module designed to consolidate the results of all relevant qa checks in transit and have them readily available in a single report document. the qa report is divided into different main error categories. each category provides distinct and valuable information, e.g. user-defined protected strings missing for translation, error categories and severity for quality rating j2450, or the preferred term from the project dictionary for incorrect terminology. each error or inconsistency is accurately logged with the file name, segment number, source language and target language segment content to make it very easy for translators to evaluate and correct the mt output. ## new smart features for post-editing for translators, transit nxt already features a plethora of options to enhance the post-editing experience. the internal repetitions (ir) mode helps them identify and correct identical segments that were not translated consistently by mt to avoid unwanted variants. tm validation for mt segments compares the mt output to a highly similar tm translation and visualises the differences between both versions for convenient editing. a new feature introduced with the latest transit nxt update is the integration of deepl glossaries that allows translators to upload a stripped-down copy of their transit nxt project terminology for supported language combina-tions and have the preferred terms applied directly to the mt output. this reduces the overall effort needed to correct terminology errors in the mt output and provides greater consistency. as a complement to this, transit nxt's built-in terminology checks add an extra layer of convenience. term recognition allows users to visually distinguish whether a preferred term from the project terminology was used in the mt output or not. the second feature to be released in 2023 is referred to as the mt confidence score foot_2 , which is based on a combination of the modern com-et foot_3 metric and proprietary ai algorithms. before editing, translators can visually distinguish between mt suggestions that require a higher or lower level of attention. ## future challenges based on our own experience, we are firmly convinced that using the evaluation methods and smart features mentioned above enables us to get a better understanding of the benefits and shortcomings of mt systems in real-world scenarios. moreover, we argue that even though automated mtqe metrics have become much more reliable over time, evaluation of real-world mt projects still provides us with better insights into the augmented translation experience. however, analysing these parameters is often timeconsuming. that said, we see a definite need for developing an advanced ped model that does not only measure edit distance as such but puts it into the bigger context by applying weighting coefficients -based on cognitive load -to the existing error categories. our first step in developing such a model will be the implementation of ai to automatically classify the changes made during the editing process.