Improving MT -enabled Triage Performance with Multiple MT Outputs (2025)

Abstract: Recent advances in Machine Translation (MT) quality may motivate adoption in a variety of use cases, but the success of MT deployment depends not only on intrinsic model quality but on how well the model, as deployed, helps users meet the objectives of their use case. This work focuses on a specific triage use case, MT-enabled scanning in intelligence analysis. After describing the use case with its objectives and failure modes, we present a user study to establish a baseline performance level and measure the mitigating effects of a simple intervention, providing additional MT outputs. We find significant improvements in relevance judgment accuracy with outputs from two distinct neural MT models and significant improvements in relevant entity identification with the addition of a rule-based MT. Users also like seeing multiple MT outputs, making it an appealing way to improve MT-enabled scanning performance.

## user study design to establish a baseline risk level for mt-enabled triage in intelligence analysis and to measure the mitigating effects of practical interventions, we conducted a user study with intelligence analysts (ias) from a us intelligence agency in the washington, dc area with significant experience (at least three months) performing triage tasks with the aid of mt and little or no knowledge of the source language. in the next sections, we describe our interventions and design a scenario and tasks for the user study that mimic real triage tasks. we then address the format of the user study and analysis methods. ## results the discussion of the user study results is structured as follows. first, we validate the user study scenario and tasks. we then establish the baseline performance using output from one nmt system and demonstrate the mitigating effects on performance from providing additional outputs, followed by the effects on confidence. after summarizing these quantitative results, we briefly address acceptability of the interventions as indicated by responses on the pre-and post-task surveys. ## conclusions we conducted a user study to establish a baseline level of ia performance on mt-enabled triage tasks and to measure the potential mitigating effects of a simple intervention, providing additional mt outputs. the user study found significant improvements in relevance judgment accuracy with output from two distinct nmt models and significant improvements in relevant entity identification with the addition of motrans rbmt. the availability of additional mt outputs had little effect on analyst accuracy for the task that required the deepest comprehension of the text, identifying the sentiment towards the identified entity. adding motrans rbmt output had little effect on analyst confidence, but providing a second nmt output significantly improved it. this does not appear to be overconfidence, as confidence remained a strong predictor of accuracy across all three types of accuracy. analysts also expressed a preference for seeing multiple mt outputs even when they felt that nmt1 provided better translations and praised the availability of multiple outputs in their openended post-task survey responses.