I n D eep × NMT : Empowering Human Translators via Interpretable Neural Machine Translation (2022)

Abstract: Neural machine translation (NMT) systems are nowadays essential components of professional translation workflows. Consequently, human translators are increasingly working as post-editors for machine-translated content. The NWO-funded InDeep project aims to empower users of Deep Learning models of text, speech, and music by improving their ability to interact with such models and interpret their behaviors. In the specific context of translation, we aim at developing new tools and methodologies to improve prediction attribution, error analysis, and controllable generation for NMT systems. These advances will be evaluated through field studies involving professional translators to assess gains in efficiency and overall enjoyability of the post-editing process.

## introduction in recent years, the widespread adoption of deep learning systems in neural machine translation (nmt) led to substantial performance gains across most language pairs. consequently, the focus of human professionals gradually shifted towards the post-editing of machine-generated content. despite the indisputable quality of nmt, the question of why and how these systems can effectively encode and exploit linguistic information stands unanswered. indeed, nmt systems are intrinsically opaque due to their multi-layered nonlinear architecture. this fact significantly hinders our ability to interpret their behavior @xcite , an essential prerequisite to their application in real-world scenarios requiring accountability and transparency. for this reason, the interpretability of neural models has grown into a prolific field of research, developing multiple ap-proaches aimed at analyzing models' predictions and learned representations @xcite . while most explainable nmt studies focus on analyzing model learning and predictive behaviors to gain theoretical insights, interpretability approaches have seldom been applied from a usercentric perspective. this criticality was highlighted by exponents of the interpretability field, among which the necessity of grounding future research in practical applications found broad consensus @xcite . in light of this, the development of methods that are selfcontained, generalizable, and scalable would enable the identification of widespread issues characterizing nmt predictions such as hallucinations @xcite , under-and overtranslation, and inadequate terminology @xcite . ## project description as part of the broader consortium 'indeep: interpreting deep learning models for text and sound' funded by the dutch research council (nwo) 1 , we aim to build upon the latest advances in interpretability studies to empower end-users of nmt via the application of interpretability techniques for neural machine translation. the indeep project will run from 2021 to 2026, involving a number of academic and industrial partners such as the universities of groningen and amsterdam, kpn, deloitte and hugging face. central to this project is improving the subjective post-editing experience for human professionals, promoting a shift from a passive proofreading routine to an active role in the translation process by employing interactive and intelligible computational practices, driv-ing further enhancements in the quality and efficiency of post-editing in real-world scenarios. on the methodological side, this entails developing and adapting tools and methodologies to improve prediction attribution, error analysis, and controllable generation for nmt systems. we will evaluate our approaches using automatic metrics, and via a field study surveying professionals in collaboration with globaltextware. 2 the focus for the first part of the project will be on identifying approaches that could be generalized to conditional text generation tasks @xcite . feature and instance attribution methods let us establish the importance of input components and training examples, respectively, in driving model predictions. these techniques are interesting due to their practical applicability in standard translation workflows. in particular, we find it essential to assess the relationship between importance scores produced by these methods and different categories of translation errors. evaluating the faithfulness for model attributions, i.e., how they are causally linked to the system's outputs, is another fundamental component of our investigation and will be pursued by employing a mix of existing and new techniques @xcite . the second part of the project will involve a field study combining behavioral and subjective quality metrics to empirically estimate the effectiveness of our methods in real-world scenarios. for the behavioral part, we intend to use a combination of keylogging and possibly eye-tracking and mouse-tracking to collect granular information about the post-editing process. our analysis will benefit from insights from recent interactive nmt studies @xcite @xcite to present translators with useful information while avoiding visual clutter. our preliminary inquiry involving professionals highlighted sentence-level quality estimation and adaptive style/terminology constraints as promising directions to increase post-editing productivity and enjoyability, supporting the potential of combining interpretable and interactive modules for nmt.