The MAKE - NMTV iz Project: Meaningful, Accurate and Knowledge-limited Explanations of NMT Systems for Translators (2024)

Abstract: This paper describes MAKE-NMTViz, a project designed to help translators visualize neural machine translation outputs using explainable artificial intelligence visualization tools initially developed for computer vision.

## introduction in their meta-review @xcite distinguish opaque, interpretable, and comprehensible systems across various fields including computer vision and natural language processing. neural machine translation (nmt) falls into the category of comprehensible systems, provided that adequate visualisation systems are implemented. they argue that "confidence in an interpretable learning system is a function of the user's ability to understand the machine's input/output mapping behaviour". following the nist report on explainable artificial intelligence (xai) @xcite , we will explore their four principles of xai, which we have rearranged to spell out our make paradigm: meaningfulness we want our visualisation system to be understandable and comprehensible to the translator, putting meaning back at the heart of the nmt process. accuracy: our system will improve our understanding of the input-process-output mapping. it may also identify aspects that are not visually apparent by utilizing visualizations that demonstrate the nmt system's internal working. knowledge limits: the visualizations will be limited to the nmt system's knowledge, operating only under conditions for which it was designed © 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. and when it reaches sufficient confidence in its output, thus revealing to the user the uncertainties inherent in the results. this will be achieved by including different metrics, such as confidence and quality estimation for mt. explanation: we hope to provide visual evidence of the workings of the different steps (subtokenisation/encoding/decoding), thus accounting e.g. for some nmt hallucinations on the basis of the frequencies of the subtokens. in this project, we propose to develop a platform that offers meaningful, accurate, and knowledgelimited explanations of nmt systems for translators. while current neural network visualization primarily focuses on analyzing activation patterns for classification tasks, our project expands its scope to investigate the effects of linguistic structures and neural representations. through our platform, translators will have the capability to translate, post-edit, and evaluate while simultaneously analyzing and explaining nmt model results, thereby bridging the gap between complex artificial intelligence (ai) algorithms and human understanding in translation. ## expected results we intend to develop and utilize a python-based system that leverages state-of-the-art tools to provide a comprehensive approach for analyzing input, process, and output in nmt. we draw inspiration from the seq2seqvis system @xcite , adapting its functionalities to analyze the decision-making process of nmt systems. we incorporate attribution methods for feature importance explanations from the inseq system @xcite (e.g. saliency maps), enabling interoperability with fairseq @xcite models for analyzing relationships between nmt model components. furthermore, we integrate attention weight analysis strategies based on bertviz visualization @xcite , and other methods that attempt to inspect the model's internal data, thus enabling a comprehensive visualization of the data cycle for nmt. our platform supports fairseq models with the addition of unobtrusive probes, called decorators, to traceable parts of the nmt architecture, enabling flexible integration across various nmt architectures. ## visualiser the visualizer is composed of four interoperable modules: translation, analysis, post-editing, and evaluation. the translation module is responsible for loading the model and source text, generating the translated text, and storing the internal data of the system's working (e.g., attention weights, sequence generation probabilities). once the translation is completed, the analysis module steps in to explain how the nmt model arrives at the proposed translation. the explanation comprises three parts: input, process and output, guiding the user process the system took to create the translation step-by-step (or token-by-token). the input analysis focuses on its representation and the subtokenization used. the process describes the workings of the model and its interaction with the input, connecting the model's internal data and features to the output. finally, the output displays in the target language all the different alternatives the model generates and explains why it chose the final proposed translation. to give the translation control to the translator we incorporate a post-editing module, which gives the translator the possibility to use other alternatives proposed by the nmt system, but also to force the generation of specific tokens to update the analysis module. the evaluation module complements the post-editing module. a post-edited text serves as the ground truth, enabling evaluation of the initially generated proposal. this reference can be replaced by standard datasets to conduct stand-alone evaluations. the evaluation contextualise the explanation of the analysis module. ## conclusion in this project, we propose to change the perspective of current explainability tools and systems which do not target the translator audience. instead, we explore the use of these systems in the translation pipeline. while existing systems are too complex to be installed and too experimental to reach the corpus linguistics and translation studies communities, our system intends to be user-centred, for a genuinely human-centred ai, and intends to add functionalities and serve as an all-in-one wrapper. the visualising tool current functionalities can be tested on hugging face spaces. 1