Exploring NMT Explainability for Translators Using NMT Visualising Tools (2024)

Abstract: This paper describes work in progress on Visualisation tools to foster collaborations between translators and computational scientists. We aim to describe how visualisation features can be used to explain translation and NMT outputs. We tested several visualisation functionalities with three NMT models based on Chinese-English, Spanish-English and French-English language pairs. We created three demos containing different visualisation tools and analysed them within the framework of performance-explainability, focusing on the translator’s perspective.

## introduction the development of machine translation (mt) is influenced by a wide range of actors and agents, ranging from the investors to general public. a stakeholder approach to mt enables us to examine the effects of mt on each of the different interest groups, with particular reference to levels of involvement with mt (e.g., translators, students and trainees, end users, mt investors and developers, translation agencies, and academic researchers) @xcite . upon refining the landscape of mt to include the directly associated stakeholders, several distinct categories emerge. the primary category consists of mt developers, typically computer scientists, whose focus lies in enhancing the accuracy and fluency of translations. in contrast, a second group of stakeholders, comprised of linguists and translators with expertise in translation studies, may argue that translation quality cannot be regarded as a static or absolute concept; instead, it is influenced by both subjective and objective factors that may change over time, as evidenced by semantic/communicative and functional translation approaches. their concern centres on how mt fits into their practical translation workflow, and mt's ability to handle cultural-specific items and nuances -a realm in which translators take great pride. moreover, industry surveys, such as the freelance translator survey 2023 by inbox translation, and ciol insights 2022, foot_0 have also demonstrated that translators are primarily concerned about the effects of mt in their professional status, i.e., decreased translation/post-editing rates, clients' unrealistic expectations and other people's perception of their professionalism. moreover, end-users constitute another critical group, prioritising usability, speed, and the cost-effectiveness of translations @xcite . although these three groups of stakeholders are closely related to the development of mt, they do not always understand each other's work or demands, underscoring the need for continuous dialogue between each group. this work is of part of the make-nmtviz project, which, by bringing together key stakeholders in mt development, aims to connect mt researchers with professional translators, taking into consideration their needs and preferences, whilst improving translators' mt literacy. this project is a starting point for facilitating communication ensuring that mt is developed and utilised effectively. central to our investigation is the role of vi-sualisation systems in explainable artificial intelligence (xai) for neural machine translation (nmt). we aim to assess the utility of various nmt visualisation tools for professional translators, examining how these tools contribute to their understanding of nmt models' decisions. through a comprehensive review of existing explainability visualisation systems, we implement selected ones in the form of three demos available on huggingface spaces, in order to determine their effectiveness in helping translators comprehend whether mt models produce accurate translations for appropriate reasons. by facilitating communication and understanding among key stakeholders, our project aims to promote the effective development and use of nmt systems. the contributions of this paper are the following: (i) a revision and typology of state-of-the-art visualisation functionalities for the explainability for nmt; (ii) a set of ready-to-use explainability and visualisation tools available in the hugging face spaces for the translator's use; 2 and, (iii) a translator-focused evaluation of explainability visualisations for nmt. the paper is structured as follows: section 2 provides an overview of previous research on nmt explainability methods and existing visualisation systems. in section 3, a typology of functionalities is examined. section 4 details the methodology for assessing explainability functionalities for translators. section 5 presents an analysis of the visualisation systems from the translator's perspective. discussion and conclusions are presented in sections 6 and 7. ## previous research on visualisation tools for nmt in this section, we present the state-of-the-art of visualisation methods and tools employed for explainable nmt (xnmt). visualisation is a key component of xnmt methods identified by @xcite in his survey paper. it is used in model-intrinsic interpretability methods, posthoc interpretability methods (interpreting predictions with input analysis), and the analysis of confidence estimation in translation. in a more recent survey by @xcite , several post-hoc methods for nlp interpretability were reviewed, which has been further specialised by @xcite , who specifically reviewed methods for nmt metrics. in this work, we focus on the review 2 https://huggingface.co/gabrielanicole of existing systems that implement such methods, aiming to make xnmt accessible for translators. ## a typology of implemented functionalities in this section, we present a typology of implemented functionalities in state-of-the-art systems. to exemplify each functionality, we map them to the transformer architecture (figure 1 , original figure taken from @xcite ). our survey of xnmt as a task and process follows an input, process, output analysis of the functionalities: the described functionalities comprehensively map the transformer architecture, although ongoing improvements are needed to develop better methods. in this work, we focus on evaluating them from the translator's perspective. ## conclusion in this paper, we have summarised the main visualisation tools adapted for xnmt, detailing the functionalities implemented in a prototype and discussing the potential benefits for translators. our innovation lies in highlighting the translator's viewpoint and utilising xnmt to provide accountability for translators. this entails gaining a better grasp of the training data, monitoring the learning phase, or finding ways to understand the entire nmt process. as future work, we will continue exploring additional visualisation tools and evaluating their use, specifically focusing on one of the following translation moments: (i) initiating use of a new technology to understand nmt system workings during training, (ii) beginning a project to comparing, trusting, and selecting the best nmt, (iii) analyzing the translation process to identify reasons for poor output, and (iv) evaluating translation results e.g. to test alternatives.