Re-design of the Machine Translation Training Tool ( MT 3) (2020)

Abstract: We believe that machine translation (MT) must be introduced to translation students as part of their training, in preparation for their professional life. In this paper we present a new version of the tool called MT3, which builds on and extends a joint effort undertaken by the Faculty of Languages of the University of Córdoba and Faculty of Translation and Interpreting of the University of Geneva to develop an open-source web platform to teach MT to translation students. We also report on a pilot experiment with the goal of testing the viability of using MT^3 in an MT course. The pilot let us identify areas for improvement and collect students’ feedback about the tool’s usability.

## introduction machine translation (mt) has made enormous progress over the past few years with the development of neural systems @xcite , and the translation industry has therefore been increasingly integrating it into daily workflow processes. this may have a direct impact on translators, who need to learn how to work with mt in a step called post-editing @xcite . it is therefore important to help translation students understand the underlying concepts of mt, including how mt systems are trained from bilingual corpora, what distinguishes systems from one another in terms of the algorithms they use, as well as the impact of their internal functioning on translation quality and, hence, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. on the post-editing task. even if the idea behind these systems sounds simple (producing the most likely translation), it can be challenging to explain to non-tech savvy students how these systems produce a final output. currently, there are many open-source tools available to train statistical and neural mt models (the most widely used are moses @xcite for statistical mt and tensorflow foot_0 or opennmt @xcite for neural mt) or commercial tools (such as kan-tanmt foot_1 or microsoft custom translator foot_2 ). however, it is difficult to integrate them in the classroom because open-source tools are mostly designed for it professionals, who do not need a graphical user interface (gui), as opposed to translation students; as for commercial tools, the costs are far too steep for some institutions. the faculty of languages of the university of córdoba (fl-unc) and the faculty of translation and interpreting of the university of geneva (fti-unige) have been collaborating since 2017 to design and prototype a tool that will support lecturers in the classroom. in this paper, we present a new version of the tool called m t 3 , which builds and extends on a joint effort undertaken by both institutions to develop an open-source web platform to teach mt to non-technical students. we also report on a pilot experiment with the goal of testing the viability of using m t 3 for an ma course on mt. after the lecturer presented the topic to students, they had to carry out an exercise using the tool and then complete a questionnaire. the goal of the pilot was not only to test the tool and identify areas for improvements, but also to get subjective feedback from the students on its usability. ## mt in the classroom mt is one of the topics that should be introduced to translation students in preparation for their professional life, as lacking the necessary skill-set to take on post-editor roles or other technical roles, such as language engineer or mt engineer, could have a negative impact on their future employability, as many researchers and lecturers have observed @xcite @xcite @xcite @xcite . the emt expert group (2017, p. 7) "recognises that the ability to interact with mt in the translation process is now an integral part of professional translation competence". furthermore, as part of the technological competence (p. 9), they highlight the inclusion of "basic knowledge of mt technologies and the ability to implement mt according to potential needs". this reflects the current need of many translation and localization companies to hire specialists with a background in translation or linguistics who know how to train and assess the quality of mt engines. as lara (2019), @xcite and @xcite explain, mt does not depend exclusively on informatics, but requires a close collaboration with language experts for the creation of corpora, labelling, quality assessment, terminology management, controlled-language definition and text preprocessing. the importance of introducing these concepts as part of translator training lies in having a set of specific tools that can serve as a playground on which translation students can gain hands-on experience, while also learning about the concepts related to mt. as pym (2013, p. 494) points out, "students should not learn just one tool step-by-step. they have to be left to their own devices, as much as possible, so they can experiment and become adept at picking up a new tool very quickly, relying on intuition, peer support, online help groups, online tutorials, instruction manuals, and occasionally a human instructor to hold their hand". however, a major roadblock when teaching technical content such as mt to non-technical audiences (like translation students) is a lack of suitable platforms that allow users to create mt engines without having to deal with low-level pro-gramming languages, unix consoles or command lines. this is the main motivation for prototyping m t 3 . ## support tools for teaching mt/pe to our knowledge, there are only few active opensource platforms that can be used for teaching mt: joey nmt @xcite , mtradumàtica @xcite and interactive teaching tool (itt) @xcite . however, they do not provide all the desired features, e.g. mtradumàtica and itt do not provide neural mt (nmt) models, nor the possibility to visualize intermediate results, and joey nmt does not have a gui to train and test models. a valuable resource for practicing post-editing and evaluating mt systems is pet @xcite , however, this tool does not offer an integrated mt module. using several tools in an unintegrated way may present new challenges to students, such as compatibility (e.g. file formats) or confidentiality issues. it would therefore be desirable to work on the same platform with intranet restrictions. the lack of free tools that meet our desiderata has fuelled the joint development of an opensource web platform called machine translation training tool -m t 3 . this platform aims at providing a playground for translation students and non-tech savvy users, helping them get hands-on experience with two main kinds of mt technology: statistical mt (smt), which is less used in the market, but is helpful for pedagogical and comparative purposes, and neural mt, which is the current state-of-art and is more difficult to teach since it involves more complex algorithms and intermediate processing steps. the main goal of this tool is to make abstraction of the technical details by letting users focus on the important processing steps and helping them understand the internals by visualizing intermediate processing results. additionally, we consider m t 3 to be a teaching aid that may be useful for developing and testing a syllabus that focuses on mt topics, including practical activities that can be carried out using this platform; hence, it might be possible to measure the acquisition of competences through selfefficacy measures @xcite @xcite . at the time of writing there is a similar initiative called multitrainmt -machine translation training for multilingual citithe first version of this tool was created almost three years ago @xcite . in this initial stage, the tool worked only as a desktop application and consisted of a python module that executed different scripts and processes in the background through a gui. this gui, shown in figure 1 , consisted of a series of tabs associated with the well-defined steps of creating a baseline engine with the moses toolkit. 5 each tab consisted of a set of fields where the user inserted the arguments; a button that activated the task in question, for which the internal engine executed one or more sub-programs; and an area on which the output of these sub-programs was printed. with this version, it was possible to train and evaluate statistical models, as well as post-edit the raw output by using a simple table with source and raw mt. the obvious drawbacks of this version are that the computer can become blocked due to the memory-intensive processes, and it is also highly dependent on electrical power (possibly for several hours or even days) to ensure that the computations that have been carried out are not lost. an improved version consisted of adapting the previous work to the web, while keeping the original distribution of tabs and a very similar look and feel. besides overcoming the roadblocks of a desktop version, this version also incorporated the use of docker @xcite to run all heavy processes on the server inside a docker container; this in addition to the above-mentioned functionalities, a new functionality was included to visualize the modifications made by the post-editor to the raw output, and some basic statistics, like the total amount of time spent on a segment and the number of edit operations, were also added. despite the improvements, some structural issues remained, since most of the work was done by students in computer science as part of their masters' graduation projects. this version provided the starting point for the current version, which is presented in the next section. ## current version of m t 3 for the new version, the back-end was completely re-designed to provide a more reliable infrastructure that would be suitable for neural models. the gui was also re-designed to include key functionalities that were missing in previous versions, such as user registration and authentication, or model sharing. figure 3 shows the new architecture of the server with the major additions: the use of several (as opposed to only one) dedicated persistent docker containers for special services (web api, database, task manager) and non-persistent containers for smaller tasks. the following sections provide some details about the server and the client (user gui). ## pilot use in the classroom in november 2019, a pilot test was carried out in the classroom as part of an mt course ("traduction automatique 1") at the fti-unige. this experiment was aimed at evaluating the performance and usefulness of the tool for mt classes. given that the tool is still under development, we decided to start testing parts separately and, at the time of the pilot, we focused on the functionalities related to statistical mt. as part of the development roadmap we will include a second pilot to test the nmt functionalities and add a post-editing tab. for the first pilot, students were provided with a publicly available statistical model previously trained by the lecturer for the language direction french-to-english. they had to explore the files generated during the training phase and perform quick translations (tests) in order to answer the following questions: at the end of the class, students were invited to complete a survey on their experience with the tool. the questionnaire addressed different aspects, such as the previous mt experience of each user. given that the users were translation students, technical concepts related to the software were presented in an accessible way, striving to maintain a balance between simplicity and precision. the survey was distributed via google forms, and the most relevant questions were: the following are the results of the survey, obtained from the responses received. we are aware that the number of respondents is small (9 valid replies in total) as the survey was not mandatory. nonetheless, it provides valuable initial feedback and the whole pilot served as an assessment of the development process, helping us to identify a roadmap for future use and development. of the total number of respondents, three had prior knowledge of mt and post-editing. all of them worked at least with the french-english language pair, while two of them also worked with the french-german language pair, two others with french-spanish, one with french-italian and one with french-russian. replies to question 4 indicate that 4 of the respondents had only seen the teacher using the tool, without using it themselves, while another 4 had used it in class. the final respondent said that he had also used it from home. as to whether the tool was useful in supporting the theoretical content, 3 agreed, while another 3 were neutral; 2 disagreed and 1 strongly disagreed. these results are shown in figure 8 . as to whether the interface was friendly and intuitive, 4 respondents agreed, while 3 disagreed, and 2 were neutral. these results are shown in figure 9 . regarding the reliability of the tool, 3 respondents considered the tool to be sufficiently reliable, as opposed to 1, who thought otherwise, while 5 respondents remained neutral, as shown in only one respondent reported having problems using the tool, particularly with the expiration of the session forcing him to log in again. finally, a student suggested highlighting some relevant files in the model, to avoid searching through all the files, which may generate some confusion. ## conclusions and future work we have presented a new version of m t 3 , an open-source web platform intended for teaching mt to translation students and others interested in the topic, who have little or no technical skills. it allows the students to create statistical models using moses and neural models using opennmt, which can be shared among other users. it also includes user management, such as authentication and a basic system of privileges to perform actions on certain resources. user activities, such as translations and evaluations, can be stored. in addition, it aims to show mt models as white boxes, enabling navigation and access to the files generated during the training process. we have also reported on a pilot experiment based on a real-life mt course for translators, given at the fti-unige. although the pilot gave us insights on the usefulness of the tool, the clear limitation was the small number of participants. the result of this experience was overall very encouraging, demonstrating that the tool can be installed and used to a certain extent as a support for practical activities in order to apply theoretical concepts. the work carried out so far was not intended to achieve a final definitive version, but rather an intermediate step towards a more reliable and extensible platform that will serve as a starting point for future developments. these include the visualization of data created during the training of neural engines, the integration of a post-editing tab and a better logging mechanism to provide more specific error messages. as for the utilization of the tool, we are planning another pilot to further test it with students. finally, we intend to make the platform available in the future to the translation teaching community.