Training and integration of neural machine translation with MTUOC (2023)

Abstract: In this paper the goals and main objectives of the project MTUOC are presented. This project aims to ease the process of training and integrating neural machine translation (NMT) systems into professional translation environments. The MTUOC project distributes a series of auxiliary tools that allow to perform parallel corpus compilation and preprocessing, as well as the training of NMT systems. The project also distributes a server that implements most of the communication protocols used in computer assisted translation tools.

## introduction to train statistical systems, a lot of ram is necessary. to train neural systems, one or more powerful gpus (graphical processing units) are essential. the mtuoc project offers solutions for the first two problems. regarding the technical skills it provides a number of easy-to-use python and bash scripts for corpus pre-processing and system training. all these scripts are well documented and can be easily adapted to specific needs. regarding integration issues, a fully configurable server is provided that supports many communication protocols. this ensures compatibility with numerous cat tools. for example, the server can use a neural engine based on marian, but behave as a moses server, so it can be directly integrated with omegat 1 , a very popular free cat tool. the project also provides a translation client that can translate text and files in a variety of formats, including xliff. the client can also generate translation memories in tmx with the content of the machine translation, which facilitates integration into almost all cat tools. finally, with reference to the hardware problem, it should be noted that the computing requirements for training are much higher than for translation with already trained systems. once a system is trained, it can be used for translation on any average consumer computer. in this way, many potential users can benefit from the various free engines distributed in the mtuoc project. to train customised systems, powerful gpu units are required. it is possible to purchase these units at affordable prices and install them in consumer computers. in addition, the uoc offers the possibility to establish technology transfer agreements for the training of customised systems at very competitive prices. this service is free of charge for ngos. since all the components of the mtuoc project are distributed under free license, any institution or company can offer customised training services using our components. ## components of the mtuoc project the mtuoc project offers a series of programs and scripts covering all steps from the creation of parallel corpora, their cleaning and corpus combination, to a program that allows to evaluate the trained systems using the most common automatic metrics. it also offers a series of scripts for corpus preparation and pre-processing and training using the toolkits presented in section 1.2. it also provides a complete server that is able to communicate with the servers provided by the main toolkits and to communicate with the client using a large number of protocols. the programs and scripts of the mtuoc project are programmed in python version 3 or are bash scripts. all components can be downloaded from the project's github page and are distributed under a free licence. in this section we present a brief description of each of these components. ## free mt engines the mtuoc project provides a growing number of freely downloadable mt engines. most of the engines are based on marian with a transformer configuration and guided-alignment. at the time of writing this paper, the following engines were available: the complete and up-to-date list of mt engines can be seen in the mtuoc project wiki. 25 4 conclusions and future work in this paper we have presented the mtuoc project, aiming to make the process of training and integrating nmt systems easier. with the components of this project, any professional or translation company will be able to use their own mt systems. one of the goals of the project is to make the developed tools usable for most users, regardless of their technical skills. we are now working on converting the scripts into programs with gui interfaces to make them even easier to use. this set of tools is being actively used in teaching activities at the bachelor and master levels at the uoc, and students with no previous knowledge of the use of terminal and scripts are able to perform all the processes involved in the training of statistical and neural systems. the tools are also being used in production environments, where custom nmt systems have been trained and used in translation projects. in the future we plan to train and make freely available more nmt systems. we also want to offer nmt systems for low resourced language pairs, starting for some romance languages in spain: asturian, aragonses and aranese.