YANMTT : Yet Another Neural Machine Translation Toolkit (2023)

Abstract: In this paper, we present our open-source neural machine translation (NMT) toolkit called “Yet Another Neural Machine Translation Toolkit” abbreviated as YANMTT - https://github.com/prajdabre/yanmtt , which is built on top of the HuggingFace Transformers library. YANMTT focuses on transfer learning and enables easy pre-training and fine-tuning of sequence-to-sequence models at scale. It can be used for training parameter-heavy models with minimal parameter sharing and efficient, lightweight models via heavy parameter sharing. Additionally, it supports parameter-efficient fine-tuning (PEFT) through adapters and prompts. Our toolkit also comes with a user interface that can be used to demonstrate these models and visualize various parts of the model. Apart from these core features, our toolkit also provides other advanced functionalities such as but not limited to document/multi-source NMT, simultaneous NMT, mixtures-of-experts, model compression and continual learning.

## introduction neural machine translation (nmt) @xcite is an end-to-end machine translation (mt) approach known for obtaining stateof-the-art results for a variety of language pairs. thanks to publicly available toolkits such as open-nmt, fairseq, tensor2tensor, etc.; for nmt, and natural language generation (nlg) in general, model training has become easier. additionally, fine-tuning of large pre-trained models such as mbart @xcite and mt5 @xcite have led to significant advances for lowresource languages. recently, the transformers library from huggingface has made fine-tuning an accessible option. parameter-efficient finetuning via adapters and prompts has recently gained popularity. this has led to the development of adapterhub, which allows people to use pretrained adapters for many tasks or easily train their custom adapters. however, when it comes to pre-training sequence-to-sequence models from scratch, we noticed that there is very little support foot_1 . this is, presumably, because pre-training is computationally intensive, especially when the number of parameters is large, and not everyone can do this. however, not all situations need large models, and with the advancements in low-precision training and parameter-efficient architectures, we feel that democratizing pre-training is necessary. additionally, relying on separate libraries for pre-training and fine-tuning can be quite exhausting. to this end, we decided to develop a publicly available toolkit to bring both into one place while simultaneously focusing on parameter efficiency. we call our toolkit yanmtt which stands for "yet another neural machine translation toolkit" which is built on top of the transformers library. yanmtt relies on a substantially modified version of the mbart model's code and contains simple scripts for nmt pre-training and fine-tuning. our modifications revolve around parameter efficiency by implementing heavy parameter sharing and incorporating adapters and prompts into the model. in order to enable users to better understand, and modify if needed, the flow of pre-training and finetuning, we heavily annotate our code with comments. yanmtt also comes with a user interface that can be used to demo any developed models as well as analyze them by visualizing their attentions and embedding representations. we hope that yanmtt will help entice more researchers into advancing the field of efficient sequence-to-sequence pre-training and fine-tuning. while the main focus is on nmt, readers should note that this toolkit can also be used for general purpose nlg. easy model demo, inspect visualization, inspect embeddings transformers modeling mbart mbart generation modified pytorch yanmtt pre-training fine-tuning generation user interface scratch or continual training, multi-gpu and multi-node distributed training, parameter efficient fine-tuning, model compression, tied layer modeling, flexibly decodable modeling decoding, attention and representation extraction, forced decoding, translation pair scoring ## related work tensor2tensor foot_2 is a deprecated library for training recurrent, convolutional as well as transformer models for a variety of sequence-to-sequence applications. it has been replaced by trax foot_3 . while ten-sor2tensor uses tensorflow as a backend, fairseq foot_4 is based on pytorch and it also allows one to train a variety of nmt models. unlike tensor2tensor, fairseq contains all necessary functionality for pretraining nmt models, but there is a severe lack of instructions for the same. opennmt @xcite , originally developed for recurrent nmt models, is based on tensorflow as well as pytorch. thumt foot_5 is an nmt training toolkit based on tensorflow, pytorch and theano. most recently, the transformers @xcite library by huggingface, based on pytorch and tensorflow has become popular as it allows users to share trained models easily. in transformers, the instructions for fine-tuning pre-trained models are abundant, but at the time of yanmtt's development (early 2021), there was no complete script for pre-training. on the huggingface hub, the central repository for all models trained with transformers, it is possible to load and run models, but enabling users to locally demo and inspect their own models is also important from the perspective of privacy. finally, for parameter-efficient finetuning, adapterhub @xcite builds on top of transformers and enables users to leverage existing or custom-trained adapters. all of the above toolkits and libraries are invaluable, but none appear to be a complete solution for sequence-to-sequence pre-training, parameter efficient fine-tuning and model demoing and inspection, a gap which yanmtt aims to fill. ## the toolkit: yanmtt yanmtt, yet another neural machine translation toolkit, relies on the transformers library and uses pytorch. we use only the mbart implementation (for now) from transformers and write several wrapper scripts enabling multilingual sequence-to-sequence pre-training, parameter efficient fine-tuning, decoding and attention and representation extraction. to enable users to quickly demonstrate and visually inspect trained models, we provide a user interface. we also modify the mbart modelling code to provide several advanced features. we provide the modified code along with our toolkit. we also provide example data and usage instructions in the form of example scripts. we encourage the reader to look at our toolkit foot_6 and watch the demo video foot_7 . figure 1 contains an overview of yanmtt architecture. ## conclusion we have presented our open-source toolkit called "yet another neural machine translation toolkit", also known as yanmtt. yanmtt allows users to pre-train and fine-tune their own multilingual sequence to sequence models. our toolkit can be used for training models at a reasonable scale, as well as to perform parameter efficient fine-tuning via adapters and prompts. we provide a convenient user interface for model demonstration and inspection, as well as the ability to visualize attention and model representations. we also implemented functionalities for compressing large models via selective parameter transfer and knowledge distillation approaches. additionally, we have provided basic functionalities for simultaneous and document/multi-source nmt. yanmtt appears to be modestly adopted by researchers, and we plan to further specialize it for better at-scale training and efficient fine-tuning.