Mixing and Matching: Combining Independently Trained Translation Model Components (2024)

Abstract: This paper investigates how to combine encoders and decoders of different independently trained NMT models. Combining encoders/decoders is not directly possible since the intermediate representations of any two independent NMT models are different and cannot be combined without modification. To address this, firstly, a dimension adapter is added if the encoder and decoder have different embedding dimensionalities, and secondly, representation adapter layers are added to align the encoder’s representations for the decoder to process. As a proof of concept, this paper looks at many-to-Estonian translation and combines a massively multilingual encoder (NLLB) and a high-quality language-specific decoder. The paper successfully demonstrates that the sentence representations of two independent NMT models can be made compatible without changing the pre-trained components while keeping translation quality from deteriorating. Results show improvements in both translation quality and speed for many-to-one translation over the baseline multilingual model.

## introduction as the availability of pre-trained models continuously increases, there is a growing need to investigate how to use them efficiently. previous works have looked at effectively using pre-trained neural machine translation (nmt) models by effective fine-tuning @xcite as well as using pre-trained language models in nmt model training @xcite @xcite @xcite . this paper examines the feasibility of combining together components (like encoders and decoders) of independent pre-trained nmt models without any retraining or fine-tuning. we investigate how representations of independently trained models can be made compatible and evaluate the resulting translation quality and efficiency. surprisingly, our evaluation shows that the resulting combined model can surpass the original models in translation quality and speed. combining any pre-trained encoder and decoder poses two problems. firstly, their representation spaces will not be compatible, as the models are trained independently. secondly, the embedding dimension of the representation can also differ across any two pre-trained models. we propose a method that solves both issues and allows the encoder and decoder of any pre-trained nmt models to be combined. specifically, in our architecture (figure 1 ), we use a small adapter to convert the dimensionality and representation space of the encoder to something the decoder is trained to process. in order for the adapter to learn its weights, the whole pipeline (encoder a -adapter -decoder b) is trained in an end-to-end fashion, except both the encoder and decoder are frozen. thus, the only part changing the weights is the adapter itself while the original components remain intact. as a proof of concept, we investigate combining encoders and decoders of multiple different pre-trained nmt models, focusing on an output language-specific scenario. in other words, a highly multilingual encoder is combined with a monolingual decoder, tuned to high performance on a single language. since highly multilingual models often suffer from the capacity bottleneck @xcite @xcite , we hypothesize that adding a high-quality languagespecific decoder can improve the translation quality to the language of the decoder. furthermore, translation to one language requires less capacity than many-to-many scenarios and thus would potentially require fewer parameters, resulting in faster translation. using nllb @xcite as the multilingual model and mtee @xcite as the language-specific estonian model, we demonstrate significant improvements in translation quality over the baseline nmt model for many-to-estonian translation and show competitive results to pivoting and fine-tuning. our method is not only effective to train compared to traditional fine-tuning but also provides a reduction in running costs of the translation model thanks to the number of parameters being reduced by 40% compared to the baseline nllb model. the main contributions of this work are: ## related work to the best of our knowledge, creating new nmt models by connecting encoders and decoders of different pre-trained nmt models has not been explored yet. similar approaches have been tested in speech translation @xcite . similarity between independently learned representations has been explored between linguistic, image representations as well as brain waves @xcite , however we attempt direct conversion and exploitation of these representations. ## conclusion we have demonstrated that different pre-trained models can be successfully combined even if they have different architectures that wouldn't be directly compatible. with our method, the pre-trained models can remain unchanged while the added dimension adapter and adapter layers align the embeddings. however, in our experiments, the best results were obtained by continuing decoder training after initial adapter training. this might differ in other scenarios depending on the dataset, pre-trained models, and desired translation domain. our method allowed for a 40% reduction in parameters, efficient training, fast translation, and increased translation quality compared to the original models. with this in mind, we can think of pretrained translation model encoders and decoders as modules that can be combined depending on the desired outcome.