I Have an Attention Bridge to Sell You: Generalization Capabilities of Modular Translation Architectures (2024)

Abstract: Modularity is a paradigm of machine translation with the potential of bringing forth models that are large at training time and small during inference. Within this field of study, modular approaches, and in particular attention bridges, have been argued to improve the generalization capabilities of models by fostering language-independent representations. In the present paper, we study whether modularity affects translation quality; as well as how well modular architectures generalize across different evaluation scenarios. For a given computational budget, we find non-modular architectures to be always comparable or preferable to all modular designs we study.

## introduction machine translation (mt) has historically been under two influences that seem a prima facie contradictory. one of the goals of mt research is to provide means of converting sentences from any language to any other. on the one hand, generalization capabilities hinge on our systems producing language agnostic representations. on the other hand, mt models ought to be apt at encoding the specifics of source languages @xcite . the former of these trends has deeply marked this field-the concept of an 'interlingua' runs through most of the history of mt research, from @xcite to @xcite . the latter has recently motivated the development of modular approaches, where network parameters are specifically tied to a specific language. how can we reconcile these two seemingly paradoxical trends? one promising approach is the inclusion of fully-shared subnetworks in modular architectures, and especially bridge components: they have been argued to foster languageindependent representations @xcite as well as zero-shot generalization capabilities @xcite . our aim is to carefully assess whether modular architectures in general and bridges do indeed foster greater generalization capabilities. we therefore study six architectures, five of which modular, with a particular focus on how they generalize-both to unseen translation directions, and to novel domains. we find that modular systems still struggle to remain competitive with fullyshared mt systems in scenarios when not all translation directions are available-a conclusion that affects systems with and without fixed-size bridges equally. while encoder-sharing modular designs can rival or outperform non-modular settings in a wide range of scenarios, all other systems we study struggle in zero-shot and out-of-distribution conditions, strongly questioning that fully-shared sub-networks in modular mt systems can improve their generalization capabilities. ## related work the full span of multilingual nmt (mnmt) architectures rely in the implicit assumption that the systems leverage the multilingual data by creating a shared encoding space via sharing: from fully-shared models @xcite , to fullymodular systems, where sharing occurs only at dataset level @xcite . in this work, we assess those two extreme cases, focusing in the modular nmt systems that incorporate some parameter-sharing bridging layers. @xcite introduced an attentional neural interlingua, which processes language-specific encoder embeddings to produce language-agnostic representations. @xcite proposed a language-aware interlingua that transforms the encoder representation to a shared semantic space, showcasing practical means of fostering the semantic consistency of translations. @xcite integrated a shared inner-attention mechanism, referred to as "attention bridge", based on the work of @xcite , to generate fixed-size sentence representations. fur-ther studies by @xcite and @xcite , whose work we specifically build upon, emphasized the advantages of using multiple attention heads on the semantic quality of the translation-as well as challenges, particularly with translating longer sentences. @xcite explored the effects of sharing encoder parameters vs. increasing the number of languages in modular mnmt. more recently, @xcite used layers shared by language groups to enhance translation, @xcite proposed a variable-length bridge that uses a classification layer to predict its length, and in pires et al. (2023) the encoder is built with interspersed fully-shared and language-specific layers. ## results the primary metric used for evaluating the performance of our models is bleu @xcite . 3 results are shown in ## conclusions in this work, we study the claim that bridge layers in modular architectures foster greater generalization capabilities. given a carefully controlled computational budget, bridge architectures never clearly outperform bridge-less architectures, be they modular or not. in particular, we find nonmodular architectures exhibit strong competitiveness, as they are only outperformed by modular architectures with language independent encoders and modular language-specific decoders. additionally, we note that training conditions, such as the translation direction accessible to a model during training, have a significant impact. these results suggest that current modular architectures, especially those using bridging layers, have limited potential insofar mt is concerned. in most cases, a default non-modular transformer fares better or just as well than the most effective modular system. our study focused on modular architectures in a small-scale, well controlled experimental protocol; we leave questions such as whether these remarks carry on at a larger scale, both of model parameter counts and number of languages concerned, for future work.