When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale (2024)

Abstract: Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.

## introduction the need for large supervised corpora remains a major bottleneck in neural machine translation (nmt) @xcite . sufficient bilingual data is scarce for most languages and limited to religious texts for the lowest-resource languages. to compensate for this lack of data, one effective approach is to leverage related parallel data from other languages via multilingual machine translation (mmt) that enables positive transfer from high-resource to low-resource languages @xcite . additionally, we can use monolingual data, either through pretraining with denoising autoencoding(dae; conneau and lample 2019; @xcite , or with backtranslation (bt; @xcite . driven by the success of these methods, recent works are converging toward a unified approach, that jointly trains mmt with monolingual data using auxiliary dae objectives @xcite @xcite and/or bt. however, the literature contains contradictory results about the effectiveness of these methods, particularly dae. early studies indicated combining mmt with dae led to improvements across all settings @xcite . these studies, however, were limited in scope, as they only considered moderately-sized models and used few languages (10 to 15), with training and test data drawn from similar domains. by contrast, @xcite found that dae helped only in very low-resource directions in mmt experiments with 200+ languages, while @xcite reported that dae produced mixed results in experiments with (mostly) african languages. to resolve this conflict, we present a systematic analysis of different methods that integrate monolingual data into mmt, focusing on bt and two dae objectives, mass @xcite and @xcite . first, we carefully investigate the role of the domain. to align with prior work, we focus on the englishcentric setting (i.e., concatenation of @xmath0. we use a realistic and diverse multilingual translation dataset with 100 directions and run controlled experiments using different monolingual splits with single-and mixed-domain data. then, we evaluate models across four widecoverage multilingual test sets from wikipedia, news, medical, and mixed domains. our results with medium-sized models (370m) show that while bt outperforms both dae objectives in most settings, the effectiveness of all methods varies signif-icantly, as they are surprisingly brittle to domain mismatches. bt is more sensitive to the domain than dae, and can underperform the parallel-only baseline when the monolingual and test data are not similar. however, increasing the diversity of the monolingual data by mixing different sources improves domain robustness to some extent. we also discover that both dae methods are less effective than previously reported, and they are mainly helpful in low-resource and @xmath1. of the two, mass consistently outperforms bart, although by a narrow margin. next, we study the role of model capacity and discover that it is crucial and can even change the ranking between methods. we hold all other factors constant and train models with sizes from 90m up to 1.6b parameters. when the scale is small, both bt and dae yield poor results, especially in out-of-domain settings. however, as model capacity grows, all methods quickly improve compared to the parallel-only baseline, and also become more robust to domain mismatches. scale affects dae the most, which transitions from underperforming the parallel-only baseline at the 90m scale to becoming competitive with bt at 1.6b and even outperforming it in low-resource. our contributions are: (i) we present a large-scale systematic analysis of how the domain and model scale affect the effectiveness of methods that incorporate monolingual data into mmt. (ii) we show that bt and dae are sensitive to domain mismatches between the monolingual and test data, particularly on small scales. bt is best in most settings. also, prior works have overestimated dae, and when comparing the two methods, mass outperforms bart. (iii) we discover that model capacity is key for the effectiveness of both methods, especially dae. when the scale is small, dae can even harm mmt, but it quickly improves with scale, and eventually becomes competitive with bt. ## related work monolingual data with multi-task learning early works on dae+mmt report universal gains in all settings. @xcite use wmt parallel data from 15 languages and large monolingual corpora from many sources, like news crawl, wikipedia, and common crawl, with mass. @xcite explore bart-like objectives with a subset of 10 languages from @xcite and news crawl monolingual data. however, more recent works that use larger and/or less uniform datasets, report less favourable results. to extend mmt to very low-resource languages, @xcite show that models learn to translate from/into languages with only monolingual data if there are sufficient parallel data in other languages to enable transfer from the dae to the mt task. @xcite explore a similar idea, but report that, in supervised translation, dae (bart) is effective only for very lowresource. @xcite compare all aforementioned dae methods and find that they often fail to outperform the parallel-only baseline. our study probes confounding factors in these prior works. large language models large language models (llms) trained on massive datasets achieve impressive results in many tasks @xcite @xcite . to adapt llms to downstream tasks including translation @xcite @xcite @xcite @xcite , the dominant approach is to use prompting, an ability enabled by model scale @xcite . our work, however, is orthogonal and presents an analysis of methods that integrate monolingual data into encoder-decoder mmt models trained from scratch. also, it is questionable whether these models are unsupervised with respect to translation, as recent work suggests that they have consumed parallel data during pretraining @xcite . model scale a growing literature investigates the scaling laws of different aspects of a model @xcite . in nmt, @xcite explore scaling laws related to model capacity, fernandes et al. (2023) ## (multi-task) multilingual nmt we follow the universal mmt training method of @xcite and train a single dense encoder ---@xmath2: illustration of the bart objective. transformer-based @xcite model on the concatenation of parallel data from multiple language pairs. we prepend a special token ⟨2xx⟩ to the source sequences, that informs the model about the translation direction (e.g., ⟨2es⟩ for spanish). ## experimental setup parallel data we use ml50 @xcite , a multilingual translation dataset between english and 50 other languages. ml50 is more representative of real-world multilingual datasets as it contains typologically diverse languages, including high, medium, and (extremely -less than 10k) low resource pairs, and with data from different domains. it is also more multilingual than the datasets from @xcite and @xcite , that use 15 and 10 languages, respectively. to reduce training time, we cap the parallel data at 10m sentences per language, similar to wang et al. 2020, which affects only few high-resource languages. ## conclusion this work presents a systematic analysis of widely used methods that include monolingual data in mmt, specifically bt, and two dae objectives. it does not negate findings from prior works but rather highlights confounding factors that explain the mixed results found in the literature. these factors range from the characteristics of the experimental setup, like the data mixture, to the effective model capacity. the main takeaway is that one should not expect gains from dae or bt in all settings but carefully consider all aspects of the system to reach optimal performance. we compare models across different data conditions and combinations of monolingual and test data, and discover that all methods are very sensitive to domain mismatches. bt overall yields the most gains, but it can fail in out-of-domain and lowresource settings. as for dae, we conclude that it can be helpful, particularly in low-resource and @xmath3, but the universal gains reported from early works can only be achieved in ideal conditions, where the parallel, monolingual, and test data are from the same domain. another key finding is that model capacity can make or break a method. larger models are better able to use monolingual data, with gains from both bt and dae increasing as the model scale grows. we also discover a novel connection between domain robustness and model size. scale is more important in out-of-domain settings, as all methods yield limited to no gains at small scales. in particular, mass is harmful to mmt with the 90m models, but when using 1.6b models, it becomes comparable or even better to bt. based on our findings, we provide some recommendations to practitioners: