Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Models (2024)

Abstract: Massively multilingual machine translation models allow for the translation of a large number of languages with a single model, but have limited performance on low- and very-low-resource translation directions. Pivoting via high-resource languages remains a strong strategy for low-resource directions, and in this paper we revisit ways of pivoting through multiple languages. Previous work has used a simple averaging of probability distributions from multiple paths, but we find that this performs worse than using a single pivot, and exacerbates the hallucination problem because the same hallucinations can be probable across different paths. We also propose MaxEns, a novel combination strategy that makes the output biased towards the most confident predictions, hypothesising that confident predictions are less prone to be hallucinations. We evaluate different strategies on the FLORES benchmark for 20 low-resource language directions, demonstrating that MaxEns improves translation quality for low-resource languages while reducing hallucination in translations, compared to both direct translation and an averaging approach. On average, multi-pivot strategies still lag behind using English as a single pivot language, raising the question of how to identify the best pivoting strategy for a given translation direction.

## introduction early work on multilingual neural machine translation (nmt) has explored combining source segments in different source languages @xcite , an idea that is also compatible with pivoting through intermediate languages. for example, one could translate from dutch to ukrainian by first translating the dutch source to english and russian, and then making a combined prediction to ukrainian. in the simplest case, this combination is achieved by predicting probability distributions for each source language and averaging these predictions in an ensemble-like manner @xcite . with massively multilingual nmt models @xcite @xcite @xcite @xcite , one can in principle translate directly in any translation direction. while early models relied on zero-shot generalization for many directions, recent improvements include massive data collection efforts @xcite and synthetic data creation via back-translation @xcite . however, these models still have low performance on many low-resource translation directions foot_0 and pivottranslation via high-resource languages remains a strong baseline. @xcite also investigate the combination of multiple translation paths, which they call multi-source self-ensemble, that slightly improves over the direct translation and a single pivot for zero-shot language pairs. in this paper, we investigate this multi-source self-ensembling strategy more closely, with a focus on preventing completely defunct translations such as hallucinations. however, we find that simple averaging is sub-optimal and may increase the number of hallucinations in the output, a typical failure case in low-resource settings. we relate this to a recent finding that hallucinations are sticky, meaning that different models trained on the same data and architecture may produce similar hallucinations @xcite . we also find evidence of such stickiness when combining multiple translation paths, and propose a new ensembling strategy that, instead of averaging probabilities, picks the output with the maximum probability across different paths: maxens. this is partially inspired by the finding that model confidence is a good heuristic for avoiding hallucinations, which tend to be lowconfidence predictions @xcite . we perform experiments on the flores benchmark @xcite for 20 low-resource translation directions by using two massively multilingual nmt models, small100 @xcite and m2m100 @xcite . our results show that while the average ensemble outperforms the direct translation, it still underperforms using only english as a pivot, both in terms of spbleu and the number of hallucinations. maxens performs significantly better than the averaging strategy for both translation performance and hallucination. specially, maxens has competitive translation performance with english pivoting on average, but still lags behind it on the hallucination performance. to sum up, our contributions are: ## related work several approaches exploited different multipivoting methods to improve the performance of nmt models, specifically for low-resource language directions @xcite @xcite @xcite . @xcite analyzed the robustness of multi-source nmt in transcription errors. @xcite improved the performance of simultaneous nmt by translating the source language into pivot languages, then applying the multi-source translation method (zoph and knight, 2016). @xcite proposed a novel zero-resource translation approach by exploiting the multi-way multilingual nmt model, introduced by @xcite , and improved the performance over traditional pivot-based translation (wu and wang, 2007; @xcite . @xcite introduced the pivot-based nmt model by jointly training source-to-pivot and pivot-to-target directions. @xcite proposed an alternative method by applying a monolingual pivot-language data for zero-resource nmt via back-translation @xcite . ## ensembling methods when performing direct translation, the score of a translation y given a source sequence x src is computed as follows: for multi-pivot ensembling, we select a set of pivot languages @xmath0, µ 2 , ..., µ k } and generate the corresponding pivot translations the final translation is generated by ensembling predictions, conditioned on the individual pivot translations. in the following, we describe two approaches for such an ensembling: the multilingual averaging method and our maxens approach. ## conclusion we investigate more closely the multi-source selfensembling method of @xcite for combining multiple translation paths to improve translations of low-resource (or very-low) language pairs. specifically, this approach (named multiavg, here) averages the predictions of probability distributions of each source language in an ensemble-like manner. we evaluated it on 20 low-resource language pairs of flores-101 benchmark by using two massively multilingual nmt models, small100 and m2m100. the multiavg method performs better than direct translation in terms of both translation quality and hallucinations, however it lags behind applying only english as pivot. then, we proposed maxens method, a novel combination method that chooses the maximum of prediction probabilities of pivots for each designated target token. this approach results in a better translation quality compared to multiavg, while reducing hallucinations. on average, it achieves competitive performance with english pivoting with regard to the translation quality metric, but performs worse with regard to the hallucination metric. the most effective translation strategy varies depending on the translation direction, suggesting the need for future research to identify the optimal strategy for each direction independently of the specific development data. we hope our findings are a starting point for the broader integration of ensemble techniques within the context of massively multilingual nmt. the insights of our experiments, specifically the stickiness of hallucinations with different inputs, have inspired our follow-up work on source-contrastive decoding @xcite , which has empirically shown to be an effective strategy to mitigate hallucinations. future work could revisit multi-pivot ensembling in combination with source-contrastive decoding.