An Empirical Study of Translation Hypothesis Ensembling with Large Language Models (2023)

Abstract: Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. In this paper, we investigate how hypothesis ensembling can improve the quality of the generated text for the specific problem of LLM-based machine translation. We experiment with several techniques for ensembling hypotheses produced by LLMs such as ChatGPT, LLaMA, and Alpaca. We provide a comprehensive study along multiple dimensions, including the method to generate hypotheses (multiple prompts, temperature-based sampling, and beam search) and the strategy to produce the final translation (instruction-based, quality-based reranking, and minimum Bayes risk (MBR) decoding). Our results show that MBR decoding is a very effective method, that translation quality can be improved using a small number of samples, and that instruction tuning has a strong impact on the relation between the diversity of the hypotheses and the sampling temperature.

## introduction significant research effort has been devoted to taskspecific neural machine translation (nmt) models trained in a fully supervised manner with large volumes of parallel data. their performance has been enhanced through techniques such as fine-tuning on in-domain data, model ensembling, and reranking during decoding @xcite . the recent achievements of general-purpose large language models (llms) such as @xcite offer a fresh perspective on the problem, demonstrating the feasibility of generating high-quality translations without explicit training for the specific task, even in a challenging zero-shot scenario @xcite . while techniques such as greedy decoding or sampling from the distribution often prove inadequate for generating translations with task-specific models, the same cannot be said for llm-based mt. there is, however, a lack of exploration in this case. our paper fills this gap by providing a comprehensive study on ensembling translation hypotheses ( §2), encompassing multiple llms such as chatgpt, llama, and the instruction-tuned alpaca @xcite . we consider different strategies to generate hypotheses (prompt-based ensembling, temperature sampling, beam search), and several techniques to produce the final translation, including choosebest, generatebest, reranking based on quality estimation, and minimum bayes risk decoding. the last two approaches have been successful at improving translation quality with task-specific models @xcite , but it is unclear whether the findings hold for llm-based mt. our main findings can be summarized as follows. first, we demonstrate that translation quality can be enhanced with a small number of samples (e.g., 20), especially when translating out of english (fig. 1 ). in the case of chatgpt, the cost in terms of paid tokens grows sublinearly with the number of samples ( §3.2). second, we show that similar findings apply to llama and alpaca ( §3.3). we discuss in which conditions beam search remains a reliable baseline for single-hypothesis translation ( §3.3.1) and how to ensemble translations ( §3.3.2). moreover, we find that there exists a significant gap in the quality of ensembles of unbiased samples from llama and alpaca ( §3.3.3). we attribute this disparity to how instruction tuning affects the relationship between the diversity of the hypotheses and the sampling temperature, which ultimately impacts translation quality. lastly, we show that hypothesis ensembling reduces the number of generated hallucinations, thereby improving the model's robustness to source perturbations ( §3.3.4). ensembling predictions and increasing the model size narrows the quality gap between these models and chatgpt. ## ensembling hypotheses ensembling has a long history in machine learning, being well known for leveraging multiple complementary systems to improve performance on a given task and provide good/robust generalization @xcite @xcite . while there have been efforts in prompt ensembling and multi-prompt learning within the context of llms, this area is largely unexplored for text generation tasks, where the output is a string rather than just a single token @xcite . see §4 for further details. in this section, we delve into the process of generating multiple translation hypotheses ( §2.1) and explore different methods for ensembling them to produce a single translation ( §2.2). 1 ## conclusions and future work we have conducted a thorough empirical analysis on various techniques for generating translations using llms. our study encompasses eight datasets and three model classes, including closedsource and open-source models, the latter with and without instruction tuning. we have demonstrated that ensembling predictions significantly enhances translation quality and reduces hallucinations under source perturbations. additionally, we have discovered that instruction tuning affects the relationship between the diversity of sampled hypotheses and the sampling temperature, which in turn influences the final translation quality. there are several avenues for future research in addition to the ones that have already been mentioned in previous sections. while ensembling predictions produced by llms is effective at improving translation quality, it also presents opportunities for developing improved methods of uncertainty quantification and calibration @xcite , crucial for addressing the inherent opacity of black-box llms.