SCALE : Synergized Collaboration of Asymmetric Language Translation Engines (2024)

Abstract: In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. By introducing translation from STM into the triplet in-context demonstrations, SCALE unlocks refinement and pivoting ability of LLM, thus 1) mitigating language bias of LLMs and parallel data bias of STMs, 2) enhancing LLM speciality without sacrificing generality, and 3) facilitating continual learning in a LLM-tuning-free way.Our comprehensive experiments show that SCALE significantly outperforms both LLMs (GPT-4, GPT-3.5) and supervised models (NLLB, M2M) in either high-resource or challenging low-resource settings. Moreover SCALE shows great scalability by only updating the lightweight STM and witness consistent system improvement, an averaged 4 BLEURT score across 4 languages without tuning LLM. Interestingly, SCALE could also effectively exploit the existing language bias of LLMs by using an English-centric STM as a pivot to conduct translation between any language pairs, outperforming GPT-4 by an average of 6 COMET points across eight translation directions. Furthermore we provide an in-depth analysis of SCALE’s robustness, translation characteristics, latency costs and inherent language bias, providing solid foundation for future studies exploring the potential synergy between LLMs and more specialized models.

## introduction large language models (llms) have recently revolutionized the field of natural language processing @xcite @xcite and brought a paradigm shift in machine translation (mt) by delivering exceptional performance without relying on bilingual data @xcite . moreover, as a unified multi-task learner, llms represent a substantial step towards artificial general intelligence @xcite , with the potential to transcend not only the language barriers emphasized by previous mt research but also cultural boundaries @xcite . despite their advancements, llm-based translation systems still confront several challenges. firstly, there exists a significant language bias towards english (e.g., 92.1% of the gpt-3 pretraining corpus is english, while french, the second largest, represents only 1.8%), which significantly constraints multilingual ability, especially for those low-resource languages @xcite @xcite . secondly, as the go-to approach for improved system performance, fine-tuning is non-trivial for llms due to (1) the trade-off between speciality and generality @xcite , and (2) the prohibitively high cost associated with tuning large-scale models. in contrast, traditional specialized translation models (stms)-those based on encoder-decoder architecture, trained with labeled data and significantly smaller in size @xcite -serve as specialists for specific translation tasks and could be efficiently fine-tuned. nevertheless, due to restricted model capacity, these models exhibit limitations in general language capabilities and may be prone to parallel data bias, such as the memorization of low-quality samples @xcite . in this paper, we demonstrate for the first time the possibility to unify these two asymmetric translation engines in a single framework. our work, scale, connects llms and stms by utilizing the llm's most enigmatic capability: in-context learning. rather than employing source-target pairs as in conventional few-shot translation @xcite 2023; @xcite , scale would first sample translations from a stm and then use triplets as incontext demonstrations consisting of a source sentence, an stm-generated set and a target sentence, which unlocks refinement and pivoting ability of llms. with scale, we could (1) mitigate both language bias of llms by utilizing an stm that concentrates on a specific language pair, and parallel data bias of stms by using a general-purpose llm as the main body of the system; (2) enhance the speciality of llms without compromising generality; our comprehensive experiments reveal that scale considerably outperforms llms (e.g., gpt-4) and stms (e.g., nllb) even in the challenging low-resource setting. moreover, in @xmath0, scale experiences consistent improvement by a 4 bleurt score without tuning llm and surpasses few-shot gpt-4 by 2.5 comet score and 3.8 bleurt score when equipped with a compact model consisting of merely 600m parameters. interestingly, scale could exploit the existing language bias of llms by using an english-centric stm as a pivot to conduct translation between any language pairs, outperforming gpt-4 by an average of 6 comet points across eight translation directions. furthermore, we conduct an in-depth analysis of the robustness, translation characteristics, latency costs and inherent language bias associated with scale. our findings provide valuable insights and encourage further research in the synergy between llms and small specialized models. ## the scale framework in this section, we present scale and provide an overview illustrated in figure 1 . popularized by gpt-3 (brown et al., 2020), in-context learning (icl) allows llms to perform a wide variety of tasks, even newly created ones @xcite , by providing a few demonstrations. for a translation task from a source language x to a target language y, an llm with parameters θ carries out icl by conditioning on k source-target paired examples @xmath1, y 1 )⊕(x 2 , y 2 )⊕...(x k , y k ) and the test source sentence x, generating the target y in an auto-regressive manner as y t ∼ p θ (y t |e, x, @xmath2. in this scenario, the llm must analyze the provided examples to discern the input distribution, output distribution, input-output mapping, and formatting to successfully complete the task @xcite . different from conventional icl, scale introduces an intermediate variable z as reference between source x and target y, transforming each demonstration example into a triplet (x, z, y). the variable z is a generation set sampled from a specialized translation model m @xmath3. the final input to the llm consists of the instruction, demonstrations, and source sentence combined in a prompt template: t ((x 1 , z 1 , y 1 )⊕(x 2 , z 2 , y 2 )...⊕ (x k , z k , y k )), (x, z)). unlike language understanding tasks that have fixed label set @xcite , the hypothesis space of translation model is actually infinite, so we could sample multiple generation paths from stm for one single source sentence to provide a more comprehensive generation guide for llm. the scale framework, though conceptually straightforward, demonstrates several advantages over stms and llms: refinement for x to y translation task, when the intermediate variable z is from m @xmath4, scale essentially conduct few-shot learning in a multi-task way by introducing an additional refinement task. refinement has long been proved effective in mt @xcite . in this refinement process, we pass sampled sentences and their confidence score (token level probability) from stm to an llm. the llm then digests the information carried by the sampled set and infers the generation space of the stm, which guides the llm to generate the output that is more consistent with the local data distribution @xcite . and since the final translation is delivered by an llm, scale could also mitigate the parallel data bias from stms and exhibit robustness by not merely copying and pasting the draft translation from stms as shown in §5.3. pivoting considering the predominantly english-centric nature of most llms @xcite , scale could employ an intermediate variable z from m @xmath5. and here z serves as a pivot point for llms to enhance their understanding of the source sentence and yield improved translations. this can also be regarded as a form of knowledge transfer from high-resource languages to low-resource languages @xcite . scalability a significant limitation of the existing llm-based translation systems is the inherent complexity of llm continual learning. this complexity arises from several factors, including the balance between speciality and generality @xcite , the catastrophic forgetting problem @xcite , and the substantial computational demands. in contrast, scale offers a more efficient and streamlined approach for scalability. by exclusively and effectively updating the lightweight m @xmath6, scale ensures that the llm remains untouched, thus preserving its general language capabilities. this selective updating process not only mitigates the issue of catastrophic forgetting but also reduces the computational burden of fine-tuning associated with large-scale models. ## related work the use of llm for translation tasks has garnered significant interest in recent times. @xcite initially demonstrated the efficacy of prompting an llm with a few examples to achieve noteworthy results, particularly in highresource languages @xcite . following the release of chatgpt, several studies have examined its overall translation performance @xcite , along with works focusing on the issue of hallucination @xcite , literalness @xcite , multilinguality @xcite and incidental bilingualism problem @xcite . a comprehensive analysis conducted by @xcite revealed the unreasonable effectiveness of few-shot llms. furthermore, a diverse range of research has attempted to enhance llm-based translation systems through cultural awareness @xcite , refinement @xcite , retrieval-augmentation @xcite , and comparison @xcite . our work also shares similarities with a series of studies aiming to build collaboration between llms and other systems. @xcite propose equipping llms with a knowledge-guiding module to access relevant information without tuning llm. @xcite propose to use microsoft translator system as the primary translation system, and then use gpt as a fallback system. @xcite introduce supericl and achieve significant improvements in various language understanding tasks. ## conclusion in this paper, we present a novel framework scale, which effectively combines the strengths of large language models (llms) and compact specialized translation models (stms) through in-context learning. by providing triplet in-context demonstrations, scale unlocks the refinement and pivoting capabilities of llms, demonstrated by comprehensive experiments in various settings. our results offer crucial understanding for subsequent research investigating the possible synergy between llms and more specialized models.