Prefix Text as a Yarn: Eliciting Non- E nglish Alignment in Foundation Language Model (2024)

Abstract: While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely “superficial”. We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation. Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across seven languages demonstrate the efficacy of PreTTY in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can attain up to 98% of the performance metrics of their SFT counterparts. This method presents a cost-effective alternative to traditional SFT and advances the democratization of multilingual LLMs.

## introduction supervised fine-tuning (sft) refines large language models (llms) using task-specific instruction data to enhance their capability to follow instructions @xcite and to align their outputs with human preferences and safety considerations @xcite @xcite @xcite . this process is often termed "alignment", signifying the tailoring of model outputs to conform to specific downstream requirements. nevertheless, current research casts doubt on the necessity and potential adverse impacts of sft. but the alignment achieved through sft is often considered to be "superficial", with the process potentially repurposing pre-existing knowledge from pre-training to merely reshape outputs to meet specific criteria @xcite . it has been observed that even a small-scale sft training dataset can produce significant alignment effects @xcite . on the other hand, recent empirical studies @xcite have raised concerns that sft might hurt the knowledge acquired during its pre-training phase, leading to serious consequences like catastrophic forgetting. not only is there no definitive consensus on the necessity of sft, but the majority of these studies also focus on monolingual tasks. llms still encounter challenges in handling complex crosslingual generation tasks @xcite . current research on crosslingual alignment primarily seeks to extrapolate or align english capabilities to other languages using the sft paradigm @xcite @xcite , yet there remains a gap in exploring the specific impacts of sft-based cross-lingual alignment. furthermore, given the potential risk of sft leading to the forgetting of pre-training knowledge, the question of how to achieve cross-lingual alignment without training remains underexplored. to bridge these gaps, our study conducts an indepth examination of the impact of sft on crosslingual generation. we investigate the influence of sft on the decoding patterns of foundation models in cross-lingual contexts, hypothesizing that the success of sft largely hinges on the selection of initial prior tokens that are critical for eliciting taskspecific generation in the target language. furthermore, the observed decoding similarities between foundation and sft models support the extension of the superficial alignment hypothesis to crosslingual scenarios. responding to these insights, we introduce a training-free alignment method named "pretty" for cross-lingual and non-english tasks. the prefix texts act as a yarn (pretty) linking the foundation llm and the sft llm, eliciting the foundation llm to exhibit near-sft performance levels. specifically, we augment the original input with a few tokens that serve as decoding priors, and then prompt the foundation llm to resume decoding based on this modified input. in most cases, only one or two task-related prior tokens are needed, and the method for constructing these prior tokens is flexible across various kinds of language resources, fostering the democratization of multilingual llms. we conducted experiments on machine translation @xcite , cross-lingual summarization @xcite and non-english part-of-speech (pos) tagging @xcite tasks across eight languages. these tasks exemplify cross-lingual generation and multilingual language understanding, and they provide ample non-english test data to evaluate effectiveness across varying levels of resource availability. the experimental results demonstrate that pretty can ef-fectively align the foundation model to match sft model's performance without training, by merely adding two prior tokens in the decoding. ## experiments we examine the effectiveness of our proposed training-free alignment method on two distinct tasks: machine translation, cross-lingual summarization and non-english pos tagging. machine translation serves as a prototypical cross-lingual generation task, entailing the transformation of a sequence from a source language to a target language @xcite @xcite . as for cross-lingual summarization, it requires the model to generate a summary of an article in a different language @xcite . although pos tagging @xcite @xcite primarily assesses the model's ability to understand monolingual text, we include it as multilingual experiments to show the universality of our methods. ## conclusions in this paper, we investigate and analyze the decision-making discrepancies between the foundation model and the sft model within crosslingual generation contexts. drawing from our analysis, we introduce a novel cross-lingual alignment method that requires no additional training and is resource-efficient. the proposed method aligns the foundation llm to perform comparably with the sft model solely by utilizing prefix text as priors during generation. in the future, we aim to broaden our research to encompass additional alignment scenarios, such as those involving reinforcement learning from human feedback.