CM -Align: Consistency-based Multilingual Alignment for Large Language Models (2025)

Abstract: Current large language models (LLMs) generally show a significant performance gap in alignment between English and other languages.To bridge this gap, existing research typically leverages the model’s responses in English as a reference to select the best/worst responses in other languages, which are then used for Direct Preference Optimization (DPO) training.However, we argue that there are two limitations in the current methods that result in noisy multilingual preference data and further limited alignment performance: 1) Not all English responses are of high quality, and using a response with low quality may mislead the alignment for other languages. 2) Current methods usually use biased or heuristic approaches to construct multilingual preference pairs.To address these limitations, we design a consistency-based data selection method to construct high-quality multilingual preference data for improving multilingual alignment (CM-Align).Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction.Experimental results on three LLMs and three common tasks demonstrate the effectiveness and superiority of our method, which further indicates the necessity of constructing high-quality preference data.

## introduction although existing large language models (llms) generally support multiple languages, the performance across different languages is heavily imbalanced @xcite @xcite and typically shows significantly better general ability in english compared to other languages. the main reason is that english, as figure 1 : an overview of our method cm-align. given a question in different languages, we first request the model to generate multiple responses for each question. then we select the most consistent one among all english responses as en-reference, and calculate the cross-lingual consistency of multiple responses in other languages with en-reference to determine the chosen/rejected example for dpo optimization. the most popular language, benefits from abundant high-quality instruction tuning and preference data @xcite . however, collecting such high-quality multilingual datasets is challenging due to the expensive human annotation and inevitable translation errors @xcite . to address this problem, existing research @xcite uses the english responses from the model itself as the reference to select the chosen/rejected pairs for multilingual data and then utilizes direct preference optimization @xcite to achieve multilingual alignment. however, previous work usually treats all english responses as equally valid and utilizes unreliable metrics or heuristic methods to construct multilingual preference pairs, which may introduce non-negligible noise to the preference data and result in limited alignment performance. for example, mapo @xcite uses the translation probability to find the most similar/dissimilar mul-tilingual responses with the english responses to construct the chosen/rejected examples. however, due to the strong semantic constraint, translation probability may not be a suitable metric for quality in scenarios like open-ended generation, math, and code snippets. additionally, lidr @xcite directly takes the translated answers from english to other languages as the chosen examples and the original answers in other languages as the rejected examples. this heuristic design may introduce translation errors and translationese into the multilingual preference data. moreover, clr @xcite applies the self-rewarding method @xcite to the multilingual scenario, which requires another aligned llm to calculate implicit cross-lingual dpo rewards. although intuitive, english-centric llms may not be well-aligned in cross-lingual scenarios and thus offer inaccurate dpo rewards. to solve these limitations, we design a consistency-based multilingual alignment method (cm-align), which constructs high-quality multilingual preference data for dpo training (as shown in figure 1 ). our method improves the quality of multilingual preference data from two perspectives: 1) consistency-guided english reference selection to find the reliable english anchor, and 2) crosslingual consistency-based preference data construction with task-specific metrics. specifically, for each prompt, we first sample multiple responses in english and select the one that is most consistent with others as en-reference. then we choose the responses in other languages that are most consistent with en-reference as the chosen example and the most inconsistent response as the rejected example. particularly, for different tasks, we design taskspecific criteria to better measure the consistency between different responses. experimental results on math, code, and general instruction following tasks demonstrate the effectiveness and superiority of our method. moreover, our method not only shows better alignment performance for in-domain languages but also exhibits excellent generalization for out-of-domain languages, which proves that different languages can learn collaboratively for llm alignment. in summary, the major contributions of this paper are as follows 1 : reinforcement learning from human feedback. rlhf @xcite @xcite is the critical technique to align llms with human preferences and values. among multiple implementations, direct preference optimization @xcite streamlines the alignment process by directly optimizing the llm policy from preference data, which needs fewer computational requirements and shows comparable effectiveness compared to classical rlhf methods. recently, some works @xcite @xcite @xcite modify the training objectives to improve the performance of dpo. additionally, other works @xcite @xcite focus on the data selection method to improve the training performance. however, these methods only concentrate on the alignment for english without considering the challenges of multilingual scenarios. additionally, some works @xcite try to utilize the dominant english to assist the construction of multilingual preference data. mapo @xcite adopts the nllb model to calculate the translation probability scores of the answers in non-dominant languages and english, and chooses the answer with the highest/lowest score as the chosen/rejected example. lidr @xcite leverages the characteristics of translation, i.e., the answers translated with the model itself from english to nondominant languages as chosen examples and the original answers in non-dominant languages as rejected examples. clr @xcite adapts the self-rewarding method to the multilingual scenario, which needs another sft model to calculate implicit cross-lingual rewards. although these methods have an initial attempt, we argue that the native translation rewards or cross-lingual rewards are unfaithful, further leading to low-quality preference data and underperformed multilingual alignment performance. in this paper, we aim to select high-quality preference data to improve multilingual alignment. ## conclusion in this paper, we design a consistency-based data selection method to construct high-quality multilingual preference data for improving multilingual alignment. specifically, our method includes two procedures: consistency-guided english reference selection and cross-lingual consistency-based preference data construction. we conduct extensive experiments on three llms with different model sizes and three common tasks (math, code, and general instruction following). the experimental results demonstrate the effectiveness and superiority of our cm-align.