Respond in my Language: Mitigating Language Inconsistency in Response Generation based on Large Language Models (2024)

Abstract: Large Language Models (LLMs) show strong instruction understanding ability across multiple languages. However, they are easily biased towards English in instruction tuning, and generate English responses even given non-English instructions. In this paper, we investigate the language inconsistent generation problem in monolingual instruction tuning. We find that instruction tuning in English increases the models’ preference for English responses. It attaches higher probabilities to English responses than to responses in the same language as the instruction. Based on the findings, we alleviate the language inconsistent generation problem by counteracting the model preference for English responses in both the training and inference stages. Specifically, we propose Pseudo-Inconsistent Penalization (PIP) which prevents the model from generating English responses when given non-English language prompts during training, and Prior Enhanced Decoding (PED) which improves the language-consistent prior by leveraging the untuned base language model. Experimental results show that our two methods significantly improve the language consistency of the model without requiring any multilingual data. Our code, data, and models will be released.

## introduction large language models (llms) have received increasing research attention for their convincing language understanding and generation abilities @xcite @xcite . they also demonstrate intrinsic capabilities of multilingual understanding @xcite and cross-task generalization after instruction tuning @xcite . however, it is observed that llms tend to generate in the wrong language after monolingual instruction tuning @xcite . as shown in figure 1 , llama2-chat, which is an instruction-tuned model of llama2 @xcite in english, struggles to generate responses in the same/consistent language with the user instruction. to make llms follow multilingual instructions, many works @xcite @xcite have attempted to perform instruction tuning over multiple languages. despite their achievements, these approaches can be costly to implement, since they involve collecting multilingual instruction-tuning data and retraining the models over larger datasets @xcite . however, as shown in figure 1 , the model retains a notable degree of multilingual understanding even after monolingual instruction tuning, which suggests that the model should have the potential to follow multilingual instructions. therefore, instead of leveraging multilingual resources for finetuning, we focus on resolving the problem of language inconsistency in response generation caused by monolingual instruction tuning, thus enabling the model to follow multilingual instructions. to this end, we delve into the process of english instruction tuning, and find that this monolingual training process increases the model's prior probability of english responses. as a result, the model tends to assign a higher probability to english responses regardless of the instruction language. based on this observation, we propose two distinct methods, adopted during the training or inference stage respectively, to control the model's preference for english responses. specifically, we propose an auxiliary training objective pseudo-inconsistent penalization (pip) that penalizes the model from generating english responses for pseudo-inconsistent instructions. in the inference stage, we propose prior enhanced decoding (ped) that increases the language consistent probability from the untuned base model. we evaluate the proposed methods across multilingual instruction following, language understanding and machine translation. experimental results show that our proposed methods significantly increase the rate of generating language-consistent responses without hurting the ability of language understanding and machine translation. the main contributions of this work include: in contrast, we focus on addressing the language inconsistency in monolingual instruction tuning, and can enable multilingual instruction-following without the need for multilingual data. off-target problem a closely related topic to language inconsistency is the off-target problem, where multilingual machine translation models generate translations in the wrong language, disregarding the control signal @xcite @xcite @xcite @xcite . researches have been conducted to explain and solve the off-target problem. for instance, @xcite consider that the model learns spurious correlations between the control signal and decoded sentences. @xcite suggest that the offtarget problem stems from the encoder's failure to capture discriminative control signal and alleviate it by separating shared tokens across languages. zan et al. (2023) impose unlikelihood sampling on constructed off-target samples to prevent the model from generating in the wrong languages. @xcite use contrastive decoding @xcite to alleviate hallucination and offtarget problems in machine translation. the key difference between language inconsistency and the off-target problem is that language inconsistency refers to the model's failure to maintain the language of the input instruction without explicit specification. we focus on generating responses in consistent language without specifically identifying them in this paper. zero-shot cross-lingual generation zero-shot cross-lingual generation is another related topic to our study that concentrates on knowledge transferring across languages in generation tasks. it has been widely observed that zero-shot cross-lingual generation encounters challenges due to language inconsistent generation @xcite @xcite @xcite . research efforts have been made to address the issue. for example, maurya et al. (2021) and @xcite attribute this issue as catastrophic forgetting and alleviate it by freezing model components and parameter-efficient prompt tuning @xcite . @xcite find that monolingual training encourages the encoder to learn languageinvariant representations which are detrimental to cross-lingual generation. they suggest that incorporating an auxiliary language can help regularize the model and mitigate this issue. @xcite conducts experimental studies and suggests that a reduced learning rate can be beneficial for cross-lingual generation. these studies are conducted based on encoder-decoder models such as mt5 @xcite and nllb @xcite , lacking analysis of the current mainstream decoder-only large language models @xcite . in addition, their evaluation tasks contain only short multilingual sentences such as multilingual qa and summarization. in contrast, we focus on multilingual instruction following under the zero-shot setting, which presents a greater challenge in generating consistent and long-form text in multiple languages. ## conclusion in this paper, we investigate the language inconsistency problem in response generation based on large language models. we find that monolingual instruction tuning increases the prior probability of english responses, and thus leads to language inconsistent generation when providing instructions in other languages. to address this issue, we propose pseudo inconsistent penalization (pip) to prevent the models from preferring to generate english responses during training, and prior enhanced decoding (ped) to improve the likelihood of language-consistent responses during inference. experiment results demonstrate that our methods significantly enhance the language consistency rate without requiring any multilingual data while maintaining language understanding and machine translation capabilities of the large language models.