R e C o V e R the Target Language: Language Steering without Sacrificing Task Performance (2025)

Abstract: As they become increasingly multilingual, Large Language Models (LLMs) exhibit more language confusion, i.e., they tend to generate answers in a language different from the language of the prompt or the answer language explicitly requested by the user. In this work, we propose ReCoVeR (REducing language COnfusion in VEctor Representations), a novel lightweight approach for reducing language confusion based on language-specific steering vectors. We first isolate language vectors with the help of multi-parallel corpus and then effectively leverage those vectors for effective LLM steering via fixed (i.e., unsupervised) as well as trainable steering functions. Our extensive evaluation, encompassing three benchmarks and 18 languages, shows that ReCoVeR effectively mitigates language confusion in both monolingual and cross-lingual setups while at the same time—and in contrast to prior language steering methods—retaining task performance. Our data code is available at https://github.com/hSterz/recover.

## introduction large language models (llms) are becoming increasingly multilingual @xcite , progressively demonstrating more and more of their abilities across a broader set of natural languages. broadening language support, however, increases the risk of language confusion @xcite : the phenomenon where an llm answers in a language that is different from the language that the user-explicitly or implicitly-requested, or switches the language mid-reply. for instance, @xcite report gpt4o has a line-level pass rate of 88% for the cross-lc portuguese subset of the lcb. thus, a non-negligible proportion of the generated lines are not in the expected language. while language confusion is not a frequent phenomenon in all languages, it has a detrimental effect on the user experience. switching to a language that the user did not specify is likely to completely prevent the user from understanding the generated response. an inability to interact with users in their preferred language(s) hinders the global adoption of llms and-given that confusion is more prevalent for low-resource languages-exacerbates the exclusivity of state-of-the-art language technology. such inconsistencies also complicate evaluation, as metrics that assume stable language output may overestimate true model performance. intuitively, language confusion can be somewhat mitigated with in-context examples (i.e., few-shot in-context learning, icl) in the desired response language as well as via multilingual instructiontuning @xcite , but these assume existence of labeled data in each target language of interest. in contrast, @xcite propose an unsupervised inference-time intervention to mitigate language confusion based on language vectors that are added to hidden representations in the forward pass. while an inference-time solution, this approach suffers from two key drawbacks: (1) language vectors are computed jointly, i.e., dependent to one another, which means that all language vectors need to be recomputed from scratch when adding a new language; (2) while it improves language fidelity, it actually harms task performance (e.g., answer accuracy in question answering). contributions. in this work, we introduce a novel lightweight approach for reducing language confusion with language vectors, dubbed recover (reducing language confusion in vector representations), illustrated in figure 2 . we use multiparallel data to pre-compute (1) language vectors for each transformer layer (as average representations over all in-language input samples and all token positions) and then (2) language-agnostic, content vectors as averages of all language-specific vectors. by subtracting the content vectors from the language vectors, we obtain the language-specific representations (i.e., the steering vectors), which finally allow us to mitigate language confusion at inference time by means of simple arithmetic operations. we also experiment with trained interventions that learn how to compute the steering vector from the individual language vectors. conceptually, like the concurrent work of @xcite , recover adds (and subtracts) language-specific representations from intermediate token representations, but the way we compute and apply language vectors allows for seamless addition of new languages, without the need to recompute language vectors for all existing languages (see figure 1 for example generations of llama). our extensive evaluation focuses on language confusion in both monolingual and cross-lingual setups and encompasses 18 typologically diverse languages. recover effectively reduces language confusion across the board, and crucially-unlike the concurrent approach (yunfan et al., 2025)largely retains task performance. ## background and related work we first provide a brief overview of the body of work on manipulating llms via steering vectors and then describe the existing work on using steering vectors for mitigating language confusion. ## lightweight language steering we introduce recover (reducing language confusion in vector representations), a lightweight language steering approach that addresses the shortcomings of the existing approaches. recover uses a (readily available) multi-parallel dataset with a large language coverage to estimate languagespecific vectors and, unlike icl-and fine-tuning-based mitigation of lc @xcite , does not require labeled task data in target languages. recover computes language representations in relation to the average representations across all languages: as such, recover is able to support cross-lc with non-english prompts. importantly, recover steering vectors for new languages can easily be computed post-hoc, without the need to recompute the steering vectors of existing languages, which is one of the key drawbacks of lsi @xcite . steering comprises two sub-problems. the first one is isolating the vector that captures how a concept (in our case, a language) is encoded in llm's hidden representations ( §3.1). the second subproblem is designing the steering strategy, i.e., how to use the concept (language) vectors to manipulate the hidden representations so that the model generates text with the desired properties (in our case, in the desired target language). to this end, we explore both unsupervised ( §3.2) and supervised strategies (i.e., learning how to combine language vectors for optimal steering; §3.3). ## experimental setup obtaining steering vectors. to isolate the language representations r l ( §3.1), we need a multiparallel corpus d l : here, we resort to @xcite , a sentence-level multi-parallel dataset covering 200 languages. we additionally require a training dataset for our learnable steering function ( §3.3). this dataset should (1) contain both monolingual as well as cross-lingual prompt-response pairs and (2) cover a wide range of typologically diverse languages (and a wide range of tasks), in order to enable cross-lingual generalization of our learned steering function. we obtain such a dataset by first sampling 4400 single-turn instances (i.e., prompt-response pairs) from the instruction-tuning dataset tulu v3 @xcite and then translate each of them (with gpt-4o) by sampling the translation language for the prompt and (independently) for the response from the following set of languages: english, spanish, french, german, portuguese, russian, chinese, japanese, arabic, hindi, indonesian, hebrew, tamil, farsi, thai, polish, dutch, bengali. we increase the likelihood of sampling monolingual pairs to ensure that they are sufficiently represented. we deliberately exclude from translation four other languages present in the evaluation benchmarks (italian, korean, turkish, and vietnamese) in order to test our learned steering for (zero-shot) cross-lingual generalization. foot_0 for details on the dataset and translation quality see a.3. evaluation benchmarks. we employ three different benchmarks for measuring language confusion. two of them come with task-specific annotations, allowing us to also measure the impact of our language steering on task performance. language confusion benchmark (lcb) (marchisio et al., 2024) covers 15 languages and serves to measure language confusion. the corresponding metrics are line-level pass rate (lpr) and word-level pass rate (wpr). for lpr, the llm generations are split into lines and each line is classified by the language identification classifier; lpr is then simply the percentage of lines in the requested/expected language. wpr is the proportion of words in the correct lines (identified for lpr) in the correct script foot_1 . lcb consists of two portions: monolingual (llm expected to reply in the language of the prompt) and cross-lingual (llm instructed to reply in the specified language, different from the prompt language). multiq @xcite ) is a multi-parallel qa dataset covering 137 languages. we use it to evaluate language confusion and qa accuracy 4 . for the latter, as in the original work, we eval-uate with llm-as-a-judge (gpt4o). we select 5 multiq languages for the mono-lc evaluation. we also create cross-lingual instances with non-english prompts by using prompts in german, basque, farsi, french, swahili, turkish, and chinese, including the translation of the cross-lingual instruction 'answer in x!', with x as the target language. crosssum @xcite is a crosslingual summarization benchmark. encompassing more than 1500 language pairs. due to computational constraints, we limit our evaluation to four language pairs: english-spanish, spanish-french, french-turkish, and turkish-swahili. models. we experiment with three different open multilingual instruction-tuned llms of varying size and declared language support: llama 3.1 @xcite besides against the original model (i.e., without any language steering), we compare the performance of recover against the steering with lsi @xcite and language-specific neurons @xcite . for lsi, we obtain the language masks m l for with samples from wikilingua @xcite and for languages not covered by wikilingua, we use texts from wikipedia foot_3 . we provide further details on lsi parameters in ## results and discussion we first show and discuss the language confusion and task performance in mono-lc and cross-lc setups and then provide further analyses for some of the design dimensions for language steering. ## conclusion we introduced recover, a novel lightweight language steering approach. we first isolate language vectors using a multi-parallel corpus and then leverage those vectors for effective language steering of llms via (i) unsupervised steering arithmetic as well as (ii) learnable steering intervention, trained in a sample-efficient manner. our extensive evaluation, encompassing three benchmarks and 18 languages, shows that, in contrast to prior approaches, recover effectively mitigates language confusion without jeopardizing task performance. future work could leverage linguistic information to obtain more effective language steering, e.g., vectors conditioned by (i) syntactic roles, exploiting multilingual dependency parsers @xcite or (ii) typological features, e.g., uriel @xcite , to facilitate cross-lingual generalization (i.e., transfer) for learning steering vectors.