Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignment (2025)

Abstract: Ensuring consistent safety across multiple languages remains a significant challenge for large language models (LLMs). We introduce Soteria, a lightweight yet powerful strategy that locates and minimally adjusts the “functional heads” most responsible for harmful content generation in each language. By altering only a fraction of parameters, Soteria drastically reduces policy violations without sacrificing overall model performance, even in low-resource settings. To rigorously evaluate our approach, we also present XThreatBench, a specialized multilingual dataset capturing fine-grained harmful behaviors drawn from real policy guidelines. Experiments with leading open-source LLMs (e.g., Llama, Qwen, Mistral) show that Soteria consistently improves safety metrics across high-, mid-, and low-resource languages. These findings highlight a promising path toward scalable, linguistically attuned, and ethically aligned LLMs worldwide.

## introduction a major obstacle to robust multilingual safety lies in the limitations of early tokenizers @xcite , which were not designed properly to capture the rich morphological and script diversity in global languages @xcite . as a result, llms built on these tokenizers struggle to generate linguistically relevant and accurate outputs in non-english settings, undermining the effectiveness of any safety measures. while newer models incorporate more sophisticated multilingual tokenizers foot_0 , prior efforts largely treated multilingual support as an afterthought added later via fine-tuning rather than integrated as a core capability @xcite . this approach often relies on "bridging strategies," such as translating queries into english before applying moderation filters, a practice that can distort content classification @xcite . even extensive fine-tuning typically fails to address deeper, english-dominant architectural constraints, especially for languages with multiple scripts or highly complex morphology. moreover, creating large-scale multilingual datasets for each fine-tuning cycle is prohibitively expensive and time-intensive @xcite . although scaling up to larger-parameter models can bolster multilingual proficiency, such approaches may be infeasible in low-resource or time-sensitive contexts @xcite . building on these insights, we focus on recently introduced models, which offer improved multilingual capability. we curate a specialized dataset xthreatbench of prohibited categories, derived from meta's content guidelines to identify safety concerns more accurately. using this dataset, we propose soteria, a novel strategy for safe multilingual generation that locates language-specific "functional heads" and selectively tunes only about ∼3% of the model parameters. by redirecting these heads away from harmful outputs, soteria effectively suppresses toxic or policy-violating responses without degrading overall model performance. through this precise calibration of multilingual fluency and safety, we demonstrate that llms can be both linguistically adaptive and ethically grounded. our contributions are as follows. ☞ to the best of our knowledge, we are the first to introduce a multilingual parameter-efficient safety mechanism -soteria -that modifies only about ∼3% of the model's languagespecific "functional heads," effectively reducing harmful outputs without compromising overall performance. ☞ we introduce xthreatbench, a multilingual dataset covering harm categories derived from meta's content guidelines, closing critical gaps in existing safety benchmarks. ☞ our experiments encompass a broad linguistic spectrum from high-to low-resource to demonstrate that these safety enhancements are not confined to english or high-resource settings. ## methodology in this section, we present our methodology for identifying and mitigating harmful behavior in llms. we first introduce the underlying components of autoregressive llms (section 3.1), focusing on their transformer decoder layers and attention mechanisms. we then describe our framework (section 3.3) for identifying important attention heads that are crucial for task-solving and languagespecific processing, followed by the procedure to remove harm-inducing directions from these heads. ## language and dataset languages: following (deng et al., 2024a), we consider twelve languages across high-, mediumand low-resource categories. from the high-resource language category, we consider english (en), chinese (zh), german (de), french (fr), and spanish (es). for the medium-resource language category, arabic (ar), thai (th), bulgarian (bg), and hindi (hi). for low-resource language category, we include tamil (ta), bengali (bn), and telugu (te). datasets: we assess soteria using two established datasets, multijail @xcite and xsafety @xcite . in addition, we introduce a new multilingual safety dataset xthreat-bench, constructed based on the policy violations outlined by meta @xcite . a detailed description of each dataset follows. we include the dataset details of xsafety and the corresponding experimental results in the appendix d due to space constraints. multijail: this dataset is the first multilingual translated jailbreak benchmark designed to assess the safety vulnerabilities of large language models across multiple languages. it contains 3150 manually translated queries across 10 languages, covering high-resource (english, chinese, italian, vietnamese), medium-resource (arabic, korean, thai), and low-resource (bengali, swahili, javanese) languages. built from harmful queries in the gpt-4 report @xcite and anthropic's red-teaming dataset @xcite , it explores unintentional and intentional jailbreaks, where translation itself serves as a jailbreak method. for our experiments, we use google translate foot_1 to translate english queries into other languages when they are not present in the dataset. xthreatbench: to comprehensively evaluate multilingual safety vulnerabilities in llms, we introduce xthreatbench, a novel benchmark of harmful prompts grounded in real-world moderation policies. unlike prior resources that rely on direct translations of english queries, xthreatbench is systematically constructed to ensure policy alignment, adversarial robustness, and linguistic diversity across 12 languages. ## experimental setup in this section, we first introduce the language models used in our evaluation, selected for their multilingual capabilities and diverse linguistic distributions. next, we define our evaluation metric, attack success rate (asr), to quantify safety violations. subsequently, we describe the jailbreak attack baselines. to benchmark our proposed safety mechanism, we compare it against existing english language-centric safety alignment approaches. ## language universals we extend our experiments by applying the soteria framework across all languages together, rather than treating each language independently. however to do so, one needs to identify a set of attention heads that are active for all languages, i.e., capturing the universal characteristics of languages, aka language universals @xcite . for each language ℓ ∈ l , we first measure the average indirect effect (aie) of each attention head, aie ℓ (atn l i ), and select the top k heads based on these values. we then compile a consensus across languages by identifying the heads that rank in the top k for at least 75% of the languages. this majority-based criterion ensures that we capture heads consistently important across the different languages. finally, we use this refined set of heads in the harm-direction removal phase, thereby reinforcing the safety alignment in a way that remains robust across all the different languages. we call this version of the model sote-riau indicating its universal nature. results: we observe that the soteriau consistently produces lower asr compared to three base models across all tested languages and model backbones (see ## conclusion we introduce soteria, a lightweight yet powerful safety alignment method that fine-tunes languagespecific "functional neurons" in multilingual llms. by adjusting only a fraction of parameters, soteria effectively curbs policy violations across high-, mid-, and low-resource languages without compromising overall performance. our xthreatbench dataset, derived from real-world policy violations, demonstrates that this targeted parameter steering outperforms baseline safety approaches. these results highlight the value of language-aware interpretability and the practicality of scalable multilingual safeguards, advancing inclusive and ethically responsible ai.