UA lign: LLM Alignment Benchmark for the U krainian Language (2025)

Abstract: This paper introduces UAlign, the comprehensive benchmark for evaluating the alignment of Large Language Models (LLMs) in the Ukrainian language. The benchmark consists of two complementary components: a moral judgment dataset with 3,682 scenarios of varying ethical complexities and a dataset with 1,700 ethical situations presenting clear normative distinctions. Each element provides parallel English-Ukrainian text pairs, enabling cross-lingual comparison. Unlike existing resources predominantly developed for high-resource languages, our benchmark addresses the critical need for evaluation resources in Ukrainian. The development process involved machine translation and linguistic validation using Ukrainian language models for grammatical error correction. Our cross-lingual evaluation of six LLMs confirmed the existence of a performance gap between alignment in Ukrainian and English while simultaneously providing valuable insights regarding the overall alignment capabilities of these models. The benchmark has been made publicly available to facilitate further research initiatives and enhance commercial applications.Warning: The datasets introduced in this paper contain sensitive materials related to ethical and moral scenarios that may include offensive, harmful, illegal, or controversial content.

## introduction recent advancements in llms have demonstrated near-human proficiency across diverse domains, leading to widespread implementation in daily applications. this expansion has generated significant concerns regarding their ethical behavior and safety implications @xcite . consequently, the alignment of llms -ensuring that model responses are not only accurate and coherent but also safe, ethical, and aligned with the values of developers and users @xcite -has emerged as a critical research focus in recent years. however, most such studies have concentrated primarily on english or chinese languages. this imbalance introduces risk for all llm users @xcite , underscoring the necessity of extending llm alignment research beyond high-resource languages. to the best of our knowledge, no comprehensive benchmarks currently exist for evaluating llm alignment in the ukrainian language. to address this limitation, we introduce a novel benchmark designed to facilitate the standardized evaluation of ethical alignment for ukrainian language models. this benchmark comprises two principal components: 1,700 ethical scenarios and 3,682 social norms, adapted from established english-language datasets. ## related work the domain of llm alignment encompasses multiple dimensions and can be categorized into five distinct areas: factuality, ethics, toxicity, stereotype and bias, and general evaluation @xcite . each domain is represented by numerous benchmarks for english language evaluation, with the most prominent being truthfulqa @xcite , ethics @xcite , social chemistry 101 @xcite , @xcite , bold @xcite , and hh-rlhf @xcite . our comprehensive review of existing ukrainian datasets and adaptations of english datasets for low/mid-resource languages revealed limited resources in this domain: aya evaluation suite @xcite : this collection comprises 26,750 open-ended, conversational prompts for evaluating multilingual generation capabilities. the dolly-machine-translated subset includes 200 ukrainian-language examples. however, our analysis confirms the authors' obser-vations that the machine translation quality is insufficient for a meaningful evaluation of ukrainian language capabilities. please refer to appendix a. multilingualholisticbias @xcite and @xcite : these datasets adapt the holisticbias @xcite dataset to measure likelihood bias across language models. while reportedly including ukrainian language adaptations, these datasets are not publicly accessible, limiting their utility for comparative research. kornat @xcite : this benchmark evaluates llm alignment with korean cultural contexts through social values and common knowledge assessment. its creation methodology combines retrieval-augmented-generation (rag) with human-in-the-loop approaches, enhanced by multiple rounds of human revision to ensure quality and cultural relevance. ## benchmark development methodology our research prioritizes the ethics domain as the initial focus for ukrainian language evaluation due to its relatively concise textual components and inherent complexity. ethical reasoning necessitates comprehension of social norms and moral principles, which, despite cultural nuances, frequently present scenarios with broader cross-cultural interpretability. the development methodology, illustrated in figure 1 , comprises multiple sequential phases, including dataset selection, filtration procedures, and adaptation protocols. ## intended use the ualign benchmark is designed to facilitate several research applications: ## conclusion in this paper, we introduced ualign, the first comprehensive benchmark for evaluating llm alignment within the ukrainian linguistic context. the benchmark focuses on models' capabilities in understanding and evaluating ethical scenarios of varying complexity. we believe that it will become a cornerstone for llm alignment researches and will advance the ethical integration of artificial intelligence systems in ukraine. the benchmark is released under the mit license, ensuring accessibility for both academic research and commercial applications. looking forward, we identify two principal directions for future work: (1) enhancing benchmark quality through expert human curation and evaluation to improve both translation quality and and cultural relevance of ethical scenarios within the ukrainian context; (2) expanding the benchmark's scope to encompass additional dimensions of value alignment beyond ethical reasoning.