O pen H u E val: Evaluating Large Language Model on H ungarian Specifics (2025)

Abstract: We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs’ generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRMs with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at https://github.com/opendatalab/OpenHuEval .

## introduction recent advancements in large language models (llms) @xcite represent significant strides toward artificial general intelligence (agi). however, notable performance gaps remain between english and other languages in both language-agnostic tasks (e.g., math reasoning, code generation) (huang et al., 2025; zhang * equal contribution. to address this gap, we introduces openhueval, the first evaluation benchmark for llms focused on hungarian language and specifics. the comparison of openhueval with the existing related benchmarks is shown in v data construction process openhueval automatic quality check of questions by llm tasks emphasizing generation capabilities with comprehensive evaluation method similarity matching rule llm-as-judge hungarian specific dimension comprehensive coverage of hungarian language scenarios gyakorikerdesek.hu mek.oszk.hu official hungarian education website and real user forum nkp.hu hu.wikipedia • huwildbench • husimpleqa • huproverbrea • humatchingfib • hustandardfib obtain questions from multiple data sources manual review of question quality gpt4o language; history; life,culture and customs; education; geography and place; figure; politics policy and law; business and finance hungarian annotator multiple types of tasks ing a variety of scenarios that users may encounter when querying in hungarian. guided by these dimensions, we collected a vast amount of hungarian specific material from multiple sources and used this to construct the corresponding evaluation tasks. based on openhueval, we evaluated the performance of mainstream llms on hungarian language and specifics. we compared the performance differences of these models on the typical datasets of openhueval with corresponding datasets in other languages. the results indicate a significant necessity for evaluation and model optimization specifically for hungarian language and specifics. moreover, large reasoning models (lrms), like o1, mark a new direction in llm development. through extensive reasoning, self-reflective negation, and exploring multiple reasoning paths, they greatly improve reasoning abilities, adhering to the test-time scale law-a crucial step toward agi. recent studies @xcite have analyzed these reasoning processes, offering insights for optimization, but have largely focused on englishlanguage contexts, neglecting hungarian language and specific scenarios. building on openhueval, we developed the framework for dissecting the reasoning processes of lrms. using hungarian as the example, we uncovered intrinsic patterns of the representative lrms in non-english languages. these findings provide valuable insights for the research community to further advance the development of lrms. in summary, the contributions of this paper include the following three points: ## openhueval openhueval is the benchmark specifically designed to evaluate the performance of llm in handling hungarian language and specifics. the overview of openhueval is in figure 1 . examples are shown in figure 2 . ## framework for analyzing the thinking process of lrm when responding to the user's query, the lrm's response typically consists of two parts: the thinking process and the answer. 8 we developped the framework for the in-depth analysis of the lrm's thinking process on openhueval. for lrm, we chose deepseek-r1 and qwq, as these are the only two models with accessible reasoning processes. ## conclusion in this paper, we constructed the first benchmark for llms focusing on the hungarian language and its specifics. the results highlight the significant need for evaluation and model optimization tailored to hungarian language and specifics. we also developed the framework for analyzing the thinking processes of the cutting-edge lrms. our work not only advances llm technology in hungarian but also provides valuable insights for studying languages of other countries and regions.