H u GME : A benchmark system for evaluating H ungarian generative LLM s (2025)

Abstract: In this study, we introduce the Hungarian Generative Model Evaluation (HuGME) benchmark, a new framework designed to assess the linguistic proficiency of large language models (LLMs) in Hungarian. HuGME evaluates models across a diverse set of linguistic and reasoning skills, including bias, toxicity, faithfulness, relevance, summarization, prompt alignment, readability, spelling, grammaticality, and domain-specific knowledge through tasks like TruthfulQA and MMLU. We applied HuGME to a range of Hungarian LLMs, including those developed in-house as well as several publicly available models that claim Hungarian language proficiency. This paper presents the comparative results of these evaluations, shedding light on the capabilities of current LLMs in processing the Hungarian language. Through our analysis, we aim to both showcase the current state of Hungarian linguistic processing in LLMs and provide a foundational resource for future advancements in the field.

## introduction language benchmarks are essential for evaluating the proficiency of large language models (llms). current benchmarks often overlook the specific requirements of languages like hungarian, especially in generative tasks. this study addresses the gap in existing benchmarks by focusing on a range of linguistic skills, including bias, toxicity, spelling, readability, and other aspects crucial for assessing llms. most tools are designed with languages like english in mind and do not perform adequately when applied to hungarian. our goal is to introduce a set of benchmarks tailored to hungarian. we evaluate various llms to see how well they manage these aspects, providing insights into their performance and highlighting areas that need improvement. ## related work state-of-the-art english-centric benchmarks, such as mmlu @xcite big-bench @xcite , and bbq @xcite , are widely used to evaluate the performance of generative language models. these are complemented by task-specific datasets, like e-bench @xcite , which assesses a model's ability to handle incorrect prompts, and truthfulqa @xcite , which focuses on the truthfulness of a model's output, as well as domainspecific benchmarks such as clinicbench @xcite , which evaluates model performance in clinical settings. beyond english, comprehensive and taskspecific evaluation frameworks are also emerging for a variety of languages, including korean (ko-dialogbench, @xcite , chinese (cdqa, @xcite , arabic (aradice, @xcite , and thai (thai-h6 and thai-cli, @xcite . benchmarks have also been developed for smaller languages, such as basque @xcite and norwegian @xcite , as well as for low-resource language groups, such as scandinavian @xcite , indonesian @xcite and iberian @xcite . however, many monolingual benchmarks are direct translations of their english counterparts, such as the dutch, spanish, and turkish versions of bbq @xcite , or @xcite , the finnish version of big-bench. as a result, they often lack tasks that address the cultural and linguistic subtleties specific to these languages. the same can be said about the practice of omitting country-specific sentences to ensure cross-lingual transferability, as in the case of veritasqa @xcite , the multilingual equivalent of truthfulqa. for hungarian, no dedicated comprehensive evaluation framework has been developed for generative language models so far. multilingual benchmarks such as alm-bench @xcite and mega @xcite are limited in scope, containing little hungarian data, or excluding the language entirely, which is also the case for mmmlu 1 and global-mmlu @xcite , the multilingual versions of mmlu @xcite . the only comprehensive hungarian benchmarks currently available are hulu @xcite , which primarily assesses language understanding and processing through classification tasks, and milqa @xcite , which focuses on question-answering. ## evaluated models in our evaluation, we assess a diverse set of large language models, including popular commercial models (e.g., gpt variants), open-source systems (e.g., llama and gemma models), models developed by hungarian enterprises, and our in-house models developed at hun-ren. ## results and discussion in the bias module, gpt models and the larger llama-based systems (such as llama-3.3-70b) demonstrated the strongest bias mitigation, whereas puli models generally struggled, suggesting potential issues in their training data. a similar trend was observed in toxicity detection, where gpt models led the performance, while puli models and some of the smaller llama versions exhibited comparatively weaker filtering capabilities. regarding relevance, both gpt systems and highparameter llama models maintained strong contextual awareness, in contrast to the puli models, which showed inconsistent performance, indicating difficulties in staying on topic. the gemma models, positioned between the small and large models, achieved competitive toxicity and prompt alignment scores but did not match the overall relevance and faithfulness levels of the top-performing systems. for faithfulness, llama-3.3-70b achieved a nearperfect or perfect score, while most other models in the truthfulqa module, llama-3.3-70b and gpt-4o-mini secured the top rankings, ensuring high factual accuracy, while puli llumix and sambalingo-7b performed less effectively, highlighting risks related to misinformation. finally, the mmlu evaluations demonstrated that llama-3.3-70b and gpt-4o-mini possessed the strongest domain-specific reasoning, whereas the puli models had a more limited grasp of broad knowledge areas. global observations indicate that gpt models consistently lead across most tasks, particularly in bias mitigation, toxicity filtering, instruction fol-lowing, and general knowledge. llama-3.3-70b emerges as a standout, rivaling gpt systems in faithfulness, grammatical accuracy, and domainspecific reasoning. in contrast, the puli models tend to struggle overall, especially in handling bias, summarization, and factual correctness. sambalingo-7b and -70b show mixed performance, with good results in faithfulness and relevance, yet falling short in factual accuracy (hutruthfulqa) and bias moderation. the novel puli llumix 3.1 model shows strong performance in hungarian linguistic aspects (spelling and grammaticality) but still has room for improvement in factual alignment. ## conclusion in this study, we introduced hugme, a comprehensive benchmark designed to evaluate the linguistic proficiency of hungarian large language models (llms) across various capabilities. hugme is the first benchmark that systematically assesses not only the factual accuracy and general performance of hungarian llms but also their linguistic competence, including spelling, grammaticality, readability, and their ability to follow prompts fluently in hungarian. 7 we applied hugme to a diverse set of models, ranging from hungarian-centric puli models to state-of-the-art gpt, llama-based, and intermediate-scale gemma systems providing a broad comparative analysis. our evaluation shows that gpt models generally excel in mitigating bias and filtering toxicity, as well as in maintaining high factual accuracy. large llama-based models (e.g., llama-3.3-70b) and our newly introduced puli llumix 3.1 model perform strongly in hungarian-specific linguistic aspects, such as spelling, grammatical accuracy, and readability. in contrast, the puli models, de-7 a part of the hugme benchmark and the expanded hungarian truthfulqa and mmlu datasets will be released under a cc-by 4.0 license. other parts of these data will not be publicly distributed to serve as evaluation tools. other datasets and models used in this study follow their respective original licenses. future work will focus on developing an inhouse judge model specifically optimized for hungarian. we also intend to extend the benchmark to more thoroughly test cultural knowledge. incorporating tasks that assess familiarity with hungarian proverbs, historical references, and other cultural artifacts will provide a more comprehensive evaluation of language models' capabilities in handling culturally rich content. finally, future iterations of hugme will integrate language exam tests derived from standardized hungarian assessments.