ITALIC : An I talian Culture-Aware Natural Language Benchmark (2025)

Abstract: We present ITALIC, a large-scale benchmark dataset of 10,000 multiple-choice questions designed to evaluate the natural language understanding of the Italian language and culture. ITALIC spans 12 domains, exploiting public tests to score domain experts in real-world scenarios. We detail our data collection process, stratification techniques, and selection strategies. ITALIC provides a comprehensive assessment suite that captures commonsense reasoning and linguistic proficiency in a morphologically rich language. We establish baseline performances using 17 state-of-the-art LLMs, revealing current limitations in Italian language understanding and highlighting significant linguistic complexity and cultural specificity challenges. ITALIC serves as a benchmark for evaluating existing models and as a roadmap for future research, encouraging the development of more sophisticated and culturally aware natural language systems.

## introduction and motivation the rapid advancement of large language models (llms) has significantly transformed the field of natural language processing, with models now exhibiting impressive capabilities across a wide array of tasks @xcite . as these models approach near-human-level performance in various domains, there is a rising need for robust and comprehensive evaluation methods. assessing model performance is crucial yet challenging, as multiple key factors must be considered. these include the model's accuracy, robustness, fairness, and computational efficiency, among others @xcite . the creation and assessment of language models capable of operating proficiently in multiple languages worldwide continue to pose substantial difficulties for researchers @xcite . the document by the italian government on the strategy for ai for 2024 to 2026 foot_0 highlights the risk of cultural homogenisation when using anglocentric llms. the currently available models often perform highly in english but are lacking in underrepresented languages @xcite . this is due to factors such as the scarce and lower quality available data @xcite , smaller contributing communities, and anglo-centric cultural bias in development @xcite . evaluation datasets for italian are often insufficient, as they are translated from english rather than properly localised. moreover, specifically, italian datasets tend to focus on simple tasks, lacking coverage of common sense and cultural knowledge. to evaluate whether an llm aligns with italian culture, we propose italian cultural benchmark (italic), a novel dataset using a set of cultural questions taken from exams for public competitions in italy. these tests assess candidates' cultural knowledge, and applying them to llms would provide a useful tool to identify models not aligned with italian culture. fig. 1 shows illustrative examples from italic. standardised aptitude and knowledge assessments are used in public domains such as university admissions, military recruitment, public sector positions, and medical licensure examinations, among others. the primary aim of these tests is to evaluate applicants' domain knowledge, general knowledge, and linguistic and analytical capabilities, which are deemed essential skills that public institutions and employers prioritise by italian national standards requirements. these assessments are crucial for a merit-based selection process across diverse fields and positions @xcite . in addition to evaluating general knowledge and analytical skills, many standardised assessments include a dedicated section focused on language and culturally relevant topics, reflecting italy's rich cultural heritage. it is an essential criterion for public and professional roles where cultural awareness and communication skills are paramount, and candidates must be aligned with the cultural norms and values embedded within the italian professional and public spheres. since these culturally relevant sections present a unique challenge for language models, our study seeks to create a standard evaluation suite for advanced large language models by utilising the structure of italian public exams as a comprehensive testing framework. the tests' structured and standardised nature makes them an excellent benchmark for comparing different llms with questions that are contextually relevant to italian culture. ## related work large language models are deep learning models trained on vast amounts of multilingual text data, capable of understanding and generating language in a sophisticated manner @xcite . while exhibiting emerging capabilities across multiple languages such as german, french, spanish, and italian, multilingual models do not perform as well as in the primary training language @xcite . therefore, it is essential to evaluate their performance using language-specific metrics. available english benchmarks. several benchmarks have been developed to assess language models' general intelligence and reasoning capabilities. notably, mmlu (massive multitask language understanding) @xcite is a comprehensive benchmark that evaluates language models across 57 diverse subjects, including stem, humanities, and social sciences. agieval @xcite , on the other hand, is designed to assess the capabilities of models in artificial general intelligence across a broad range of tasks and skills. cultural benchmarks. cultural benchmarks have emerged as crucial tools in evaluating the adaptability and fairness of language models across different social and cultural contexts @xcite . several studies have aimed to quantify how well these models perform across diverse demographic groups and cultural settings. normad (norms and adaptability of language models) @xcite focuses on the cultural adaptability of llms. by utilizing 2,600 stories from 75 countries, normad assesses how well these models understand and replicate social norms and etiquette as proxies for cultural comprehension. in a complementary effort, the blend benchmark @xcite expands the scope by offering a dataset of 52.6k question-answer pairs spanning 13 languages and 16 countries/regions. this large-scale study reveals a significant imbalance in how llms process cultural knowledge, with models often performing better on questions related to highly represented cultures. blend highlights the need for more equitable training approaches that ensure underrepresented cultures are not neglected in model performance. lastly, click (cultural and linguistic intelligence in korean) @xcite provides a localised examination by testing models on their understanding of korean culture and language. based on 1,995 qa pairs from official exams and textbooks, the benchmark covers 11 categories related to korean cultural knowledge, thereby addressing the linguistic and cultural nuances specific to the region. each benchmark focuses on a different aspect-social biases, adaptability to diverse social norms, cultural representation, or linguistic specificity-yet they all contribute to a broader understanding of the limitations and potentials of llms in culturally complex environments. while the above benchmarks have been instrumental in advancing the field, they primarily focus on english or specific cultural contexts. some may include some tests in italian but are not focused on italian cultural commonsense and language capabilities. italian benchmarks. compared to the english community, the italian nlp community lacks the depth of original language evaluation benchmarks. @xcite present a unified benchmark for italian natural language understanding, which includes tasks such as textual entailment, event detection and classification, factuality classification, sentiment polarity classification, irony detection, and hate speech detection. @xcite propose a collaborative benchmark covering 13 tasks. despite their contributions, both benchmarks mainly address classification-based tasks and do not examine llm capabilities like commonsense reasoning. @xcite provides an italian benchmark for news text summarization. @xcite introduces an adaptation of the invalsi standardised tests for the automated evaluation of llms, aiming to improve the assessment of these models within diverse italian linguistic and cultural contexts. our work aims to build upon these foundations by developing a specialised benchmark tailored for cultural knowledge assessment specific to italian. italic contains 10,000 carefully curated questions selected from an initial corpus of 2,110,643 questions. the dataset curation process overview is summarised in fig. 2 . the initial data was sourced from various files in pdf, html, doc, and other formats published by official bodies that announce individual competitive public examinations 3 . the preprocessing process involved standard crawling techniques, including rate-limited requests to ensure politeness and acquire publicly available documents from official governmental sources and public registries. document collection was followed by format conversion from various source formats (html, pdf) using regex and standard libraries. the corpus comprises questions and tasks from real-world exams, professional assessments, and domain-specific challenges. this ensures the dataset reflects the knowledge and skills required in italy's various professional and academic contexts, like medical and military. given that the data originates from institutional sources, it is expected to maintain a high standard of quality and accuracy, as domain experts crafted it for public evaluations. the collected data, including the questions, answers, relevant metadata, and associated images, were normalised into json. each entry follows a standard multiple-choice format with a single correct option, providing a clear structure for model evaluation. ## results and discussion in this section, we present the performance of llms on the italic benchmark, assessing their proficiency in italian cultural and linguistic aspects. we evaluated the open-source models on an nvidia a100 80gb pcie gpu. for this evaluation, we utilised vllm with its default openaicompatible server, running models in bf16 format. the entire process, including setup and evaluation, took approximately 5 hours. closed-source models and models with over 70 billion parameters were accessed from api services, costing around 100$. tab. 2 shows the accuracy (%) of each model across several categories within the italian cultural and linguistic context. ## conclusion and future work italic represents a comprehensive and nuanced benchmark for evaluating italian language understanding and cultural knowledge in ai models, providing diverse tasks across multiple domains and a robust tool for assessing and driving progress in italian nlp. our evaluation of 17 state-of-the-art llms revealed that while models perform reasonably well in commonsense and culturally relevant tasks, they face significant challenges in mastering the linguistic intricacies of italian, particularly in tasks involving morphology, syntax, and orthography. we observed a consistent improvement in few-shot settings compared to zero-shot, demonstrating that providing task-specific examples greatly enhances model performance. italic highlights several areas for improve-ment in italian language models, such as enhanced cultural training data to capture better italianspecific knowledge and contexts and specialised fine-tuning for professional domains relevant to the italian market. as the field continues to advance, italic will serve as a valuable resource for researchers and practitioners working on italian language ai, helping to ensure that these technologies can effectively serve the needs of italian-speaking users and applications.