I ta E val: A CALAMITA Challenge (2024)

Abstract: In recent years, new language models for Italian have been spurring.However, evaluation methodologies for these models have not kept pace, remaining fragmented and often limited to the experimental sections of individual model releases. This paper introduces ItaEval, a multifaceted evaluation suite designed to address this gap. By reviewing recent literature on the evaluation of contemporary language models, we devise three overarching task categories—natural language understanding, commonsense and factual knowledge, and bias, fairness, and safety—that a contemporary model should be able to address. Next, we collect a set of 18 tasks encompassing existing and new datasets. The so-compiled ItaEval suite provides a standardized, multifaceted framework for evaluating Italian language models, facilitating more rigorous and comparative assessments of model performance. We release code and data at https://rita-nlp.org/sprints/itaeval.

## mc ## mc ## mc ami20 m la seguente frase è misogina? rispondi sì o no.\nfrase: {{text}}\nrisposta: a collection of about 5,400 anonymized tweets annotated with binary labels for misogyny (referred to as "ami20 m" in our experiments) and aggressiveness (referred to as "ami20 a" in our experiments)-e.g., non ci posso credere sono queste le cose che mi fanno incazzare p*rca p*ttana stiamo scherzando è uno @xmath0. @xcite