UINAUIL : A Unified Benchmark for I talian Natural Language Understanding (2023)

Abstract: This paper introduces the Unified Interactive Natural Understanding of the Italian Language (UINAUIL), a benchmark of six tasks for Italian Natural Language Understanding. We present a description of the tasks and software library that collects the data from the European Language Grid, harmonizes the data format, and exposes functionalities to facilitates data manipulation and the evaluation of custom models. We also present the results of tests conducted with available Italian and multilingual language models on UINAUIL, providing an updated picture of the current state of the art in Italian NLU.

## introduction large language models (llm) have revolutionized the field of natural language processing. in the span of a few years, the common practice for most nlp tasks shifted from building ad-hoc models trained on task-specific data to fine-tuning general-purpose language models trained in a selfsupervised fashion. the focus of the evaluation practices shifted accordingly, from measuring the impact of different features and neural architectures on the prediction performance, to assessing the predictive power of llms applied to a variety of nlp tasks. this has been possible, at least in part, due to the standardization proposed in @xcite , where four general task formats are claimed to represent the structure of most nlp tasks: text classification, sentence pair classification, sequence labeling, and question answering. in this scenario, benchmarks have emerged that collect a number of tasks falling into the four mentioned categories, with the purpose of evaluating llms in a fair and reproducible environment. perhaps the best known of such benchmarks is glue @xcite , a set of nine sentence classification and sentence pair classification natural language understanding tasks. superglue @xcite was presented not long after glue with the goal of proposing a harder set of nlu tasks, given the high performance reached by llms on glue not long after its release. while english is covered by benchmarks such as glue and superglue, the situation differs sensibly when we turn to other languages, with only a few nlu benchmarks available for non-english languages @xcite @xcite @xcite . these are useful resources for the international nlp community, and a great example of language equality @xcite . however, these benchmarks are mostly static collection of datasets with no additional tool to facilitate data gathering and management, evaluation, and automation in general (except for leaderboards, which are usually maintained by the respective authors). in this paper, we present uinauil (unified interactive natural understanding of the italian language), an integrated benchmark for italian nlu, with three main goals: g1 filling the gap in italian nlu evaluation in the era of llms by proposing one integrated benchmark as opposed to a myriad of individual shared task datasets. g2 raising the bar on the automation level of nlu benchmarks, in order to create more accessible and user-friendly benchmarks. g3 set the example via a use case to encourage scholars and nlp practitioners to publish modern, integrated benchmarks for underrepresented languages. ## a unified benchmark for italian nlp uinauil is a set of six tasks originally proposed as shared tasks for evaluation of italian nlp. in this section, we describe the background of the tasks and how they are integrated into uinauil. ## the uinauil library in addition to defining the benchmark, we created a software library to access the data and functionalities of uinauil. we developed a python module for downloading the task data and metadata, represented in a structurally consistent way. furthermore, the library provides an implementation of evaluation metrics. the uinauil package is available on pip / pypi, and can be installed with the command: the package depends only on two non-standard packages: elg, the library maintained by the elg project for accessing to the resources of the european platform 6 (see section 2.2), and scikit-learn, a well known library for machine learning and data analysis 7 . otherwise, since the uinauil library is fully contained into a single file, developers can directly download the source file and save it in the working folder. once installed, the library can be used to download the resources of the tasks described in section 2.3, and to evaluate the predictions of a personal model through standard performance metrics selected by us for each task. the data are contained in python standard structures (lists and dictionaries) and are divided into training and test sets, according to the original split for each task represented in the elg repository. the list of available tasks is stored into a proper attribute: the tasks variable contains information in a dictionary format, where each key is the name of a task (used to acces the task data in uinauil), while the value contains its identifier on the elg platform and the type of task. an example of usage of the library for a learning and evaluation pipeline is the following: in addition to these core functionalities, the uin-auil library contains several metadata that helps programmers to understand the resources of each task, including the list of key names of features and target, a brief description of the meaning of each feature, the list of possible values of the target and their meaning, and others. the complete list of variables and methods of uinauil library is available on the github repository of the project foot_5 . also present on the repository are several examples of use of the library on several common machine learning models in form of python notebooks, and the complete leaderboards by task. ## evaluation in order to test the library, and to offer the community a first set of results on italian nlu tasks, we conducted a series of experiments with the aim of setting a baseline for all the tasks of the benchmark. the experiments consist in fine-tuning pre-trained language models for italian (plus a multilingual one) on the training data of each task, and testing their prediction against the corresponding test data, computing the appropriate evaluation metrics. ## conclusions in this paper we presented uinauil (unified interactive natural understanding of the italian language), an integrated benchmark for italian nlu. its purposes are manifold: to fill the gap in italian nlu evaluation by proposing one integrated benchmark; to create more accessible and user-friendly benchmarks for italian nlu; to encourage scholars to publish modern, integrated benchmarks for under-represented languages. uinauil is implemented in python library, publicly available via pip/pypi, that permits to easily download resources in italian language for six different nlu tasks, that can be used by programmers and researchers to train and evaluate their nlp models. uinauil is built with automation as principle, with the main goal of minimizing the overhead for a user who wants to evaluate a nlu model for italian. in effect, only a few lines of code are sufficient before and after the main logic of a model, in order to retrieve the data and evaluate the model. in this paper, we presented in details each task included into uinauil and the main features of the python library, including a sample quickstart code for its most common functionalities. furthermore, we evaluated the performances of several common nlp models for italian language on each task of uinauil. the results show that current models represent a high-performing baseline, especially for sentence classification tasks, while there is still room for improvement for italian nlu, as shown by the performance on the textual entailment task. however, it should be noted that the goal of this paper is mainly to present the uinauil benchmark. a thorough analysis of the results of all available models, while out of our present scope, is a natural next development of this work. as further developments, we plan to add other tasks to the project, accordingly to the future developments in the field of nlp for italian language. we also plan to implement the leaderboard as a service in elg, besides the github repository, so that users can submit their results autonomously, leveraging the provided authentication.