Monolingual and Cross-Lingual Acceptability Judgments with the I talian C o LA corpus (2021)

Abstract: The development of automated approaches to linguistic acceptability has been greatly fostered by the availability of the English CoLA corpus, which has also been included in the widely used GLUE benchmark. However, this kind of research for languages other than English, as well as the analysis of cross-lingual approaches, has been hindered by the lack of resources with a comparable size in other languages. We have therefore developed the ItaCoLA corpus, containing almost 10,000 sentences with acceptability judgments, which has been created following the same approach and the same steps as the English one. In this paper we describe the corpus creation, we detail its content, and we present the first experiments on this new resource. We compare in-domain and out-of-domain classification, and perform a specific evaluation of nine linguistic phenomena. We also present the first cross-lingual experiments, aimed at assessing whether multilingual transformer-based approaches can benefit from using sentences in two languages during fine-tuning.

## introduction the ability to judge whether a sentence is perceived as natural and well-formed by a native speaker is called acceptability judgment. despite several open issues concerning methods for collecting and evaluating them @xcite @xcite , these judgments have been the most significant source of data in linguistics throughout the history of the discipline @xcite @xcite . with the rise of neural language models, several works have tried to assess how much a model can encode linguistic information @xcite , ranging from specific phenomena @xcite to a general grammar knowledge @xcite . acceptability judgments have proven to be a promising area to test the acquisition of linguistic knowledge by neural language models @xcite . in particular, with the creation of the corpus of linguistic acceptability (cola) @xcite several approaches have been proposed that cast acceptability as a binary classification task and address it by finetuning transformer-based models on the corpus @xcite @xcite . unfortunately, most classification experiments on acceptability judgments have focused on english, mainly because of the lack of large corpora in other languages. in this work, we therefore describe the creation of a novel corpus of acceptability judgments in italian, following the methodology used in cola for english. we collect 10k sentences extracted from linguistic literature and labelled by experts as acceptable or not. furthermore, we enrich around 30% of the sentences with additional labels describing nine linguistic phenomena. we also present a set of experiments aimed at testing the performance of a bert-based classifier on the data and comparing it with results obtained on english. additionally, cross-lingual experiments using xlm-roberta @xcite show the potential of this approach, even if it is outperformed by monolingual models. the main contributions of this work are therefore i) the creation and release of the italian corpus of acceptability judgments (itacola), 1 that to our knowledge is the largest resource of its kind in a language other than english; ii) a set of experiments to assess the performance of bert-based models on the whole corpus and on specific phenomena. iii) a set of experiments using a massive multilingual language model on italian and english, with the potential to open up novel cross-language research perspectives. ## itacola: italian corpus of linguistic acceptability in this section we introduce the italian corpus of linguistic acceptability (itacola), built with the purpose of representing a large number of linguistic phenomena while distinguishing between acceptable and not acceptable sentences. the methodology of corpus creation and its size are similar to those proposed for the english cola in @xcite , i.e. we have collected examples from different manuals covering several linguistic phenomena. this fulfills a dual purpose: the size of the corpus allows the application of deep learning approaches to acceptability judgment, while its structure paves the way to cross-language comparative analyses. concerning acceptability annotation, for the creation of itacola we have chosen to keep a boolean judgment in line with several previous works @xcite @xcite . this choice ensures robustness and simplifies classification, while allowing us to keep the original judgments as formulated by an expert (i.e. the authors of the different data sources). ## monolingual experiments the monolingual experiments are aimed at presenting the first classification results on itacola and at defining standard training, validation and test split, to be used also in future experiments with the corpus. we compare two classifiers: one using lstm and fasttext embeddings, which we consider our baseline, and the other using an italian version of bert @xcite , which we fine-tune using itacola training dataset. the two classifiers are evaluated in an in-domain and an out-of-domain setting, similar to the evaluation performed on english cola. @xcite . for the in-domain evaluation, we divide the ita-cola corpus into a training, a validation and a test split, including respectively 7,801, 946 and 975 examples. we create the splits so that each source is equally represented in each split and the acceptability/not acceptability ratio is preserved. for the out-of-domain setting, training is performed on the same split used for the in-domain experiments. validation and test, instead, are carried out using the accomplit dataset @xcite . in particular, for validation we use the training set released for the evalita shared task and for testing we use the official accomplit test set. we consider this dataset out-of-domain not only because it comes from different sources compared to itacola, but also because it was created using crowd-sourcing, i.e. following a completely different approach than ours, which relies on linguistic literature. baseline lstm: as baseline classifier, we implement a bidirectional lstm with two layers (64 and 32 neurons) and a dropout of 0.3. each sentence is represented as a sequence of word embeddings, obtained with the italian model of fasttext @xcite trained on common crawl and wikipedia with size 300. 4 the network is implemented with keras @xcite (adam optimizer, learning rate 0.01, loss function: binary crossentropy, 15 epochs). we perform 10 restarts. reported results represent the mean performance obtained over the restarts. bert: among the italian bert-like versions available, we select bert-base-italian-xxl-cased, available on huggingface. foot_4 it is a model pretrained on a total general-purpose corpus of 81gb. after randomizing the order of instances in our training set, we fine-tune the model using pytorch, 6 with a maximum sequence length of 64, a batch size of 32 for 12 epochs. we perform 10 restarts. also in this case, reported results are the mean across the repeated classifications. ## analysis of specific linguistic phenomena acceptability judgments involve a number of different linguistic phenomena, which we tried to cover as much as possible by selecting different sources for the creation of the dataset. however, in order to fully understand how well classifiers can judge acceptability in the presence of these phenomena, we perform also a fine-grained evaluation focused on specific linguistic constructions. ## cross-lingual experiments given that itacola has been created following the same principles of english cola and that monolingual results on italian are in line with results obtained on the english dataset using a similar bert-based approach, we perform a first set of cross-lingual classification experiments, to serve as baseline results for future improvements. we rely on xlm-roberta-base @xcite , a large multi-lingual language model, trained on 2.5tb of filtered commoncrawl data. we exper-iment with different classification settings, which are all evaluated both on itacola and on cola in-domain test sets. this means that, starting from the same multilingual model, we classify english and italian sentences. we implement the model in pytorch, using a batch size of 32 and a max sequence length of 64. the learning rate is set to 2e-5, and training goes for 12 epochs. three restarts are performed for each experiment. the number of restarts was constrained by the fact that evaluation of the english test set was possible only through kaggle, which limits the number of runs that can be submitted for evaluation. results are reported in ## conclusions in this paper we present the italian corpus of linguistic acceptability, a novel dataset including al- the corpus is released in three splits (training, development and test set) so to make replicability and further experiments easier. part of the dataset has also been manually annotated with 9 linguistic phenomena, enabling a fine-grained evaluation of the classifier performance on specific dimensions. the process to create the corpus has followed as much as possible the one adopted to collect the english cola, which has become the de facto standard dataset for linguistic acceptability and has greatly fostered the development of automated systems for acceptability judgments. itacola can represent a first step towards the creation of multilingual benchmarks for acceptability, in line with recent efforts to create massive multilingual resources covering different tasks @xcite . in the future, we plan to further explore the differences between itacola and accomplit @xcite , the other existing dataset for acceptability in italian. we will also experiment with the swedish corpus for acceptability studies presented in @xcite , to check whether the findings in our work, in particular the cross-lingual experiments, hold also for swedish when paired with english and italian. furthermore, we plan to explore classification approaches that yield stateof-the-art results on cola. while some of them are not applicable to the new corpus, because of the lack of many massive lms for italian, recent studies showed that with smaller language models it should be possible to achieve better results after reformulating nlp tasks as entailment @xcite . we will explore whether this research direction is promising also for acceptability studies for languages with limited resources.