R u C o LA : R ussian Corpus of Linguistic Acceptability (2022)

Abstract: Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers.However, the application scope of LA in languages other than English is limited due to the lack of high-quality resources.To this end, we introduce the Russian Corpus of Linguistic Acceptability (RuCoLA), built from the ground up under the well-established binary LA approach. RuCoLA consists of 9.8k in-domain sentences from linguistic publications and 3.6k out-of-domain sentences produced by generative models. The out-of-domain set is created to facilitate the practical use of acceptability for improving language generation.Our paper describes the data collection protocol and presents a fine-grained analysis of acceptability classification experiments with a range of baseline approaches.In particular, we demonstrate that the most widely used language models still fall behind humans by a large margin, especially when detecting morphological and semantic errors. We release RuCoLA, the code of experiments, and a public leaderboard to assess the linguistic competence of language models for Russian.

## introduction recent nlp research has approached the linguistic competence of language models (lms) with acceptability judgments, which reflect a sentence's well-formedness and naturalness from the perspective of native speakers @xcite . these judgments have formed an empirical foundation in generative linguistics for evaluating humans' grammatical knowledge and language acquisition @xcite . borrowing conventions from linguistic theory, the community has put much effort into creating @xcite and itacola @xcite . %=percentage of acceptable sentences. to this end, we introduce the russian corpus of linguistic acceptability (rucola), a novel benchmark of 13.4k sentences labeled as acceptable or not. in contrast to related binary acceptability classification benchmarks in the main contributions of this paper are the following: (i) we create rucola, the first largescale acceptability classification resource in rus-sian. (ii) we present a detailed analysis of acceptability classification experiments with a broad range of baselines, including monolingual and cross-lingual transformer @xcite lms, statistical approaches, acceptability measures from pretrained lms, and human judgements. (iii) we release rucola, the code of experiments foot_0 , and a leaderboard to test the linguistic competence of modern and upcoming lms for the russian language. applications of acceptability acceptability judgments have been broadly applied in nlp. in particular, they are used to test lms' robustness @xcite and probe their acquisition of grammatical phenomena @xcite @xcite . la has also stimulated the development of acceptability measures based on pseudoperplexity @xcite , which correlate well with human judgments @xcite and show benefits in scoring generated hypotheses in downstream tasks @xcite . another application includes evaluating the grammatical and semantic correctness in language generation @xcite @xcite . ## experiments we evaluate several methods for acceptability classification ranging from simple non-neural approaches to state-of-the-art cross-lingual models. ## results and analysis ## cross-lingual transfer given the availability of acceptability classification corpora in other languages, one might be curious about the possibility of knowledge transfer between languages for this task. this is particularly important in the case of estimating sentence acceptability for low-resource languages, which are an important focus area of nlp research @xcite . however, the nature of the task makes successful transfer an open question: for instance, specific grammar violations in one language might not exist in another. with this in mind, we explore the zero-shot cross-lingual transfer scenario, in which the training and validation datasets are provided in one language and the test data in a different one. we use four multilingual models: mbert @xcite , xlm-r base , xml-r, and rembert. we study the transfer between three datasets: cola, itacola, and rucola, containing examples in english, italian and russian, respectively. as shown in due to the space constraints, we defer a detailed description of the experimental setup and results to appendix e; here, we overview the key findings of this study. specifically, we find that the monolingual scenarios outperform cross-lingual transfer by a large margin, which confirms and extends the results of @xcite . also, we observe that rembert performs best in monolingual and cross-lingual setups. for the in-domain set, we observe a cross-lingual transfer gap: there is little difference in language to transfer from, and for rucola, the zero-shot results are as poor as those of a linear "monolingual" classifier. however, the cross-lingual setup performs on par with the monolingual setup for out-of-domain data. ## conclusion and future work this work introduces rucola, the first largescale acceptability classification corpus in the russian language. the corpus consists of more than 13.4k sentences with binary acceptability judgments and provides a coarse-grained annotation of four violation categories for 3.7k unacceptable sentences. rucola covers two types of data sources: linguistic literature and sentences produced by generative models. our design encourages nlp practitioners to explore a wide range of potential applications, such as benchmarking, diagnostic interpretation of lms, and evaluation of language generation models. we conduct extensive experimental evaluation by training baselines that cover a broad range of models. our results show that lms fall behind humans by a large margin. finally, we explore the cross-lingual generalization capabilities of four cross-lingual transformer lms across three languages for acceptability classification. the preliminary results show that zero-shot transfer for in-domain examples is hardly possible, but the discrepancy between monolingual and cross-lingual training results for out-of-domain sentences is less evident. in our future work, we plan to explore the benefits and limitations of rucola in the context of applying acceptability classifiers to natural language generation tasks. another direction is to augment the in-domain and out-of-domain validation sets with fine-grained linguistic annotation for nuanced and systematic model evaluation. in the long run, we hope to provide valuable insights into the process of grammar acquisition by language models and help foster the application scope of linguistic acceptability.