R u BL i MP : R ussian Benchmark of Linguistic Minimal Pairs (2024)

Abstract: Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and decontaminating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used LMs for Russian are sensitive to morphological and agreement-oriented contrasts, but fall behind humans on phenomena requiring the understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available.

## introduction acceptability judgments are the main empirical test in generative linguistics for assessing humans' linguistic competence and language acquisition @xcite . one of the wellestablished approaches to judging a sentence's acceptability is a forced choice between minimal pairs of sentences, where a native speaker is expected to prefer a grammatical sentence to an ungrammatical one, as in example 1. the cat is on the mat. (grammatical) b. *the cat are on the mat. (ungrammatical) @xcite , climp @xcite , jblimp @xcite , sling @xcite , nocola zero @xcite , dalaj @xcite , lindsea @xcite , clams @xcite , and rublimp (ours). the paradigm of minimal pairs has been widely adopted for evaluating the grammatical knowledge of language models (lms) across various linguistic phenomena @xcite @xcite @xcite . the evaluation design implies that an lm assigns a higher probability to the grammatical sentence than the ungrammatical one if it is sensitive to the isolated phenomenon. over the last few years, a broad range of lms has been analyzed via this paradigm in typologically diverse languages, except for russian (e.g., @xcite @xcite . this paper introduces the russian benchmark of linguistic minimal pairs (rublimp), which consists of 45 datasets, each including 1k minimal pairs. our benchmark covers morphological, syntactic, and semantic phenomena well-represented in russian theoretical linguistics. in contrast to existing benchmarks of linguistic minimal pairs (see our method allows for generating minimal pairs at scale and ensures high customizability w.r.t. domain, dataset size, and lms. validating rublimp by 20 native speakers with a background in linguistics confirms that the generated minimal pairs unambiguously isolate the target phenomenon and contrast in grammaticality. our main contributions are: (i) we create rublimp, the first diverse and large-scale benchmark of minimal pairs in russian, (ii) we conduct ablation studies to analyze the effect of pretraining data decontamination on the model performance, (iii) we evaluate 25 monolingual and cross-lingual transformer lms @xcite and crowdsourcing workers, (iv) we release rublimp 1 , our codebase 2 , and all data collection, data annotation, and other materials. 1 hf.co/datasets/russiannlp/rublimp 2 github.com/russiannlp/rublimp 2 rublimp figure 1 outlines our approach to generating minimal pairs for rublimp, which includes the following stages: sentence extraction and annotation ( §2.1), minimal pair generation ( §2.2) and curation ( §2.4). our framework allows the user to customize each component and provides the foundation to mitigate the limitations of static benchmarks @xcite through continuous generation of minimal pairs for a domain of interest and decontaminating the data for specific russian lms. ## min-k%: ablation studies we begin with ablation studies on the effect of the minimal pair curation stage and the hyperparam-eter k% ∈ {30, 40, 50, 60}. for each paradigm in rublimp, (i) we randomly sample 1k generated minimal pairs and evaluate the lms to get the reference scores (the accuracy scores are averaged over 100 runs), and (ii) decontaminate the generated minimal pairs through a greed search for t and select 1k pairs with the maximum min-k% prob as described in §2.4 and evaluate the lms' performance. we then compute the ∆-score between (i) and (ii) for each lm, which measures the performance drop when using min-k% prob with certain k%. higher k% is more effective figure 3 shows that min-k% prob ensures adversarial filtering of the pool of generated minimal pairs. in general, the higher k% value, the lower the ∆-score for most lms. we find that the overall performance can drop from 2.9% to 12% and the ∆-score can depend on the model size (e.g., rugpt, bloom, and llama-2). however, the ∆-scores for rembert and mdeberta are positive; we relate it to the fact that these lms perform close to random guessing on rublimp ( §5) and other related benchmarks ( §6). we select k% of 60 to create rublimp. ## results on rublimp this section describes the empirical evaluation results on rublimp. we report the results by phenomenon in analyzing the results for the monolingual and multilingual lms, we observe that the former generally perform better, and the latter can achieve the random baseline performance (e.g., rembert, mdeberta, xglm-1.7b). we evaluate the multilingual lms on five related blimp-style benchmarks to explore this behavior in more detail ( §6). below, ## multilingual analysis to analyze the multilingual lms in more detail, we evaluate their sensitivity to linguistic phenom-ena in six benchmarks: blimp, climp, sling, jblimp, clams, and rublimp (see ## conclusion and future work this work introduces rublimp, the first largescale multidomain benchmark of 45k minimal pairs for the russian language. rublimp covers 45 minimal pair types grouped into 12 linguistic phenomena in morphology, syntax, and semantics. the rublimp creation approach ensures the linguistic diversity and high quality of the minimal pairs and minimizes the data contamination risk. we conduct an extensive empirical evaluation of 25 widely used monolingual and multilingual lms for russian and analyze their performance w.r.t. various criteria. our results show that the lms are better at identifying morphological and agreementoriented contrasts than violations of structural relations, negation, transitivity, and tense. furthermore, we analyse the 17 multilingual lms in seven languages and find that no single lm performs well in all languages. our future work includes (i) comparison of pretraining data detection methods (ii) implementation of new phenomena (e.g., islands), and (iii) a more detailed multilngual study of the lms' linguistic abilities. by releasing rublimp, we hope to foster further research on how the russian language is acquired by lms.