JBL i MP : J apanese Benchmark of Linguistic Minimal Pairs (2023)

Abstract: In this paper, we introduce JBLiMP (Japanese Benchmark of Linguistic Minimal Pairs), a novel dataset for targeted syntactic evaluations of language models in Japanese. JBLiMP consists of 331 minimal pairs, which are created based on acceptability judgments extracted from journal articles in theoretical linguistics. These minimal pairs are grouped into 11 categories, each covering a different linguistic phenomenon. JBLiMP is unique in that it successfully combines two important features independently observed in existing datasets: (i) coverage of complex linguistic phenomena (cf. CoLA) and (ii) presentation of sentences as minimal pairs (cf. BLiMP). In addition, JBLiMP is the first dataset for targeted syntactic evaluations of language models in Japanese, thus allowing the comparison of syntactic knowledge of language models across different languages. We then evaluate the syntactic knowledge of several language models on JBLiMP: GPT-2, LSTM, and n-gram language models. The results demonstrated that all the architectures achieved comparable overall accuracies around 75%. Error analyses by linguistic phenomenon further revealed that these language models successfully captured local dependencies like nominal structures, but not long-distance dependencies such as verbal agreement and binding.

## introduction the past few years have seen a remarkable success of neural language models, and some language models based on transformer @xcite have achieved the state-of-the-art performance in various natural language processing (nlp) tasks @xcite . in fact, recent neural language models are extremely successful in solving a variety of downstream tasks, but it remains to be understood how well these neural language models understand the syntax of natural languages. in order to address this question, some studies investigated the syntactic knowledge of language models with a specially designed dataset for targeted syntactic evaluations (e.g., @xcite @xcite @xcite . however, most of these studies have focused on english and other european languages, and only few studies extended this investigation to non-european languages @xcite . importantly for the purpose here, even fewer studies have dealt with a wide variety of linguistic phenomena in non-english languages @xcite . in this paper, we introduce jblimp (japanese benchmark of linguistic minimal pairs), a novel dataset for targeted syntactic evaluations of language models in japanese. 1 jblimp consists of 331 minimal pairs, which are created based on acceptability judgments extracted from journal articles in theoretical linguistics. these minimal pairs are grouped into 11 categories, each covering a different linguistic phenomenon. jblimp is unique in that it successfully combines two important features independently observed in existing datasets: (i) coverage of complex linguistic phenomena (cf. cola; @xcite and (ii) presentation of sentences as minimal pairs (cf. blimp; @xcite . we evaluate the syntactic knowledge of several language models on jblimp: gpt-2 @xcite , lstm @xcite and n-gram language models. the results demonstrated that all the architectures achieved comparable overall accuracies around 75%. error analyses by linguistic phenomenon further revealed that these language models successfully captured local dependencies like nominal structures, but not long-distance dependencies such as verbal agreement and binding. ## related work evaluation of language models has been mainly performed by computing metrics such as perplexity. this gives us an objective standard of the performance of language models, but doesn't provide insight into their performance on specific downstream tasks. while recent large-scale benchmarks like glue @xcite and superglue @xcite are informative in this respect, many recent studies have sought to provide evidence that language models have learned the syntax of natural languages. in a pioneering work by @xcite , minimal pairs were employed to investigate whether language models are sensitive to subject-verb agreement in english. for instance, they tested whether language models assign a higher probability to are than is in (1). their results suggested that lstm language models are fairly sensitive to english subject-verb agreement. however, this and related studies (e.g., @xcite only covered a limited range of linguistic phenomena like subject-verb agreement. in order to tackle this problem, more recent studies have introduced large-scale datasets for comprehensive syntactic evaluations @xcite . one such dataset is cola (corpus of linguistic acceptability; @xcite , which consists of 10,000 sentences with binary acceptability labels extracted from linguistics journals and textbooks. cola is incorporated into glue benchmark @xcite and has been used to evaluate the sensitivity of language models to the syntax of natural languages. while cola has enabled the comprehensive syntactic evaluations of language models, this dataset is not without its limitation, as noted by @xcite themselves. the limitation lies in the need to train a supervised classifier on cola for evaluation. in short, cola is designed for binary classification of acceptability judgements, but there is no clear way to map the probability of the sentence estimated by language models to binary acceptability judgements. unfortunately, "the use of supervision prevents making strong conclusions about the sentence encoding component, since it is not possible to distinguish what the encoder knows from what is learned through supervised training on acceptability data" @xcite . ## jblimp in order to overcome all the limitations mentioned above, we introduce jblimp (japanese benchmark of linguistic minimal pairs), a novel dataset for targeted syntactic evaluations of language models in japanese. jblimp is unique in that it successfully combines two important features independently observed in existing datasets ( ## conclusion in this paper, we introduced jblimp (japanese benchmark of linguistic minimal pairs), a novel dataset for targeted syntactic evaluations of language models in japanese. jblimp consists of 331 minimal pairs, which are created based on acceptability judgments extracted from journal articles in theoretical linguistics. these minimal pairs are grouped into 11 categories, each covering a different linguistic phenomenon. jblimp is unique in that it successfully combines two important features independently observed in existing datasets: (i) coverage of complex linguistic phenomena (cf. cola) and (ii) presentation of sentences as minimal pairs (cf. blimp). in addition, jblimp is the first dataset for targeted syntactic evaluations of language models in japanese, thus allowing the comparison of syntactic knowledge of language models across different languages. we then evaluated the syntactic knowledge of several language models: gpt-2, lstm and n-gram language models. the results demonstrated that all the architectures achieved comparable overall accuracies around 75%. error analyses by linguistic phenomenon further revealed that these language models successfully captured local dependencies like nominal structures, but not long-distance dependencies such as verbal agreement and binding. finally, these detailed analyses of language models' knowledge on complex linguistic phenomena using minimal pairs are only possible with the unique design of jblimp. this paper will hopefully encourage the development of the datasets with jblimp's two important features in other languages.