SLING : S ino Linguistic Evaluation of Large Language Models (2022)

Abstract: To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics (SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. Each pair demonstrates the acceptability contrast of a specific syntactic or semantic phenomenon (e.g., The keys are lost vs. The keys is lost), and an LM should assign lower perplexity to the acceptable sentence. In contrast to the CLiMP dataset (Xiang et al., 2021), which also contains Chinese minimal pairs and was created by translating the vocabulary of the English BLiMP dataset, the minimal pairs in SLING are derived primarily by applying syntactic and lexical transformations to naturally-occurring, linguist-annotated sentences from the Chinese Treebank 9.0, thus addressing severe issues in CLiMP’s data generation process. We test 18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and multi-lingual (e.g., mT5, XLM) language models on SLING. Our experiments show that the average accuracy for LMs is far below human performance (69.7% vs. 97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested LMs, even much larger ones. Additionally, we find that most LMs have a strong gender and number (singular/plural) bias, and they perform better on local phenomena than hierarchical ones.

## introduction while large-scale pretrained language models (lms) have achieved considerable downstream success (devlin et al., 2019; xue et al., 2021; brown et al., 2020, a.o.), it remains challenging to evaluate how much linguistic knowledge they have acquired. one approach is to design minimal pairs consisting of two sentences that differ only in a @xcite the sling data and code can be found https:// github.com/yixiao-song/sling_data_code . a: 他们在吃饭了。 critical word or phrase, which renders only one of the sentences acceptable (e.g., the keys are lost vs. the keys is lost). if an lm is sensitive to the phenomenon exemplified by the minimal pair (in this case, plurality), it should assign a lower perplexity to the acceptable sentence. this methodology can be used to test an lm's understanding of a wide range of linguistic phenomena; for example, the blimp dataset (warstadt et al., 2020) contains 67k minimal pairs automatically generated via manually-constructed grammars that span 12 high-level english phenomena. can we create similar datasets to study linguistic phenomena in a different language, such as chinese? as a first step in this direction, xiang et al. (2021) introduce climp, a chinese dataset of minimal pairs. however, we identify two major issues with climp's construction process: (1) its vocabulary is translated from blimp's vocabulary, which due to morphological differences between english and chinese (e.g., the latter lacks numeral or verbal inflections) results in a large number of unin-telligible sentences; and (2) the grammatical templates for several phenomena (anaphor agreement, classifier-noun agreement, and filler-gap dependencies) are inadequately designed, which along with the vocabulary issue results in minimal pairs that do not have any clear contrast. @xcite to address these issues, we introduce sling (sino linguistics benchmark), a dataset of 38k minimal pairs to study nine chinese linguistic phenomena, many of which are unique to the chinese language. instead of translating blimp, we construct sling primarily using the chinese treebank 9.0 (xue et al., 2016), which was annotated by trained linguists (see we evaluate a total of 18 publicly-available pretrained lms on sling, including monolingual chinese (e.g., bert-base-chinese, pangu-α) and multilingual models (e.g., mt5, xlm-r). our results reveal that: (1) no lm consistently outperforms others on sling; (2) larger lms do not necessarily outperform smaller ones; (3) monolingual chinese lms generally perform better than multilingual ones; and (4) humans significantly outperform all lms (97.1% vs 69.7% average across lms). we observe that the ranking of models on climp differs from that on sling: for example, bert-chinese-base is the best-performing model on sling (average accuracy 84.8%), while chinese-pert-base performs best on climp (81.2%). this result is due in part to the issues in climp's construction process, as well as the different phenomena that we test in sling. additionally, sling is more discriminative than climp (i.e., lms vary more across the phenomena in terms of accuracy), which makes it more useful as a diagnostic benchmark especially given the large gap climp sling vocab. source blimp's vocab. translated chinese treebank 9.0 vocab. size actual 1272 types (w/ 230 proper names) 11988 types (claimed 3456) grammar 9 syntax phenomena 3 semantics + 6 syntax (16 paradigms) (5 syntax differ from climp) (38 paradigms) evaluated lms monolingual only 10 mono-& 8 multilingual 1 bert-base-chinese 1 lstm 3 lstm 3 causal lms 2 5-gram 14 masked lms using minimal pairs to detect a function of a single element (e.g., phoneme, affix, or word) is a common practice in linguistics. in figure 1 , by changing the position of 了, sentence a is transformed into the ungrammatical sentence b, and we know how the two aspect markers 在 and 了 interacts. in this paper, following blimp and climp, we call each major grammatical category a phenomenon, and minimal pair types within each phenomenon paradigms. the a and b sentences in figure 1 form a minimal pair of a paradigm in the aspect phenomenon of sling. @xcite created climp to evaluate 9 chinese syntactic phenomena with 16 paradigms. however, the dataset suffers from two major issues: (1) faulty minimal pair generation templates and (2) its translated vocabulary. in this section, we discuss the issues in detail and show why they hamper climp's utility as a diagnostic dataset for lms. climp's minimal pairs often do not show the desired acceptability contrast. this problem is especially prominent in the ba construction, binding/anaphor, and filler-gap dependency phenomena, on which xiang et al. (2021) conclude that lms perform poorly. the templates used to generate data for these phenomena are the primary cause of these errors, as we show below. the primary reason for the low quality of these pairs is that climp does not carefully control the source of unacceptability (abrusán, 2019), which we discuss further in the limitations section. specific to the ba construction, climp does not include essential information about thematic relations foot_3 in the vocabulary. another contributing factor is the small size of the climp vocabulary, which is translated from that of blimp despite many annotated features of blimp not applying to chinese (e.g., number features, verb forms, or cases). for example, the english verb buy has six forms in blimp, listed in chinese english features mai buy bare zheng zai mai buying ing mai le bought finite, past mai le bought en mai buy finite, pres mai buys finite, pres, 3sg binding and anaphor paradigms: these two paradigms test whether the gender feature of the object anaphor agrees with that of the subject. issues in the binding and anaphor paradigms stem from the fact that climp uses proper names, which were added to climp's vocabulary in addition to the one translated from blimp. however, chinese proper names do not always unambiguously show gender. if the gender of the subject is ambiguous as in (1) where ye zi can be either gender (similarily for alex in english), the performance of the lms is not representative of whether they know the function of the reflexive anaphor, which is exactly what the binding and anaphor paradigms want to test. other issues with these two paradigms are discussed in detail in appendix d.2. filler-gap paradigm: to create minimal pairs for the filler-gap paradigm in climp, xiang et al. (2021) use what they call the topicalization construction. however, (2a), taken from climp, does not contain a filler-gap topicalization dependency. a real topicalization filler-gap structure should be the one in (2b), in which the direct object of the verb buy is topicalized and moved to the beginning of the sentence, leaving a (gap) at its base generated position (huang et al., 2009, section 6.1). unfortunately, the minimal pairs associated 三套小说 三户小说 三本家庭小说 三户家庭小说 三户小说家庭 三本小说家庭 dt m nn m dt nn clauses should be in correct order 这本小说 本这小说 这户家庭 户这家庭 cd m 1 nn 1 cd m 2 nn 1 classifier should be compatible with noun cd m 1 nn 2 nn 1 cd m 2 nn 2 nn 1 classifier should match farther noun ………. 三户家庭 三套家庭 ………. ………. nn m 小说 本, 套 家庭 户 ………. 1.2k nn-m pairs found in treebank, stored as one to many map with this paradigm are generated based on an erroneous template, which means no conclusions can be drawn from model performance on it. ## creating the sling benchmark this section describes our process of generating minimal pairs for sling. we make use of the chinese treebank 9.0 (xue et al., 2016), a chinese corpus with linguist-annotated constituency parses that contains 2,084,387 words. this treebank allows us to use naturally-occurring sentences to construct our minimal pairs, unlike the synthetic and sometimes nonsensical sentences of climp. also, unlike climp, whose linguistic templates rely solely on one grammar book (po-ching and rimmington, 2015), our linguistic templates are constructed by a native chinese linguist (the first author of this paper) based on multiple works in linguistics. details of the construction of each phenomenon and the cited works can be found in appendix d. the general minimal pair generation process is to identify a linguistic pattern, search for relevant linguistic structures in the treebank, and form minimal pairs by applying hand-crafted transformation rules on the extracted structures. figure 2 provides an overview of this process, with the same running example as this section. ## results & analysis ## conclusion we present sling, a new benchmark for evaluating chinese linguistic knowledge in large scale pretrained lms. unlike the existing climp dataset, in which we identify several critical issues, we construct sling from naturally-occurring sentences in the chinese treebank. our results show that monolingual chinese lms achieve better performance on sling than multilingual lms. we find that lms are better at handling local dependencies than long-range dependencies or with distractors, and that they are better at syntactic rather than semantic phenomena. overall, there remains a large gap between lm and human performance.