CL i MP : A Benchmark for C hinese Language Model Evaluation (2021)

Abstract: Linguistically informed analyses of language models (LMs) contribute to the understanding and improvement of such models. Here, we introduce the corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire. CLiMP consists of sets of 1000 minimal pairs (MPs) for 16 syntactic contrasts in Chinese, covering 9 major Chinese linguistic phenomena. The MPs are semi-automatically generated, and human agreement with the labels in CLiMP is 95.8%. We evaluate 11 different LMs on CLiMP, covering n-grams, LSTMs, and Chinese BERT. We find that classifier–noun agreement and verb complement selection are the phenomena that models generally perform best at. However, models struggle the most with the ba construction, binding, and filler-gap dependencies. Overall, Chinese BERT achieves an 81.8% average accuracy, while the performances of LSTMs and 5-grams are only moderately above chance level.

## introduction language models (lms) are crucial parts of natural language processing (nlp) systems for a large variety of tasks, including summarization, machine translation, and dialog generation. more recently, they have become popular in the form of pretrained models, foot_0 which are then fine-tuned on downstream tasks and often obtain state-of-the-art performance @xcite @xcite . however, which linguistic phenomena language models can or cannot learn is still poorly understood for many languages. resources for the syntactic evaluation of lms, such as blimp @xcite have focused mainly on english, and non-english resources currently only cover a small set of phenomena @xcite @xcite . in order to spur the analysis and subsequent improvement of lms in chinese, we introduce the corpus of chinese linguistic minimal pairs (climp), which can be used to evaluate lms' knowledge of chinese grammar. climp consists of 16 individual datasets that are semi-automatically generated from grammar templates. each set-or paradigm-contains 1,000 minimal pairs (mps). together, they cover 9 core linguistic phenomena in chinese. human agreement on this corpus is 95.8%, confirming that climp represents robust contrasts in chinese grammar. high performance on climp thus implies high correlation with human acceptability judgments across these phenomena. we use climp to study chinese bert @xcite ), foot_1 6 lstm @xcite lms, and 4 5-gram lms. we evaluate for each mp whether the lm assigns a higher probability to the grammatical or the ungrammatical sentence. our results show that chinese bert is closest to human performance, achieving an 81.8% accuracy on average over all phenomena, while the performances of lstms and 5-grams, regardless of the training data size, are only moderately above chance level. classifier-noun agreement and verb complement selection are the phenomena that models generally perform best at, suggesting that chinese lms are better at acquiring knowledge of local selectional restrictions. the bǎ construction, binding, and filler-gap dependencies are the phenomena models have the most difficulties with. this indicates that they struggle to learn hierarchical syntax and to identify long-distance dependencies. ## climp our main contribution is climp, a corpus of chinese mps designed to evaluate chinese lms. climp consists of 1,000 mps for each of 16 grammatical contrasts, covering 9 major chinese linguistic phenomena. example mps for each phenomenon are shown in ## models and methods we use accuracy for evaluation. a mp in climp is classified correctly if a lm assigns a higher probability to the grammatical sentence than to the ungrammatical one. we evaluate statistical and neural lms, including masked lms. corpora which contain 0.4m, 2m, and 21.5m sentences are used for further exploration. we also investigate the effect of different tokenizations. 5 chinese bert bert @xcite ) is a transformer-based neural model @xcite . here, we evaluate chinese bert. 6 this phenomenon n acceptable example unacceptable example anaphor agreement 1 王玉珍 震惊-了 她自己。 jane.f shock-pst herself. 'jane shocked herself.' 王玉珍 震惊-了 他自己。 jane.f shock-pst himself. 'jane shocked himself.' binding 1 杨颖 治疗 吴宇涛 之后 佩服-过 她自己。 yang.f cure wu.m after admire-pst herself 'yang admired herself after she cured wu.' 杨颖 治疗 吴宇涛 之后 佩服-过 他自己。 yang.f cure wu.m after admire-pst himself 'yang admired himself after she cured wu.' bǎ construction 1 王鑫 把 自行车 扔 了。 wong.m ba bike throw pst 'wong threw away the bike.' 王鑫 被 自行车 扔 了 。 wong.m pass bike throw pst 'wong was thrown away by the bike.' coverb 3 李文清 乘 卡车 到达-了 咖啡店。 lee.m ride truck arrive-pst coffee shop 'lee went to the coffee shop by truck.' 李文清 于 卡车 到达-了 咖啡店。 lee.m at truck arrive-pst coffee shop 'lee went to the coffee shop at truck.' np head finality 1 王梦 正在 卖 张红梅 清洗-过-的 推车。 wong.f prog sell may.f clean-prf-adj trolley 'wong is selling the trolley that mel has cleaned.' 王梦 正在 卖 推车 张红梅 清洗-过-的。 wong.f prog sell trolley may.f clean-prf-adj 'wong is selling the trolley that mel has cleaned.' classifier 2 张杰 正在 穿过 一 家 艺术画廊。 jay.m prog pass one cl:institution art gallery 'jay is passing through an art gallery.' 张杰 正在 穿过 一 段 艺术画廊。 jay.m prog pass one cl:length art gallery 'jay is passing through an art gallery.' filler gap 1 图书馆， 我 开车 去-过 这个地方。 the library, i drive to-prf this place 'the library, i have driven to this place.' 图书馆， 我 开车 去-过 博物馆。 the library, i drive to-prf the museum 'the library, i have driven to the museum.' passive 1 这些 患者 被 转移-了。 these patient pass transfer-pst 'these patients were transferred.' 这些 患者 被 下降-了。 these patient pass fall-pst 'these patients were fell.' verb complement 5 王慧 的 文章 吓 坏 了 包曼玉 。 wong.f poss article frighten badly pst bao.f. 'wong's article frightened bao badly.' 王慧 的 文章 吓 开 了 包曼玉 。 wong.f poss article frighten openly pst bao.f. 'wong's article frightened bao openly.' ## results all results are shown in phenomenon-specific results our lms perform best on classifier-noun agreement and verb complement selection: chinese bert's accuracy is only 6.8% and, respectively, 3% lower than that of humans on these two phenomena. lstms and 5grams remain around 30% behind humans, but still perform better on these phenomena than on others in climp. this indicates that chinese lms acquire local selection knowledge better than the linguistic knowledge needed to master other phenomena. our lms stuggle most with the bǎ construction, binding, and filler-gap dependencies. all models perform close to chance level for binding, suggesting that they lack the hierarchical knowledge necessary to correctly resolve the structural relationship between a reflexive and its binder. similarly, most models perform near chance on filler-gap dependencies. this suggests that they do not robustly represent long-distance dependencies. foot_7 on the head-final construction, chinese bert performs surprisingly poorly as compared to the other models: only 53.1% accuracy as compared to an average accuracy of 81% by the lstms. the coverb construction, in contrast, is easy for chinese bert: it achieves 87.9% accuracy, while the highest accuracy among all other models is 47%. ## conclusion we introduced climp, a suite of diagnostic test sets aimed at evaluating which syntactic phenomena chinese lms learn, and used it to evaluate 11 different models. all lms appeared to have learned local selectional restrictions, but struggled with argument structure alternations, hierarchical structure, and long-distance dependencies. chinese bert performed best on climp overall. however, it obtained a 14% lower accuracy than humans, suggesting there is still much room for improvement. we hope that climp will serve as a linguistically informed resource for benchmarking and analyzing future progress on chinese lms. climp is available at https://nalacub.github.io/resources .