OCNLI : O riginal C hinese N atural L anguage I nference (2020)

Abstract: Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g.,SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ~56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (~12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese NLU. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language.

## introduction in the last few years, natural language understanding has made considerable progress, driven largely by the availability of large-scale datasets and advances in neural modeling @xcite . at the center of this progress has been natural language inference (nli), which focuses on the problem of deciding whether two statements are connected via an entailment or a contradiction. nli profited immensely from new datasets such as the stanford nli (snli, @xcite ) and multi-genre nli (mnli, @xcite ) datasets. however, as often the case, this progress has centered around the english language given that the most well-known datasets are limited to english. efforts to build comparable datasets for other languages have largely focused on (automatically) translating existing english nli datasets @xcite . but this approach comes with its own issues (see section 2). to overcome these shortcomings and contribute to ongoing progress in chinese nlu, we present the first large-scale nli dataset for chinese called the original chinese natural language inference dataset (ocnli). unlike previous approaches, we rely entirely on original chinese sources and use native speakers of chinese with special expertise in linguistics and language studies for creating hypotheses and for annotation. our dataset contains ∼56,000 annotated premise-hypothesis pairs and follows a similar procedure of data collection to the english mnli. following mnli, the premises in these sentence pairs are drawn from multiple genres (5 in total), including both written and spoken chinese (see additionally, we establish baseline results based on a standard set of nli models @xcite tailored to chinese, as well as new pre-trained chinese transformer models @xcite . we find that our strongest model, based on roberta @xcite genre level ## related work natural language inference (nli), or recognizing textual entailment (rte), is a long-standing task in nlp. since we cannot cover the whole field, we focus on existing datasets and current systems. data: to date, there exists numerous datasets for english, ranging from smaller/more linguistics oriented resources such as fracas @xcite , to larger ones like the rte challenges @xcite and sick @xcite . perhaps the most influential are the two large-scale, human-elicited datasets: the stanford natural language inference corpus (snli) @xcite , whose premises are taken from image captions, and the multi-genre natural language inference corpus (mnli) @xcite , whose premises are from texts in 10 different genres. both are built by collecting premises from pre-defined text, then having annotators come up with possible hypotheses and inference labels, which is the procedure we also employ in our work. these large corpora have been used as part of larger benchmark sets, e.g., glue @xcite , and have proven useful for problems beyond nli, such as sentence representation and transfer learning @xcite @xcite , automated question-answering @xcite and model probing @xcite @xcite . the most recent english corpus adversarial nli @xcite uses human-and-model-in-the-loop enabled training (hamlet) method for data collection. their annotation method requires an existing nli corpus to train the model during annotation, which is not possible for chinese at the moment, as there exists no high-quality chinese data. premise hypothesis a. louisa may alcott和nathaniel hawthorne 住在pinckney街道，而 那 个被oliver wendell holmes称为 "晴 天街道 的beacon street街道住着有 些喜欢自吹自擂的历史学家 william prescott eng.: louisa may alcott and nathaniel hawthorne lived on pinckney street, but on beacon street street, which is named "sunny street by oliver wendell holmes, lived the bragging historian william prescott. [sic] hawthorne住 在main street上 eng.: hawthorne lived on main street. b. 运行 slient，运行deep，运行答案 eng.: run slient, run deep, run answer. [sic] 悄悄的逃走 eng.: while automatically translated data have proven to be useful in many contexts, such as cross-lingual representation learning @xcite , there are well-known issues, especially when used in place of human annotated, quality controlled data. one issue concerns limitations in the quality of automatic translations, resulting in incorrect or unintelligible sentences (e.g., see a related issue is that a translation approach also copies the cultural context of the source language, such as an overemphasis on western themes or cultural situations. the latter two issues are shown in systems: as inference is closely related to logic, there has always been a line of research building logic-based or logic-and-machine-learning hybrid models for nli/rte problems (e.g. @xcite @xcite @xcite . however, in recent years, large datasets such as snli and mnli have been almost exclusively approached by deep learning models. for examples, several transformer architectures achieve impressive results on mnli, with current state-of-the-art t5 @xcite reaching 92.1/91.9% accuracy on the matched and mismatched sets. re-implementations of these transformer models for chinese have led to similar successes on related tasks. for example, @xcite report that a large roberta model @xcite , pre-trained with whole-word masking, achieves the highest accuracy (81.2%) among their transformer models on xnli. in the clue benchmark @xcite , the same roberta model also achieves the highest aggregated score from eight tasks. we will use this model to establish baselines on our new dataset. biases: the advances in dataset creation have led to an increased awareness of systematic biases in existing datasets @xcite , as measured through partial-input baselines, e.g., the hypothesis-only baselines explored in @xcite where a model can achieve high accuracy by only looking at the hypothesis and ignoring the premise completely (see also @xcite ). these biases have been mainly associated with the annotators (crowd workers in mnli's case) who use certain strategies to form hypotheses of a specific label, e.g., adding a negator for contradictions. there have been several recent attempts to reduce such biases @xcite @xcite . there has also been a large body of work using probing datasets/tasks to stress-test nli models trained on datasets such as snli and mnli, in order to expose the weaknesses and biases in either the models or the data @xcite @xcite . for this work, we closely monitor the hypothesis-only and other biases but leave systematic filtering/biasreduction/stress-testing for future work. an interesting future challenge will involve seeing how such techniques, which focus exclusively on english, transfer to other languages such as chinese. ## creating ocnli here, we describe our data collection and annotation procedures. following the standard definition of nli @xcite , our data consists of ordered pairs of sentences, one premise sentence and one hypothesis sentence, annotated with one of three labels: entailment, contradiction, or neutral (see examples in following the strategy that williams et al. (2018) established for mnli, we start by selecting a set of premises from a collection of multi-genre chinese texts, see section 3.1. we then elicit hypothesis annotations based on these premises using expert annotators (section 3.2). we develop novel strategies to ensure that we elicit diverse hypotheses. we then describe our verification procedure in section 3.3. ## experimental investigation of ocnli 4.1 experimental setup to demonstrate the difficulty of our dataset, we establish baselines using several widely-used nli models tailored to chinese foot_4 . this includes the baselines originally used in @xcite such as the continuous bag of words (cbow) model, the bilstm encoder model and an implementation of esim @xcite 7 . in each case, we use chinese character embeddings from @xcite in place of the original glove embeddings. we also experiment with state-of-the-art pretrained transformers for chinese @xcite using the fine-tuning approach from @xcite . specifically, we use the chinese versions of bert-base @xcite and robertalarge @xcite with whole-word masking (see details in @xcite ). in both cases, we rely on the publicly-available tensorflow implementation provided in the clue benchmark @xcite 8 . following bowman et al. (2020), we also fine-tune hypothesis-only variants of our main models to measure annotation artifacts. to measure human performance, we employed an additional set of 5 chinese native speakers to annotate a sample (300 examples) of our ocnli test set. this follows exactly the strategy used in @xcite for measuring human performance in glue, and provides a conservative estimate of human performance in that annotators were provided with minimal amounts of task training (see appendix e for details). ## conclusion in this paper, we presented the original chinese natural language inference (ocnli) corpus, the first large-scale, non-translated nli dataset for chinese. our dataset is composed of 56,000 premise-hypothesis pairs, manually created by university students with a background in language studies, using premises from five genres and an enhanced protocol from the original mnli anno-tation scheme. results using bert and roberta show that our dataset is challenging for the current best pre-trained transformer models, the best of which is ∼ 12 percentage-points below human performance. we also demonstrate the relative advantage of using our human constructed dataset over machine translated nli such as xnli. to encourage more progress on chinese nlu, we are making our dataset publicly available for the research community at https://github.com/ cluebenchmark/ocnli and will be including it in the chinese natural language understanding (clue) @xcite ) benchmark (https:// www.cluebenchmarks.com/ ). given the wide impact that large-scale nli datasets, such as snli and mnli, have had on recent progress in nlu for english, we hope that our resource will likewise help accelerate progress on chinese nlu. in addition to making more progress on chinese nli, future work will also focus on using our dataset for doing chinese model probing (e.g., building on work such as 2020 )) and sentence representation learning @xcite , as well as for investigating bias-reduction techniques @xcite @xcite for languages other than english.