V i NLI : A V ietnamese Corpus for Studies on Open-Domain Natural Language Inference (2022)

Abstract: Over a decade, the research field of computational linguistics has witnessed the growth of corpora and models for natural language inference (NLI) for rich-resource languages such as English and Chinese. A large-scale and high-quality corpus is necessary for studies on NLI for Vietnamese, which can be considered a low-resource language. In this paper, we introduce ViNLI (Vietnamese Natural Language Inference), an open-domain and high-quality corpus for evaluating Vietnamese NLI models, which is created and evaluated with a strict process of quality control. ViNLI comprises over 30,000 human-annotated premise-hypothesis sentence pairs extracted from more than 800 online news articles on 13 distinct topics. In this paper, we introduce the guidelines for corpus creation which take the specific characteristics of the Vietnamese language in expressing entailment and contradiction into account. To evaluate the challenging level of our corpus, we conduct experiments with state-of-the-art deep neural networks and pre-trained models on our dataset. The best system performance is still far from human performance (a 14.20% gap in accuracy). The ViNLI corpus is a challenging corpus to accelerate progress in Vietnamese computational linguistics. Our corpus is available publicly for research purposes.

## introduction although over 98 million people speak vietnamese globally foot_1 , vietnamese is considered a low-resource language for natural language processing (nlp) research because of the lack of human-annotated corpora. to help accelerate nlp progress, @xcite and @xcite collected a large number of human-annotated data to benchmark vietnamese nlp tasks. we built the vinli corpus for evaluating natural language inference (nli) models. nli is an emerging and important task in natural language understanding which is to predict the semantic relation between two sentences. several english corpora were released for the nli task @xcite . recently, nli has also witnessed corpuscreation efforts in other languages such as ocnli @xcite , kornli @xcite , and indonli @xcite . @xcite proposed the vietnamese-english nli task foot_2 with three labels: agree, disagree, and neutral. to contribute to the progress of nlp research for vietnamese, we introduce a high-quality, opendomain corpus for vietnamese nli. inspired by the success of nli corpora @xcite @xcite , we follow a strict annotation process and design the guidelines specific to vietnamese characteristics to make the corpus realistic and high-quality. however, snli @xcite uses image captions as the main data resource, which are often short, simple texts and limited in linguistic phenomena. therefore, we use a practical resource with various topics to capture diverse inferences of the vietnamese nli task. the premises in our corpus are sentences extracted from 800 news articles on 13 different topics. in addition, most of the previous corpora require annotators to create only one hypothesis sentence for each premise, for example, snli @xcite , multinli @xcite , and xnli @xcite . however, in reality, human reasoning is very diverse in many semantic ways from a given sentence. hence, we asked annotators to generate two hypothesis sentences for a premise sentence to capture many layers of semantic inference in our corpus. a similar approach has been implemented in the ocnli @xcite and indonli @xcite corpora. vinli not only has three inference labels as in most previous corpora but also has one more label. the other label is added to separate the inference types, which is different from the meaning of the neutral label because its purpose is to distinguish pairs of sentences that are unrelated in terms of semantic information, such as events, subjects, and objects. vinli is created and annotated by vietnamese native speakers with solid linguistic backgrounds. annotators are trained carefully to familiarize themselves with the corpus creation guidelines following a strict training process. to evaluate the challenge of the corpus to models, we employ three deep neural networks, including cbow @xcite , bil-stm @xcite , and esim @xcite . we also evaluate the sota pre-trained language models: bert @xcite , xlm-r @xcite , and phobert @xcite , which have achieved impressive performances on different nlp tasks. contributions of this study are as follows: (1) we introduce the vinli corpus, an open-domain, highquality corpus consisting of over 30,376 humanannotated sentence pairs for evaluating the nli task. (2) we conduct experiments on nli models including neural network-based and pre-trained transformer-based models. (3) we analyze the corpus and the experimental results in different linguistic aspects to gain more insights into vietnamese nli. ## related work in this section, we review existing corpora and sota models for natural language inference. ## result analysis to gain more insights, we analyze the two bestperformance models, including xlm-r large and phobert large on different linguistic aspects. in this section, xlm-r and phobert stand for xlm-r large and phobert large , respectively. ## conclusion and future work in this paper, we introduced vinli, an opendomain, high-quality corpus for evaluating vietnamese nli models. by a strict annotation scheme with high annotator agreements, 30,376 premisehypothesis pairs of the corpus were annotated by humans with solid linguistic backgrounds, which is the largest vietnamese nli corpus to date. the performances of the two powerful models (xlm-r and phobert) illustrate that our corpus is chal-lenging for the pre-trained language models in vietnamese, the best of which underperforms humans by over 14% (on three labels). vinli is available freely for research purposes in developing vietnamese nlu models. taking advantages of state-of-the-art models on large-scale, high-quality nli corpora (snli, multinli, xnli, ocnli, kornli, and indonli), we hope that our corpus will accelerate progress on vietnamese nli and other nlp tasks. based on the findings of our work, we continue to enhance the quality and quantity of the corpus with different data sources and expand our corpus with adversarial samples @xcite . moreover, future studies will concentrate on exploiting vietnamese models using bertology @xcite ) (e.g., sbert @xcite ) and improving nlp applications such as vietnamese machine reading comprehension models and retriever-reader question answering systems.