I ndo NLI : A Natural Language Inference Dataset for I ndonesian (2021)

Abstract: We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We adapt the data collection protocol for MNLI and collect ~18K sentence pairs annotated by crowd workers and experts. The expert-annotated data is used exclusively as a test set. It is designed to provide a challenging test-bed for Indonesian NLI by explicitly incorporating various linguistic phenomena such as numerical reasoning, structural changes, idioms, or temporal and spatial reasoning. Experiment results show that XLM-R outperforms other pre-trained models in our data. The best performance on the expert-annotated data is still far below human performance (13.4% accuracy gap), suggesting that this test set is especially challenging. Furthermore, our analysis shows that our expert-annotated data is more diverse and contains fewer annotation artifacts than the crowd-annotated data. We hope this dataset can help accelerate progress in Indonesian NLP research.

## introduction indonesian language or bahasa indonesia is the 10th most spoken language in the world with more than 190 million speakers. in line with their effort, we introduce indonesian nli (indonli), a natural language inference dataset for indonesian. natural language inference (nli), also known as recognizing textual entailment (rte; @xcite is the task of determining whether a sentence semantically entails another sentence. nli has been used extensively as a benchmark for nlu, especially with the availability of large-scale english datasets such as the stanford nli (snli; @xcite and the multi-genre nli (mnli; @xcite datasets. recently, there have been efforts to build nli datasets in other languages. the most common approach is via translation @xcite . one exception is the work by @xcite which uses a human-elicitation approach similar to mnli to build an nli dataset for chinese @xcite . until now, there are two indonesian nli datasets available. the first one is @xcite , which is created using revision history from indonesian wikipedia. the second one, inarte @xcite , is created automatically based on question-answer pairs taken from web data. both datasets have relatively small number of examples (400 pairs for wrete and ∼1.5k pairs for inarte) and only uses two labels (entailment and non-entailment). furthermore, since the hypothesis sentence is generated automatically from the premise, they tend to be so similar that arguably they will not be effective as a benchmark for nlu @xcite . on the other hand, indonli is created using a human-elicited approach similar to mnli and ocnli. it consists of ∼18k annotated sentence pairs, making it the largest indonesian nli dataset to date. indonli is annotated by both crowd workers (layperson) and experts. lay-annotated data is used for both training and testing, while expertannotated data is used exclusively for testing. our goal is to introduce a challenging test-bed for indonesian nli. therefore the expert-annotated test data is explicitly designed to target phenomena such as lexical semantics, coreference resolution, idioms expression, and common sense reasoning. we also propose a more efficient label validation protocol. instead of selecting a consensus gold label from 5 votes as in mnli data protocol, we incrementally annotate the label starting from 3 annotators. we only add more label annotation if consensus is not yet reached. our proposed protocol is 34.8% more efficient than the standard 5 votes annotation. we benchmark a set of nli models, including multilingual pretrained models such as xlm-r @xcite and pretrained models trained on indonesian text only @xcite . we find that the expert-annotated test is more difficult than lay-annotated test data, denoted by lower model performance. the hypothesis-only model also yields worse results on our expert-annotated test, suggesting fewer annotation artifacts. furthermore, our expert-annotated test has less hypothesispremise word overlap, signifying more diverse and creative text. overall, we argue that our expertannotated test can be used as a challenging test-bed for indonesian nli. we publish indonli data and model at https://github.com/ir-nlp-csui/indonli . for nli datasets in other languages, the crosslingual nli (xnli) corpus extends mnli by manually translating sampled mnli test set into 15 other languages @xcite . the original chinese natural language inference (ocnli) is a large-scale nli dataset for chinese created using data collection similar to mnli @xcite . other works contribute to creating nli datasets for persian @xcite and hinglish @xcite . some corpora are created with a mix of machine translation and human participation. the turkish nli (nli-tr) corpus is created by machinetranslating snli and mnli sentence pairs into turkish, which are then validated by turkish native speakers @xcite . for dutch, wijnholds and moortgat (2021) introduce the sick-nl by machine-translating the sick dataset @xcite . it is then manually reviewed to maintain the correctness of the translation. americasnli, an extension of xnli to 10 indigenous languages of the americas, is created with the primary goal to investigate the performance of nli models in truly low-resource language settings @xcite . the dataset is built by translating spanish xnli into the target languages, and vice versa, using transformerbased sequence-to-sequence models. research conducted by tsuchiya (2018); poliak et al. (2018b); gururangan et al. (2018) show that there are hidden biases in the nli corpus, such as word choices, grammaticality, and sentence length, which allow models to predict the correct label only from the hypothesis. ## experiments we experiment with several neural network-based models to evaluate the difficulty of our corpus. as our baseline, we use a continuous bag of words (cbow) model, initialized with indonesian fast-text embeddings @xcite . the others are transformers-based models: multilingual bert @xcite , xlm-r @xcite , and two pre-trained models for indonesian, indobert and indobert-lite @xcite . the first two are multilingual models which are trained on multilingual data, which includes indonesian text. indobert uses bert large architecture with 335.2m parameters, while indobertlite uses albert @xcite architecture with fewer number of parameters (11.7m). ## cross-lingual transfer performance prior work @xcite has demonstrated the effectiveness of crosslingual transfer when training data in the target language is not available. to evaluate the difficulty of our test sets in this setting, we experiment with two cross-lingual transfer approaches, zero-shot learning, and translate-train. in this experiment, we only use xlm-r as it obtains the best performance in our nli evaluation. in the zero-shot learning, we employ an xlm-r model trained using a concatenation of mnli training set and xnli validation set, which covers 15 languages in total. 7 in the translate-train setting, we machine-translate mnli training and validation sets into indonesian and fine-tune the pre-trained xlm-r on the translated data. our english to indonesian machine translation system uses the standard transformer architecture @xcite with 6 layers of encoders and decoders. following guntara et al. (2020), we train our translation model on a total of 13m pairs of multi-domain corpus from news articles, religious texts, speech transcripts, wikipedia corpus, and back-translation data from oscar (ortiz suárez et al., 2020). we use marian (junczys-dowmunt et al., 2018) toolkit to train our translation model. translate-train outperforms a model trained on our training data (indonli) on test expert . translate-train obtains the best performance with 5.4 points over the indonli model. we further investigate if the performance gap comes from the larger training data used for training our translatetrain model. we train another model (translatetrain-s), using a subset of translated training data such that the size is comparable to indonli training data. we find that the model performance using the same training data is also higher, although the gap is smaller (0.8 points). since we do not include expert data in the indonli training data, this result indicates that the translated training data might contain more examples with similar characteristics with examples in test expert than indonli training data. overall, we observe that the best performance on our test expert is still relatively low (e.g., 75.7 compared to the best performance in ocnli, 78.2), indicating that the test set is still challenging. we see that many premise-hypothesis pairs in in-donli diagnostic set apply lexical semantics (e.g., synonyms, antonyms, and hypernyms-hyponyms) or require common-sense knowledge (i.e., a basic 8 we use the same examples to evaluate human baseline understanding of physical and social dynamics) to make an inference. pairs with num tag also occur with high frequency in our data, whereas the challenges of idiomatic expression is less prevalent. few phenomena are evenly distributed among labels, e.g., num, quant, and coord. on the other hand, there is only a small proportion of pairs in lsub and struct categories which have neutral labels. many examples of neg unsurprisingly have contradiction label. our analysis on diagnostic set shows that the models can handle examples tagged with morphosyntactic categories or boolean logic well. co-ord and morph are among 2 tags with highest performance for xlm-r. the translate-train model achieves 85% on lsub, indicating that our data is considerably robust with respect to syntactic heuristics, such as lexical overlap and subsequence @xcite . on the other hand, all models also have decent accuracy on inference pairs with negation; the performance remains stable in three different models (more than 70%). in contrast, the hardest overall categories appear to be num and comp, indicating that models struggle with arithmetic computation, reasoning about quantities, and dealing with comparisons @xcite . in addition, models find it difficult to reason about temporal and spatial properties, as the accuracy for examples annotated with temp and spat are also inadequate. commonsense reasoning is shown as another challenge for nli model, suggesting that there is still much room for improvement for models to learn tacit knowledge from the text. ## conclusion we present indonli, the first human-elicited nli data for indonesian. the dataset is authored and annotated by crowd (lay data) and expert annotators (expert data). indonli includes nearly 18k sentence pairs, makes it the largest indonesian nli dataset to date. we evaluate state-of-the-art nli models on indonli, and find that our dataset, especially the one created by expert is challenging as there is still a substantial human-model gap on test expert . the expert data contains more diverse hypotheses and less annotation artifacts makes it ideal for testing models beyond its normal capacity (stress test). furthermore, our qualitative analysis shows that the best model struggles in handling linguistic phenomena, particularly in numerical reasoning and comparatives and superlatives. we expect this dataset can contribute to facilitating further progress in indonesian nlp research.