K or NLI and K or STS : New Benchmark Datasets for K orean Natural Language Understanding (2020)

Abstract: Natural language inference (NLI) and semantic textual similarity (STS) are key tasks in natural language understanding (NLU). Although several benchmark datasets for those tasks have been released in English and a few other languages, there are no publicly available NLI or STS datasets in the Korean language. Motivated by this, we construct and release new datasets for Korean NLI and STS, dubbed KorNLI and KorSTS, respectively. Following previous approaches, we machine-translate existing English training sets and manually translate development and test sets into Korean. To accelerate research on Korean NLU, we also establish baselines on KorNLI and KorSTS. Our datasets are publicly available at https://github.com/kakaobrain/KorNLUDatasets .

## introduction natural language inference (nli) and semantic textual similarity (sts) are considered as two of the central tasks in natural language understanding (nlu). they are not only featured in glue @xcite and @xcite , which are two popular benchmarks for nlu, but also known to be useful for supplementary training of pre-trained language models @xcite as well as for building and evaluating fixedsize sentence embeddings @xcite . accordingly, several benchmark datasets have been released for both @xcite and sts @xcite in the english language. when it comes to the korean language, however, benchmark datasets for nli and sts do not exist. popular benchmark datasets for korean nlu typically involve question answering 12 and sentiment analysis foot_0 , but not nli or sts. we believe that the lack of publicly available benchmark datasets for korean nli and sts has led to the lack of interest for building korean nlu models suited for these key understanding tasks. motivated by this, we construct and release ko-rnli and korsts, two new benchmark datasets for nli and sts in the korean language. following previous work @xcite , we construct our datasets by machine-translating existing english training sets and by translating english development and test sets via human translators. we then establish baselines for both kornli and korsts to facilitate research on korean nlu. ## baselines in this section, we provide baselines for the korean nli and sts tasks using our newly created benchmark datasets. because both tasks receive a pair of sentences as an input, there are two different approaches depending on whether the model encodes the sentences jointly ("cross-encoding") or separately ("bi-encoding"). 5 ## effect of translation on data quality as noted in @xcite , translation quality does not necessarily guarantee that the semantic relationships between sentences are preserved. we also translated each sentence independently and took the gold labels from the original english pair, so the resulting label might no longer be "gold," due to both incorrect translations and (in rarer cases) linguistic differences that make it difficult to translate specific concepts. fortunately, it was also pointed out in @xcite that annotators could recover the nli labels at a similar accuracy in translated pairs (83% in french) as in original pairs (85% in english). in addition, our baseline experiments in section 4.1 show that supplementary training on kornli improves korsts performance (+1% for roberta and +4-11% for xlm-r), suggesting that the labels of kornli are still meaningful. another quantitative evidence is that the performance of xlm-r fine-tuned on kornli (80.3% with cross-lingual transfer) is within a comparable range of the model's performance on other xnli languages (80.1% on average). nevertheless, we could also find some (not many) examples the gold label becomes incorrect after translating input sentences to korean. for example, there were cases in which the two input sentences for korsts were so similar (with 4+ similarity scores) that upon translation, the two inputs simply became identical. in another case, the english word sir appeared in the premise of an nli example and was translated to ᄉ ᅥ ᆫᄉ ᅢ ᆼᄂ ᅵ ᆷ, which is a correct word translation but is a gender-neutral noun, because there is no gender-specific counterpart to the word in korean. as a result, when the hypothesis referencing the entity as the man got translated into ᄂ ᅡ ᆷᄌ ᅡ (gender-specific), the english gold label (entailment) was no longer correct in the translated example. more systematically analyzing these errors is an interesting future work, although the amount of human efforts involved in this analysis would match that of labeling a new dataset. ## conclusion we introduced kornli and korsts-new datasets for korean natural language understanding. using these datasets, we also established baselines for korean nli and sts with both cross-encoding and bi-encoding approaches. looking forward, we hope that our datasets and baselines will facilitate future research on not only improving korean nlu systems but also increasing language diversity in nlu research. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bertnetworks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992, hong kong, china. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. alex wang, yada pruksachatkun, nikita nangia, amanpreet singh, julian michael, felix hill, omer levy, and samuel r bowman. 2019. superglue: a stickier benchmark for general-purpose language understanding systems. arxiv preprint arxiv:1905.00537. alex wang, amanpreet singh, julian michael, felix hill, omer levy, and samuel bowman. 2018. glue: a multi-task benchmark and analysis platform for natural language understanding. in proceedings of the 2018 emnlp workshop black-boxnlp: analyzing and interpreting neural networks for nlp, pages 353-355, brussels, belgium. association for computational linguistics. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122. association for computational linguistics. yinfei yang, daniel cer, amin ahmad, mandy guo, jax law, noah constant, gustavo hernandez abrego, steve yuan, chris tar, yun-hsuan sung, brian strope, and ray kurzweil. 2019. multilingual universal sentence encoder for semantic retrieval.