A Dog Is Passing Over The Jet? A Text-Generation Dataset for K orean Commonsense Reasoning and Evaluation (2022)

Abstract: Recent natural language understanding (NLU) research on the Korean language has been vigorously maturing with the advancements of pretrained language models and datasets. However, Korean pretrained language models still struggle to generate a short sentence with a given condition based on compositionality and commonsense reasoning (i.e., generative commonsense reasoning). The two major challenges are inadequate data resources to develop generative commonsense reasoning regarding Korean linguistic features and to evaluate language models which are necessary for natural language generation (NLG). To solve these problems, we propose a text-generation dataset for Korean generative commonsense reasoning and language model evaluation. In this work, a semi-automatic dataset construction approach filters out contents inexplicable to commonsense, ascertains quality, and reduces the cost of building the dataset. We also present an in-depth analysis of the generation results of language models with various evaluation metrics along with human-annotated scores. The whole dataset is publicly available at ( https://aihub.or.kr/opendata/korea-university ).

## introduction with the advent of transformer @xcite model, the importance of language resources and language modeling in natural language processing (nlp) has been heightened. indeed, various studies on korean language resources, such as korean morpheme analysis @xcite @xcite , natural language understanding (nlu) tasks including kornli and korsts @xcite , kmre @xcite , and klue @xcite figure 1 : example results of the kogpt2, kobart, mbart-50, mt5, and humans on korean generative commonsense reasoning task. the results are sentences that combine the given content morphemes (in red boldface). despite the prosperity of korean nlp research, two critical problems remain: (i) an absence of a research base for natural language generation (nlg) and (ii) a deficient ability for models to generate commonsense knowledge. in other words, (i) there exists neither a dataset @xcite nor standards for evaluating @xcite the results generated by language models in korean because most korean language resources are focused on nlu tasks thereby making it difficult to accelerate the development of nlg research. though people acquire commonsense knowledge and effortlessly use it in their daily lives, it is challenging for language models to imitate this ability. as shown in figure 1 , off-the-shelf korean and multilingual language models seem to lack the competence for generative commonsense reasoning. some model-generated sentences do not make sense (e.g., "개가 제트기 위를 지나고 있다. a dog is passing over the jet.", "재채기가 코를 닦 고 있다. sneeze is wiping the nose."), use inappropriate prepositions (e.g., "파인애플 에서 칼을 베고 먹는다. i cut a knife from a pineapple and eat it."), or misplace parts of speech (e.g., "러닝머신이 음악을 듣고 달 리고 있다. the treadmill is listening to music and running."). to address these issues and inspired by com-mongen @xcite , we develop a korean commongen dataset for generative commonsense reasoning. the dataset is composed of commonly used daily-life concepts and sentences made by combining those concepts. our dataset differs from the commongen dataset as follows: (i) we collect korean corpus and label it to cover korean sociocultural commonsense knowledge. for example, the sentence of the corpus contains the unique korean sociocultural terms "귀농 (return to the farm)" and "곶감 (dried persimmons)". (ii) although we adopt the image caption data from commongen, we add the summary data of daily conversations into our dataset thereby obtaining diverse sentences. (iii) because korean language models use segmented morphemes as vocabulary @xcite , we construct the concept set with content morphemes that have linguistic features and lexical meanings. @xcite we analyze the evaluation metrics including the human-annotated score to demonstrate the validity of the evaluation criteria. we reduce the cost of data construction significantly through an automated method and inspect the quality and unethical issue of korean common-gen by crowd-sourcing 1 . in addition, we conduct 1 we employ @xcite an in-depth study of our proposed dataset with various ablation experiments on morpheme segmentation and training methods. furthermore, the modelgenerated sentences are compared and analyzed by quantitative, qualitative, and human scores. we disclose the dataset used in this paper to contribute to the development of korean nlg research. compositionality compositionality is an essential element that ai systems have to solve for given conditions. for example, the ms-coco @xcite dataset is utilized for image caption task generating the description from an image data as an input data. the task demands the compositionality of the model as the model composes the natural description. moreover, scan @xcite demonstrates mapping instructions to sequence an rnn model's ability to generate continuous behavior. however, these studies have shown that ai systems still struggle to generate complete results. commonsense reasoning commonsense reasoning is the ability to infer unrecognized commonsense knowledge or relations among given concepts. in a recent nlp research, various commonsense reasoning datasets have been disclosed. commonsenseqa @xcite organizes the dataset with closed questions to commonsense based on the concept of conceptnet and analyzes the skills of commonsense reasoning required for each question by categorizing them. cosmos qa dataset @xcite introduces a question answering dataset based on the fact that is not externally revealed in the context. @xcite attempts to strengthen the commonsense training of the model by adding a person's description of the commonsense qa. the xcopa dataset @xcite mitigates the gaps in commonsense that could arise from linguistic and cultural differences, while building a dataset. ## korean commongen korean commongen is a text-generation dataset for korean commonsense reasoning and evalua-tion. as depicted in figure 2 , korean commongen consists of concept sets typically used in daily life and sentences depicting those concepts. language models are trained to generate a sentence by reasoning and combining the concepts based on humangenerated sentences. the model-generated result should include all the given concepts, be grammatically correct and make sense. korean commongen includes 43,188 training examples and 2,040 testing examples as presented in ## quantitative evaluation we conduct a quantitative analysis of our dataset as shown in second, kobart has a model parameter of 124m which is smaller than that of the mbart and mbart-50 models (whose model parameters 11 https://github.com/skt-ai/kogpt2 12 https://github.com/skt-ai/kobart are 610m) but exhibits partially competent performance. this result shows that if the models are of identical structure, the multilingual model with a high proportion of machine-translated data in the pre-training process may encounter difficulties in generating high-quality sentences based on generative commonsense reasoning. third, mbart, and mbart-50 show similar performance. in the case of mbart-50, lowresource languages have the effect of improving, but high-resource languages present partial degradation because of the curse of dimension. korean is a medium-resource language, but its performance decreases like other high-resource languages. forth, mt5-large model has the most model parameters of 1.3b, and the pre-training method using sequence-to-sequence task form with prompt engineering appears to impact it positively. additionally, the mt5-large shows the most comparable execution to human performance and higher coverage and semantic scores than barts. however, most of small size baselines still have tribulation generating sentences containing all of the given content morphemes, and overall performance is lower than humans. ## ablation study we attempt to demonstrate the validity of concept extraction and data sources. thus, we perform ablation studies with respect to the concept set and data source configuration method. ## human evaluation we further conduct a human evaluation for modelgenerated sentences, as shown in the correlation score between each evaluation criteria of human evaluation and automatic evaluation metrics is described in figure 4 . the heatmap shows that recall-based metrics have high correlations with human scores. this result corresponds to the demonstrations in @xcite and @xcite . in the case of coverage, a different trend showing low correlations with other human scores except for factuality is observed. it can be predicted that the criteria we have suggested and the basis for human judgment are somewhat consistent. the lower correlation in factuality with automatic evaluation metrics shows that containing all the given concepts does not necessarily constitute a well-crafted sentence. moreover, human-annotated scores have statistical significance in correlation with automatic evaluation metrics (p.@xmath0.05), and show considerably similar values to figure 3 in bert-based semantic scores. however, there are weak positive correlations with other automatic evaluation metrics. these results show that it is difficult to evaluate the model's generative commonsense reasoning in the traditional estimation approach, and advanced research is needed to improve it. ## conclusion in this paper, we propose a korean commongen dataset including korean sociocultural commonsense knowledge and morpheme-based linguistic features. the dataset heeds the semi-automatic dataset construction method based on automatic construction and crowd-sourcing annotation with quality assessment. we perform a comparative analysis and an ablation study to demonstrate the validity of the dataset and evaluation metrics for generative commonsense reasoning. moreover, we conduct a korean and multilingual language models' standard performance experiment to investigate the dataset's problems and competencies. in future work, the dataset will enhance evaluation metrics regarding syntax and diversity of sentences to improve the interpretation of the model-generated results. we believe that our proposed dataset will serve as a fundamental resource to korean nlg and commonsense reasoning research.