K o C ommon GEN v2: A Benchmark for Navigating K orean Commonsense Reasoning Challenges in Large Language Models (2024)

Abstract: The evolution of large language models (LLMs) has culminated in a multitask model paradigm where prompts drive the generation of user-specific outputs. However, this advancement has revealed a critical challenge: LLMs frequently produce outputs against socially acceptable commonsense standards in various scenarios. To address this gap in commonsense reasoning, we present KoCommonGEN v2, a fine-grained benchmark dataset focused on Korean commonsense reasoning. This dataset, enriched with human annotations, comprises multiple-choice questions across seven error categories. These categories include commonsense memorization, numerical commonsense, toxic speech, and more, which are vulnerable to undermining the reliability of LLMs’ commonsense reasoning capabilities. The empirical results present that LLMs struggle with Korean commonsense reasoning. With human accuracy benchmarked at approximately 85%, GPT-4’s performance lags at about 74%, and other LLMs demonstrate an average accuracy of around 42%. Our findings emphasize the need for targeted improvements in Korean commonsense reasoning within LLMs, paving the way for more socially and contextually sensitive AI models.

## introduction in the field of natural language processing (nlp) research, the significance of benchmark datasets has grown notably with the advent of transformerbased language models @xcite and the feasibility of pre-training on large corpora. language models have evolved to engage in multitask learning, emphasizing the need for overall linguistic skills and language understanding @xcite . in particular, attaining performance akin to human cogni- † corresponding author tion remains a long-standing challenge, and language models are striving to grasp commonsense reasoning. this pursuit has led to the introduction of prominent commonsense benchmarks, such as commonsenseqa @xcite , @xcite , piqa @xcite , and swag @xcite . however, the efficacy of these commonsense benchmark datasets has begun to diminish with the innovative improvements of large language models (llms), such as gpt-3 @xcite , flan @xcite , instructgpt @xcite , llama2 @xcite , mistral @xcite , and @xcite . llms can outperform upper-bound even with few-shot prompt-based tuning or zero-shot settings. however, as a wide array of tasks and scenarios are consolidated into a single llm, the new concern is the heightened risk of these models generating outputs that violate commonsense in social and cultural contexts. recently, there has been a notable incident referred to as 'the macbook throwing incident of king sejong (1397-1450) 1 ' in the korean community where the media negatively described llms as lying to make hallucinations even in commonsense knowledge. this incident shows the increasing need to recognize the complexity of commonsense required from llms based on sociocultural differences and to adopt broad-ranging approaches in commonsense reasoning that extend beyond inherent everyday life or elementary school-level knowledge. it also demonstrates the importance of including commonsense knowledge shared among speakers of the same language community. therefore, we contend that evaluating llms for commonsense reasoning, if solely reliant on universally applicable commonsense knowledge and english translated without considering sociocultural knowledge, along with approaches limited in scope to explaining commonsense, are grounded in outdated concerns. to address the challenges posed by the new generation of llms, we introduce kocommon-gen v2, a dataset aimed at fine-grained evaluation of korean commonsense reasoning. we adopt the korean commongen @xcite 2 design and enhance outdated settings for evaluating the performance of llms. firstly, we reconstruct the dataset from scratch to associate more closely with korean commonsense knowledge, moving away from depending on the translated sources and universal commonsense. secondly, we expand the scope beyond elementary-level commonsense reasoning. this benchmark dataset includes seven error types to precisely identify where llms are missing commonsense reasoning. thirdly, we alter from a natural language generation-based evaluation to a multiple-choice task format. this change standardizes the evaluation process, ensuring that language models are tested on their reasoning ability to produce results aligned with korean commonsense based on unified instructions @xcite . our contributions are as follows: ## kocommongen v2 as illustrated in ## experimental settings we introduce models used in our experiment, prompts, n-shot examples, evaluation metrics, and human evaluation. more details are in appendix a. models to enhance the robustness and diversity of our experiments, we select models that have recorded high performance and are widely used as backbones across various models in the open-llm leaderboard. among these, we determine models that have undergone additional training based on korean or are frequently used within the korean community of users as our experimental subjects. we also consider models with a capacity of up to 13b parameters and detailed information publicly disclosed. for our experiments, we employ language models pre-trained in korean, including kogpt2 foot_4 @xcite , and polyglot-ko @xcite . for models based on english, we use opt @xcite , llama @xcite , llama2 @xcite . for a model based on english-chinese, we employ qwen @xcite . additionally, we utilize llama-ko-en foot_5 and llama2-ko-en foot_6 , adaptation of the llama/llama2 model with an expanded vocabulary, and further pre-training with additional korean and english corpora. models based on polyglot-ko with instruction tuning applied include kullm foot_7 and koalpaca foot_8 , while for llama2-ko-en, we use a model with instruction tuning and dpo @xcite applied. for commercial apis, we apply gpt-3.5 @xcite , gpt-4 (openai, 2023), and hyper-clova @xcite . prompt we feed the llms prompts as shown in evaluation metric we adopt utilizing generation probabilities as the evaluation method in lmevaluation-harness. we denote x 0:p as the prompt and x p:s as the continuation sentence with a token length of |s -p|. the continuation sentence comprises an 'answer number + sentence'. we calculate the log probability for every candidate sentence, and the answer is chosen based on the option with the highest probability. the probability for each candidate sentence is calculated as follows: this approach attempts to normalize for length by computing the average log probability per token. ## related work the advancement of nlp benchmarks has been pivotal in assessing model competencies in commonsense reasoning. these benchmarks, employing a variety of task formats, are key to assessing the inherent understanding and application of commonsense knowledge in unexplored scenarios. among these are question-answering tasks such as swag @xcite , commonsenseqa @xcite , and piqa @xcite , alongside context inference like art @xcite , glu-cose @xcite , and com-fact @xcite . additionally, generative reasoning benchmarks @xcite challenge models in creating contextappropriate responses. however, llms possess a basic level of commonsense reasoning that enables them to achieve upper-level performance easily @xcite , and existing benchmarks show limited scopes in dealing with commonsense knowledge that considers the complexity of sociocultural back-grounds @xcite . to address these issues, we consider the potential threats to llms identified in previous commonsense reasoning studies, particularly those unique to korean cultural and societal nuances. our approach is in line with potential threats to commonsense reasoning mentioned in previous research: commonsense distortion: the scope and content of commonsense accepted can vary depending on geographical diversity @xcite , and factual commonsense from different cultures may appear biased or distorted based on the trained language @xcite . commonsense memorization: llms tend to memorize data during the pre-training process @xcite , and the formation of commonsense knowledge is influenced by the frequency and the complexity of relationships, leading to a tendency to recall memories based on co-occurrence and simple reasoning @xcite . toxic speech: social and cultural elements can influence the training data, potentially carrying biases or aggressiveness @xcite . grammaticality: a generalized grammatical understanding is necessary for composing given elements @xcite , and constructing complete sentences that conform to commonsense requires consideration of grammatical correction for each language @xcite . plausibility: commonsense reasoning considers retrieved evidence and judgments made through implicit abductive reasoning @xcite . numerical commonsense: language models tend to show weaknesses in handling numerical information that falls within the range of commonsense @xcite . proverb: idiomatic expressions, whose meanings are non-compositional, remain a challenging area for language models, and attempts are being made to address this through the commonsense knowledge graph @xcite . we propose a korean commonsense reasoning benchmark dataset tailored to the new era of llms and capable of interacting with sociocultural contexts, categorizing these challenges to address them effectively. ## conclusion we introduce kocommongen v2, a new benchmark dataset for finely evaluating korean commonsense reasoning. our benchmark encompasses challenges in cultural and societal contexts for assess-ing llms' ability to handle commonsense reasoning. to navigate these challenges, we define seven error categories based on criteria highlighted in previous commonsense reasoning research. our analysis includes both open-source llms and commercially available llms accessed via api. despite advancements, these models still risk generating socially unacceptable errors or aggressive outputs in the context of korean commonsense reasoning. furthermore, most llms exhibit unstable performance across multilingual contexts and in following instructions. we hope our proposed benchmark dataset will contribute to developing more commonsense-aligned and reliable llms.