K o BEST : K orean Balanced Evaluation of Significant Tasks (2022)

Abstract: A well-formulated benchmark plays a critical role in spurring advancements in the natural language processing (NLP) field, as it allows objective and precise evaluation of diverse models. As modern language models (LMs) have become more elaborate and sophisticated, more difficult benchmarks that require linguistic knowledge and reasoning have been proposed. However, most of these benchmarks only support English, and great effort is necessary to construct benchmarks for other low resource languages. To this end, we propose a new benchmark named Korean balanced evaluation of significant tasks (KoBEST), which consists of five Korean-language downstream tasks. Professional Korean linguists designed the tasks that require advanced Korean linguistic knowledge. Moreover, our data is purely annotated by humans and thoroughly reviewed to guarantee high data quality. We also provide baseline models and human performance results. Our dataset is available on the Huggingface.

## introduction the nlp field is now facing unprecedented rapid development. a major factor propelling the progress is the existence of unified benchmark datasets like glue @xcite , which are designed to assess models' language understanding capabilities. such benchmark datasets, enabled modern pre-trained language models (plms), such as bert @xcite and gpt-2 and gpt-3 @xcite , to be assessed in objective and multifaceted manners. the success of glue also lead to similar benchmark datasets in a variety of other languages, such as french @xcite , korean @xcite , chinese @xcite and indonesian @xcite . however, many recent studies reveal that the outstanding performance of plms on such benchmark datasets seems plausible but not probable. these studies have found that datasets may contain many spurious artefacts, and the performance of plms is enhanced by excessive usage of said artefacts @xcite @xcite . another line of work observed that many plms, which showed promising results in glue, fall short of expectations for more difficult tasks that require linguistic knowledge @xcite or logical reasoning @xcite . as a result, the importance of well-designed evaluation datasets with higher difficulty-level has been highlighted, and new datasets, such as check-list @xcite , and logicnli @xcite , have been proposed. most of them only support specific languages like english, and it requires large efforts to build higher difficultylevel language evaluation suits for other low resource languages, however. when it comes to the korean language, two benchmarks are widely used: korean-nli & sts @xcite and klue @xcite . the former is machine-and humantranslated from english natural language inference (nli) and semantic textual similarity (sts) datasets, which hardly reflect the characteristics of the korean language. the latter is a korean version of glue benchmark which supports eight tasks, such as nli, sts, named entity recognition (ner), and relation extraction @xcite . although these tasks are useful for assessing general language ability, it is difficult to ascertain whether a model is able to reason based on more complicated knowledge beyond text form (e.g., passage of time, meaning of text, causality). to this end, we aim to construct a new benchmark dataset in korean named kobest, which consists of five downstream tasks that require advanced knowledge of korean. we carefully constructed the data based on the following design principles: ## design principles in this section, we provide detailed descriptions about how we attempted to achieve the design principles illustrated in section 1. ## experiments in this section, we provide model and human performance results. ## conclusion a well-designed benchmark dataset is crucial for an objective and precise evaluation of lms. following the glue benchmark, more challenging benchmarks have been proposed as modern lms become more elaborate and sophisticated. however, most of these benchmarks only support english or originate from english (e.g., translation), which hardly captures important characteristics of a specific language. to this end, we propose a new korean benchmark suite named kobest, which consists of five challenging downstream tasks. to overcome the disadvantages of the previous korean benchmarks, we focused on 1) employing koreanspecific knowledge, 2) achieving high data quality and 3) removing superficial cues. to achieve these goals, we worked with professional korean linguists and collected data manually and not automatically. we also conducted human-and modeldriven review processes to eliminate superficial cues from our dataset. moreover, we were extra cautious to avoid using unethical expressions. finally, we evaluated various plms on our new benchmark and provide baseline model and hu-man performance metrics. our experimental results show that current lms need further improvements to attain human-level language ability. we hope our new benchmark can contribute to advancements in the korean nlp field.