Open K orean Corpora: A Practical Report (2020)

Abstract: Korean is often referred to as a low-resource language in the research community. While this claim is partially true, it is also because the availability of resources is inadequately advertised and curated. This work curates and reviews a list of Korean corpora, first describing institution-level resource development, then further iterate through a list of current open datasets for different types of tasks. We then propose a direction on how open-source dataset construction and releases should be done for less-resourced languages to promote research.

## introduction the korean language is less explored in terms of corpus and computational linguistics, but its prevalence is often underrated. it regards about 80 million language users and is recently adopted in multilingual research as it is bound to cjk (chinese, japanese, and korean), also handling a distinguished writing system. however, compared to the industrial need, the interest in korean natural language processing (nlp) has not been developed much in international viewpoints, which recurrently hinders the related publication and further academic extension. besides, in the recent nlp, where the benchmark practice is a trend, such systems lack at this point, deterring abroad and even native researchers who start korean nlp from finding directions. @xcite has shown a decent survey, but it seems that the techniques are mainly on the nlp pipeline. also, albeit some curations on korean nlp foot_0 and datasets foot_1 , we considered that little more organization is required, and better if internationally available. our attempts are expected to mitigate the challenges that the researchers who handle korean from a multi-or cross-lingual viewpoint may face. in this paper, we scrutinize the struggles of government, institutes, industry, and individuals to construct public korean nlp resources. first, we state how the institutional organizations have tackled the issue by making up the accessible resources, and point out the limitation thereof regarding international availability and license, to finally introduce and curate the fully public datasets along with the proposed criteria. through this, we want to find out the current state of korean corpora across the nlp tasks and whether they are freely or conditionally available. our survey is to be curated and updated in the public repository foot_2 . ## accessible resources with the increase in popularity of machine learningdriven methods in nlp, constructing a novel dataset and releasing it to the public can be considered the cornerstone of advancing research of a given language. while we believe many useful datasets exist behind industry walls, this is not particularly useful for advancing open research. fortunately, there are organizations that construct and distribute cleaned, pre-processed datasets which are occasionally accompanied by a task and the annotation. in the context of korean, there are numerous efforts in this field driven by governmentaffiliated organizations. ## open datasets all the datasets to be introduced from now on are fully open access. this means that the dataset is downloadable with a single click or cloning, or at least one can acquire the dataset with simple signing. we set three checklists for the status of the corpus, namely documentation, usage, and redistribution. the first one is on how fine-grained the corpus description is. ## summary in total, we surveyed 32 corpora, namely 18 korean text corpora, 9 multilingual corpora, and 5 speech corpora. they are composed of 7 datasets on parsing and tagging, 7 datasets on entailment, paraphrasing, and summarization, 8 datasets on (spoken language) classification, qa, and dialog, 5 datasets on machine translation/transliteraion, and 5 datasets on speech (pre-)processing foot_32 . we provide the full specification in documentation ensuring that a curated list of resources is up-to-date is a challenge. in this regard, we aim to make our work open and canonical, as an online repository of curated resources for korean. for the research community to have unconstrained access to all current open resources, while endorsing community contributions, the following criteria are crucial: we will make the registry machine parseable, so that other curated sites such as nlpprogress. org, can utilize the registry to automate updates. the project will be maintained as an open-source project, under a permissive license. a living document is a new territory for the field of academia, but we strongly believe that given the rapid progress of nlp research, this is an experiment worth attempting; and hope that a successful effort can inspire other languages to follow the same approach. our approach is to be described in the public repository, guaranteeing the accessibility for domestic and abroad researchers. also, a large portion of the data are expected to be more easily accessible via koco foot_33 and korpora foot_34 , the recently constructed dataset wrappers for korean nlp.