KOMBO : K orean Character Representations Based on the Combination Rules of Subcharacters (2024)

Abstract: The Korean writing system, Hangeul, has a unique character representation rigidly following the invention principles recorded in Hunminjeongeum. However, existing pre-trained language models (PLMs) for Korean have overlooked these principles. In this paper, we introduce a novel framework for Korean PLMs called KOMBO, which firstly brings the invention principles of Hangeul to represent character. Our proposed method, KOMBO, exhibits notable experimental proficiency across diverse NLP tasks. In particular, our method outperforms the state-of-the-art Korean PLM by an average of 2.11% in five Korean natural language understanding tasks. Furthermore, extensive experiments demonstrate that our proposed method is suitable for comprehending the linguistic features of the Korean language. Consequently, we shed light on the superiority of using subcharacters over the typical subword-based approach for Korean PLMs. Our code is available at: https://github.com/SungHo3268/KOMBO.

## introduction determining the representation of a word is the first stepping stone in building pre-trained language models. the predominant approach in english has attempted to decompose each word (e.g., pureness) into subwords (e.g., pure + ness) based on the frequency of the words, such as byte-pair encoding (bpe) @xcite , wordpiece @xcite , and sentencepiece @xcite . notably, even for the korean language, which has a significantly different linguistic structure from english, the subwordbased approach has been widely adopted for korean plms @xcite . however, the korean writing system (known as hangeul) has a unique property in representing letters. unlike english, which typically adheres to a "word-subword-character" structure, korean includes an additional "subcharacter" level consisting of chosung (initial consonants), joongsung (vowels), and jongsung (final consonants). this leads to a "word-subword-character-subcharacter" structure, giving rise to distinct korean linguistic features. this structure is even problematic when generating compound words, such as ᄀ ᅵᄎ ᅡ ᆺ ᄀ ᅵ ᆯ train track . 2 despite these nuances in subcharacters with the same meaning, subword-based methods are blind to this information, typically parsing the compound word as separate tokens: ᄀ ᅵ, ᄎ ᅡ ᆺ, and ᄀ ᅵ ᆯ. such structural differences raise questions about the suitability of the subword-based method for the korean plms, considering that it may overlook important linguistic information beyond characters. in this work, given that it is crucial to ground the linguistic nuances and unique structures of the korean language, we draw our attention towards subcharacter in building korean plms. to accurately reflect the linguistic information of subcharacters, we bring the historical document hunminjeongeum, which provides comprehensive insights into the design principles of subcharacters and their combination rules. the following are the two essential statements related to subcharacters (national hangeul museum, 2021): building on these principles, we propose a novel framework for korean plms, referred to as (korean character representations based on the combination rules of subcharacters). to instill the design principles and combination rules of subcharacters into plms, starts with subcharacters as the initial representations. these representations are progressively combined to form a character through a merging layer guided by the combination principles. additionally, we also introduce a masking strategy tailored to subcharacters, aimed at learning the structural knowledge of characters during pre-training. consequently, such a comprehensive and fundamental approach enables plms to better comprehend the unique structure and composition of the korean language. to demonstrate the efficacy of the proposed method, we perform extensive experiments over a wide range of nlp tasks and compare ours with a variety of korean tokenization methods in building plms. the results convincingly show that considering subcharacter, especially for jamo, in a principled manner brings substantial improvement to korean plms. moreover, the in-depth analysis supports that can better understand the korean features, such as conjugations and offensive language, compared to existing methods. in summary, the contributions of this paper include the following: ## experiments in this section, we conduct a comparative analysis between the proposed method and existing korean tokenization models on both standard korean datasets and noisy korean datasets. moreover, to assess the robustness of models in noisy korean settings, we synthetically injected the typos into the korean nlu datasets, such as kornli, korsts, nsmc, and paws-x. we randomly generated typos using four different typo methods, following @xcite ; insertion: randomly add a letter which is adjacent of the letter on the keyboard, transposition: randomly switches a letter with one of its neighbor letter, substitution: randomly changes a letter with one of its neighbor letters on the keyboard, and deletion: drops a random letter. the other settings, unless specified, follow the same manner as korean nlu tasks. ## ablations in ## analysis in this section, we analyze two korean linguistic characteristics. the primary objective is to investigate the resilience of predicates against variations arising from korean conjugations, while the secondary aim involves evaluating the model's robustness to korean offensive languages. ## conclusion although there is crucial linguistic information and clear invention principles in korean letters, the existing korean plms have overlooked them and just opted to use subword-level tokenization methods due to their high performance. in this paper, we first bring attention to the overlooked design principles and combination rules of subcharacters, as specified in hunminjeongeum. we have introduced a novel framework for korean plms called