Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings (2021)

Abstract: Word embedding is considered an essential factor in improving the performance of various Natural Language Processing (NLP) models. However, it is hardly applicable in real-world datasets as word embedding is generally studied with a well-refined corpus. Notably, in Hangeul (Korean writing system), which has a unique writing system, various kinds of Out-Of-Vocabulary (OOV) appear from typos. In this paper, we propose a robust Hangeul word embedding model against typos, while maintaining high performance. The proposed model utilizes a Convolutional Neural Network (CNN) architecture with a channel attention mechanism that learns to infer the original word embeddings. The model train with a dataset that consists of a mix of typos and correct words. To demonstrate the effectiveness of the proposed model, we conduct three kinds of intrinsic and extrinsic tasks. While the existing embedding models fail to maintain stable performance as the noise level increases, the proposed model shows stable performance.

## introduction word embedding refers to the process of generating vectors that contain semantic and syntactic information of languages. machines can understand the meaning of words through the word embeddings. several embedding methodologies have been proposed to improve various natural language processing (nlp) models' performance, such as machine translation, sentence classification, and text generation @xcite @xcite . word2vec @xcite generates word vectors using the contextual data of words based † equal contribution on the distributional hypothesis. word2vec assigns a unique vector to each word, and has proved its effectiveness by improving various systems' performance. similarly, glove @xcite that uses co-occurrence information of each word found throughout the corpus, generates word vectors that contain semantic and syntactic characteristics. however, word2vec and glove suffer from being dependent on the training corpus. in particular, they do not include the word embeddings that did not appear in the training process. to tackle this oov problem, replacing unseen words with a unique token "unk" is widely used as a solution. this method can allow the model to deal with oov words. however, this countermeasure reduces the performance of the system. this is because several unseen words are recognized as having the same meaning. hence, the importance of each word is ignored and recognized as noise. this oov problem amplifies when dealing with real-world datasets that contain a large number of typos and intended slang that has not been included in the training phase. in particular, if words relevant to the core meaning of a sentence are replaced as oov, the system's performance will drastically reduce. for example, suppose an evaluation set for a sentiment classification task contains typos such as "funnnny" and "coooool". when predicting the sentiment of the sentence, the original meanings of the two words, "funny" and "cool", serve as decisive information. however, the sentiment classification accuracy drastically reduces when these words are processed as "unk" and encoded to have the same meaning. therefore, to increase the effectiveness of nlp models, robust word embeddings are strongly needed in noisy text. meanwhile, fasttext @xcite ) presents a method to address the oov problem. fasttext follows the training process of word2vec, but understands a word as a set of n-gram subword information. for example, the meaning of the word 'funny' is learned by decomposing the characters into n-grams such as 'fun', 'unn', and 'nny.' in this way, fasttext can generate embeddings for oov words that did not appear during the training phase by utilizing this subword information. fasttext is suitable for languages that have characteristics of isolated words (e.g., english). this is because the meaning of the word can be inferred through the summation of subword information: prefix, root, and postfix. however, in the case of agglutinative languages such as hangeul (korean writing system), it is not easy to fully understand the meaning of a word by merely referring to the summation of the subword information. a word in hangeul consists of at least one to ten or more different morphemes, which make extracting internal information complicated. therefore, to decipher a hangeul word, it is necessary to be sensitive when looking for the vital subword information within the word. cnn captures locational and morphological information of words and characters, and many studies using cnn have achieved good performance in various nlp tasks @xcite @xcite . inspired by this, in this paper, we propose a cnn-based channel attention word embedding model that generates robust hangeul embeddings for noisy text. the proposed model utilizes cnn to extract n-gram pairs from korean words. the model then applies an attention mechanism to features from each channel representing the inherent information in the n-gram pairs. an input word for training is replaced with intentionally generated typos with a certain probability of adapting to typos. this generated word vector enhances the semantics of a word by predicting the context words. in summary, the contributions of this paper are three-fold: the remainder of the paper is organized as follows. in section 2, we investigate related work and discuss key differences from ours. in section 3, we describe the proposed methodology in detail. furthermore, we present the evaluation results and analyze the performance in section 4 and 5, respectively. finally, section 6 contains the conclusion. ## related work in this section, we review previous studies on 1) embedding generation methods that are robust to typos and 2) embedding generation methods that consider the unique characteristics of hangeul. ## methodology in this section, we describe the architecture of the proposed model. the model consists of three parts. the first part is the generation of korean typos. we create rule-based typo datasets, and they reinforce the robustness of the embedding model for typos. next, we create a jamo-level cnn with channel attention to represent the word vector. by applying channel attention to features extracted through convolution operation from each jamo, we selectively enhance the importance of the primary ngram features. based on a certain probability, the cnn learns to generate the original word vector from the typos, reducing the difference between the typos and the actual word embeddings. finally, we perform the task of predicting the surrounding context words from the generated word embeddings. through this process, we can proceed with the reinforcement of the semantics of words. figure 1 shows the whole architecture of our model. ## experiments in this section, we present our experimental results by comparing the performance of the proposed model with other embedding methods. we measure the performance of the models through word analogy, language modeling, and sentiment classification tasks. ## analysis in this section, we analyze the proposed methodology. we find the nearest neighbor words mapped close to the misspelled word to evaluate robustness of the typo embeddings qualitatively. ## conclusion in this paper, we proposed a robust hangeul embedding model against typos. for this purpose, we intentionally generated typos to train word embeddings. a channel attention mechanism is used to utilize the structure information found in korean words selectively. this reflects the distinctive characteristics of hangeul, which combines several morphemes to form a single word, without depending on the performance of a morpheme analyzer. based on this, the training progresses from the extracted jamo n-gram feature of the target words through the convolution layer to aim the pre-trained word embeddings. the proposed embedding model outperformed the current state-of-the-art in the intrinsic task. also, we verify that the original research goal was achieved by functioning a stable embeddings even in the extrinsic tasks. we plan to apply the proposed methodology to various downstream tasks (e.g., pos tagging and machine translation) and other agglutinative languages, such as japanese and turkish.