UA - KO at S em E val-2022 Task 11: Data Augmentation and Ensembles for K orean Named Entity Recognition (2022)

Abstract: This paper presents the approaches and systems of the UA-KO team for the Korean portion of SemEval-2022 Task 11 on Multilingual Complex Named Entity Recognition.We fine-tuned Korean and multilingual BERT and RoBERTA models, conducted experiments on data augmentation, ensembles, and task-adaptive pretraining. Our final system ranked 8th out of 17 teams with an F1 score of 0.6749 F1.

## introduction named entity recognition (ner) is the task of recognizing and classifying named entities in unstructured text. ner is a critical component for nlp tasks such as question answering and relation extraction. recent advances in neural ner have allowed state-of-the-art systems to perform well in recognizing persons, locations, and organizations in a variety of benchmark datasets. however, these systems still struggle to recognize complex named entities such as titles of creative works. semeval task 11 @xcite asks participants to build systems to identify both classic and complex named entities in multiple languages. in this paper, we describe the ua-ko team's approach to deal with the challenge of recognizing complex named entities in korean. we used both monolingual and multilingual transformerbased models. to improve the performance of the models, we conducted experiments on data augmentation, ensembles, and task-adaptive pretraining. our final performance on the test set ranked 8th out of 17 teams with an f1 score of 0.675. code to replicate our experiments is available at https://github.com/hyunssong/semeval2022task11 . semeval-2022 task 11 asks participants to build systems that can perform well in real-world settings, providing a dataset with complex named entities in short sentences. the task is divided into three tracks: multi-lingual, monolingual, and codemixed. we participated in the korean portion of the monolingual track. korean is an agglutinative language, where each word (eojeol) is formed by combining a root morpheme with a bound morpheme (josa) or postposition (eomi) @xcite . more than 60 different forms can thus be created from each root morpheme. a model that is not aware of this morphological richness may end up significantly increasing the vocabulary size to represent all these different forms, at the costs of having only sparse data to learn each form and encountering a high rate of out-of-vocabulary (oov) forms. recent studies have explored different representations to account for the agglutinative characteristics of korean. @xcite explored both the syllable level and sub-character level representations of the text, achieving similar results to multilingual bert with 1/10 of the training data. @xcite proposed a deep learning based ner system that operates over syllables rather than words, resulting in a speedup by removing the need for morphological analysis. @xcite al. (2021) ## data the semeval-2022 task 11 dataset follows the conll format. each token is classified into six entity types: person, location, group, corporation, product, or creative work. the size of the korean language portion of the dataset is shown in our data augmentation experiments (see section 4.2) utilized additional data. for locations, we used the geonames geographical database, which provides countries and place names in many different languages. we selected the 170 country names provided in the korean language. for person names, we used the encyclopedia of korean culture @xcite , which is a korean language encyclopedia. we selected their 18,506 names of korean historical figures. ## methodology we used both monolingual and multilingual pretrained models for the task. for the monolingual models, we used kobert foot_0 , kr-bert @xcite , ko-electra foot_1 , and klue-robertalarge @xcite , which all have shown successful performance on korean nlp tasks. for the multilingual model, we used xlm-robertalarge @xcite , which was provided as the baseline of the shared task. we fine-tuned these models on the training set. monolingual models were fine-tuned with a learning rate of 5e-5 for 5 epochs with a weight decay of 0.01. the multilingual model was trained with the same configuration but for 3 epochs. we also experimented with different methodologies to improve performance: task-adaptive pretraining, data augmentation, and ensembles. these approaches are described in the following sections. gururangan et al. (2020) showed that further pretraining models on the unlabeled task data, called task-adaptive pretraining, can improve model performance when the dataset is curated to capture language appropriate to the task. we follow the training environment as @xcite . to avoid catastrophic forgetting, we additionally reduced the number of epochs following @xcite 's approach of applying task-adaptive pretraining on small datasets. specifically, we conduct task-adaptive pretraining with a batch size of 256 for 50 epochs. for the koelectra model, we further train for 7k steps. xlm-r was not included in these experiments due to computing resource limitations. ## results on dev given these results, for our official submissions on the test set, we applied data augmentation with @xmath3, and we included several types of ensembles. we did not apply any task adaptive pre-training ## results on test annotation errors: we observed inconsistent labeling in named entities, especially in the creative work named entities. the development data contained many creative works enclosed in 《 and 》, such as example a1 in token boundary issues: we see that often tokens are misclassified due to token boundary issues. for instance, in example a3 in foreign names: the model had difficulty recognizing transliterated named entities, such as names of foreign people or groups as in example sentence a4 of ## conclusion we have presented a description of our different approaches for identifying complex named entities in korean language data. a monolingual model that considers characteristics of the korean language performs well, and an ensemble of monolignual models and a multilingual model further improves performance. though we also explored task-adaptive pretraining and data augmentation, task-adaptive pretraining did not help on the development data, and data augmentation helped on the development data but hurt on the test data. our results suggest that while ensembles yield reliable gains for korean named entity recognition, further research is needed to utilize external knowledge when dealing with complex named entity recognition.