C ode NLP at S em E val-2023 Task 2: Data Augmentation for Named Entity Recognition by Combination of Sequence Generation Strategies (2023)

Abstract: In the article, we present the CodeNLP submission to the SemEval-2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition. Our approach is based on data augmentation by combining various strategies of sequence generation for training. We show that the extended procedure of fine-tuning a pre-trained language model can bring improvements compared to any single strategy. On the development subsets, the improvements were 1.7 pp and 3.1 pp of F-measure, for English and multilingual datasets, respectively. On the test subsets our models achieved 63.51% and 73.22% of Macro F1, respectively.

## introduction in this study, we address the topic of named entity recognition from the field of natural language processing. the task is to identify sequences of words in a text that refer to some categories of entities -people, locations, organizations, objects, events, etc. there is no one firm definition of what a named entity is. the definition and the range of categories may vary from one application to another. named entity recognition is a challenging task due to several factors. one is that many named entities are proper names, and proper names are rigid designators @xcite . proper names refer to entities. they do not describe the entities by definition. this implies that you should know that a given term is a named entity to identify a named entity. the set of named entities is unlimited. to make a model recognize a term as a named entity, you can train on recognition terms from the annotated dataset, provide some form of a common-sense knowledge base about the world, or infer some characteristics from training data. for instance, in the sample sentence "mark works in xax.", the term xax is probably a named entity because it is capitalized. however, semantic categorization might require some additional information about the term, as it might refer to a location (city or country) or a company name. for example, the following sentence, "he loves this city." might clarify that xax is a name of a city. the less information we have from the input text, the more information we need in the training dataset or external sources. in the paper, we deal with the named entity recognition task called multiconer ii @xcite . the main challenges of this task are: a) fine-grained categories of named entities -6 main categories and 36 subcategories; b) all texts are lowercase, and c)the dataset consists of short sentences. ## related work current state-of-the-art methods in the field of named entity recognition are based on either lstm networks @xcite , or pretrained language models in a transformers architecture @xcite @xcite . these methods are characterized by different approaches to improving performance by dealing with specific problems. an essential element is a way in which the processed text is represented. @xcite combines different vector representations, while @xcite propounds an entirely new approach to delimiting areas representing entities. in addition, @xcite proposes an extended presentation of entities in the context. @xcite describes the use of a special loss function for unbalanced data. instead, the @xcite authors propose to search for additional contexts for under-represented data to improve the quality of the system. the best-performing methods also vary depending on the dataset on which they are evaluated. thus, the current best performing solutions @xcite @xcite @xcite on the ontonotes v5 dataset @xcite are a disjoint set from the best performing solutions @xcite @xcite @xcite on the wnut 2017 dataset @xcite . the topics solved in this edition of the semeval event @xcite are a continuation of the topics from the previous edition @xcite . the main challenge of the previous edition was to perform the ner task in a small context. finally, we decided to use the poldeepner2 @xcite package as the foundation for our research. it is based on a transformer architecture and thus allows the use of pretrained language models. it differs from traditional token classification models in the way of representation subject to classification -classification is done on tokens, which represent single words, not subwords. in addition to this, it has different methods for adding a context representation to the sentence being analyzed. ## data in our research, we used solely the dataset provided by the shared task organizers @xcite . the dataset was provided in the conll format and was limited to two columns: the first one contains tokens text form and the fourth contains tokens label in the iob2 format. samples sentenced are presented in figure 1 . ## methodology our system employs a pre-trained masked language model and a fully-connected layer performing token head classification. the classification layer features dropout regularization. we used the mlukelarge @xcite ) model as the underlying pre-trained masked language model. the model is trained solely on the data provided by the organizers for this shared task. we used data augmentation on the level of sequence representation using different strategies of feeding data to the network. ## results in tables 4 and 5 , we present results obtained on the development subsets for english and multilingual datasets, respectively. to verify our hypothesis that data augmentation by simple sentence concatenation with different sentences can improve performance, we trained the models in two setupssingle and union. in the single setup we trained the model using single sentences as vectors. in the union setup we combined all three strategies, i.e. single, merged, and context. for english and multilingual datasets, the highest score was obtained for the union setup with the experiments' results confirmed that combining various strategies for sequence generation during training can improve the model's performance even when using the single strategy on inference. for english, when we used single strategy for training and inference, we obtained 69.55% of f-measure. for the combination of various strategies (union) and single on inference we got 71.24%. for the multilingual dataset, we obtained an even greater improvement from 76.44% to 79.51% on the test datasets, our models got 63.51% and 73.22% of f-measure, respectively, for english and multilingual datasets. ## conclusion our experiments showed that we could improve the performance of a model for named entity recognition using data augmentation on the sequence generation level without any additional data sources. we benefit from training the model with and without context, even when the context was unrelated to the sentence, and on the inference, we processed the sentence separately. the presented data augmentation technique helped improve the f-measure on the english development subset by 1.7 pp and the multilingual dataset by 3.1 pp.