KDDIE at S em E val-2022 Task 11: Using D e BERT a for Named Entity Recognition (2022)

Abstract: In this work, we introduce our system to the SemEval 2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER) competition. Our team (KDDIE) attempted the sub-task of Named Entity Recognition (NER) for the language of English in the challenge and reported our results. For this task, we use transfer learning method: fine-tuning the pre-trained language models (PLMs) on the competition dataset. Our two approaches are the BERT-based PLMs and PLMs with additional layer such as Condition Random Field. We report our finding and results in this report.

## introduction in today's world there is an ever-growing supply of unstructured data, a lot of it in the form of free text. named entity recognition (ner) is a process that seeks to gather information from free text, by extracting and labeling the named entities. semeval 2022 task 11 is a competition where ner systems are trained and then they compete against each other for the best scores @xcite ). the task is split into 13 tracks: one for each of 11 different languages, a multi-lingual track, and a mixed language track @xcite ). in this paper we will be focusing on track 1, which is monolingual english ner. for this semeval task participants were asked to have their system both identify named entities and then label them with one of the six provided labels. the six labels to be used were person, location, group, corporation, product, and creative work @xcite ). figure 1 shows an example of these words being tagged with their appropriate labels. for this paper our strategy involved trying various transformer models from the huggingface library @xcite ) and training them on the training data provided. we then tried adding a conditional random field layer to some different models to see if that would improve the scores that we received. with all the models we trained we also fine-tuned many different training parameters to obtain the best scores possible. we tried using different number of epochs, different learning rates, different batch sizes, and changing many additional parameters. in the research we have done leading up to this paper we have learned many things. one is that adding a crf layer to a bert model @xcite ) helps its performance, but adding a crf layer to a deberta model ( he et al. (2021) ) doesn't help and actually hurts its performance for ner. we also learned through our testing that a deberta model @xcite ) is one of the best transformer models for ner, specifically on the semeval 2022 task 11 dataset @xcite ). ## related work there are many challenges that can make ner extremely difficult. in @xcite they explain that named entity recognition is especially difficult in situations with low-context or in scenarios where the named entities are exceptionally complex and unique. also as stated in @xcite ner requires well annotated data and lots of it. this a major obstacle because annotating data can be extremely time consuming and expensive. some older approaches to ner include rule-based systems and unsupervised learning systems, however both of these fall short of the performance of modern feature-based supervised methods, which often use transformer models @xcite ). according to @xcite using pretrained transformer models with other possible layers and then fine tuning these transformers is becoming the standard for ner. within this paper we will be using pre-trained transformer models to help us achieve the best results. another related work @xcite ) shows the use of a bert transformer model ( devlin et al. (2018) ) for named entity recognition. they also added a conditional random field (crf) layer on top the bert model which ended up improving results. this paper ( souza et al. (2019) ) shows that both bert is good for ner and that adding a crf layer can also help a transformer model perform better. ## system overview for this paper our main strategy will be fine-tuning some large feature-based transformer models on the training data provided. we used the hugging-face library @xcite ) to download and train the transformer models. to train our models we used the huggingface default optimizer called adamw. for the most part, we left the adamw optimizer parameters at the default values. also, for all of the models we used a linear learning rate scheduler with zero warm up steps. ## results based on the results shown in ## conclusion we tried many different types of models for this se-meval 2022 task 11 competition. we tried training each of these models on the training data, a bert model, a bert-crf model, a deberta-large model, a deberta-xlarge model, and lastly a deberta-crf model. in the end the fine-tuned deberta-xlarge model achieved the highest f1score. we also found out that adding a crf layer to a deberta pre-trained model didn't help its performance at all. for any future work or improvements on our current work, we would like to try adding some other layers to the deberta model and work on fixing some commmon errors our model has. since the deberta model seems to be the best at ner, we would like to modify it in some way or modify the data in some way to help it perform better. with further adjustments and experimentation, we believe that the performance of our deberta model could improve.