KDDIE at S em E val-2023 Task 2: External Knowledge Injection for Named Entity Recognition (2023)

Abstract: This paper introduces our system for the SemEval 2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II) competition. Our team focused on the sub-task of Named Entity Recognition (NER) for the language of English in the challenge and reported our results. To achieve our goal, we utilized transfer learning by fine-tuning pre-trained language models (PLMs) on the competition dataset. Our approach involved combining a BERT-based PLM with external knowledge to provide additional context to the model. In this report, we present our findings and results.

## introduction there is a large and ever-increasing supply of unstructured data present in today's world. much of this data is in the form of free text. named entity recognition (ner) is the task of labeling named entities within the text. this allows us to gather structured data from the free text. semeval 2023 task 2 is a competition in which groups compete to build the best ner system for the provided data @xcite ). this task contains 13 different tracks: one for each of 12 different languages and a multilingual track, which combines all the other languages @xcite ). in this paper, we will focus on track 1 which is monolingual english ner. the task requires ner systems to identify 36 different labels. the six coarse labels to be used were person, location, group, medical, product, and creative work @xcite ). each of these coarse-level labels is then split into several fine-grain labels. for example, location is split into facility, human settlement, station, and other locations. figure 1 shows some examples of words being labeled with their corresponding tags. in this paper, we used external knowledge injection from outside data sources to provide additional context to our plms, as described in (wang christoph haberland designed a new marble pulpit for the church which was built in italy in 1793. ## related work the work in this competition is a continuation of the research done in the previous multilingual ner task done last year @xcite ). this year's competition is more difficult because there are many more possible labels. our team also submitted a report @xcite ) to last year's competition where we made use of deberta @xcite ) for the ner competition. there are many different challenges that make ner difficult. it explained in @xcite that identifying named entities is especially difficult when there is not much context or in situations where the entities are exceptionally complex. these issues were key problems that made last year's competition dataset @xcite so difficult to get good scores on. this problem is partially solved by using external knowledge injection to add more context as done by @xcite which we make use of in the work for this paper. as explained in @xcite ner also requires large amounts of well-annotated data. this can be a problem because it can be quite expensive to annotate data. ## data the competition data was given in conll format @xcite ). for the english data, there were 16778 examples for training and 871 examples for validation @xcite ). the data was provided in bio format. bio is a labeling scheme in which, if a word is at the beginning of an entity it is labeled b, if the word is inside the entity it is labeled i, if the word is outside of an entity it is labeled o. we started processing the data by splitting the data in each example and then splitting each example into lists of labels and tokens. next, we assigned each label a number, 0-72, to represent it. finally, we create a hugging face dataset object from these lists of lists @xcite ). for our experiments, we used the default train/eval splits which have 16778 examples for training and 871 examples for validation. for this paper, we used the metrics of macro precision, recall, and f1-score to evaluate our models. to calculate these metrics, we make use of seqeval @xcite ) a python library. seqeval makes it simple to calculate these metrics, we just need to give it the predicted values and the ground truth. the equations seqeval @xcite ) uses to calculate these metrics are shown below with tp meaning true positives, fp meaning false positives, and fn meaning false negatives: m acro p @xmath0 ## methodology our main approach for this paper is to fine-tune large feature-based transformer models on the provided training data, using external knowledge to enhance our approach. we then combine the predictions from all models to get the final results. we used the huggingface library @xcite ) to download and train the transformer models. we followed in the work of @xcite to extract external knowledge to provide our models with extra context. figure 2 shows a pipeline diagram of our system. the first step is feeding the input sentence into the knowledge injection module, which produces a new sentence with extra context appended on the end. then as seen in figure 2 this new sentence is given to three different transformer models to predict an output on. after this is done the output of the three models is then combined to produce a final prediction. each of these steps is described in more detail in the upcoming sections. ## results as seen in ## conclusion throughout the competition, we experimented with several different models and found that the de-berta model performed the best individually. additionally we found that adding the extra context improved the performance of all the models. moreover, we also discovered that even though the de-berta model performed the best on its own, we could improve its scores by combining the predictions with the other models. in future work, we could explore using different data sources for exter-nal knowledge injection to determine whether they can further enhance our results.