SRCB at S em E val-2023 Task 2: A System of Complex Named Entity Recognition with External Knowledge (2023)

Abstract: The MultiCoNER II shared task aims at detecting semantically ambiguous and complex named entities in short and low-context settings for multiple languages. The lack of context makes the recognition of ambiguous named entities challenging. To alleviate this issue, our team SRCB proposes an external knowledge based system, where we utilize 3 different types of external knowledge retrieved in different ways. Given an original text, our system retrieves the possible labels and the descriptions for each potential entity detected by a mention detection model. And we also retrieve a related document as extra context from Wikipedia for each original text. We concatenate the original text with the external knowledge as the input of NER models. The informative contextual representations with external knowledge significantly improve the NER performance in both Chinese and English tracks. Our system win the 3rd place in the Chinese track and the 6th place in the English track.

## introduction the task of multilingual complex named entity recognition (multiconer) @xcite aims to deal with the complex named entity recognition problem. unlike the ordinary entities in most named entity recognition (ner) tasks, these complex entities can be composed of any form of a language. compared with the 6 categories defined in multiconer @xcite , multiconer ii @xcite furtherly defines 33 fine-grained categories for the complex entities and provides them with short and uncased texts (in short and low-context settings). moreover, at the stage of test, some spelling mistakes are added into the test set, causing data disturbance. recognizing complex named entities in such settings is challenging for ner systems. in practice, for a professional annotator, a solution to deal with these complex entities is using external knowledge. the external knowledge includes professional knowledge of their own, information presented by search engines and contents in databases. retrieving such related external knowledge can help to determine whether a fragment of sentence is an entity or not and eliminate the ambiguity of complex entities @xcite . meanwhile, @xcite proposed a general knowledge-based ner system retrieve the related documents of the input sentence as external knowledge which is proved to be very effective in short and low-context named entity recognition. therefore, we believe that introducing external knowledge to ner systems is still a straight-forward and effective way to improve the performance of complex entity recognition. in this paper, we propose an external knowledge based system, which is composed of a knowledge retrieval module and an ner module. in the knowledge retrieval module, given an original text, we try to detect potential entities with a high-recall mention detection model and project the detected entities to the corresponding wikidata entities to collect entity types and entity descriptions from wikidata. besides, we also retrieve the most related document as extra context for the given text. we refer to the collected entity types, the entity descriptions and the retrieved extra context as prompt, description and context respectively. and then in the ner module, during training and predicting, we concatenate the original texts with these external knowledge as the input of a range of ner models, which use different pretrained language models or different model structures. finally, we use the model ensemble method of voting, which shows a certain improvement in both chinese and english track. ## system description in this section, we introduce our external knowledge based ner system. given an original text of n tokens @xmath0, x 2 , ..., x n }, the knowledge retrieval module first use a mention detection model to detect potential entities within the text as many as possible. then it takes the mentions of potential entities or the original text as the query to retrieve three types of external knowledge from different sources: prompt which is the possible labels of the potential entities retrieved from a knowledge base constructed from wikidata; description which is the concise descriptions for the potential entities retrieved from mediawiki api; context which is the related document of the original text retrieved from wikipedia. the system concatenates the original text with these three types of external knowl-edge into a new readable input text, and feed it into the ner module. although the models we implemented in our ner module differ in the output format, the outputs of these models are finally converted to ŷ = { ŷ1 , ŷ2 , ..., ŷn } ## results at the test stage, there are some spelling mistakes and typos added to the test set, which causes the knowledge retrieval module fails to retrieve external knowledge for some of the potential entities due to the matching problem.to deal with this problem, we use the predictions of the models which are trained without using external knowledge to supplement the final predictions on the tokens which are predicted as 'o' by the ensemble models. we won the 3rd place in the chinese track and the 6th place in the english track. for the test results of the chinese and the english track, our system obtains overall macro f1 of 75.86 and 75.62 respectively. although our system shows good performance on the clean subset, our system also suffers a great performance degradation on the noisy subset compared with that on the clean set, which shows the impact of spelling mistakes and typos is huge for our system. ## analysis to evaluate the quality and coverage of prompt for train, validate and test set retrieved from our knowledge base, we define a special recall which counts a tp if the correct label is one of the retrieved labels of a real entity. the evaluation results of the training set and the development set in the chinese track are 92.27 and 93.40. for the english track, the numbers are 86.45 and 84.65. from the evaluation results in the chinese track and the english track, we can observe that the performance in the english track falls significantly compared with that in the chinese track. this is caused by the fact that there are more potential entity string fail to match any of the wikidata entities due to the absence of certain mention of it. and due to the noise of spelling mistakes and typos in the test set, we observe a decrease in the number of potential entities for which at least 1 label has been retrieved. description also suffers a decrease in the number of potential entities for which at least 1 description has been retrieved. we didn't prepare fuzzy match for both english and chinese language due to the limit of the time. less prompt and description cause performance degradation of final ner performance on the test set, compared with that on the development set. fortunately, we applied fuzzy match to the retrieval of context, so that it is not influenced much as we observed. the reason for performance degradation on the test set is not only caused by the quality reduction of external knowledge. we expected that the spelling mistakes and typos only appear in the entity tokens, while they also appear in the tokens other than entity tokens. this makes a negative impact on the contextual representation, which causes the performance of ner models drop regardless of the external knowledge. comparing to majority voting, the method of random voting obtain 1.06 improvement on the development set and obtains 0.26 improvement on the test set. this method is not so effective on the test set as it does on the development set, and we assume the reason is that fitting the development set by random voting may lead to the loss of generalization ability on the test set. ## conclusion in conclusion, we describe our external knowledge based system which utilizes 3 different types of external knowledge for the multiconer ii task. and we use different model structures, methods and pretained language models for the ner task. our models benefits from the external knowledge and obtain great improvement on the development set. however, they also show the weakness of relying too much on the external knowledge from the performance degradation due to the quality decline of external knowledge retrieval on the test set. the result of model ensemble shows the methods of random voting is effective both on the development set and the test set. for future work, we plan to improve the performance of our knowledge retrieval module when faced with noise of spelling mistakes and typos, and explore different ways of external knowledge utilization that involve the modification of model structures. moreover, we will concentrate more on the importance of contextual information for ner models, in order to reduce the dependence on external knowledge.