PAI at S em E val-2022 Task 11: Name Entity Recognition with Contextualized Entity Representations and Robust Loss Functions (2022)

Abstract: This paper describes our system used in the SemEval-2022 Task 11 Multilingual Complex Named Entity Recognition, achieving 3rd for track 1 on the leaderboard. We propose Dictionary-fused BERT, a flexible approach for entity dictionaries integration. The main ideas of our systems are:1) integrating external knowledge (an entity dictionary) into pre-trained models to obtain contextualized word and entity representations 2) designing a robust loss function leveraging a logit matrix 3) adding an auxiliary task, which is an on-top binary classification to decide whether the token is a mention word or not, makes the main task easier to learn. It is worth noting that our system achieves an F1 of 0.914 in the post-evaluation stage by updating the entity dictionary to the one of (CITATION), which is higher than the score of 1st on the leaderboard of the evaluation stage.

## introduction name entity recognition (ner) is a fundamental task in natural language processing. processing complex and ambiguous named entities (nes) and in low-context situations is a challenging nlp task in practical and open-domain settings, which was recently outlined by @xcite . other work has extended this to multilingual and codemixed settings @xcite since codemixed queries, with entities in a different language than the rest of the query, pose a particular challenge in domains like e-commerce (e.g. queries containing movie or product names). semeval-2022 task 11 multilingual complex named entity recognition @xcite focuses on dealing with the challenges above: detecting complex entities in short, low-context, and code-mixed settings. for this task, we propose dictionary-fused bert to integrate external dictionaries into the ner model, and it is compatible with emerging entities and user-defined entities, without retraining the model. ## related work named entity recognition (ner) is a core task in knowledge extraction and is important to various downstream applications such as question answering and dialogue systems. ner is the task of detecting mentions of real-world entities from text and recognizing their types (e.g., locations, persons). however, the ner task is facing many challenges outlined by @xcite , such as short texts like search queries @xcite , emerging entities @xcite , long-tail entities, complex entities, and entities in code-mixed queries @xcite . for example, complex nes, like the titles of creative works (movie/book/song/software names) are not simple nouns and are harder to recognize. the neural models driven by memorization perform well on "easy" entities (person names) but fail to recognize complex/unseen entities when entities overlap less between train/test sets. a lot of works have been proposed to address the challenges above. ## data we leverage luke entity dictionary @xcite since our model needs external knowledge. we train and test on multiconer dataset @xcite . multiconer.dataset for the semeval-2022 task 11, containing a training set of size 15300, a dev set of size 800, and a test set of size 217818, consisting of 6 entity types: person (per for short, names of people), location (loc, locations/physical facilities), corporation (corp, corporations and businesses), groups (grp, all other groups), product (prod, consumer products), and creative-work (cw, movie/song/book/etc. titles). all data are uncased. luke entity dictionary consists of 500k entities retrieved from english wikipedia data. ## methodology we propose dictionary-fused bert, which adopts a multi-layer bidirectional transformer @xcite and has the ability to encode not only words of the sentence but also information of matched entities. we experiment with transformerbased models which produce contextualized word representations. the overall architecture is shown in figure 1 , the components are detailed below. ## conclusion in this paper, we focused on how to integrate external dictionaries into ner models to deal with ner challenges in open-world settings and design a robust loss function to prevent overfitting. we proposed dictionary-fused bert, a flexible and simple approach that includes a contextualized entity representation encoder and a robust loss function leveraging a logit matrix. more importantly, our approach can infuse entity dictionaries into any transformer-based pre-trained models and is compatible with any emerging entities and user-defined entities without retraining the model. although we have already succeeded in injecting entity information into transformer-based models, entity spans are implicitly informed through a special token $. in the future, we will attempt to explore other ways to explicitly encode entity span information.