Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledge (2023)

Abstract: Although named entity recognition (NER) helps us to extract domain-specific entities from text (e.g., artists in the music domain), it is costly to create a large amount of training data or a structured knowledge base to perform accurate NER in the target domain. Here, we propose self-adaptive NER, which retrieves external knowledge from unstructured text to learn the usages of entities that have not been learned well. To retrieve useful knowledge for NER, we design an effective two-stage model that retrieves unstructured knowledge using uncertain entities as queries. Our model predicts the entities in the input and then finds those of which the prediction is not confident. Then, it retrieves knowledge by using these uncertain entities as queries and concatenates the retrieved text to the original input to revise the prediction. Experiments on CrossNER datasets demonstrated that our model outperforms strong baselines by 2.35 points in F1 metric.

## introduction named entity recognition (ner) helps us to extract entities from text in various domains such as biomedicine @xcite , disease @xcite , and covid-19 @xcite . however, accurate neural ner requires a massive amount of training data @xcite @xcite . as well, the annotation of a domain-specific ner dataset costs a lot of money because it requires the involvement of domain experts. to compensate for the lack of training data in ner, researchers have utilized external knowledge. traditional feature-based ner uses features based on gazetteers or name lists @xcite @xcite as external knowledge. although recent neural ner methods can even benefit from gazetteers and name lists @xcite et al., 2020), only a few domains with structured knowledge bases (gazetteers) have this merit. thus, several studies have resorted to using raw text (unstructured knowledge) to perform weaklysupervised learning on general-domain structured knowledge @xcite @xcite . ## task settings we developed sa-ner to solve the problems of ner with unstructured knowledge. ner is a sequence tagging task in which the model inputs a token sequence @xmath0, where v is the vocabulary and l is the maximum sequence length. the model outputs a bio label sequence of the same length. let c be the number of types. then, the number of the bio labels is 2c + 1. sa-ner assumes a corpus as an unstructured knowledge, which is split into token sequences of length l, following the existing retrievalaugmented language model (lm) @xcite , in order to store a large corpus efficiently. we retrieve m pieces of knowledge and concatenate them into x. we feed the concatenated text x + ∈ v (m+1)l to the model. ## related work here, we review ner that uses raw text (unstructured knowledge) without structured knowledge, with in-domain structured knowledge, with generaldomain structured knowledge, and for pre-training of billion-scale lms . also, we review the retrievalaugmented lms. ## method here, we present sa-ner. we explain the construction of the unstructured knowledge base ( §4.1), the encoder architecture ( §4.2), the two-stage ner algorithm which revises the prediction using the unstructured knowledge ( §4.3), the training method ( §4.4), and the pre-training method ( §4.5). ## evaluation we conducted experiments on three ner datasets to evaluate the effectiveness of our self-adaptive ner with unstructured knowledge. we used the entitylevel f 1 as the metric, following the literature. ## conclusions we proposed sa-ner, which is designed for ner to retrieve knowledge from the labeled and unlabeled ukbs by using unconfident entities and given inputs as queries. it encodes many pieces of knowledge efficiently with sparse attention. in experiments, sa-ner outperformed distilbert and bert baselines pre-trained on the conll03 and nerbert datasets by 1.22 to 2.35 points. we found that the entity-level retrieval, the focus on the unconfident entities, the labeled and unlabeled ukbs, and the large m that is enabled by the sparse attention all contribute to sa-ner's performance. we believe that sa-ner can help application providers to develop ner services in their target domain with domain-specific entity types that they have defined, even if they do not have an annotated dataset sufficiently.