S cd NER : Span-Based Consistency-Aware Document-Level Named Entity Recognition (2023)

Abstract: Document-level NER approaches use global information via word-based key-value memory for accurate and consistent predictions. However, such global information on word level can introduce noise when the same word appears in different token sequences and has different labels. This work proposes a two-stage document-level NER model, ScdNER, for more accurate and consistent predictions via adaptive span-level global feature fusion. In the first stage, ScdNER trains a binary classifier to predict if a token sequence is an entity with a probability. Via a span-based key-value memory, the probabilities are further used to obtain the entity’s global features with reduced impact of non-entity sequences. The second stage predicts the entity types using a gate mechanism to balance its local and global information, leading to adaptive global feature fusion. Experiments on benchmark datasets from scientific, biomedical, and general domains show the effectiveness of the proposed methods.

## introduction named entity recognition (ner) is an important task for many natural language processing applications @xcite @xcite . most ner models process sentences independently, leading to a risk of inconsistent prediction for the mentions of the same entity. that is, they can produce different predictions for the same entity mentions in different sentences depending on the contextual information for each sentence. consideration of document-level features in entity representation learning is likely to obtain more consistent and accurate predictions since different occurrences of a specific token sequence within a document are likely to have the same entity types @xcite . based on this idea, some document-level ner (dl-ner) approaches @xcite proposed to fuse global features of tokens via token-based key-value memory, where the features of the same token are fused to produce documentlevel encoding for token label prediction. however, this token-level feature-sharing strategy may introduce noise in global feature fusing, as the same token occurrences in different contexts may not carry the same labels. for instance, labels for token "type" in the entity span "breast cancer type 1" and the token sequence "a unique type of cancer" are gene-i and o, respectively (here, we call an entity span as a token sequence with an entity type). fusing features of these two tokens will bring noise in predicting their labels. to address the problem of token-level feature sharing, this work proposes a span-based consistency-aware dl-ner model (scdner) to enable context feature fusing on the span level. in particular, this work proposes a document-level context feature fusion technique for the same entity mentions. given a document, we first extract document-level contextual encoding for each enumerated token sequence. based on contextual encoding, we propose a two-stage prediction framework. in stage 1, a binary classifier is trained to predict if a token sequence is an entity span with a probability. stage 2 employs a span-based key-value memory to enable context feature fusing of the same entity spans using probabilities from stage 1 to reduce the impact of non-entity spans. the entity span features are updated by fusing global information with a gate to balance local and global features. finally, the entity type is predicted for an entity span. dl-ner approaches use document-level contextualized information for more accurate and consistent predictions. qian et al. (2019) use word sequence dependency structures to model non-local and nonsequential dependency. akbik et al. (2019) propose to pool contextualized embeddings of each unique this paper presents a formal analysis … an approximation of the formal analysis … x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 encoder span classification candidate spans surface name span features probability cspan0 this @xmath0.1 … … … … cspan12 analysis @xmath1.2 … … … … cspan17 formal analysis @xmath2, h5) @xmath3.8 … … … … cspan24 formal analysis @xmath4, h12) @xmath5.9 … … … … @xmath6. formal analysis hs17 0.8 hs24 0.9 … … … entity span feature update h0 h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 h11 h12 h13 mlp method figure 1: illustration of the proposed scdner. in this example, "formal analysis" is a surface name shared by two entity spans. their features are updated via fusing the other's contextual features, resulting in a "method" prediction. string. luo et al. (2020) employ a word-based key-value memory to memorize word encoding, and an attention mechanism computes each token's document-level representations. gui et al. (2021) design a two-stage label refinement network via a word-based key-value memory network. however, most existing works try to fuse word-level global information, which introduces noise when the same word in different entity spans has different labels. this work proposes to use a span-level key-value memory for global feature fusion. our model. scdner is a two-stage dl-ner model. in the first stage, a binary span classifier is trained to predict if a span is an entity. the second stage uses a span-based key-value memory to adaptively fuse global features of spans with the same surface names for better span feature representation learning. figure 1 illustrates the proposed scdner. ## experiments we evaluate the proposed scdner model on benchmark ner datasets from various domains. we aim to study if span-level global feature fusing improves span feature learning and ner results. experimental setups. we use biobert @xcite , scibert (cased) @xcite , and bert base (cased) @xcite based on huggingface's transformers @xcite as document encoder for biomedical, scientific, and general domain datasets, respectively. all hyper-parameters are tuned on the validation sets (see appendix a for details). we report the average over 5 runs for scdner. evaluation metrics include precision, recall, and f1 scores. ## conclusion this work proposes a two-stage dl-ner model, scdner, to enable more accurate and consistent ner prediction via global feature fusion of the same entity spans. in stage 1, scdner trains a binary span classifier to classify a span with a probability. stage 2 uses a span-based key-value memory to adaptively fuse global features of the same entity spans. the probabilities in stage 1 are reused to reduce the impacts of non-entity spans. empirical results on document-level ner datasets from various domains demonstrate the effectiveness.