Deep Span Representations for Named Entity Recognition (2023)

Abstract: Span-based models are one of the most straightforward methods for named entity recognition (NER). Existing span-based NER systems shallowly aggregate the token representations to span representations. However, this typically results in significant ineffectiveness for long entities, a coupling between the representations of overlapping spans, and ultimately a performance degradation. In this study, we propose DSpERT (Deep Span Encoder Representations from Transformers), which comprises a standard Transformer and a span Transformer. The latter uses low-layered span representations as queries, and aggregates the token representations as keys and values, layer by layer from bottom to top. Thus, DSpERT produces span representations of deep semantics. With weight initialization from pretrained language models, DSpERT achieves performance higher than or competitive with recent state-of-the-art systems on six NER benchmarks. Experimental results verify the importance of the depth for span representations, and show that DSpERT performs particularly well on long-span entities and nested structures. Further, the deep span representations are well structured and easily separable in the feature space.

## introduction as a fundamental information extraction task, named entity recognition (ner) requires predicting a set of entities from a piece of text. thus, the model has to distinguish the entity spans (i.e., positive examples) from the non-entity spans (i.e., negative examples). in this view, it is natural to enumerate all possible spans and classify them into the entity categories (including an extra non-entity category). this is exactly the core idea of spanbased approaches @xcite @xcite . analogously to how representation learning matters to image classification @xcite @xcite , it should be crucial to construct good span representations for span-based ner. however, existing models typically build span representations by shallowly aggregating the top/last token representations, e.g., pooling over the sequence @xcite @xcite , or integrating the starting and ending tokens @xcite . in that case, the token representations have not been fully interacted before they are fed into the classifier, which impairs the capability of capturing the information of long spans. if the spans overlap, the resulting span representations are technically coupled because of the shared tokens. this causes the representations less distinguishable from the ones of overlapping spans in nested structures. inspired by (probably) the most sophisticated implementation of attention mechanism -transformer and bert @xcite , we propose dspert, which stands for deep span encoder representations from transformers. it consists of a standard transformer and a span transformer; the latter uses low-layered span representations as queries, and token representations within the corresponding span as keys and values, and thus aggregates token representations layer by layer from bottom to top. such multilayered transformer-style aggregation promisingly produces deep span representations of rich semantics, analogously to how bert yields highly contextualized token representations. with weight initialization from pretrained language models (plms), dspert performs comparably to recent state-of-the-art (sota) ner systems on six well-known benchmarks. experimental results clearly verify the importance of the depth for the span representations. in addition, dspert achieves particularly amplified performance im-provements against its shallow counterparts foot_0 on long-span entities and nested structures. different from most related work which focuses on the decoder designs @xcite @xcite , we make an effort to optimize the span representations, but employ a simple and standard neural classifier for decoding. this exposes the pre-logit representations that directly determine the entity prediction results, and thus allows further representation analysis widely employed in a broader machine learning community (van der @xcite . this sheds light on neural ner systems towards higher robustness and interpretability @xcite . ## related work the ner research had been long-term focused on recognizing flat entities. after the introduction of linear-chain conditional random field @xcite , neural sequence tagging models became the de facto standard solution for flat ner tasks @xcite @xcite @xcite . recent studies pay much more attention to nested ner, which a plain sequence tagging model struggles with @xcite . this stimulates a number of novel ner system designs beyond the sequence tagging framework. hypergraph-based methods extend sequence tagging by allowing multiple tags for each token and multiple tag transitions between adjacent tokens, which is compatible with nested structures @xcite . span-based models enumerate candidate spans and classify them into entity categories (sohrab and miwa, 2018; eberts and ulges, 2020; @xcite . @xcite reformulates nested ner as a reading comprehension task. @xcite borrow the methods from image object detection to solve nested ner. @xcite propose a generative approach, which encodes the ground-truth entity set as a sequence, and thus reformulates ner as a sequenceto-sequence task. @xcite describe the entity set by word-word relation, and solve nested ner by word-word relation classification. the span-based models are probably the most straightforward among these approaches. however, existing span-based models typically build span representations by shallowly aggregating the top token representations from a standard text encoder. here, the shallow aggregation could be pooling over the sequence dimension @xcite , integrating the starting and ending token representations @xcite , or a concatenation of these results @xcite . apparently, shallow aggregation may be too simple to capture the information embedded in long spans; and if the spans overlap, the resulting span representations are technically coupled because of the shared tokens. these ultimately lead to a performance degradation. our dspert addresses this issue by multilayered and bottom-to-top construction of span representations. empirical results show that such deep span representations outperform the shallow counterpart qualitatively and quantitatively. ## analysis of pre-logit representations for a neural classification model, the logits only relate to the pre-defined classification categories, while the pre-logit representations contain much richer information @xcite . hence, the analysis of pre-logit representations has become a popular tool in machine learning research @xcite @xcite . however, such analysis is incompatible with most neural ner systems except for the span-based ones. our dspert employs a standard classification head, which exposes the pre-logit representations and thus allows the analysis. in this section, we investigate the pre-logit span representations, i.e., z ij in eq. ( 6 ), providing more insights into why the deep span representations outperform the shallow counterparts. ## discussion and conclusion neural nlp models have been rigidly adhere to the paradigm where an encoder produces tokenlevel representations, and a task-specific decoder receives these representations, computes the loss and yields the outputs @xcite . this design works well on most, if not all, nlp tasks; and it may also deserve a credit for facilitating nlp pretraining @xcite , since such common structure in advance bridges the pretraining and downstream phases. however, this paradigm may be sub-optimal for specific tasks. in span-based ner (or information extraction from a broader perspective), the smallest modeling unit should be spans instead of tokens; in conclusion, deep and span-specific representations can significantly boost span-based neural ner models. our dspert achieves sota results on six well-known ner benchmarks; the model presents pronounced effect on long-span entities and nested structures. further analysis shows that the resulting deep span representations are well structured and easily separable in the feature space. 10573 to some extent, dspert pursues performance and interpretability over computational efficiency. the major computational cost of a transformer encoder is on the multihead attention module and ffn. for a t -length input and a d-dimensional transformer encoder, the per-layer complexities of the multihead attention and ffn are of order o(t 2 d) and o(t d 2 ), respectively. when the maximum span size k ≪ t , our span transformer brings additional o(k 2 t d) complexity on the attention module, and o(kt d 2 ) complexity on the ffn. empirically, training a dspert consumes about five times the time for a shallow model of a same scale. however, this issue can be mitigated if we use fewer layers for the span transformer (subsection 4.3). as noted, we empirically choose the maximum span size k such that it covers most entities in the training and development splits. from the perspective of f 1 score, this heuristic works well, and dspert performs favourably on long-span entities as long as they are covered. however, the entities with extreme lengths beyond k will be theoretically irretrievable.