GNN er: Reducing Overlapping in Span-based NER Using Graph Neural Networks (2022)

Abstract: There are two main paradigms for Named Entity Recognition (NER): sequence labelling and span classification. Sequence labelling aims to assign a label to each word in an input text using, for example, BIO (Begin, Inside and Outside) tagging, while span classification involves enumerating all possible spans in a text and classifying them into their labels. In contrast to sequence labelling, unconstrained span-based methods tend to assign entity labels to overlapping spans, which is generally undesirable, especially for NER tasks without nested entities. Accordingly, we propose GNNer, a framework that uses Graph Neural Networks to enrich the span representation to reduce the number of overlapping spans during prediction. Our approach reduces the number of overlapping spans compared to strong baseline while maintaining competitive metric performance. Code is available at https://github.com/urchade/GNNer .

## introduction named entity recognition (ner) is an information extraction task that aims to identify named entities such as locations, organizations and person names from textual data. frequently, ner is designed as a sequence labelling task where each word is classified into its respective label using an annotation scheme such as bio @xcite . such schemes are used to encode segment information on the token level. recently, span-based ner has gained a lot of popularity by handling segments, instead of individual words, as the basic units for labelling @xcite . specifically, span-based ner enumerates every segment in a text and classifies them by their entity label, whereby non-entity segments are classified into an allocated null label. while this method has shown good empirical results, it often assigns entity labels to overlapping spans, which is not desirable, especially for flat ner tasks. therefore, to ensure that entities do not overlap, a constraint must be explicitly applied during decoding through, for example, semi-markov crfs @xcite . recent work by @xcite and @xcite address overlapping entities using heuristic decoding: conflict between overlapping spans is resolved by retaining the span with the highest prediction probability, dropping the others. this approach has proven effective, however, the no-overlap constraint is not imposed during learning, which is sub-optimal. in this work, we consider that the no-overlap constraint could be optimized directly by injecting inductive biases into the model. in this regard, we propose a new approach to reduce overlapping in span-based ners without affecting the efficiency of heuristic-based decoding. the idea is to make the representation of each span directly influenced by other spans overlapping with it. specifically, we encode overlapping information as a graph and feed it into the span representation using an equivariant graph neural network layer. in this way, we bias the model towards predictions that implicitly respect the constraints without explicitly modelling them. our results demonstrate that injecting this graph during model training significantly reduces the number of overlaps compared to our baseline model while achieving better performance. we propose, in this paper, two variants of our model, gnner-conv based on the graph convolution network @xcite and gnner-at based on the graph attention network @xcite . we observe that gnner-at is best at preventing span overlaps at the cost of a low recall, while gnner-conv provides a better trade-off between the number of violated constraints and metric performance (precision, recall and f-score). ## model given an input sequence, our task involves enumerating and classifying every span. the architecture of our model, summarized in figure 1 , includes the following components: token representation layer, span representation layer, gnn layer and span classification layer. our model is similar to the vanilla span-based ner models @xcite , to which we add the gnn layer. ## limitations there are several limitations to our approach. first, the addition of gnn does not completely remove the overlapping spans in contrast to heuristic approches. moreover, the inclusion of gnn layer bring more comptation to the model which result into a slower model than the baseline span-based ner. in fact since, the overlaping span graph is dense (contains many egde), the model does not really benefit of efficient sparse operations of gnn layers. ## related works approaches for ner ner is an important tasks in natural language processing and is used in many downstream information extraction applications. usually, ner tasks are designed as sequence labelling @xcite @xcite @xcite . the goal is to predict bio tags in which a word is labelled as b-tag if it is the beginning of an entity, i-tag if it is within but not the first in the entity and o for nonentity words. recently, different approaches have been proposed to perform ner tasks that go beyond traditional sequence labelling. one approach that has been widely adopted is the span-based approach @xcite @xcite where the representation of each segment is computed using a neural network, then fed to a classifier. to prevent overlapping span, priors works either used heuristic decoding @xcite @xcite or structured decoding using semi-crfs @xcite . however, to the best of our knowledge, no work have used gnn for the purpose of reducing span overlap for ner. some work @xcite has also approached ner as a question answering task in which named entities are extracted by retrieving answer spans. in addition, with the growing popularity of prompt-based learning, recent work such as @xcite considers ner as template filling by fine-tuning a bart @xcite encoder-decoder model. in contrast we focus on learning appropriate span representations. gnn for nlp gnns have gained a lot of popularity recently due to their powerful ability to represent arbitrary shapes of data @xcite @xcite . specifically, gnns provide a way to inject prior knowledge into nlp systems through, for example, dependency graphs @xcite , constituency graphs @xcite or knowledge graphs @xcite . as a result, gnns have been widely applied to different nlp tasks such as neural machine translation @xcite , semantic parsing @xcite , information extraction @xcite ) and text classification @xcite . more relevant to our work, dygie @xcite used gnns to refine the span representation for joint ner and re extraction, but in contrast, they learn their graph dynamically during training while we used a static span graph. for a detailed review of gnns for nlp, please refer to @xcite . ## conclusion in this work, we investigated new span-based ner method using graph neural networks. our best approach, built on a graph convolution network, significantly reduces the number of overlapping spans compared to a strong baseline (up to 2 times less) while maintaining competitive metric performance. in future work, we will explore ways to integrate gnn-enhanced representations into architectures for joint named entity recognition and relation extraction tasks.