Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks (2023)

Abstract: Entity and Relation Extraction (ERE) is an important task in information extraction. Recent marker-based pipeline models achieve state-of-the-art performance, but still suffer from the error propagation issue. Also, most of current ERE models do not take into account higher-order interactions between multiple entities and relations, while higher-order modeling could be beneficial.In this work, we propose HyperGraph neural network for ERE (HGERE), which is built upon the PL-marker (a state-of-the-art marker-based pipleline model). To alleviate error propagation, we use a high-recall pruner mechanism to transfer the burden of entity identification and labeling from the NER module to the joint module of our model. For higher-order modeling, we build a hypergraph, where nodes are entities (provided by the span pruner) and relations thereof, and hyperedges encode interactions between two different relations or between a relation and its associated subject and object entities. We then run a hypergraph neural network for higher-order inference by applying message passing over the built hypergraph. Experiments on three widely used benchmarks (ACE2004, ACE2005 and SciERC) for ERE task show significant improvements over the previous state-of-the-art PL-marker.

## introduction entity and relation extraction (ere) is a fundamental task in information extraction (ie), compromising two sub-tasks: named entity recognition (ner) and relation extraction (re). there is a long debate on joint vs. pipeline methods for ere. pipeline decoding extracts entities first and predicts relations solely on pairs of extracted entities, while joint decoding predicts entities and relations simultaneously. recently, the seminal work of @xcite shows that pipeline decoding with a frustratingly simple marker-based encoding strategyi.e., inserting solid markers @xcite around predicted subject and object spans in the input text -achieves state-of-the-art re performance. modified sentences (with markers) are fed into powerful pretrained large language models (llm) to obtain more subject-and object-aware representations for re classification, which is the key to the performance improvement. however, current markerbased pipeline models (e.g., the recent state-of-theart ere model pl-marker @xcite ) only send predicted entities from the ner module to the re module, therefore missing entities would never have the chance to be re-predicted, suffering from the error propagation issue. on the other hand, for joint decoding approaches (e.g. table filling methods @xcite @xcite ))-though they do not suffer from the error propagation issue-it is hard to incorporate markers for leveraging llms, since entities are not predicted prior to relations. our desire is to obtain the best of two worlds, being able to use marker-based encoding mechanism for enhancing re performance and meanwhile alleviating the error propagation problem. we adopt pl-marker as the backbone of our proposed model and a span pruning strategy to mitigate error propagation. that is, instead of sending only predicted entity spans to the re module, we over-predict candidate spans so that the recall of gold entity spans is nearly perfect (but there also could be many non-entity spans), transferring the burden of entity classification and labeling from the ner module to the re module of pl-marker. the number of over-predicted spans is upper-bounded, balancing the computational complexity of marker-based encoding and the recall of gold entity span. empirically, we find this simple strategy by itself clearly improves pl-marker. we further incorporate a higher-order interaction module into our model. most previous ere models either implicitly model the interactions between instances by shared parameters @xcite @xcite or use a traditional graph neural network that models pairwise connections between a relation and an entity @xcite . it is difficult for these approaches to explicitly model higher-order relationships among multi-instances, e.g. the dependency among a relation and its corresponding subject and object entities. many recent works in structured prediction tasks show that explicit higher-order modeling is still beneficial even with powerful large pretrained encoders @xcite @xcite , motivating us to use an additional higherorder module to enhance performance. a common higher-order modeling approach is by means of probabilistic modeling (i.e., conditional random field (crf)) with end-to-end mean-field variational inference (mfvi), which can be seamlessly integrated into neural networks as a recurrent neural network layer @xcite , and has been widely used in various structured prediction tasks, such as dependency parsing @xcite , semantic role labeling @xcite , and information extraction @xcite . however, the limitations of crf modeling with mfvi are i): crf's potential functions are parameterized in log-linear forms with strong independence assumptions, suffering from low model capacities @xcite , ii) mfvi uses fully-factorized bernoulli distributions to approximate the otherwise multimodal true posterior distributions, oversimplifying the inference problem and thus is sub-optimal. therefore we need more expressive tools to improve the quality of higher-order inference. fortunately, there are many recent works in the machine learning community showing that graph neural networks (gnn) can be used as an inference tool and outperform approximate statistical inference algorithms (e.g., mfvi) @xcite @xcite (see @xcite for a survey). inspired by these works, we employ a hypergraph neural network (hypergnn) instead of mfvi for high-order inference and propose our model hgere (hypergraph neural network for ere). concretely, we build a hypergraph where nodes are candidate subjects and objects (obtained from the span pruner) and relations thereof, and hyperedges encode the interactions between either two relations with shared entities or a relation and its associated subject and object entity spans. in contrast, existing gnn models for ie @xcite only model the pairwise interactions between a relation and one of its corrsponding entity. we empirically show the advantages of our higher-order interaction module (i.e., hypergraph neural network) over mfvi and tranditional gnn models. our contribution is three-fold: i) we adopt a simple and effective span pruning method to mitigate the error propagation issue, enforcing the power of marker-based encoding. ii) we propose a novel hypergraph neural network enhanced higher-order model, outperforming higher-order crf-based models with mfvi. iii) we show great improvements over the prior state-of-the-art plmarker on three commonly used benchmarks for ere: ace2004, ace2005 and scierc. ## method overview. our method is built upon the state-ofthe-art pl-marker. we employ a high-recall span pruner to obtain candidate entity spans, similar to the ner module in pl-marker. however, instead of aiming to accurately predict all possible entity spans, our pruner focuses on removing unlikely candidates to achieve a much higher recall. then we feed the candidate span set to the re module to obtain entity and relation representations, which are used to initialize the node representations of our hypergraph neural network for higher-order inference with a message passing scheme. finally, we perform ner and re based on the refined entity and relation representations. fig. 1 depicts the neural architecture of our model. ## experiment datasets we experiment on scierc (luan et al., 2018), ace2004 @xcite and ace2005 @xcite . we follow @xcite to split ace2004 into 5 folds and split ace2005 and scierc into train/dev/test sets. see appendix a.1 for detailed dataset statistics. evaluation metrics we report micro labeled f1 measures for ner and re. for re, the difference between rel and rel+ is that the former requires correct prediction of subject and object entity spans and the relation type between them, while the latter additionally requires correct prediction of subject and object entity types. ## conclusion in this paper, we present hgere, a joint entity and relation extraction model equipped with a span pruning mechanism and a higher-order interaction module (i.e., hgnn). we found that simply using the span pruning mechanism by itself greatly improve the performance over prior state-of-the-art pl-marker, indicating the existence of the error propagation problem for pipeline methods. we compared our model with prior tranditional gnn-based models which do not contain hyperedges connecting multiple instances and showed the improvement, suggesting that modeling higher-order interactions between multiple instances is beneficial. finally, we compared our model with the most popular higher-order crf models with mfvi and showed the advantages of hgnn in higher-order modeling.