A Multi-Gate Encoder for Joint Entity and Relation Extraction (2022)

Abstract: “Named entity recognition and relation extraction are core sub-tasks of relational triple extraction. Recent studies have used parameter sharing or joint decoding to create interaction between these two tasks. However, ensuring the specificity of task-specific traits while the two tasks interact properly is a huge difficulty. We propose a multi-gate encoder that models bidirectional task interaction while keeping sufficient feature specificity based on gating mechanism in this paper. Precisely, we design two types of independent gates: task gates to generate task-specific features and interaction gates to generate instructive features to guide the opposite task. Our experiments show that our method increases the state-of-the-art (SOTA) relation F1 scores on ACE04, ACE05 and SciERC datasets to 63.8% (+1.3%), 68.2% (+1.4%), 39.4% (+1.0%), respectively, with higher inference speed over previous SOTA model.”

## introduction extracting relational facts from unstructured texts is a fundamental task in information extraction. this task can be decomposed into two sub-tasks: named entity recognition (ner) @xcite , which aims to recognize the boundaries and types of entities; and relation extraction (re) @xcite , which aims to extract semantic relations between entities. the extracted relational triples in the form of (subject, relation, object) are basic elements of large-scale knowledge graphs @xcite . traditional approaches perform ner and re in a pipelined fashion @xcite @xcite . they first extract all the entities in a given text, and then identify pairwise relations between the extracted entities. however, because the two sub-tasks are modeled independently, pipelined methods are vulnerable to error propagation issue. since the interaction between ner and re is neglected, the errors accumulated in the previous ner stage cannot be corrected in the subsequent re stage. to resolve this issue, some joint models have been proposed to model these two tasks simultaneously. early feature-based joint models @xcite rely on complicated feature engineering to build interaction between entities and relations. more recently, neural joint models have attracted increasing research interest and have demonstrated promising performance on joint entity and relation extraction. in existing neural joint models, there are mainly two ways to build the interaction between ner and re: parameter sharing and joint decoding. in parameter sharing methods @xcite @xcite , ner model and re model are built on top of a shared encoding layer to achieve joint learning. however, approaches based on parameter sharing implicitly rather than explicitly model the inter-task interaction, leading to insufficient excavation of the inherent association between the two tasks. moreover, these two tasks focus on different contextual information @xcite , but methods of sharing representations cannot provide task-specific features with enough specificity for the two tasks. in terms of error propagation, parameter sharing methods alleviate the error propagation between tasks, but to a limited extent, because these models still perform pipelined decoding. ## related work the task of extracting relational triples from plain text can be decomposed into two sub-task: named entity recognition and relation extraction. the two tasks can be performed in a pipelined manner @xcite @xcite or in a joint manner @xcite @xcite . traditional pipelined methods @xcite @xcite firstly train a model to extract entities and then train another model to classify the relation type between subject and object for each entity pair. recent pipelined approaches @xcite still follow this pattern and adopt marker-based span representations to learn different contextual representations between entities and relations, and between entity pairs, which sheds some light on the importance of feature specificity. although @xcite and @xcite achieve better performance than previous pipelined methods and some joint methods, they still run the risk of error propagation and do not adequately account for interactions between tasks. to ease these issues, some joint models that extract entities and relations jointly has been proposed. joint entity and relation extraction is a typical multi-task scenario, and how to handle the interaction c c l 2 0 2 2 between tasks is a frequently discussed topic. early joint models @xcite rely on feature engineering to build task interaction. more recently, many neural joint models have been proposed and show promising performance. @xcite builds a sequence tagging model for ner and a dependency tree model for re separately on top of a shared lstm layer and performs joint learning, achieving task interaction through parameter sharing. zeng et al. (2018) uses sequence-to-sequence learning framework with copy mechanism to jointly extract entities and relations. bekoulis et al. (2018b) builds a crf layer for ner and a sigmoid layer for re on a shared lstm layer. eberts and ulges (2020) proposes a span-based joint model for entity and relation extraction. they performs span classification and span filtering to extract entity spans and then performs relation classification based on the contextual span representations from bert (devlin et al., 2019) encoder. all these approaches construct the interaction between ner and re through parameter sharing. another class of methods adopts joint decoding to fuse the two tasks together. li and ji (2014) uses structured perceptron with beam search to extract entities and relations simultaneously. wang et al. (2018) proposes a transition system to convert the joint task into a directed graph. wang et al. (2020b) introduces a novel handshaking tagging scheme to formulate joint extraction as a token pair linking problem. zhang et al. (2017) and ren et al. (2021) convert the task into a table-filling problem. in addition to building interaction between tasks, another important issue is the specificity of task features. as recent studies @xcite have shown, generating specific contextual features for different tasks can achieve better results on the overall task than sharing input features. @xcite and @xcite both use a pre-trained language model (e.g., bert) for ner and another for re to obtain different contextual representations for specific task. however, fine-tuning distinct pre-trained encoders for the two task separately is computationally expensive. in our work, we adopts gating mechanism to balance the flow of feature information, taking into account both the interaction between tasks and the specificity of task features. ## method in this section, we first formally define the problem of joint entity and relation extraction and then detail the structure of our model. finally, we discuss how our model differs from the approach we follow and explain why our method performs better. ## conclusion in this paper, we propose a multi-gate encoder for joint entity and relation extraction. our model adopts gate mechanism to build bidirectional task interaction while ensuring the specificity of task features by c c l 2 0 2 2