Recurrent Interaction Network for Jointly Extracting Entities and Classifying Relations (2020)

Abstract: The idea of using multi-task learning approaches to address the joint extraction of entity and relation is motivated by the relatedness between the entity recognition task and the relation classification task. Existing methods using multi-task learning techniques to address the problem learn interactions among the two tasks through a shared network, where the shared information is passed into the task-specific networks for prediction. However, such an approach hinders the model from learning explicit interactions between the two tasks to improve the performance on the individual tasks. As a solution, we design a multi-task learning model which we refer to as recurrent interaction network which allows the learning of interactions dynamically, to effectively model task-specific features for classification. Empirical studies on two real-world datasets confirm the superiority of the proposed model.

## introduction the extraction of entities and relations from textual data comprises of two sub-tasks: entity recognition (er) and relation classification @xcite . the er task aims at extracting all entities in a given text. the rc task aims at classifying the relation between any pair of entities in the text. in practice, both tasks are required to be solved jointly, and have been observed to contribute significantly in extracting structured knowledge from unstructured text for several applications, including knowledge base construction @xcite @xcite . for instance, consider the sentence john was born in sheffield, a city in the north of england. the goal of a joint entity and relation extraction task is to identify all the factual relational triples (or relational facts) (sheffield, * corresponding author birth place of, john) and (england, contains, sheffield). the simplest approach to solve this joint task is to utilize a pipeline-based approach by firstly extracting all entities in the sentence and then classifying the relation between all entity pairs @xcite @xcite . however, pipeline-based approaches disregard the correlation between er and rc tasks, leading to error propagation in these methods. recently, researchers have exploited multi-task learning (mtl) @xcite techniques to capture the correlation between the er and rc tasks, and have successfully improved the performance of the individual tasks @xcite @xcite . these methods have a flat structure @xcite . figure 1(a) shows a flat structure for multitask learning. methods using a flat structure learn interactions between tasks through a shared network, and extract a shared common representation which is exploited by task-specific networks independently. we refer to mtl methods utilizing a flat structure as conventional mtl methods. a conventional mtl method is effective to an extent because they help to improve generalization performance on all the tasks. however, it is based on the strong assumption that the shared network is sufficient to capture the correlations between the tasks. even so, identifying the relational facts in sentences is a difficult problem. reason being that several relational facts may overlap in a sentence @xcite . although a conventional mtl method may learn task-specific features and has been successfully applied in a wide variety of scenarios @xcite @xcite @xcite @xcite , its flat structure restricts the model to effectively learn the correlations between tasks. for example in figure 1 (a), the model cannot explicitly learn correlations between the two tasks. without modeling explicit interactions, as shown in a sequence learning task @xcite , the existing mtl-based methods @xcite @xcite cannot effectively capture the correlation between the er and the rc tasks. in this paper, we overcome the aforementioned limitation of previous mtl-based methods by proposing a recurrent interaction network (rin) to effectively capture the correlations between the er and rc tasks. rin has a multi-task learning architecture which allows interactions between the er and rc tasks to be learned explicitly, with the aim to improve the performance on the individual tasks. more specifically, rin has a recurrent structure comprising of multiple interaction layers, allowing the model to progressively learn complex interactions while refining predictions for er and rc. the rin structure is an example of a multitask learning network with a graph structure @xcite . we show the graph structure in figure 1 ## related work previous neural methods proposed for jointly extracting entities and relations can generally be categorized into three classes. the first class models the joint extraction task as a sequence labeling problem @xcite @xcite . among the proposed works, @xcite was the first to introduce a tagging strategy to address the problem, transferring the joint extraction task to a sequence labelling problem. however, this method has the fundamental weakness of addressing the overlapping problem of relational facts in the text. to meet it, @xcite proposed a position-attentive tagging scheme to solve the overlapping problem. meanwhile, @xcite approach the problem by decomposing the joint extraction task into two sequence labeling sub-tasks, to address the joint entity and relation extraction problem. the second class of works use a sequence-tosequence (seq2seq) approach to address the problem @xcite . @xcite employs a seq2seq model to directly extract relational facts from the sentence by decoding the first entity, second entity, and relation in that order. but, their approach is limited to extracting a predefined number of relational facts from the text. in extracting relational triples, the order of extraction is key to identify the relational facts. as such, @xcite proposed a seq2seq approach which utilizes a reinforcement learning model to learn the order of extracting the relational triples. although effective, the proposed seq2seq models @xcite only decode a single word for an entity. the third class design a multi-task learning model to extract relational facts @xcite @xcite @xcite . among these works, @xcite consider the dependency structure of the sentence to model the common features between the er and rc tasks, while @xcite employ convolutional neural networks (cnn) to encode the common features. @xcite proposed an mtl model which comprises of an er model to extract entities with multi-tokens, and a seq2seq model to extract relational facts. their approach solves the entity extraction problem faced by models which are solely seq2seq based. @xcite exploits a bidirectional recurrent neural network and graph convolutional network to extract the common features. despite the substantial efforts and great successes in the design of these mtl-based methods, these methods follow the conventional mtl approach @xcite . thus, they only capture implicit interactions by means of the shared network of the er and rc tasks. modelling explicit interactions between multiple tasks in an mtl architecture has been explored to improve predictions in several domains @xcite @xcite @xcite . modeling explicit interactions have also been explored for the joint extraction task @xcite . however, @xcite models the interaction in a successive way by using the output of one task to benefit the prediction of the other task. the method we propose considers the output of the tasks simultaneously, and model their interaction in a dynamic way with a recurrent structure, allowing the tasks to gradually refine their predictions and benefit from each other. to the best of our knowledge, this is the first work to model explicit interactions in a multitask learning architecture for the joint extraction of entities and relations in text. ## problem statement in this section, we formally describe the joint entity and relation extraction problem. for a set @xmath0, • • • , t l } of pre-defined l relation types, and a given sentence @xmath1, w 2 , • • • , w n } of n words, the problem is to extract all relational facts for the given sentence. in this paper, a single relational triple is of the form w i , t, w j , where w i , w @xmath2, and w @xmath3, and the relation @xmath4. the goal of the relation classification task is to predict the probability y t (i,j) that the relational triple w i , t, w j is factual given the word pair (w i , w j ). moreover, the entity recognition task takes each word w @xmath5. when no confusion will arise, we will remove the subscript i from the prediction y i for clarity and brevity. clearly, the er task can help identify the head or tail words of multi-token entities for the joint extraction task. ## model in this section, we describe the recurrent interaction network (rin) for extracting relational facts in text. the rin model is composed of an entity recognition (er) module and a relation classification (rc) module. we start by presenting an overview of the rin model, showing the interaction between the er and rc tasks. next, we elaborate the er and rc modules and define the training objective. the framework of rin is shown in figure 2 . ## experiment we conduct experiments to evaluate rin on two public datasets nyt @xcite and webnlg @xcite . the nyt dataset consists of 1.18m sentences with 24 predefined relation types. the webnlg dataset was created by natural language generation (nlg) tasks, and adapted by @xcite for a relational triple extraction task. we directly use the preprocessed datasets released by @xcite ). foot_0 it is worth mentioning that only the tail word of an entity is marked in the preprocessed dataset released by @xcite . to properly distinguish entities, we take a further step of tagging entities with the conventional bioes tagging scheme as the one used in @xcite . we report precision (prec), recall (rec) and micro-f1 (f1) scores on our model and other recent models @xcite @xcite @xcite for the partial match task and the exact match task. for our proposed method, we report the mean results over five runs using different random seeds, along with its standard deviation to show the stability of our results. the statistics of datasets are summarized in ## conclusion in this paper, we tackle the weakness of existing mtl-based methods proposed for the joint extraction of entities and relation in unstructured text. specifically, these methods assume that a shared network is sufficient to capture the correlations between the entity recognition task and the relation classification task, and that the shared features derived from this network can be passed into models for the task-specific tasks to make predictions independently. instead, we show that dynamically learning the interactions between the tasks may capture complex correlations which improves the taskspecific feature for classification. we proposed multi-task learning model which allows explicit interactions to be dynamically learned among the subtasks. our experiments on benchmark datasets validates clear advantage over the existing proposed methods. we note that our model can be adapted for other nlp tasks, including aspect level sentiment classification and slot filling. as future work, we intend to explore its application in those fields.