Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders (2020)

Abstract: Named entity recognition and relation extraction are two important fundamental problems. Joint learning algorithms have been proposed to solve both tasks simultaneously, and many of them cast the joint task as a table-filling problem. However, they typically focused on learning a single encoder (usually learning representation in the form of a table) to capture information required for both tasks within the same space. We argue that it can be beneficial to design two distinct encoders to capture such two different types of information in the learning process. In this work, we propose the novel table-sequence encoders where two different encoders – a table encoder and a sequence encoder are designed to help each other in the representation learning process. Our experiments confirm the advantages of having two encoders over one encoder. On several standard datasets, our model shows significant improvements over existing approaches.

## introduction named entity recognition @xcite and relation extraction (re, @xcite @xcite are two fundamental tasks in information extraction (ie). both tasks aim to extract structured information from unstructured texts. one typical approach is to first identify entity mentions, and next perform classification between every two mentions to extract relations, forming a pipeline @xcite . an alternative and more recent approach is to perform these two tasks jointly @xcite @xcite , which mitigates the error propagation issue associated with the pipeline ap-1 our code is available at https://github.com/ lorrinwww/two-are-better-than-one . proach and leverages the interaction between tasks, resulting in improved performance. among several joint approaches, one popular idea is to cast ner and re as a table filling problem @xcite @xcite . typically, a two-dimensional (2d) table is formed where each entry captures the interaction between two individual words within a sentence. ner is then regarded as a sequence labeling problem where tags are assigned along the diagonal entries of the table. re is regarded as the problem of labeling other entries within the table. such an approach allows ner and re to be performed using a single model, enabling the potentially useful interaction between these two tasks. one example foot_0 is illustrated in figure 1 . unfortunately, there are limitations with the existing joint methods. first, these methods typically suffer from feature confusion as they use a single representation for the two tasks -ner and re. as a result, features extracted for one task may coincide or conflict with those for the other, thus confusing the learning model. second, these methods underutilize the table structure as they usually convert it to a sequence and then use a sequence labeling approach to fill the table. however, crucial structural information (e.g., the 4 entries at the bottom-left corner of figure 1 share the same label) in the 2d table might be lost during such conversions. in this paper, we present a novel approach to address the above limitations. instead of predicting entities and relations with a single representation, we focus on learning two types of representations, namely sequence representations and table representations, for ner and re respectively. on one hand, the two separate representations can be used to capture task-specific information. on the other hand, we design a mechanism to allow them to interact with each other, in order to take advantage of the inherent association underlying the ner and re tasks. in addition, we employ neural network architectures that can better capture the structural information within the 2d table representation. as we will see, such structural information (in particular the context of neighboring entries in the table) is essential in achieving better performance. the recent prevalence of bert @xcite has led to great performance gains on various nlp tasks. however, we believe that the previous use of bert, i.e., employing the contextualized word embeddings, does not fully exploit its potential. one important observation here is that the pairwise self-attention weights maintained by bert carry knowledge of word-word interactions. our model can effectively use such knowledge, which helps to better learn table representations. to the best of our knowledge, this is the first work to use the attention weights of bert for learning table representations. we summarize our contributions as follows: our proposed method achieves the state-of-theart performance on four datasets, namely ace04, ace05, conll04, and ade. we also conduct further experiments to confirm the effectiveness of our proposed approach. ## related work ner and re can be tackled by using separate models. by assuming gold entity mentions are given as inputs, re can be regarded as a classification task. such models include kernel methods @xcite , rnns @xcite , recursive neural networks @xcite , cnns @xcite , and transformer models @xcite . another branch is to detect cross-sentence level relations @xcite , and even document-level relations @xcite . however, entities are usually not directly available in practice, so these approaches may require an additional entity recognizer to form a pipeline. joint learning has been shown effective since it can alleviate the error propagation issue and benefit from exploiting the interrelation between ner and re. many studies address the joint problem through a cascade approach, i.e., performing ner first followed by re. @xcite use bi-lstm @xcite and tree-lstm @xcite for the joint task. @xcite formulate it as a head selection problem. @xcite apply biaffine attention @xcite for re. @xcite , @xcite , and @xcite use span representations to predict relations. @xcite tackle joint ner and re as from a table filling perspective, where the entry at row i and column j of the table corresponds to the pair of i-th and j-th word of the input sentence. the diagonal of the table is filled with the entity tags and the rest with the relation tags indicating possible relations between word pairs. similarly, @xcite employ a bi-rnn structure to label each word pair. @xcite propose a global optimization method to fill the table. @xcite investigate cnns on this task. recent work @xcite @xcite @xcite usually leverages pre-trained language models such as elmo @xcite , bert @xcite , roberta @xcite , 2019), and albert @xcite . however, none of them use pre-trained attention weights, which convey rich relational information between words. we believe it can be useful for learning better table representations for re. ## problem formulation in this section, we formally formulate the ner and re tasks. we regard ner as a sequence labeling problem, where the gold entity tags y ner are in the standard bio (begin, inside, outside) scheme @xcite . for the re task, we mainly follow the work of @xcite to formulate it as a table filling problem. formally, given an input sentence @xmath0, we maintain a tag table y @xmath1,j ] 1≤i,@xmath2. suppose there is a relation with type r pointing from mention x i b , .., x i e to mention x j b , .., x j e , we have y re i,@xmath3 we use ⊥ for word pairs with no relation. an example was given earlier in figure 1 . ## model we describe the model in this section. the model consists of two types of interconnected encoders, a table encoder for table representation and a sequence encoder for sequence representation, as shown in figure 2 . collectively, we call them tablesequence encoders. figure 3 presents the details of each layer of the two encoders, and how they interact with each other. in each layer, the table encoder uses the sequence representation to construct the table representation; and then the sequence encoder uses the table representation to contextualize the sequence representation. with multiple layers, we incrementally improve the quality of both representations. ## training and evaluation we use s l and t l to predict the probability distribution of the entity and relation tags: the goal is to minimize both losses l ner + l re . during evaluation, the prediction of relations relies on the prediction of entities, so we first predict the entities, and then look up the relation probability table p θ (y re ) to see if there exists a valid relation between predicted entities. specifically, we predict the entity tag of each word by choosing the class with the highest probability: the whole tag sequence can be transformed into entities with their boundaries and types. relations on entities are mapped to relation classes with highest probabilities on words of the entities. we also consider the two directed tags for each relation. therefore, for two entity spans (i b , i e ) and (j b , j e ), their relation is given by: we evaluate our model on four datasets, namely ace04 @xcite , ace05 @xcite , conll04 @xcite and ade @xcite . more details could be found in appendix b. following the established line of work, we use the f1 measure to evaluate the performance of ner and re. for ner, an entity prediction is correct if and only if its type and boundaries both match with those of a gold entity. 5 for re, a relation prediction is considered correct if its relation type and the boundaries of the two entities match with those in the gold data. we also report the strict relation f1 (denoted re+), where a relation prediction is considered correct if its relation type as well as the boundaries and types of the two entities all match with those in the gold data. relations are asymmetric, so the order of the two entities in a relation matters. ## conclusion in this paper, we introduce the novel tablesequence encoders architecture for joint extraction of entities and their relations. it learns two separate encoders rather than one -a sequence encoder and a table encoder where explicit interactions exist between the two encoders. we also introduce a new method to effectively employ useful information captured by the pre-trained language models for such a joint learning task where a table representation is involved. we achieved state-of-the-art f1 scores for both ner and re tasks across four standard datasets, which confirm the effectiveness of our approach. in the future, we would like to investigate how the table representation may be applied to other tasks. another direction is to generalize the way in which the table and sequence interact to other types of representations.