TPL inker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking (2020)

Abstract: Extracting entities and relations from unstructured text has attracted increasing attention in recent years but remains challenging, due to the intrinsic difficulty in identifying overlapping relations with shared entities. Prior works show that joint learning can result in a noticeable performance gain. However, they usually involve sequential interrelated steps and suffer from the problem of exposure bias. At training time, they predict with the ground truth conditions while at inference it has to make extraction from scratch. This discrepancy leads to error accumulation. To mitigate the issue, we propose in this paper a one-stage joint extraction model, namely, TPLinker, which is capable of discovering overlapping relations sharing one or both entities while being immune from the exposure bias. TPLinker formulates joint extraction as a token pair linking problem and introduces a novel handshaking tagging scheme that aligns the boundary tokens of entity pairs under each relation type. Experiment results show that TPLinker performs significantly better on overlapping and multiple relation extraction, and achieves state-of-the-art performance on two public datasets.

## introduction extracting entities and relations from unstructured texts is an essential step in automatic knowledge base construction @xcite . traditional pipelined approaches first extract entity mentions and then classify the relation types between candidate entity pairs. however, due to the complete separation of entity detection and relation classification, these models ignore the interaction and correlation between the two subtasks, being susceptible to cascading errors @xcite . in the last few years, there has been increasing research interest in building joint models to simultaneously extract entities and relations. recent works show that joint learning approaches can effectively integrate the information of entity and relation, and therefore achieve better performance in both subtasks @xcite . @xcite proposed a unified tagging scheme to convert joint extraction to a sequence labeling problem but lacks the elegance to identify overlapping relations: one entity may participate in multiple relations in the same text (figure 1 ). most existing models in handling entitypairoverlap (epo) and singleentiyoverlap (seo) cases can be categorized into two classes: decoder-based and decomposition-based. decoder-based models use encoder-decoder architecture where the decoder extracts one word or one tuple at a time like machine translation models @xcite . decomposition-based models first distinguish all the candidate subject entities that may be involved with target relations, then label corresponding object entities and relations for each extracted subject @xcite @xcite . although these methods have achieved reasonable performance, they all suffer from the same problem: exposure bias. for the decoder-based method, at training time, the ground truth tokens are used as context while at inference the entire sequence is generated by the resulting model on its own, and hence the previous tokens generated by the model are fed as context. as a result, the predicted tokens at training and inference are drawn from different distributions, namely, from the data distribution as opposed to the model distribution @xcite . similarly, the decomposition-based method uses the gold subject entity as specific input to guide the model extract object entities and relations during the training process while at inference the input head-entity is given by a trained model, leading to a gap between training and inference. in this paper, we present a one-stage method for joint extraction of entities and overlapping relations, namely tplinker, which bridges the gap between training and inference. tplinker transforms the joint extraction task as a token pair linking problem. given a sentence, two positions p 1 , p 2 and a specific relation r, tplinker is to answer three yes/no pseudo questions: "whether p 1 and p 2 are the start and end positions of the same entity respectively?", "whether p 1 and p 2 are the start positions of two entities with r relation respectively?" and "whether p 1 and p 2 are the end positions of two entities with r relation respectively?" to this end, we design a handshaking tagging scheme that annotates three token link matrices for each relation to answer the above three questions. these link matrices are then used to decode different tagging results, from which we can extract all entities and their overlapping relations. intuitively, tplinker does not contain any inter-dependency extraction steps, so it avoids the dependence on ground truth conditions at training time, realizing the consistency of training and testing. we evaluate our method on two public datasets: nyt @xcite and webnlg @xcite . experimental results show that tplinker outperforms previous works and achieves the stateof-the-art results on the benchmark datasets. further analysis demonstrates that tplinker significantly improves the performance on normal, seo, epo, and multiple relation extraction. ## related work researchers proposed several methods to extract both entities and relations. traditional pipelined methods @xcite neglect the relevance of entity extraction and relation prediction. to resolve this problem, several joint models have been proposed. feature-based works @xcite need a complicated process of feature engineering and heavily depend on nlp tools for feature extraction. neural models for joint relation extraction are investigated in recent studies @xcite , they show promising results but completely giving up overlapping relations. to address this problem, a variety of neural networks for joint extraction of entities and overlapping relations are proposed. @xcite extracted triplets by tagging one sentence for n times with a position-aware attention mechanism. @xcite al. (2019) these methods can only recognize seo relations in the sentence and fail to extract epo triplets. to handle the epo cases, @xcite proposed a sequence-to-sequence model to decode overlapping relations but fail to generate multi-word entities. as the improvement, nayak and ng (2020) employed an encoder-decoder model where the decoder extracts one word at a time like machine translation models. besides, @xcite proposed a novel cascade binary tagging framework that first identifies all possible subject entities in a sentence then identifies all possible relations and object entities for each subject entity. actually, these methods decompose the extraction of overlapping relations into several inner-dependency steps, since the decoder needs a recursive decoding process and cascade tagging has to identify subject entities in advance. such decomposition makes the task easy to conduct but inevitably causes the exposure bias problem, which leads to error accumulation. at training time, they predict the triplets with the ground truth tokens or subjects while at inference they have to rely on the predicted results. in this paper, we propose a unified tagging method to extract entities and overlapping relations. different from previous methods, our model performs in one stage and generates triplets without a gap between training and inference. ## methodology in this section, we first introduce our handshaking tagging scheme and its decoding algorithm. then we detail the tplinker model structure. ## conclusion in this paper, we propose an end-to-end sequence labeling model tplinker for joint extraction of entities and relations based on a novel handshaking tagging strategy, by which the joint extraction task is converted to a token pair linking game. to the best of our knowledge, tplinker is the first one-stage joint extraction model that can extract all kinds of overlapping relations without the influence of exposure bias. experimental results show that our model outperforms all baselines and achieves a new state-ofthe-art on two public datasets. further analysis especially demonstrates the capabilities of our model on handling sentences with overlapping relations and multiple relations. the results also prove that it is of benefit to close up the gap between training and inference. in the future, we would like to generalize the token linking idea and explore its performance on other information extraction problems, such as nested name entity extraction and event extraction.