TDEER : An Efficient Translating Decoding Schema for Joint Extraction of Entities and Relations (2021)

Abstract: Joint extraction of entities and relations from unstructured texts to form factual triples is a fundamental task of constructing a Knowledge Base (KB). A common method is to decode triples by predicting entity pairs to obtain the corresponding relation. However, it is still challenging to handle this task efficiently, especially for the overlapping triple problem. To address such a problem, this paper proposes a novel efficient entities and relations extraction model called TDEER , which stands for T ranslating D ecoding Schema for Joint E xtraction of E ntities and R elations. Unlike the common approaches, the proposed translating decoding schema regards the relation as a translating operation from subject to objects, i.e., TDEER decodes triples as subject + relation \rightarrow objects . TDEER can naturally handle the overlapping triple problem, because the translating decoding schema can recognize all possible triples, including overlapping and non-overlapping triples. To enhance model robustness, we introduce negative samples to alleviate error accumulation at different stages. Extensive experiments on public datasets demonstrate that TDEER produces competitive results compared with the state-of-the-art (SOTA) baselines. Furthermore, the computation complexity analysis indicates that TDEER is more efficient than powerful baselines. Especially, the proposed TDEER is 2 times faster than the recent SOTA models. The code is available at https://github.com/4AI/TDEER .

## introduction extraction of entities and relations from unstructured texts is one of the most essential information extraction tasks. it aims to extract entities and their corresponding semantic relations from unstructured texts, which are usually presented in a triple form of (subject, relation, object), e.g., (microsoft, co-founder, bill gates). it is also a crucial step in building a large-scale kb and exerts an important role in the development of web search @xcite ), question answering @xcite , biomedical text mining @xcite , etc. traditional approaches @xcite @xcite handle this task in a pipeline manner, i.e., extracting the entities first and then identifying their relations. the pipeline framework simplifies the extraction task, but it ignores the relevance between entity identification and relation prediction. to address this problem, several joint learning models have been proposed and can be categorized into feature-based models and end-to-end deep models. feature-based models @xcite introduce a complex process of feature engineering and profoundly depend on natural language processing tools for feature extraction. more recently, the end-to-end neural network models @xcite @xcite @xcite have become the mainstream method for relation extraction tasks. such models utilize the learned representation from pre-trained language models and are a more promising approach than manual features. more research interests have been concerned with complicated entity and relation extraction problems, such as the overlapping triple problem. @xcite summarized the overlapping triple problem into three categories, i.e. normal, seo, and epo, which are depicted in figure 1 . many methods have been proposed to address the overlapping issue, for instance, encoder-decoder framework @xcite and decomposition approaches @xcite . however, such approaches still suffer from setbacks when handling the overlapping triple problem. more specifically, the encoder-decoder framework can only resolve the one-word entity overlapping problem and fail to handle the multi-word entity overlapping problem. meanwhile, the decomposition approaches suffer error accumulation between dependent stages. to address these problem, @xcite presented a one-stage method, tplinker, that transforms the joint extraction task into a token pair linking problem to resolve the overlapping triple problem. tplinker does not contain any inter-dependent stages, hence it can alleviate error accumulation. however, processing all token pairs at encoder layers suffers from high computational complexity, which is an obstacle for tplinker to encode long text. we present a novel framework tdeer to jointly extract the entities and relations by a translating decoding schema to handle the overlapping triple problem. more concretely, tdeer interprets the relation as a translating operation from subject entity to object entities, i.e., it decodes triples by subject + @xmath0. the proposed translating decoding schema can effectively resolve the overlapping triple problem. tdeer iterates all pairs of subjects and relations to recognize objects (or no object), hence all possible triples, including overlapping or non-overlapping triples, can be considered. we propose a negative sample strategy to detect and alleviate error propagation in different stages. this strategy can enable tdeer to alleviate error accumulation to achieve higher results. tdeer is an efficient approach as it first retrieves all possible relations and entities, then uses distinguished entities and relations to decode triples. by doing this, the search space can be reduced, thus it is more efficient than previous works. the computational complexity of the proposed translating decoding schema is o(n + sr), where n is the sequence length, s is the number of subjects in the input sentence, r denotes the number of relations in the input sentence. extensive experiments illustrate that tdeer achieves better results than sota models in most datasets and is competent in handling the overlapping triple problem. in summary, our contributions are as follows: (1) we propose a novel translating decoding schema for joint extraction of entities and relations from unstructured texts. (2) tdeer can handle the intractable overlapping triple problem effectively and efficiently. (3) notably, tdeer is about 2 times faster than the current sota models. ## related work the pipeline approach and joint approach are the two mainstream methods for extracting entities and relations from unstructured texts. traditionally, extracting entities and relations to form triples has been studied as two separated independent tasks: named entity recognition (ner) and relation extraction. @xcite introduced a distant supervision model, @xcite used a weak supervision method to extract entities and relations. the features of distant supervision and weak supervision approaches are often derived from natural language processing (nlp) tools. it suffers from data labeling errors that inevitably exist in nlp tools. to address this problem, @xcite employed a multi-instance learning approach to tackle the problem of data labeling errors. @xcite applied reinforcement learning for extraction of entities and relations. although the pipeline models produced promising results, they neglect the triple-level dependencies between entities and relations. recently, @xcite presented a pipeline approach incorporating entity information for entity and relation extraction. to exploit the dependencies between entities and relations, multiply joint extraction models have been proposed. @xcite introduced a unified tagging scheme and transformed the relationship extraction problem into a sequence labeling problem. @xcite applied a sequenceto-sequence model with a copy mechanism to solve the overlapping triple problem. trisedya et al. (2019) employed the encoder-decoder framework to jointly extract triples from sentences and map them into an existing kb. fu et al. (2019) applied graph convolutional networks to jointly learn named entities and relations. dai et al. (2019) presented a unified joint extraction model to tag entity and relation labels directly according to a query word position, which can simultaneously extract all entities and their types. wei et al. (2020) proposed a cascade binary tagging framework. wang et al. (2020) formulated the joint extraction as a token pair linking problem. moreover, some knowledge representation models @xcite @xcite are adopted to refine the triple extraction model via scoring candidate facts by knowledge graph embedding. although some of them also use the "translation" idea, the function is different from ours. in their setting, they use the "translation" idea to construct rank-based knowledge graph embedding models. they cannot be used to extract entities and relations from texts directly. in our setting, the "translation" idea is applied to end-to-end joint extract entities and relations from text. the proposed translating decoding schema is a novel approach to solve the overlapping triple problem effectively and efficiently, which makes our model crucially different from previous works. ## methodology this paper proposes a three-stage model, tdeer. in the first stage, tdeer uses a span-based entity tagging model to extract all subjects and objects. in the second stage, tdeer employs the multilabel classification strategy to detect all relevant relations. in the third stage, tdeer iterates the pairs of subjects and relations to identify respective objects by the proposed translating decoding schema. figure 2 shows the generic framework of tdeer. in subsequent sections, we will describe the three stages of tdeer in detail. ## conclusion & future work in this paper, we have proposed a novel translating decoding schema for joint extraction of entities and relations, namely tdeer. it models the relation as a translating operation from subjects to objects, which can handle the overlapping triple problem naturally. we have conducted extensive experiments on widely used datasets to demonstrate the effectiveness and efficiency of the proposed model. the proposed negative sample strategy is used to alleviate the error accumulation problem. though it is effective, it may increase training time. for future work, we plan to explore more efficient approaches to alleviate error accumulation.