Span-based Joint Entity and Relation Extraction with Attention-based Span-specific and Contextual Semantic Representations (2020)

Abstract: Span-based joint extraction models have shown their efficiency on entity recognition and relation extraction. These models regard text spans as candidate entities and span tuples as candidate relation tuples. Span semantic representations are shared in both entity recognition and relation extraction, while existing models cannot well capture semantics of these candidate entities and relations. To address these problems, we introduce a span-based joint extraction framework with attention-based semantic representations. Specially, attentions are utilized to calculate semantic representations, including span-specific and contextual ones. We further investigate effects of four attention variants in generating contextual semantic representations. Experiments show that our model outperforms previous systems and achieves state-of-the-art results on ACE2005, CoNLL2004 and ADE.

## introduction this paper considers intra-sentence joint entity and relation extraction. for joint extraction mode can alleviate cascading errors and promote information utilization compared to pipelined one, this mode has drawn much attention. typically, the joint extraction task is solved by sequence tagging based methods @xcite . rather than using sequence tagging based methods, recent works attempt to solve the task with spanbased joint extraction mode @xcite . typically, this mode first processes sentence text into text spans, which are span-based candidate entities ("spans" for short); then, calculates span semantic representations and performs span classification on them to obtain predicted entities; next, forms span-based candidate relation ("relation" for short) tuples with spans, and calculates relation semantic representations with corresponding span semantic representations; at last, performs relation classification on relation semantic representations and obtains predicted relation triples. this mode further improves joint extraction performance, whereas exists three problems. first, different tokens in span should contribute differently to span representation, which we call spanspecific features. but existing methods treat each span token equally important @xcite or just consider span head and tail tokens @xcite , ignoring these significant features. take the span "a palestinian youth" in sentence 1 of figure 1 as an example, the "youth" should contribute much greater to the span representation than "a" and "palestinian" when classifying the span into "per". second, local contextual information of relation tuples is omitted @xcite or just calculated by max pooling way @xcite when performing relation classification, which do not sufficiently capture information contained in it. whereas local context may contain crucial information to help predict the relations that relation tuples hold. a case study is shown in sentence 2 of figure 1 , the "ownership" (in red font) can greatly help to determine the relation ("part-whole") of the relation tuple (@xmath0, @xmath1. third, sentence-level contextual information is ignored in both span and relation classifications, which may be important compensation information for both ones. an example is given in sentence 3 of figure 1 , "state" (in red font) benefits the relation classification of the relation tuple (@xmath2, @xmath3, while "state" is neither contained in the relation tuple nor in the local context (namely "claim"), but contained in other part of sentence 3, which is sentence-level. to address above issues, we introduce a span-based joint extraction model with attention-based spanspecific and contextual semantic representations. specifically, 1) mlp attention is used to calculate spanspecific semantic representation; 2) attention-based sentence-level contextual semantic representation for span is calculated by taking span-specific semantic representation as query and sentence token sequence semantic representations as key, value respectively; 3) local and sentence-level contextual semantic representations for relation are obtained by attention calculation with relation tuple semantic representations and corresponding token sequence semantic representations. the advantage of this approach is that we can capture the most useful information to constitute efficient span and relation semantic representations. we take bert @xcite as the default backbone network, and explore the three research questions. moreover, we investigate effects of multi-head attention @xcite , dot-product attention @xcite , general attention @xcite and additive attention @xcite in generating contextual semantic representations. extensive experiments on three benchmark datasets show that our model consistently outperforms previous systems. in addition, the multi-head attention firmly improves over other attention variants. ## related works traditionally, pipelined entity and relation extraction method is divided into two subtasks, namely entity detection and relation classification. various neural networks have been widely investigated for the two subtasks, such as rnns @xcite , cnns @xcite for entity detection, and rnns @xcite , cnns @xcite , transformer @xcite for relation classification. as discussed in §1, joint entity and relation extraction is typically formulated as a sequence tagging based task. traditionally, table-filling methods have been widely explored @xcite , where token labels and relation labels fill the diagonal and off-diagonal of the table respectively. recently, many works concentrate on leveraging deep neural networks to tackle this task, e.g., stacked bidirectional lstm @xcite , combination of bidirectional lstm and cnn @xcite , and combination of bidirectional lstm and attention mechanism @xcite . in addition, a novel machine reading comprehension based approach @xcite is proposed, which formulates the task as a question & answer task but still in sequence tagging based mode. recently, span-based joint extraction methods have been investigated to tackle problems existing in sequence tagging based methods, e.g., inability to detect overlapping entities. specially, dixit and al-onaizan (2019) realize this method by obtaining span semantic representations through a bilstm over concatenated elmo, word and character embeddings, then share them in both span and relation classifications. luan et al. (2018) obtain span semantic representations generally the same as lee et al. (2017), but reinforce them by introducing coreference task. follow luan et al. (2018), luan et al. (2019) propose dygie, which can capture span interactions through a span graph constructed dynamically. wadden et al. (2019) deliver further performance increases on dygie by replacing the bilstm with bert and introduce dygie++. more recently, eberts and ulges (2019) propose spert, a simple but effective span-based model that takes bert as backbone and use two ffnns to classify span and relation respectively. unlike previous works, spert dramatically reduces model training complexity by adopting negative sampling. our work follows spert, but differs in span-specific and contextual semantic representations. specifically, our model obtains these semantic representations with attention mechanism. by calculating the matching degree between target sequence semantic representations and source sequence semantic representations, attention mechanism obtains attention scores on the source sequence, which are weight scores in essence. and the more important the information, the higher weight score it holds. classified by implementation manners of score function, attention mechanism has multiple variants, e.g., content-base attention @xcite , additive attention @xcite , general attention @xcite , dot-product attention @xcite , multi-head attention @xcite . ## approach in the rest, we abbreviate "semantic representation" as "representation". figure 2 gives an overview of our model, which uses bert as encoder following spert: we map word embeddings into bert embeddings using pre-trained transformer blocks @xcite . based on the representations, we calculate span representations and perform span classification & filtration ( §3.1); then, we organize relation tuples, calculate relation representations and perform relation classification & filtration ( §3.2); third, we investigate effects of multiple attention variants in generating contextual representations ( §3.3); at last, we introduce model training settings ( §3.4). define a sentence and a span from the sentence to help introduce the rest, where t denotes tokens and subscripts (e.g., 1, 2, 3...) denote token indexes, as: ## ablation study based on span m ulti-head , we conduct ablations on the ace2005 dev set to analyze effects of different model components. ## conclusion we introduce attention-based semantic representation generating methods in span-based joint entity and relation extraction method. we apply mlp attention to capture span-specific features aiming to obtain semantic rich span representation, and calculate task-specific contextual representations with attention architecture to further reinforce span and relation representations. our approach firmly outperforms both the sequence tagging based and span-based sota methods on three benchmark datasets, creating new state-of-the-art results. as future work, we would like to consider further improving relation classification performance by reducing span classification errors. we also plan to explore more advanced methods for encoding efficient span and relation representations.