Packed Levitated Marker for Entity and Relation Extraction (2022)

Abstract: Recent entity and relation extraction works focus on investigating how to obtain a better span representation from the pre-trained encoder. However, a major limitation of existing works is that they ignore the interrelation between spans (pairs). In this work, we propose a novel span representation approach, named Packed Levitated Markers (PL-Marker), to consider the interrelation between the spans (pairs) by strategically packing the markers in the encoder. In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information. Furthermore, for those more complicated span pair classification tasks, we design a subject-oriented packing strategy, which packs each subject and all its objects to model the interrelation between the same-subject span pairs. The experimental results show that, with the enhanced marker feature, our model advances baselines on six NER benchmarks, and obtains a 4.1%-4.3% strict relation F1 improvement with higher speed over previous state-of-the-art models on ACE04 and ACE05. Our code and models are publicly available at https://github.com/thunlp/PL-Marker

## introduction recently, pre-trained language models (plms) @xcite have achieved significant improvements in named entity recognition (ner, @xcite ; @xcite ) and relation extraction @xcite ; @xcite ), two key sub-tasks of information extraction. recent works @xcite ) regard these two tasks as span classification or span pair classification, and thus focus on extracting better span representations from the plms. david assaulted a pair of restaurant workers during a night out with national squad teammates in @xcite copenhagen[/s]. david assaulted a pair of restaurant [o]workers[/o] during a night out with national squad teammates in @xcite copenhagen[/s]. david assaulted a pair of restaurant workers during a night out with national squad @xcite teammates[/o] in [s]copenhagen[/s]. soild marker packed levitated marker [o david][/o] [o workers][/o] [o teammates][/o] levitated marker [s copenhagen][/s] [o david][/o] david assaulted a pair of restaurant workers during a night out with national squad teammates in copenhagen. [s copenhagen][/s] [o workers][/o] [s copenhagen][/s] [o teammates][/o] training process [s copenhagen][/s] [o david][/o] [s copenhagen][/s] [o workers][/o] [s copenhagen][/s] [o teammates][/o] inference process figure 1 : an example in the re task. solid marker separately processes three pairs of spans with different insertions of markers. levitated marker processes the span pairs independently during training and processes them in batches during inference. our proposed packed levitated marker packs three objects for the same subject into an instance to process. three span representation extraction methods are widely used: (1) t-concat @xcite concatenates the representation of the span's boundary (start and end) tokens to obtain the span representation. it collects information at the token level but ignores the connection between boundary tokens of a span when they pass through the network; (2) solid marker @xcite explicitly insert two solid markers before and after the span to highlight the span in the input text. and it inserts two pair of markers to locate the subject and object of a span pair. however, the method cannot handle multiple span pairs at the same time because of its weakness in specifying the solid markers of a span pair from more than two pairs of markers in the sequence. (3) levitated marker @xcite ) first sets a pair of levitated markers to share the same position with the span's boundary tokens and then ties a pair of markers by a directional attention. to be specific, the markers within a pair are set to be visible to each other in the attention mask matrix, but not to the text token and other pairs of markers. existing work @xcite ) simply replaces solid markers with levitated markers for an efficient batch computation, but sacrifices the model performance. as the re example shown in figure 1 , to correctly identify that david, workers and teammates are located_in copenhagen, it is important to separate out that david attacked the restaurant workers and he had social relation with his teammates. however, prior works with markers @xcite independently processes the span pairs with different insertions of markers in the training phrase, and thus ignore interrelation between spans (pairs) @xcite @xcite . in this work, we introduce packed levitated marker (pl-marker), to model the interrelation between spans (pairs) by strategically packing levitated markers in the encoding phase. a key challenge of packing levitated markers together for span classification tasks is that the increasing number of inserted levitated markers would exacerbate the complexity of plms quadratically @xcite . thus, we have to divide spans into several groups to control the length of each input sequence for a higher speed and feasibility. in this case, it is necessary to consider the neighbor spans integrally, which could help the model compare neighbor spans, e.g. the span with the same start token, to acquire a more precise entity boundary. hence, we propose a neighborhood-oriented packing strategy, which packs the spans with the same start token into a training instance as much as possible to better distinguish the entity boundary. for the more complicated span pair classification tasks, an ideal packing scheme is to pack all the span pairs together with multiple pairs of levitated markers, to model all the span pairs integrally. however, since each pair of levitated markers is already tied by directional attention, if we continue to apply directional attention to bind two pairs of markers, the levitated marker will not be able to identify its partner marker of the same span. hence, we adopt a fusion of solid markers and levitated markers, and use a subject-oriented packing strategy to model the subject with all its related objects integrally. to be specific, we emphasize the subject span with solid markers and pack all its candidate object spans with levitated markers. moreover, we apply an object-oriented packing strategy for an intact bidirectional modeling @xcite . we examine the effect of pl-marker on two typical span (pair) classification tasks, ner and endto-end re. the experimental results indicate that pl-marker with neighborhood-oriented packing scheme performs much better than the model with random packing scheme on ner, which shows the necessity of considering the neighbor spans integrally. and our model also advances the t-concat model on six ner benchmarks, which demonstrates the effectiveness of the feature obtained by span marker. moreover, compared with the previous state-of-the-art re model, our model gains a 4.1%-4.3% strict relation f1 improvement with higher speed on ace04 and ace05 and also achieves better performance on scierc, which shows the importance of considering the interrelation between the subject-oriented span pairs. ## related work in recent years, span representation has attracted great attention from academia, which facilitates various nlp applications, such as named entity recognition @xcite , relation and event extraction @xcite , coreference resolution @xcite , semantic role labeling @xcite and question answering @xcite . existing methods to enhance span representation can be roughly grouped into three categories: span pre-training the span pre-training approaches enhance the span representation for plms via span-level pre-training tasks. sun et al. (2019); lewis et al. (2020); raffel et al. (2020 ) mask and learn to recover random contiguous spans rather than random tokens. joshi et al. (2020) further learns to store the span information in its boundary tokens for downstream tasks. knowledge infusion this series of methods focuses on infusing external knowledge into their models. zhang et al. (2019); peters et al. (2019); wang et al. (2021a) learn to use the external entity embedding from the knowledge graph or the synonym net to acquire knowledge. soares et al. (2019); xiong et al. (2020); wang et al. (2021b); yamada et al. (2020) conduct specific entity-related pre-training to incorporate knowledge into their models with the help of wikipeidia anchor texts. ## method in this section, we first introduce the architecture of the levitated marker. then, we present how we pack the levitated marker to obtain the span representation and span pair representation. ## conclusion in this work, we present a novel packed levitated markers, with a neighborhood-oriented packing strategy and a subject-oriented packing strategy, to obtain the span (pair) representation. considering the interrelation between spans and span pairs, our model achieves the state-of-the-art f1 scores and a promising efficiency on both ner and re tasks across six standard benchmarks. in future, we will further investigate how to generalize the markerbased span representation to more nlp tasks. chen, 2021) to obtain the pure w. invrel. model. as shown in