UTC - IE : A Unified Token-pair Classification Architecture for Information Extraction (2023)

Abstract: Information Extraction (IE) spans several tasks with different output structures, such as named entity recognition, relation extraction and event extraction. Previously, those tasks were solved with different models because of diverse task output structures. Through re-examining IE tasks, we find that all of them can be interpreted as extracting spans and span relations. They can further be decomposed into token-pair classification tasks by using the start and end token of a span to pinpoint the span, and using the start-to-start and end-to-end token pairs of two spans to determine the relation. Based on the reformulation, we propose a Unified Token-pair Classification architecture for Information Extraction (UTC-IE), where we introduce Plusformer on top of the token-pair feature matrix. Specifically, it models axis-aware interaction with plus-shaped self-attention and local interaction with Convolutional Neural Network over token pairs. Experiments show that our approach outperforms task-specific and unified models on all tasks in 10 datasets, and achieves better or comparable results on 2 joint IE datasets. Moreover, UTC-IE speeds up over state-of-the-art models on IE tasks significantly in most datasets, which verifies the effectiveness of our architecture.

## introduction information extraction (ie) aims to identify and classify structured information from unstructured texts @xcite . ie consists of a wide range of tasks, such as named entity recognition (ner), joint entity relation extraction (re) 2 and event extraction (ee) . in the last decade, many paradigms have been proposed to solve ie tasks, such as sequence label-ing @xcite @xcite , span-based classification @xcite @xcite , mrc-based methods @xcite @xcite and generation-based methods @xcite @xcite . the above work mainly concentrates on solving individual tasks, but it is desired to unify all ie tasks without designing dedicated modules, as tackling all ie tasks with one model can facilitate knowledge sharing between different tasks. therefore, various attempts have been made to unify all ie tasks with one model structure. @xcite ; @xcite ; @xcite encode all ie tasks' target structure as graphs and design graph-based methods to predict them; @xcite ; @xcite solve general ie tasks in a generative way with text-to-text or text-to-structure frameworks. however, graph-based models tend to be complex to design, while generative models are time-consuming to decode. in our work, we creatively propose a simple yet effective paradigm for unified ie. inspired by @xcite , we re-examine ie tasks and consider that all of them are fundamentally span extraction (entity extraction in ner and re, trigger extraction and argument span detection in ee) or relational extraction foot_0 (relation extraction in re and argument role classification in ee). based on this perspective, we further simplify and unify all ie tasks into tokenpair classification tasks. figure 1 shows how each task can be converted. specifically, a span is decomposed into start-to-end and end-to-start token pairs. as depicted, the entity "school of computer science" in figure 1 (a) is decomposed into indices of (school, science) and (science, school). as for for the span extraction, we use the start-to-end and end-to-start token pairs to pinpoint the span, such as entity spans e 1 , e 2 , argument spans a 1 , a 2 and trigger span t (cells with pure color). for the relational extraction, we use the start-to-start and end-to-end token pairs to represent the relation, such as r and rol 1 , rol 2 (cells with gradient color). it is worth mentioning that both relations and event roles are regarded as directional, namely from start entity to end entity and from event trigger to argument spans. therefore, all ie tasks can be decomposed into token-pair classifications. after the reformulation, the local dependency and interaction from the plus-shaped orientation (as the orange and blue dotted lines depict) can provide vital information to classify the central token pair. based on the above decomposition, we propose a unified token-pair classification architecture for information extraction (utc-ie). specifically, we first apply biaffine model on top of the pre-trained language model (plm) to get representations of token pairs. then we design a novel transformer to obtain interactions between them. as the plusshaped dotted lines depicted in figure 1 , token pairs in horizontal and vertical directions cover vital information for the central token pair. for span extraction, token pairs in the plus-shaped orientation are either clashing or nested with the central token pair, for example, e 2 is contained by e 1 in figure 1 @xcite ; for relational extraction, the central token pair's two constituent mentions locate in the plus-shaped orientation, such as in figure 1 (b), r is determined by e 1 and e 2 . therefore, we make one token pair only attend horizontally and vertically in the token pair feature matrix. additionally, position embeddings are incorporated to keep the token pairs position-aware. moreover, neighboring token pairs are highly likely to be informative to determine the types of the central token pair, so we apply convolutional neural network (cnn) to model the local interaction after the plus-shaped attention. since the attention map for one token pair is intuitively similar to the plus operator, we name this whole novel module as plusformer. we conduct numerous experiments in two settings. when training separately on each task (named as single ie task), our model outperforms previous task-specific and unified models on 10 datasets of all ie tasks. when training a single model simultaneously on all ie tasks in one dataset (named as joint ie task), utc-ie achieves better or comparable results than 2 joint ie baselines. to thoroughly analyze why utc-ie is useful under the token-pair paradigm, we execute several ablation studies. we observe that cnn module in plusformer plays a significant role in ie tasks by the abundant local dependency between token pairs after the reformulation. besides, owing to the good parallelism of self-attention and cnn, utc-ie is one to two orders of magnitude faster than prior unified ie models and some task-specific work. to summarize, our key contributions are as follows 1. we introduce utc-ie, which decomposes all ie tasks into token-pair classification tasks. in this way, we can unify all single ie tasks under the same task formulation, and use one model to fit all ie tasks without designing taskspecific modules. besides, this unified decomposition is much faster than recently proposed generation-based unified frameworks. 2. after the reformulation of different ie tasks, we propose the plusformer to model interaction between different token pairs. the plusshaped self-attention and cnn in plusformer are well-motivated and effective in the reformulated ie scenario. experiments in 12 ie datasets all achieve state-of-the-art (sota) performance which justifies the superiority of plusformer in ie tasks. 3. the reformulation enables us to use one model to fit all ie tasks concurrently. therefore, we can train one model on three ie tasks, and results on two joint ie datasets show that the proposed unification can effectively benefit each ie task through multi-task learning. 4. extensive ablation experiments reveal that components in plusformer are necessary and beneficial. among them, cnn module in plusformer can be essential to the overall performance. analysis shows that this performance gain is well-explained because when reformulating ie tasks into token-pair classifications, the adjacent token pairs can be informative and cnn can take good advantage of the local dependency between them. ## task decomposition and decoding we first introduce how we decompose ie tasks to conduct training, then present the decoding procedure for decomposition. more discussions about the decomposition are presented in appendix a. ## method figure 2 shows an overview of the architecture. firstly, we present biaffine @xcite 3x3 conv gelu + 3x3 conv plusattention plm biaffine model plusformer mlp x n layernorm + layernorm figure 2: an overview of the utc-ie model. ## related work information extraction tasks, which consist of named entity recognition, relation extraction, and event extraction, have long been a fundamental and well-researched task in the natural language processing (nlp) field. previous researches mainly only focus on one or two tasks. recently, building joint neural models of unified ie tasks has attracted increasing attention. some of them incorporate graphs into ie structure. @xcite propose a unified framework called dygie++ to we unify all ie tasks as several token-pair classification tasks, which are fundamentally similar to the span-based methods on the ie task, for the start and end tokens can locate a span. numerous ner studies emerge on span-based models, which are compatible with both flat and nested entities and perform well @xcite @xcite . in addition to entities, the span-based method is also used in re. some models @xcite only leverage span representations to locate entities and simply calculate the interaction between entity pair, while others @xcite ; zhong and chen, 2021) encode span pair information explicitly to extract relations. with regard to event extraction, as far as we know, there is little work on injecting span information into ee explicitly. @xcite leverage span representations on general ie, but their model is complicated and only considers span at the embedding layer without further interaction. conceptually, jiang et al. (2020)'s work is similar to ours, but they need a two-stage model to determine the span type and span relations, respectively. detailed analysis are depicted in appendix g. although many spanbased ie models exist, they are task-specific and lack interaction between token pairs. decomposing ie tasks as token-pair classification and conducting interaction between token pairs can uniformly model span-related knowledge and advance sota performance. the most novel component of plusformer is the plus-shaped attention mechanism, which can make token pairs interact with each other in an efficient way. a similar structure called axial transformers @xcite is proposed in the computer vision (cv) field, which is designed to deal with data organized as high-dimension tensors. @xcite incorporate axial attention into relation classification to improve the performance on two-hop relation. however, cnn was not used in these works, while cnn has been proven to be vital to the ie tasks. another similar structure named twin transformer @xcite used in cv, where they encode pixels of image from row and column sequentially, and leverage cnn on top of them. but the position embeddings, which are important for ie tasks, are not used in the twin transformer. besides, we want to point out that the usage of plus-shaped attention and cnn originates from the reformulation of ie tasks, any other modules which can directly enable interaction between constituent spans of a relation and between adjacent token pairs should be beneficial. ## conclusion in this paper, we decompose ner, re and ee tasks into token-pair classifications. through the decomposition, we unify all ie tasks under the same formulation. after scrutinizing the token-pair feature matrix, we find the adjacent and plus-shaped interactions between token pairs should be informative. therefore, we propose plusformer, which uses an axis-aware plus-shaped self-attention followed by cnn layers to help token pairs interact with each other. experiments on 10 single ie datasets and 2 joint ie datasets all outperform or approach the sota performance. besides, owing to the parallelism of self-attention and cnn, our model's inference speed is substantially faster than previous sota models in re and ee. lastly, most of the previous ie models limit the interaction in the 1-d sequential dimension, while the reformulation of ie tasks opens a new angle to broaden the communication to the 2-d feature matrix.