UTC-IE: A U NIFIED T OKEN - PAIR C LASSIFICATION A RCHITECTURE FOR I NFORMATION E XTRACTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Information Extraction (IE) spans several tasks with different output structures such as named entity recognition, relation extraction and event extraction
Previously those tasks were solved with different models because of diverse task output structures
Through re -examining IE tasks , we find that all of them can be interpreted as extracting spans and span relations.
We propose using the start and end token of a span to pinpoint the span in texts, and using the start -to-start and end -to-end token pairs of two spans to determine the relation.
Hence, we can unify all IE tasks under the same token-pair classification formulation.
Based on the reformulation, we propose a U nified T oken-pair C lassification architecture for
I nformation E xtraction ( UTC-IE ), where we introduce Plusformer on top of the token-pair feature matrix.
Specifically, it models axis-aware interaction with plusshaped self-attention and local interaction with Convolutional Neural Network over token pairs.
Experiments show that our approach outperforms task-specific and unified models on all tasks in 10 datasets, and achieves better or comparable results on 2 joint IE datasets.
Moreover, UTC-IE speeds up over state-of-the-art models on IE tasks significantly in most datasets, which verifies the effectiveness of our architecture.
I NTRODUCTION
Information Extraction (IE) aims to identify and classify structured information from unstructured texts (Andersen et al., 1992; Grishman, 2019).
IE consists of a wide range of tasks, such as named entity recognition (NER), joint entity relation extraction (RE) 1 and event extraction (EE) 2 .
In the last decade, many paradigms have been proposed to solve IE tasks, such as sequence labeling (McCallum & Li, 2003;
Huang et al., 2015; Zheng et al., 2017; Yu et al., 2020a), span-based classification (Jiang et al., 2020; Yu et al., 2020b; Wang et al., 2021; Ye et al., 2022), MRC-based methods (Levy et al., 2017; Li et al., 2020; Liu et al., 2020) and generation-based methods (Zeng et al., 2018; Yan et al., 2021a;
Hsu et al., 2022).
The above work mainly concentrates on solving individual tasks, but it is desired to have a unified model to solve all IE tasks without designing dedicated modules.
Besides, tackling all IE tasks with one model can facilitate knowledge sharing between different tasks.
Therefore, various attempts have been made to unify all IE tasks with one model structure.
Wadden et al.
; Lin et al.
Nguyen et al.
(2021) encode all IE tasks’ target structure as graphs and design graph-based methods to predict them; Paolini et al.
Lu et al.
(2022) solve general IE tasks in a generative way with a text-to-text or text-to-structure framework.
However, graph-based models tend to be complex to design, and generative models are time-consuming to decode.
In our work, we creatively propose a simple yet effective paradigm for unified IE.
Inspired by Jiang et al. (2020), we re-examine IE tasks and consider that all of them are fundamentally span extraction (entity extraction in NER and RE, trigger classification and argument span detection in EE) 𝑎 !
𝑎 !
𝒓𝒐𝒍 𝟏 𝒓𝒐𝒍𝑡 𝒓𝒐𝒍𝒓𝒐𝒍𝑎 $ 𝑎 $
The girl finallyreturned to New York City
The finally returned to
New
York
City girl 𝒂𝒂𝑎 !
𝑎 " The girl
New York City 𝑡 Transport 𝒓𝒐𝒍
Artifact
Destination 𝒕 𝑒 !
𝒓 𝑒 !
𝒓 𝑒 $ 𝑒 $
British author J.K. Rowling wrote Harry Potter novels
British
J.K.
Rowling wrote
Harry
Potter novels author 𝒆𝒆𝑒 !
𝑒 " J.K. Rowling
Harry Potter novels
Author 𝑒 !
𝑒 $ 𝑒 !
𝑒 $
School of Computer Sciencehas the best teachers
School
Computer
Science has the best teachers of 𝒆 𝟏 𝒆𝑒 !
𝑒 " School of Computer Science
Computer Science
[Figure]
𝒓 𝒓𝒐𝒍 or relational extraction 3 (relation extraction in RE and argument role classification in EE). Based on this perspective, we further simplify and unify all IE tasks into token-pair classification tasks .
Figure 1 shows how each task can be converted.
Specifically, a span is decomposed into start-to-end and end-to-start token pairs.
As depicted, the entity “School of Computer Science” in Figure 1(a) is decomposed into indices of (School, Science) and (Science, School).
As for detecting the relation between two spans, we convert it into start-to-start and end-to-end token pairs from head mention to tail mention.
For example, in Figure 1(b), the relation “Author” between “J.K. Rowling” and “Harry Potter novels” is decomposed into indices of (J.K., Harry) and (Rowling, novels).
Based on the above decomposition, we propose a U nified T oken-pair C lassification architecture for I nformation E xtraction ( UTC-IE ).
Specifically, we first apply Biaffine model on top of the pretrained language model to get representations of token pairs.
Then we design a novel Transformer to obtain interactions between them.
As the plus-shaped dotted lines depicted in Figure 1, token pairs in horizontal and vertical directions cover vital information for the classification on the central token pair.
For span extraction, token pairs in the plus-shaped orientation are either clashing or nested with the central token pair, for example, e 2 is contained by e 1 in Figure 1(a); for relational extraction, the central token pair’s two constituent spans locate in the plus-shaped orientation, such as in Figure 1(b), r is determined by e 1 and e 2 .
Therefore, we make one token pair only attend horizontally and vertically in the token pair feature matrix.
In addition, position embeddings are incorporated to keep the token pairs position-aware.
Moreover, neighboring token pairs are highly likely to be informative to determine the types of the central token pair, so we apply Convolutional Neural Network (CNN) to model the local interaction after the plus-shaped attention.
Since the attention map for one token pair is intuitively similar to the plus operator, we name the novel module as Plusformer .
We conduct numerous experiments in two settings.
When training separately on each task, our model outperforms previous task-specific and unified models on 10 datasets of all IE tasks.
When training a single model simultaneously on all IE tasks in one dataset (named as joint IE task), UTCIE achieves better or comparable results than 2 joint IE baselines.
To thoroughly analyze why
our
UTC-IE architecture is useful in IE tasks under the token-pair paradigm, we execute several ablation studies.
We observe that CNN module in Plusformer plays a significant role in IE tasks because of the abundant local dependency between token pairs after the reformulation.
Furthermore, owing to the good parallelism of self-attention and CNN, UTC-IE is one to two orders of magnitude faster than prior unified IE models and some task-specific work.
To summarize, our key contributions are as follows
We introduce UTC-IE, which decomposes all IE tasks into token-pair classification tasks
In this way, we can unify all IE tasks under the same task formulation.
Henceforth, we can use one model to fit all IE tasks without designing task-specific modules.
Besides, this unified decomposition is much faster than recently proposed generation-based unified frameworks.
After the reformulation of different IE tasks, we propose the Plusformer to model interaction between different token pairs.
The plus-shaped self-attention and CNN in Plusformer are well-motivated and effective in the reformulated IE scenario.
Experiments in 12 IE datasets all achieve state-of-the-art (SOTA) performance which justifies the superiority of Plusformer in IE tasks.
The reformulation enables us to use one model to fit all IE tasks.
Therefore, we can train one model on three IE tasks, and results on two joint IE datasets show that the proposed unification can effectively benefit each IE task through multi-task learning.
Extensive ablation experiments reveal that components in Plusformer are necessary and beneficial.
Among them, CNN module in Plusformer can be essential to the overall performance.
Analysis shows that this performance gain is well-explained because when reformulating IE tasks into token pair classifications, the adjacent token pairs can be informative and CNN can take good advantage of the local dependency between them.
T ASK D ECOMPOSITION AND D ECODING
We first introduce how we decompose IE tasks to conduct training, and then present the decoding procedure for the decomposition.
More discussions about the decomposition are presented in
Appendix A.
2. T ASK D ECOMPOSITION
Formally, given an input sentence of L tokens x =[ x 1 , x 2 , ..., x L ] , the potential token pairs can form a score matrix Y ∈ R L × L × ( |S| + |R| ) , where S is span classes, R is relational classes.
We stipulate
• When a span ( s, e ) is of type t , then Y [ s, e, t ]
=
Y [ e, s, t ] = 1 , where s, e ∈ [1 , L ] and t ∈ [1 , |S| ] are the start, end token indices and span type; • When the span ( s 1 , e 1 ) forms the relation r ∈ [ |S| +1 , |S| + |R| ] with another span ( s 2 , e 2 ) , then Y [ s 1 , s 2 , r ]
= Y [ e 1 , e 2 , r ] = 1 .
NER aims to extract all entities { ( s i , e i , t i ) } , where t i ∈ S e and S e is pre-defined entity types.
Therefore, in NER, S = S e and R = ϕ .
RE aims to extract all relations { (( s hi , e hi , t hi ) , r i , ( s ti , e ti , t ti )) } , where the superscript h and t denotes the head and tail entities, t hi , t ti ∈ S e , r i ∈ R r and S e , R r are pre-defined entity types, and relation types.
Therefore, in RE, S = S e and R = R r .
EE aims to extract all events {{ ( s i , e i , t i ) , ( s 1 ia , e 1 ia , rol 1 i ) , . . .
, ( s kia , e kia , rol ki ) }} , where ( s i , e i ) means the trigger span, t i ∈ S t is the event type, S t is pre-defined event types; s ia , e ia ∈ [1 , L ] are the start and end token indices of an argument span, k is the number of arguments of the trigger, to extract argument spans, we set the argument type as S a , and |S a | = 1 ; rol
i ∈ R o is the role type of the argument and R o is pre-defined role types.
We can view role types between the trigger and the arguments as relations.
Therefore, in EE, S = S t ∪ S a and R = R o .
Joint IE aims to jointly extract entities, relations, and events in the text.
Extracting entities and relations are generally the same as those in NER and RE.
When extracting events, there is no need to extract argument spans purposely because all argument candidates are entities.
Therefore, in joint
IE, S = S t ∪ S e and R = R r ∪ R o .
[Figure]
2. D ECODING
The decoding essentially extracts spans and relations from the score matrix Y .
If Y [ s, e, t ] =
Y [ e, s, t ] = 1 and t ∈ [1 , |S| ] , then the span ( s, e ) is of type t .
And for two spans ( s 1 , e 1 ) and ( s 2 , e 2 ) , if Y [ s 1 , s 2 , r ] =
Y [ e 1 , e 2 , r ] = 1 and r ∈ [ |S| + 1 , |S| + |R| ] , then the span ( s 1 , e 1 ) forms relation r with the span ( s 2 , e 2 ) .
The above decoding is for the ideal situation, where no span clash exists.
However, for model’s predictions, we need to first resolve the conflicts.
The decoding with model’s predictions will be presented in Appendix B.
M ETHOD
Figure 2 shows an overview of the architecture.
Firstly, we present Biaffine (Dozat & Manning, 2017) model based on pre-trained language models (PLM).
Then, we propose a novel Transformerlike structure named Plusformer to model interactions between token pairs.
Next, we describe loss functions.
3. B IAFFINE M ODEL
Given an input sentence, we first apply a PLM as our sentence encoder to obtain the contextualized representation as follows
[Equation]
where H ∈ R L × d , d is the PLM ’s hidden size.
Next, we use the Biaffine mechanism to get features for each token pair as follows
[Equation]
where MLP start , MLP end are multi-layer perceptron layers, H s , H e ∈ R L × d , W
1 ∈ R d × c × d ,
W 2 ∈ R c × 2 d , b ∈ R c , ⊕ refers to concatenation; S ∈ R L × L × c provides features for all possible token pairs, and c is the feature dimension size.
3. P LUSFORMER
As illustrated in Section 1, when modeling the interaction between token pairs, the plus-shaped and local interaction should be beneficial.
Therefore, we introduce the axis-aware plus-shaped selfattention and position embeddings to conduct plus-shaped interaction, we name this self-attention
PlusAttention.
Then, we leverage CNN to model local dependencies.
We name this whole structure
Plusformer .
PlusAttention.
We first apply the self-attention mechanism (Vaswani et al., 2017) horizontally and vertically as follows
[Equation]
where W Qh , W Kh , W Vh , W Qv , W Kv , W Vv ∈ R c × c , Z h , Z v ∈ R L × L × c .
After the self-attention, we use the following method to merge Z h , Z v
[Equation]
where
S
′ ∈
R L × L × c .
We make the plus-shaped self-attention axis-aware by using two groups of attention parameters and using concatenation instead of an addition to merge Z h , Z v .
Position Embeddings.
Although the model should be able to distinguish between horizontal and vertical directions through axis-aware plus-shaped attention, it still lacks the sense of distances between token pairs and the area the token pair locates.
Therefore, we utilize two kinds of position embeddings to enable the model with these abilities.
• Rotary Position Embedding (RoPE)
(Su et al., 2021) can encode the relative distance between two token pairs.
It is utilized in both horizontal and vertical self-attention.
• Triangle position embedding is incorporated to mark the position of token pairs in the feature map, which means cells in the upper and lower triangles will use different position embeddings.
It adds to S in Eq.(3) before Attention.
CNN Layer.
After the PlusAttention, we apply CNN with kernel size 3 × 3 on the S ′ to help the model exploit the local dependency between neighboring token pairs.
The formulation is as follows
[Equation]
where S ′′ ∈ R L × L × c , and σ is the activation function; and the bias term of CNN is not used to avoid result inconsistencies for a sample when it is in batches of different lengths.
The Plusformer layer will be repeatedly used to interact fully between token pairs.
Layer normalization (Ba et al., 2016) is ignored in the formulation for brevity.
3. L OSS F UNCTION
Finally, we get final scores as follows
[Equation]
where ˆ Y ∈ R L × L × ( |S| + |R| +1) ; ˆ Y S ∈ R L × L ×|S| ,
ˆ Y R ∈
R
L × L × ( |R| +1) are scores for span extraction and relational extraction, respectively.
The +1 in ( |R| +1) is because we use the adaptive thresholding loss (ATL) from (Zhou et al., 2021) to avoid a global threshold in relational extraction.
For the span extraction, we use the binary cross-entropy (BCE) loss as follows
[Equation]
For the relational extraction, we utilize the ATL as follows
[Equation]
where P T and N T denote the positive and negative classes, ˆ Y R
[: , : , |R| +1] is the score for the threshold class TH .
Only token pairs with scores higher than their corresponding adaptive thresholds are considered when decoding.
We do not use ATL loss for span extraction because we need to sort span scores when decoding spans.
The total loss L = L 1 + L 2 is used for optimization.
[Table] [Table]
PFN (Yan et al., 2021b) - -
[Table] [Table]
[Table] [Table]
Lu
[Table]
al.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
SciERC
[Table]
Ent
[Table]
ERE
[Table]
TEXT2EVENT
[Table]
al
[Table]
[Table] [Table]
[Table] [Table]
Lu
[Table]
al
[Table] [Table]
[Table] [Table]
DEGREE
[Table]
Hsu et al
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table]
E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
We conduct experiments on 10 datasets across three IE tasks, including NER, RE, and EE, and on 2 joint IE datasets.
We evaluate NER task with CoNLL03 (Sang & Meulder, 2003) and OntoNotes (Pradhan et al., 2013) on flat NER, and with ACE04 (Doddington et al., 2004), ACE05-Ent (Walker et al., 2006) and GENIA (Kim et al., 2003) on nested NER.
As for relation extraction, we use
ACE05-R (Walker et al., 2006) and SciERC (Luan et al., 2018).
Since Wang et al.
(2021) and Ye et al.
(2022) consider symmetric relations, which shall massively influence the performance, we name this scenario Symmetric RE with datasets ACE05-R +
and SciERC + .
For event extraction, we follow Lin et al.
(2020) to perform experiments on three datasets, ACE05-E,
ACE05-E+ (Doddington et al., 2004) and ERE-EN (Song et al., 2015).
And for joint IE, we test on ACE05-E+ and ERE-EN.
Statistics of all these datasets, implementation details, evaluation metrics, baselines, pre-trained language models and hyper-parameters are described in Appendix C.
[Table]
4. R ESULTS ON S INGLE IE TASKS
In this section
[Table]
we report the UTC-IE performance in each single IE task.
Results are shown in Table 1 4 .
The table shows that UTC-IE exceeds the previous SOTA models on all IE tasks.
Particularly,
UTC-IE averagely improves the entity F1 of NER, the entity F1 and relation F1 of RE, the entity F and the relation F1 of symmetric RE for +0.18, +0.71, +1.07, +0.21, +0.94, respectively.
And for the EE tasks, UTC-IE increases the trigger F1, argument F1 for +0.35 and +1.67.
We highlight that our model is significantly helpful for relational extraction, such as relation extraction and argument extraction, which proves the effectiveness of interaction between token pairs.
Although the performance increment of span extraction is not as significant as that of relational extraction, UTC-IE consistently improves on various span extraction tasks.
Besides, we also test UTC-IE without Plusformer.
Surprisingly, this simple model surpasses previous SOTA models on four results marked with ♣ , which proves the effectiveness of the task decomposition.
The comparison between models with and without Plusformer clearly shows that Plusformer is effective in all tested datasets, and the performance improvement ranges from +0.40 (on
OntoNotes) to +3.00 (on SciERC + ).
Notably, the average performance gain of adding Plusformer on symmetric RE (+2.06) is more remarkable than that on RE (+1.03).
We presume this is because the interaction between token pairs are more beneficial for symmetric relations.
4. R ESULTS ON J OINT IE TASK
Multi-task learning has be proven to be useful in the IE area (Lin et al., 2020; Nguyen et al., 2021).
Since UTC-IE unifies all IE tasks into a token-pair classification scenario, it is natural to test whether one UTC-IE model can benefit from jointly learning all IE tasks.
In Table 2, the performance of
UTC-IE single is from the entity F1 of NER, relation F1 of RE, trigger F1 of EE and argument F1 of
EE, respectively.
Based on the comparison between UTC-IE single and UTC-IE joint , it is obvious that jointly learning these three tasks consistently improves performance in the 2 joint IE datasets.
Moreover, UTC-IE joint outperforms previous SOTA joint IE models in Table 2, the average performance enhancement is +0.69 in ACE05-E+ and +0.75 in ERE-EN.
Specifically, UTC-IE joint increases the average performance of relational extraction by +1.30.
Thusly, through unifying different IE tasks through our task decomposition, Plusformer can enjoy the benefit of multi-tasking learning, and achieve better performance than previous SOTA models.
4. S PEED C OMPARISON
To get a sense of the speed superiority of UTC-IE, we compare the inference speed of UTC-IE with previous unified models and task-specific SOTA models.
The former comparison is presented in Table 3 and the latter locates in the Appendix E.
Compared with the generative UIE (Lu et al., 2022), UTC-IE improves F1 from 0.46 (on CoNLL03) to 2.89 (on ACE05-E+), and obtains one order magnitude of speed boost.
Compared with OneIE (Lin et al., 2020), UTC-IE fundamentally enhances the performance for relational extractions (e.g., Rel. and Arg.) with an average of 4.47 F increment in joint IE.
At the same time, UTC-IE is one order of magnitude faster than OneIE.
In
[Table]
of the factor
the
CNN-IE
[Table]
a nutshell, compared with previous SOTA models (whether task-specific, unified or joint), UTC-IE achieves substantial performance gain across several datasets with a significant speed boost.
4. A BLATION S TUDY
To analyze the effectiveness of each component in Plusformer
[Table]
we ablate each of them and list the outcomes in Table
[Table]
and results on more datasets are presented in Appendix F. Besides, we study how many Plusformer layers are suitable in Appendix F.5.
Based on the ablation
[Table]
CNN is the most useful component among all IE tasks.
The reason behind this improvement is that once token pairs are organized in the square feature map, the spatial correlations between neighboring token pairs become allusive
[Table]
and CNN excels at exploiting these local interactions.
More comprehensive analysis of CNN in Plusformer locates in Appendix F.1.
To deepen our understanding of UTCIE
[Table]
we try another variant of Plusformer where the PlusAttention is discarded
[Table]
and we name this variant CNN-IE .
The bottom line of Table 4 shows that the CNN-IE model can surpass or approach previous SOTA performance in almost all datasets, which proves the universality of our proposed task formulation.
However, CNN is not a panacea for UTC-IE.
From Table 4, removing position embeddings or axisaware 5 from UTC-IE will lead to an average of 0.39 or 0.44 performance degradation, respectively.
Moreover, based on the performance of CNN-IE and UTC-IE, the average performance shrinks from 74.53 to 74.17 if the PlusAttention is deprived of Plusformer, which means the plus-shaped self-attention is a desideratum.
In addition, we present some intuitive examples and deeper analysis for position embeddings and axis-aware in Appendix F.3 and F.4.
R ELATED W ORK
Information extraction tasks, which consists of named entity recognition, relation extraction, and event extraction, have long been a fundamental and well-researched task in the natural language processing (NLP) field.
Previous researches mainly only focus on one or two tasks.
Recently, building joint neural models of unified IE tasks has attracted increasing attention.
Some of them incorporate graphs into IE structure.
Wadden et al.
(2019) propose a unified framework called DYGIE++ to extract entities, relations and events by leveraging span representations via span graph updates.
Lin et al.
(2020) and Nguyen et al.
(2021) extend DYGIE++ by incorporating global features to extract cross-task and cross-instance interactions with multi-task learning.
In addition to the graph-based models mentioned above, other studies focus on tackling general IE by generative models.
Paolini et al.
(2021) construct a framework called TANL, which enhances the generation model using augmented language methods.
Moreover, Lu et al.
(2022) regard IE task as a text-to-structure generation task, and leveraging prompt mechanism.
We unify all IE tasks as several token-pair classification tasks, which are fundamentally similar to the span-based methods on the IE task, for the start and end tokens can locate a span.
Numerous
NER studies emerge on span-based models, which are compatible with both flat and nested entities and perform well (Eberts & Ulges, 2020; Yu et al., 2020b; Li et al., 2021; Zhu & Li, 2022).
In addition to entities, the span-based method is also used in RE.
Some models (Wang et al., 2021;
Ye et al., 2022) only leverage span representations to locate entities and simply calculate the interaction between entity pair, while others (Wang et al., 2020; Zhong & Chen, 2021) encode span pair information explicitly to extract relations.
With regard to event extraction, as far as we know, there is little work on injecting span information into EE explicitly.
Wadden et al.
leveraging span representations on general IE, but their model is complicated and only considers span at the embedding layer without further interaction.
Conceptually, Jiang et al.
(2020)’s work is similar to ours, but they need a two-stage model to determine the span type and span relations, respectively.
Detailed analysis are depicted in Appendix G.
Although many span-based IE models exist, they are task-specific and lack interaction between token pairs.
Decomposing IE tasks as token-pair classification and conducting interaction between token pairs can uniformly model span-related knowledge and advance SOTA performance.
The key component of Plusformer is the plus-shaped attention mechanism, which can make token pairs interact with each other in an efficient way.
A similar structure called Axial Transformers (Ho et al., 2019) is proposed in Computer Vision (CV) field, which is designed to deal with data organized as high-dimension tensors.
Tan et al.
(2022) incorporate axial attention into relation classification, aiming to improve the performance on two-hop relation.
However, CNN was not used in these work, while CNN has been proven to be vital to the IE tasks.
Another similar structure named
Twin Transformer (Guo et al., 2021) used in CV is very much similar to Plusformer, where they encode pixel of image from row and column sequentially, and leverage CNN on top of them.
But the position embeddings, which is important for IE tasks, are not used in the Twin Transformer.
Besides, we want to point out that the usage of plus-shaped attention and CNN originates from the reformulation of IE tasks, any other modules which can directly enable interaction between constituent spans of a relation and between adjacent token pairs should be beneficial.
C ONCLUSION
In this paper, we decompose NER, RE and EE tasks into token-pair classifications.
Through the decomposition, we unify all IE tasks under the same formulation.
After scrutinizing the tokenpair feature matrix, we find the adjacent and plus-shaped interactions between token pairs should be informative.
Therefore, we propose Plusformer, which uses an axis-aware plus-shaped self-attention followed by CNN layers to help token pairs interact with each other.
Experiments in 10 single IE datasets and 2 joint IE datasets all outperform or approach the SOTA performance.
Besides, owing to the parallelism of self-attention and CNN, our model’s inference speed is substantially faster than previous SOTA models in RE and EE.
Lastly, most of the previous IE models limit the interaction in the 1-D sequential dimension, while the reformulation of IE tasks opens a new angle to broaden the communication to the 2-D feature matrix.
A D ISCUSSION ON T ASK DECOMPOSITION
In this section, we will discuss two issues of the decomposition.
The first is the inconsistency stipulation about the relation decomposition, the second is the false positive issue when decoding relations.
A. T HE INCONSISTENCY
As our stipulation in Section 2, if a span ( s, e ) has an expected span type t , both the Y [ s, e, t ] and
Y [ e, s, t ] are 1 .
If two spans ( s 1 , e 1 ) and ( s 2 , e 2 ) have relation r , this means the relation should also exist between spans ( s 1 , e 1 ) and ( e 2 , s 2 ) (the end-to-start version of the span ( s 2 , e 2 ) ), then based on our stipulation on the relation, the Y [ s 1 , e 2 , r ] and Y [ e 1 , s 2 , r ] should also equal 1, but we only define the Y [ s 1 , s 2 , r ] = 1 and Y [ e 1 , e 2 , r ] = 1 , this causes an inconsistency between the stipulations.
We ignore Y [ s 1 , e 2 , r ] and Y [ e 1 , s 2 , r ] to make the decoding less cluttered.
A. F ALSE P OSITIVE R ELATION
A potential risk of the decomposition and decoding is that it may cause false positive relations.
Given four spans p 1 =
( s 1 , e 1 ) , p 2 = ( s 2 , e 2 ) , p 3 = ( s 3 , e 3 ) , p 4 = ( s 4 , s 4 ) , if p 4 has relation r with p 1 and p 2 , and no relation exist between p 4 and p 3 .
Then Y [ s 4 , s 1 , r ]
=
Y [ e 4 , e 1 , r ] = 1 ,
Y [ s 4 , s 2 , r ]
=
Y [ e 4 , e 2 , r ] = 1 .
However, if s 1 = s 3 , e 2 = e 3 .
Namely, p 1 shares start token with p 3 and p 2 shares end token with p 3 .
Then, based on Y [ s 4 , s 1 , r ] =
Y [ e 4 , e 2 , r ] = 1 , we get
Y [ s 4 , s 3 , r ]
=
Y [ e 4 , e 3 , r ] = 1 ,, the decoding process will mistakenly think p 4 has relation r with p 3 .
However, this situation should be rare, and none is found in the tested datasets.
B D ECODING WITH MODEL ’ S PREDICTIONS
In this section, we will detail the decoding process for models’ predictions.
The process described in Section 2.2 is not directly applicable to models’ predictions since spans may conflict with each other 6 .
With prediction score matrix ˆ Y S from Eq.(6), we follow previous work (Yu et al., 2020b) to first filter out spans whose scores are less than 0.5; for the left spans, we sort the spans based on their scores, then choose spans in descending order and make sure the span has no boundary clash with chosen spans.
For relational extraction, we first decode all spans, then we get a binary matrix
¯ Y R =
ˆ Y R
[: , : , : | R | +1]
>
ˆ Y R
[: , : , | R | +1] , then we pair spans to check whether they form relations.
Take two spans ( s 1 , e 1 ) and ( s 2 , e 2 ) for instance, if ¯ Y R [ s 1 , s 2 , r ] = ¯ Y R [ e 1 , e 2 , r ] = 1 , we claim the first span has relation r with the second span.
For the RE task, we pair all entity spans to check if they form relations; for the EE task, we pair the trigger spans and argument spans to check if they form a role relationship; and for the joint IE task, we pair entity spans to check if they form relations, we pair the trigger spans and entity spans (because all argument spans are entity spans) to check if they form a role relationship.
C E XPERIMENTAL S ETTINGS
In this section, we describe all experimental settings in detail, such as the statistics of datasets, baseline models, and more implementation details.
C. D ATASETS
We conduct experiments on 10 single IE datasets and 2 joint IE datasets, and we detail the statistic of all datasets in Table 5. [Table] we experiment on Walker
To used
, we name in named entity recognition as
Specifically, we use the same preprocessing and splitting procedure on nested [-NER benchmarks. In flat NER, we adopt CoNLL
-]datasets as Yan et al.
(2022), for they fix some annotation problems to unify different versions of these datasets and make a strictly fair comparison.
Relation extraction.
We conduct experiments on two relation extraction datasets, ACE05 (Walker et al., 2006) and SciERC 10 (Luan et al., 2018).
The ACE05 dataset, named as ACE05-R in our paper, is collected from various domains, such as newswire and online forums.
The SciERC dataset provides entity, coreference and relation annotations from AI conference/workshop proceedings.
We
In typical RE, it is crucial to distinguish which entity comes first (head entity) and which comes next (tail entity).
As for symmetric relational instance, the relation exists from both head-to-tail and tail-to-head directions.
There are one such relation type in ACE05-R and two in SciERC.
Some papers (Wang et al., 2021; Ye et al., 2022) regard each symmetric relational example as two directed relations, while others regard them as one relation.
We find that this setting will hugely influence the performance.
Therefore, we name this setting Symmetric Relation Extraction and name the corresponding datasets ACE05-R +
and SciERC + .
Event extraction.
We evaluate UTC-IE on two widely used event extraction datasets, ACE (Doddington et al., 2004) and ERE (Song et al., 2015).
Following the prior preprocessing step (Wadden et al., 2019; Lin et al., 2020; Lu et al., 2021) on them, we obtain three datasets, ACE05E, ACE05-E+ and ERE-EN.
ACE05-E+ additionally takes relation arguments, pronouns and multitoken event triggers into consideration compared with ACE05-E. We use the same train/dev/test split as Lu et al.
(2021) for all datasets to ensure a fair comparison.
Furthermore, we still use ACE05-E+ and ERE-EN on joint IE, for they have annotations on all IE tasks.
C. B ASELINES
TANL (Paolini et al., 2021) and UIE (Lu et al., 2022) are both unified information extraction models in the generative way, with different input and output formats.
TANL uses T5-base as the backbone model, while UIE uses T5-large.
We compare our model with them in every IE task.
For TANL, we report single-task results for our model is trained under each task.
For UIE, we report results with pre-training, which have better performance.
In addition to these two baselines, each task also compares with a series of recently proposed task-specific methods as follows.
CNN-IE is the baseline model we design to prove the necessity of PlusAttention.
The only difference between CNN-IE and UTC-IE is the former ignores the PlusAttention in Figure 2.
We tune the number of CNN layers in CNN-IE from 2 to 6, and the best results are reported.
Named entity recognition.
We compare our model’s performance on NER with several recently proposed NER methods
• BART-NER (Yan et al., 2021a) formulates unified NER model as entity span sequence generation task.
They use BART-large as the pre-trained model.
• W 2 NER (Li et al., 2022) formulates unified NER model as word-to-word classification task.
The model employs BioBERT on GENIA and BERT-large on other datasets.
• BS (Zhu & Li, 2022): authors use span-based NER model as baseline and propose boundary smoothing as a regularization technique to improve model performance.
It leverages RoBERTa-base as the base encoder.
• CNN-NER
(Yan et al., 2022) utilizes CNN to model local spatial correlations between spans and surpass recently proposed methods on nested NER.
We report results using RoBERTa-base model.
Relation extraction.
For relation extraction, we compare our model with several SOTA models.
• UniRE (Wang et al., 2021) jointly extracts entities and relations using a table containing all word pairs.
• PURE (Zhong & Chen, 2021) adopts a pipeline approach to solve NER and RE independently, using distinct contextual representations for entities and relations.
• PFN
(Yan et al., 2021b) claims that some information should be shared between named entity recognition and relation extraction, while other information should be independent.
They propose PFN to model two-way interaction (partition and filter) between two tasks.
• PL-Marker
(Ye et al., 2022): authors consider interactions between spans and propose PL-Marker by strategically packing the markers in the encoder. [Table]
Previous models mentioned above use different RE datasets.
Specifically, UniRE and PL-Marker regard symmetric relations as two directed relations, while other work does not.
Besides, these two models utilize cross-sentence context
Event extraction.
Generative methods are popular in recently proposed event extraction papers.
• TEXT2EVENT (Lu et al., 2021) is a sequence-to-structure model which outputs a tree-like event structure with an given input sentence.
The model uses T5-large as the base model.
• DEGREE (Hsu et al., 2022) leverages manually designed prompts to generate event records in natural language.
We report the end-to-end performance of DEGREE instead of the pipeline way.
The model leverages BART-large as encoder-decoder.
Joint IE.
There are only two previous models that consider the joint IE in ACE05-E and ERE-EN datasets
• OneIE (Lin et al., 2020) proposes an end-to-end IE model, which employs global features and type dependency constraint at decoding step.
• FourIE (Nguyen et al., 2021) further improves the model by incorporating interaction dependency on representation level and label level.
For a fair comparison, we list the pre-trained model used for all baselines and our model on every
IE dataset in Table 6.
When choosing our pre-trained language model in different IE tasks’ datasets, we pick the same pre-trained model as the most recently published papers, such as BioBERT for
GENIA and RoBERTa-base for other NER datasets.
For RE and joint IE tasks, we choose the same pre-trained model as previous work.
For tasks where previous work applied a generative pre-trained model, we choose pre-trained model that has a similar size.
For example, in event extraction, we use
DeBERTa-large, whose number of parameters is 390M, which is closest to BART-large and T5-large used by previous EE papers.
C. E VALUATION M ETRICS
We report micro-F1 on all tasks:
• Entity: an entity is correct if its entity type and span offsets match a golden entity.
We use “Ent.”
to represent entity F1 through all tables.
• Relation: a relation is correct if its type and its head and tail entities are correct, and the offsets and type of entities should also match the golden instance.
We use “Rel.”
to represent relation F1 through all tables.
• Event trigger: a trigger is correct if its span offset and event type is correct.
We use “Trig.” to represent trigger F1 through all tables.
• Event argument: an argument is correct if its span offset, event type and role type all match the ground truth.
We use “Arg.” to represent argument F1 through all tables.
C. H YPER -P ARAMETERS
The detailed hyper-parameters used in each dataset are listed in Table 7.
AdamW optimizer (Loshchilov & Hutter, 2019) with weight decay 1e-2 for all datasets.
Experiments are conducted five times with five different random seeds.
We report the performance on test sets based on the model which achieves the best dev results in each dataset.
For NER, the best results are calculated by the entity F1; for RE, the best results are calculated by the sum of entity F1 and relation
F1; for EE, the best results are calculated by the best argument F1; for joint IE, the best results are calculated by the sum of relation F1 and trigger F1.
D C OMPLETE R ESULTS
We present the complete results of UTC-IE and that without Plusformer in Table 8. [Table] [Figure]
[Table]
E S PEED C OMPARISION
We test the speed of other models through their released code.
For models, such as OneIE (Lin et al., 2020), DEGREE (Hsu et al., 2022) and PL-Marker 11 (Ye et al., 2022), they also released a trained model along with their code
[Table]
and we used their released model to test the inference speed.
For UIE (Lu et al., 2022) and BS (Zhu & Li, 2022), we trained a model with their code.
The speed test is conducted in one RTX 3090 GPU and the batch size is set as 32 for all models (if the model goes out of memory, we choose the largest batch size that can accommodate the GPU); the test corpus is the test set of each datasets.
The speed is measured by the number of sentences in the test set divided by the number of seconds that elapsed.
And each inference is repeated three times, the average speed is reported.
The speed comparison can be roughly categorized into two kinds.
The first kind is the comparison with previous universal IE models, namely OneIE (Lin et al., 2020) and UIE (Lu et al., 2022), and results are depicted in Table 3.
Compared with UIE in five chosen datasets, UTC-IE is x19.7 faster and improves performance by 1.86 averagely.
Besides, for the joint IE task, UTC-IE is 18.4 times faster than OneIE and improves performance by 2.72 on average.
The second kind is the comparison between UTC-IE and SOTA models targeted for each IE task, and results are presented in Table 9.
Compared with previous SOTA models, the average performance increments for entity F1, relation
F1 and argument F1 are 0.31, 0.94 and 2.15.
In the meantime, UTC-IE speeds up for x1.0, x5.5 and x101.9 averagely.
In short, using UTC-IE for IE tasks can not only substantially enhance the performance in most cases, but also significantly speed up the inference speed in almost all datasets.
[Table] [Table]
F A BLATION S TUDY
For ablation, we will choose two datasets for each IE task to study the effect of each component in Plusformer.
We separately list the performance for span extraction (including entity extraction in NER and RE, trigger extraction in EE) in Table 10 and relational extraction (including relation extraction in RE and argument extraction in EE) in Table 11.
Besides, we also study how the performance varies with the change of the number of Plusformer layers in Figure 10.
F. CNN
Based on our ablations in Table 10 and Table 11, the CNN module in Plusformer contributes most to the performance enhancement.
To reveal why CNN is so effective in both span extraction and relational extraction, we first present an intuitive example in Figure 3 to show how CNN helps to extract entities and relations in the RE task.
Like in Figure 3(a), for NER, the entity e 2 can interact with entity e 1 and relation r 1 through CNN.
Besides, for RE, CNN can contribute in two ways.
On the one hand, CNN helps the relational token pair to directly gather information from its constituent entities, like the r 1 in Figure 3(b).
On the other hand, the start-to-start and end-to-end relational token pairs, like two r 2 cells, can directly interact with each other through CNN.
To quantitatively present the effectiveness of CNN in UTC-IE, we propose further ablations to show how the distance between the relational token pair and its constituent spans affects the relational F1, and how the distance between start-to-start and end-to-end token pairs affects the relational F1. [Table] 𝑒 !
[Table]
𝑒 ! [Table] 𝑟 [Table]
𝑒
[Table]
David Perkins is
[Table]
village doctor in California
David is village doctor in
California
Perkins a
[Table] [Table]
village doctor in
California
Perkins a
[Figure]
Furthermore, we conduct experiments on UTC-IE with different kernel sizes and choose the most proper size.
F.1. D ISTANCE B ETWEEN THE R ELATIONAL T OKEN P AIR AND I TS C ONSTITUENT SPANS VS. R ELATIONAL F
In this section, we will show how the relational F1 (relation F1 in RE and argument F1 in EE) will change when the distance between the relational token pair and its constituent spans varies.
For two spans ( s 1 , e 1 ) and ( s 2 , e 2 ) (we ignore their diagonally symmetric counterparts, since they will not affect the calculation here), the span relation from ( s 1 , e 1 ) to ( s 2 , e 2 ) is represented by two token pairs ( s 1 , s 2 ) and ( e 1 , e 2 ) .
The distance between the two token pairs and its constituent spans is calculated as follows
[Equation]
where the distance d is named as “ Span-Rel-Span Distance ”, it represents the longest distance between the relational token pairs to their constituent spans.
The relation between d and the relational
F1 is shown in Figure 4.
Without CNN, the performance for extracting relations between nearby constituent spans will drop sightly, while less affected for further ones, which proves that CNN is effective for exploiting local dependency to predict relations.
F.1. D ISTANCE B ETWEEN S TART - TO -S TART AND E ND - TO -E ND T OKEN P AIRS VS. R ELATIONAL F
As shown in Figure 3(b), if the distance between the start-to-start and end-to-end relational token pairs is small, the CNN should be helpful.
To verify this assumption, we first define the “ Inner
Relational Distance ” as follows, for two spans ( s 1 , e 1 ) and ( s 2 , e 2 ) , the relational token pairs are ( s 1 , s 2 ) and ( e 1 , e 2 ) , then the distance between two relational token pairs is calculated as follows
[Equation]
[Figure] [Figure] where d reveals the distance between start-to-start and end-to-end token pairs, and it is actually decided by the max constituent span length.
And its relation with the relational F1 is shown in
Figure 5.
It is clear that, most of the start-to-start token pair is near to its end-to-end token pair, and
CNN takes advantage of this adjacency to make better predictions.
We study the relation between the kernel size of CNN and F1 performance in Figure 6.
We observe that CNN with kernel size 3 obtains the best performance on almost all datasets and tasks. Specifically, reducing CNN kernel size to 1 significantly harms the performance on all datasets, for CNN will lose the capability of interacting with neighboring token pairs. In contrast, F1 also slightly decreases with larger CNN kernel size. We presume that CNN with a larger kernel size may introduce more noise and harm performance.
Therefore, we choose kernel size 3 for all datasets.
F. I S CNN A LL W E N EED ?
Since CNN is so effective in the Plusformer, it is natural to ask whether it is enough only to use
CNN.
Therefore, we conduct experiments on models without the plus-shaped self-attention and named this model CNN-IE.
We conduct experiments for CNN-IE in six datasets, and results are listed in Table 10 and Table 11.
Only with the CNN module can the model achieve SOTA or near
SOTA performance in all six datasets, which depicts the effectiveness of the proposed token-pair decomposition and CNN module.
However, it still lags behind the UTC-IE model, which reveals the necessity of the PlusAttention.
F. P OSITION E MBEDDINGS
The RoPE embedding aims to help token pairs be aware of the spatial relationships between each other, and the triangle position embedding tries to enable spans to be informed of their areas in the feature map.
From Table 10 and Table 11, the position embeddings enhance the span extraction [Figure] and relational extraction for 0.39, 0.64 averagely.
Besides, in Figure 7, we show that the position embeddings can help the model exploit the distance bias to improve the performance of relational extraction.
[Equation]
Lastly, we study the effect of PlusAttention.
We present an example to delineate why the axisaware is valuable for span extraction and relational extraction in Figure 8.
From Figure 8, axisaware should be worthwhile no matter what the task is, span extraction or relational extraction.
As expected, from Table 10 and Table 11, if we discard the axis-aware in Plusformer, the average performance of span extraction and relational extraction diminish 0.28 and 0.86, respectively, which reveals the necessity of axis-aware in the PlusAttention module. Besides, we show two case studies of the plus-shaped attention in Figure 9. The sentences are from the test dataset of ACE2005-Ent and ACE2005-R.
Both cases put larger attention scores on informative token pairs.
We study the relation between the number of Plusformer layers and F1 performance in Figure 10. For the NER datasets, we use two layers of Plusformer, and for the RE and EE we use three. G C OMPARISON WITH GLAD
Jiang et al.
(2020) claims that many NLP tasks can be regarded as the span prediction and prediction of relations between pairs of spans (named as span extraction and relational extraction in our paper), which is conceptually similar to our insights.
However, our work is fundamentally distinct from theirs on both formulation and model architecture.
Jiang et al.
(2020) classify various NLP tasks into two separate tasks and design different modules for them.
To contrast, we unify all traditional
IE tasks into a single formulation, namely token-pair classification.
Therefore, we only need one model for all tasks.
Besides, Jiang et al.
(2020) simply use the concatenation of the start and end
[Figure]
[Table]
token representations to represent a span, and for relations, they concatenate the head and tail span representations.
Therefore, in their work, the interaction between spans are weak.
In our work, we obtain the feature matrix of all token pairs and add well-designed Plusformer module on top of all token pairs, where token pairs can interact with others thoroughly.
In order to prove the superiority of our reformulation and UTC-IE model
[Table]
we make a fair comparison on several tasks from the GLAD benchmark (Jiang et al., 2020).
We choose 3 additional IE tasks, including Open Information Extraction (OIE), Semantic Role Labeling (SRL) and Aspect Based
Sentiment Analysis (ABSA), and NER and RE.
We use WLP (Hashimoto et al., 2017) on NER and
RE, OIE2016 (Stanovsky & Dagan, 2016) on OIE, OntoNotes (Pradhan et al., 2013) on SRL and
SemEval14 (Pontiki et al., 2014) on ABSA.
The detailed experimental settings are the same as those in GLAD, to ensure a fair comparison.
Results are present in Table 12.
The table shows that UTC-IE outperforms GLAD on all chosen tasks exceedingly, with +2.
improvement on average.
Moreover, we observe that UTC-IE without Plusformer also surpasses
GLAD benchmarks on all tasks with +0.84 improvement on average, which proves the superiority of our unified reformulation.
✘ ✘ ✘ 𝒆 ✘ ✘ ✘
Pittsburgh one of
US Airway ’ s hubs built …
Pittsburgh of
US
Airway hubs built ‘s one [Figure] 𝒆𝒓 𝒆
Pittsburgh one of
US Airway ’ s hubs built …
Pittsburgh of
US
Airway hubs built ‘s one [Figure]
[Figure]
[Figure]
[Figure]