Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model
Abstract
Universally modeling all typical information extraction tasks (UIE) with one generative language model (GLM) has revealed great potential by the latest study, where various IE predictions are unified into a linearized hierarchical expression under a GLM.
Syntactic structure information, a type of effective feature which has been extensively utilized in IE community, should also be beneficial to UIE.
In this work, we propose a novel structure-aware GLM, fully unleashing the power of syntactic knowledge for UIE.
A heterogeneous structure inductor is explored to unsupervisedly induce rich heterogeneous structural representations by posttraining an existing GLM.
In particular, a structural broadcaster is devised to compact various latent trees into explicit high-order forests, helping to guide a better generation during decoding.
We finally introduce a task-oriented structure fine-tuning mechanism, further adjusting the learned structures to most coincide with the end-task’s need.
Over 12 IE benchmarks across 7 tasks our system shows significant improvements over the baseline UIE system.
Further in-depth analyses show that our GLM learns rich task-adaptive structural bias that greatly resolves the UIE crux, the long-range dependence issue and boundary identifying .
Introduction
Information extraction (IE) is widely considered as one of the most kernel topics in natural language processing (NLP), which is defined as to identify the desired structural information from the unstructured texts [4, 65, 49, 46, 40, 30, 15].
There is a variety of IE and IE-derived tasks, yet all of which revolves around predicting two key elements: mention spans or/and their semantic relations .
For example as in Fig. 1(b), NER detects the mention spans, while RE recognizes each possible mention and its associated mention with relation.
In this regard, all the existing IE jobs can be reduced into three prototypes: span extraction, pair extraction and hyper-pair extraction, as depicted in Fig. 1(a).
In the era of deep learning, IE witnesses extraordinary developments, where especially the recent triumph of pre-trained language models (LMs) helps push the state-of-the-art (SoTA) IE performances amazingly [10, 3, 26, 82, 75].
Prior related works mostly design particular models for certain IE tasks in isolation; while the latest SoTA progress [43] is achieved by unifying all IE tasks with a single encoder-decoder GLM, i.e., UIE.
As different IE tasks essentially share the similar nature (i.e., modeling span and relation features), it is proven that universally modeling multiple IE tasks helps further learning of general sharable knowledge from varying task sources, which makes UIE great potentials in real-world scenarios.
In this work we inherit this wisdom and also focus on UIE.
ARG
ARG ARG
ORG
PER … of SpaceX, Elon Musk … at California.
The founder of SpaceX, Elon Musk, … … Elon Musk, visited Tim Cook last week at California.
ORG
PER
LOC
Founder
ARG VISIT
Visitor
Time
Place Object
NER :
Relation Extraction :
Event Extraction :
LM Encoder
LM Decoder
Input text Label prompts +
Spans & Attributes
Relations & Types 3) Hyper-pairExtraction 3) Hyper-pairExtraction
The founder of SpaceX , Elon Musk , visited Tim Cook last week at California .
2) PairExtraction 2) PairExtraction 1) SpanExtraction
[Figure]
) SpanExtraction
[Figure]
[Figure]
[Figure]
Extraction 2) PairExtraction 2) PairExtraction 1) SpanExtraction 1) SpanExtraction
The founder of SpaceX , Elon Musk , visited Tim Cook [Figure] week at California .
[Figure]
Assisting [Figure] with
[Figure]
syntax structure [Figure] constituency syntax structure [ ] [ ] [ ]
Linearized hierarchical expression
Raw text
On the other hand, previous IE research extensively employs the external syntactic structure information, such as the dependency tree, for task improvements [5, 51, 47, 25, 54, 18].
Behind the enhancements is that IE structure corresponds much with the syntax structure explicitly, where the latter can essentially offer low-level linguistic bias for better learning the high-level semantic structure.
As exemplified in Fig. 1(d), the dependency tree coincides much with structure of EE task as in
Fig. 1(b).
Importantly, some findings reveal that the LMs, being pre-trained on large corpus, capture structural syntax knowledge [70, 20, 24], which gives rise to LMs’ distinguishing promotion on IE.
Yet probing tasks show that the auto-learned structure representations is weak, which inevitably limits the LM efficacy for IE [8, 22, 31, 67].
Correspondingly, a line of researches fuse external syntax trees into LMs to reinforce the structure awareness, i.e., structure-aware LMs [73, 2, 39, 6].
Motivations.
After carefully revisiting the existing literatures, we summarize four key limitations of syntactic structure-aware LMs that hamper IE from further improvements.
First , existing structureaware LMs are mostly designed for one certain IE task (e.g., NER [71], RE [37]) instead of UIE, leaving the shared IE knowledge and the task-invariant syntax features unexploited.
Second , current structure-aware LMs merely consider making use of one standalone type of syntax structures, i.e., mostly using the dependency trees [51, 25].
We however argue that as one core grammar, constituency syntax can serve complementary contributions for IEs.
There are two common challenges of IEs: longrange dependence problem and boundary identifying , in which the dependency structure especially helps solve the former one [51, 47, 54] and the constituency syntax could mostly benefits the latter [81, 48, 56], as in Fig.
1(d)&(e).
Thus it is best to simultaneously model both two heterogeneous structures [32, 17].
Third , existing works mostly integrate supervised syntax parse trees, where unfortunately, either the amount of manually annotated syntactic data (e.g., PTB) are largely limited, or the annotation noises from third-party parsers are inevitably introduced due to such explicit injection.
Fourth , parsing syntax comes with task-irrelevant or indirect substructures (e.g., in Fig. 1(d) the black and dotted lines respectively), which would deteriorate the efficacy.
Meanwhile, different IE tasks largely demand distinct bias of structural features, while current structure-aware LMs fail to fine-tune the structure knowledge to allow the structure bias best accord with end task’s need.
Contributions.
On the above basis, we propose learning a latent adaptive structure-aware generative language model for UIE (namely LasUIE).
First of all, we reduce UIE into three uniform prototypes, upon which we transform the UIE into generative paradigm with an encoder-decoder GLM, predicting the linearized hierarchical expression (i.e., spans&attributes, relations&types, as shown in Fig. 1(c)).
Then, we adopt a three-stage of LM training procedure, where an additional structure-aware posttraining is added between the pre-training and fine-tuning stages for structure learning.
Inspired by the progress of unsupervised grammar induction [60, 61, 44, 29, 62], we design a heterogeneous structure inductor (HSI) module, where two heterogeneous syntactic structures are simultaneously measured and automatically learned.
With HSI, our GLM initialized with existing pre-trained parameters, during post-training, performs unsupervised syntax induction based on unlabeled texts without relying on external syntax parses or any annotation labor (cf.
Fig. 2). Since the induced latent structural representations may be squeezed aside by the mainstay contextual representations in LM encoder, we further enhance the utility of syntax by introducing a structural broadcaster (SB) module (cf. Fig.2).
SB compacts multiple varying latent trees from different encoding attention heads into an explicit constituency-like and a dependency-like forest respectively.
During each decoding step, two heterogeneous syntactic forests are utilized to produce high-order features at global level for guiding better content generation.
Finally, during the prompt-based fine-tuning stage we perform task-oriented structure adaptive tuning to narrow the gaps between the induced syntactic and task-specific structures (cf. Fig. 3).
With policy gradient we dynamically adjust the attributes of two heterogeneous structures according to the feedback of end task performance.
Extensive experiments are performed on 12 representative data across 7 IE tasks.
On both the supervised and low-resource settings our framework consistently shows improvements over the baseline systems.
Via further analyses we verify that 1) unifying IE tasks by further modeling structure information in LM benefits IE substantially, especially in the low-resource scenario. 2)
Integrating two heterogeneous structures brings mutual advantages for UIE, helping fully resolve the boundary identifying and long-range dependence issue.
3) Automatically inducing latent structures in LM with further task-oriented structural adaptation learning significantly consolidates the efficacy of structure knowledge for end tasks.
4) Different types of IE tasks rely subtly on varying structural bias, all of which can be flexibly learned and correctly satisfied by our system.
Our resources can be found at https://github.com/ChocoWu/LasUIE .
Related Work
IE is a long-standing research topic in NLP, which includes various tasks as well as growing derivations [4, 65, 49, 46, 36, 80].
We reveal that essentially all the IE tasks can be summarized into three main prototypes, according to the combination numbers of ‘mention span’ and ‘semantic relation’ prediction targets: 1) span extraction , e.g., named entity recognition (NER) [9], aspectbased sentiment analysis (ABSA) [66], aspect-term extraction (ATE) [38]; 2) pair extraction , e.g., relation extraction (RE) [83, 35], aspect-opinion pair extraction (AOP) [87], aspect-based sentiment triplet extraction (ASTE) [52]; and 3) hyper-pair extraction , e.g., event extraction (EE) [21], semantic role labeling (SRL) [19], opinion role labeling (ORL) [28, 63].
Mostly prior IE researches all solve one particular task exclusively (or one specific IE type) [51, 74, 41, 79, 88], while they may unfortunately ignore certain task-invariant universal IE features.
In this work, we consider the line of UIE, unifying all IE tasks to exploit the shared IE knowledge.
And based on the above UIE prototypes, we develop a LM-based unified framework with generative paradigm.
Many efforts are paid for building LMs to handle IE tasks by taking advantages of the knowledge from large-scale pre-training [10, 3, 26, 82, 75].
Another line of IE researches propose injecting external knowledge into LMs or GLMs, such as knowledge graph (KG) [42, 27, 85, 16], syntax structure information [73, 2, 39, 6].
Comparing to the integration of domain-specific KG information for certain IE tasks, syntactic information would provide much broader generic features in the scope of UIE.
The very latest research attention of LMs has been focused on the GLMs, the encoder-decoder paradigm LMs.
GLMs transform various NLP tasks into a unified seq-to-seq scheme with some properly-designed prompt texts as additional inputs [33, 57, 82].
Very recently, Lu et al.
[43] pioneer the UIE by casting the IE structure prediction into text generation with a GLM, with which our UIE modeling shares the same spirit.
We however note that our work can advance in two major aspects.
First, we consider the integration of additional structural knowledge in GLMs for UIE enhancements.
Besides, [43] require supervisedly pre-training their UIE GLM on a large-scale annotated IE corpus, while our system automatically induces structure knowledge based merely on unlabeled texts without any further annotation and labor. This work also closely relates to the line of structure-aware LMs.
On the one hand, some researches propose directly introducing external syntax trees into LMs to reinforce the structure awareness.
They mostly take the Transformer-based LMs as backbone, and fuse the syntax signals (annotations) from external parsers or PTB corpus by modifying the Transformer attentions [73, 39, 6].
Another line of structure-aware LMs directly induce syntax structure into LMs automatically, a.k.a., unsupervised grammar induction [7, 77, 60, 12, 61, 29].
We in this work borrow the success of unsupervised LM Encoder LM Decoder … [Figure]
st
[Figure]
Broadcaster
Constituency
[Figure]
Dependency
[Figure]
Dependency forest
Constituency forest
Trm
Trm
Trm 2 nd … Trm
Trm
Trm
Constituency structure induction
Dependency structure induction
Rule
Rule
LM Encoder … … …
Heterogeneous Structure Inductor
Structural Broadcaster
Cross-attention
High-order structure feature
My dog barks at the boy in red 1 46 1 468 =
[Figure]
[Figure]
constituency
[Figure]
grammar induction, inducing rich structure information for LMs for better UIE.
Inspired by the foundation of syntax distance measurements [60, 44, 13, 62], we propose to induce linguistic structures and compose both the constituency and dependency syntax structures simultaneously.
Unifying Information Extraction with Text Generative Paradigm
As aforementioned, we reduce all IE jobs into the predictions of few structural elements: 1) spans and 2) relations , and without losing the generality of UIE, we also consider the predictions of 3) span attributes and 4) relation types .
Using different combinations of the structural elements properly can construct a hierarchical IE structure.
In other words, we can arrange these elements into a sequential textual expression, from which the actual IE target can be easily restored.
Based on this we unifiedly model all IE tasks (i.e., UIE) by transforming the structure prediction into text generation.
Fig. 1(c) illustrates the main idea.
Such generative scheme also enables to take the advantage of the recent achievement of GLMs, such as BART [33] and T5 [57].
Taking an existing GLM as backbone, we reach the goal of end-to-end UIE as well as complex IE, such as overlapped and discontinuous cases [84, 14, 34].
Our LM encoder takes input text (i.e., x ), and the decoder produces the linearized hierarchical expression (LHE), i.e., y .
Fig. 3 illustrates the input and output implications.
Input.
The input text x = { w 1 , · · · , w n } includes the raw sentence and task-specific label prompts.
The label prompts contain the pre-defined task-specific labels, including span attributes (‘ Attr ’) and relational types (‘ Type ’), where each label is separated by a ‘ <SPAN> ’ or a ‘ <REL> ’ marker.
Parts of the ‘ Attr ’ and ‘ Type ’ labels will be copied and output in y .
We also insert a task identifier token ‘ <TASK> ’ to inform the model which task to predict.
Output.
The output text y is a linearized hierarchical expression that describes how the structural elements organize into the target structure, as depicted in Fig.
For example, in span extraction, y should be a list of text spans and attribute labels, i.e., ‘ { ( Span, Attr ) , · · · } ’.
In pair extraction y is a list of pairs, where a pair is represented as ‘ ( Span i , Attr i
[ Type k ]( Span j , Attr j )) ’ in which Span j is a subordinate mention of Span i with a semantic relation Type k .
For hyper-pair extraction y is a list of hyper-pairs represented as ‘ ( Span i , Attr i
[ Type k ]( Span j , Attr j ) [ Type m ]( Span k , Attr k ) · · · ) ’.
It is also noteworthy that our LHE takes a similar scheme with [43] , but with difference.
For example, in our scheme all the mention comes with an associated attribute label in any IE prototype; while in [43] the subordinate mentions have no attribute labels.
Thus, our design could be more generalized.
Learning Latent Adaptive Structure-aware Generative Language Model 4.1 Overall Framework
The overall framework is built upon a Transformer-based encoder-decoder GLM, based on which we additionally add 1) a heterogeneous structure inductor module at top of the encoder for structural learning, 2) a structural broadcaster module between GLM encoder and decoder for enhancing the structural feature utility.
Fig. 2 shows the overall architecture of our LasUIE GLM.
LasUIE takes a three-stage training process, where a structure-aware post-training is inserted between the pre-training and fine-tuning stages for structure learning.
LasUIE takes an existing well pretrained GLM parameters (e.g., BART, T5) as initiation.
During structure-aware post-training stage our GLM carries out unsupervised syntax induction based on unlabeled plain texts (cf. § 4.2).
Thereafter, LasUIE is fine-tuned on the in-house training data, along with which we perform task-oriented structure adaptive tuning (cf.
We also note that LasUIE takes a consistent paradigm of text-totext generation throughout the whole three stages, which ensures a minimum information loss from the early trainings to the final predicting.
4.2 Unsupervised Structure-aware Post-training
Heterogeneous structure inductor.
As cast earlier, although LMs are able to learn certain linguistic knowledge from generic pre-training, the signal strength of learned syntax is quite weak to contribute
IE enough [67, 31].
In the structure-aware post-training stage, we aim to unsupervisedly enrich
our
GLM with sufficient structural knowledge, reinforcing the awareness of linguistic syntax.
Inspired by Shen et al.
(2021) [62], we explore a heterogeneous structure inductor (HSI) stacked on top of GLM encoder to reach the above goal.
HSI induces linguistic structures based on the foundation of syntax distance measurements [60, 44].
We employ two heterogeneous syntax measurements, i.e., O C =
{ o c 1 , · · · , o cn − 1 } ( o c< 1 =
o c>n − 1 = ∞ ) for measuring constituency syntax, and O D
=
{ o d 1 , · · · , o dn } for measuring dependency syntax.
As illustrated in Fig. 2(b), o ci is a real value depicting the height of the lowest common ancestor between two consecutive words w i and w i +1 ; while o di is a real value describing the spanning distance between the words linking to w i .
Intuitively, bigger o ci means bigger information divergence of the split point between the two sides of phrasal span, and larger o di implies wider range of connections, i.e., longer-term dependent relations.
As revealed that the syntax features are best learned at lower layer of GLM encoder [24], HSI thus takes the first-layer encoding representations h 1 i as input and produce syntax context representations via convolution operation: h ∗ i =
Conv( h 1 i ).
Based on h ∗ i , HSI represents o ci and o di as:
[Equation]
Then, two rules are made for generating two heterogeneous syntax based on the two measurements [62], which also helps coordinate two types of structures so that they can co-exist together and legally.
▶ Rule Γ C :
A smallest constituent span C [ l,r ] of w i ( l<i<r ) should satisfy ( o cl − 1 >o di ) & ( o cr >o di ).
For example as in Fig. 2(b), o c 2 (=4) >
o d 3 (=3.5) and o c 8 (= ∞ ) >o d 3 , thus C [3 , 8] is the valid minimum span for w 3 .
▶ Rule Γ D : Generalizing w i as a potential span C [ l = i,r = i ] , the dependent head of any word in
C [ l,r ] is w j ← argmax k ∈ [ l,r ] ( o dk ) .
For example, the maximum o d in constituent span of C [3 , 8] is o d 6 =4.5, thus dependent head of the word in C [3 , 8] is w 6 .
Based on rule Γ C we first generate all possible phrasal spans and organize them into a constituency tree T C , then constructing the dependency tree T D according to rule Γ D .
We parameterize the above structure construction process so as to make it all differentiable, i.e., by describing into the probabilistic perspective.
We represent the span C [ l,r ] distribution as:
[Equation]
where c k is a short hand for C [ l,r ] , σ is a sigmoid function.
We then depict the rule Γ D , and represent the word-word dependent distribution:
[Equation]
where we use the top-layer encoder representation h Li , by which we encourage the final encoding representations to learn from the low-layer syntax-rich representations.
We note that the above structure induction is carried out in each of multi-head attention blocks (total M ) in Transformer.
This means that multiple distinct syntax trees of each type (i.e., T C and T D ) will be induced.
Structural broadcaster.
It is a high chance that in GLM encoder the mainstay contextual representations will weaken the structural features and thus hurt the structure utility at decoder.
To combat this, we propose a SB module, by which we explicitly collect varying trees of a type and compacted them into a forest, respectively, i.e., constituency forest F C and dependency forest F D .
According to prior studies [50, 45, 64], comparing to the optimal 1-best syntax tree, a compact forest advances in higher structure recall, which allows to learn a better bias for task.
In SB, the structural priors from the syntax forests are explicitly broadcast into each decoding step for guiding the generation process.
Technically, SB selects and ranks candidate tree substructures based on the probabilistic confidence (Eq.2&3), which are then compacted into a forest based on the K-best maximum spanning tree (MST) algorithm [1, 90].
We then model the two types of forests with a graph attention model (GAT) [69] respectively, during which the decoding representation e at each step is attended to spot the high-order structural feature u at global level:
[Equation]
Further via a cross-attention operation (cf. Fig. 3) we navigate the encoder representation h L and the structural feature u into the updated encoder representation e ∗ :
[Equation]
Post-training objectives.
The first objective is performing seq-to-seq style language modeling, i.e., ‘corrupting + reconstructing’ the inputs [33], which is identical to the pre-training objectives.
We denote the language modeling loss as L W .
Along with the language modeling we then promote the unsupervised structure induction, including the one for dependency syntax and the one for constituency syntax:
[Equation]
where ϕ ( c k ) =
( h c ) T · h L [ l,r ] ||
h c ||·||
h
L [ l,r ] || is a span similarity score between the constituent phrase c k and the counterpart text span derived from the top-layer encoder.
Z is for normalization.
We also perform structure diversifying regularization (SDR), putting constraints on the varying trees induced from different multi-head encoder attentions so as to ensure structure diversification.
[Equation]
where A m or A k is an attention map.
We put all the above objectives together as the post-training target: L
[Equation]
4.3 Task-oriented Structure Fine-tuning
[Figure]
After structure-aware post-training, our GLM is finally fine-tuned on a specific terminal IE task to learn the on-demand features.
This target is an empirical risk minimization with cross-entropy loss:
[Equation]
where D is the mini-batch size.
Meanwhile, we perform further task- oriented structural fine-tuning, adapting the learned structure informa- tion to the task-specific IE structures, for example, in dependency struc- ture pruning those trivial word-word connections and adjusting the range of dependent paths; in constituency structure refining the phrase widths and granularities.
Our main idea is to amend the syntax attribute (i.e., dependency links and constituent compositions) by directly taking the feedback of end task performance.
Therefore, we employ the stochastic policy gradient algorithm [78].
As shown in Fig. 3, the actions a c/di ∈ ( − 1 , 1) are real values sampled with probabilities from a Gaussian distribution, by which we maintain a continuous control over syntax measurements, i.e., O D and O C .
[Equation]
where the s c/di =( h 1 i ⊕ h ∗ i ⊕ h Li ) are the state representations as the inputs of the policy agents.
The policy agents π C ( θ c ) and π D ( θ d ) are two parameterized two-layer feedforward networks, respectively.
We design the reward of the policy as the probability of correct task prediction, such that the structure adjustments are directly supervised by terminal task’s signals: R C/D = log p ( y | x ) .
The learning target of each policy is to maximize the corresponding expected reward:
[Equation]
We summarize all the fine-tuning targets: L FT =
L Task + L FS , where L FS =
L FD + L FC .
Experiments 5.1 Setups
We take the pre-trained T5 Base 2 as our default backbone GLM.
It is a 12-layer of encoder and decoder structure respectively.
HSI module has convolutional kernel sizes of [3,4,5] with output hidden representation size of 768-d.
The high-order feature dimension of SB module is 768.
Also we follow the practice of [43], instead of directly taking the raw label abbreviations as label prompts, we use the full names of labels in natural languages, such that we can fully utilize their semantic representations in GLM.
We use the plain texts from Wikipedia 3 and BooksCorpus 4 corpora for the unsupervised post-training.
To cover all three UIE prototypes, we consider 7 representative IE tasks with corresponding data: 1) NER: CoNLL
[68], OntoNote [55], ACE04 [11], ACE05 [23]; 2) RE: CoNLL04 [59], NYT [58], ACE05 [23]; 3) AOP: Res [53]; 4) ASTE: Res14 [53]; 5) ORL: MPQA [76]; 6) SRL: CoNLL
[55]; 7) EE: ACE05 [23].
Each dataset has its own split, and we follow the same practice of the relevant prior works when using it.
We verify the IE performances under the traditional separate scheme and the recent unified scheme, respectively.
1) In separate IE , we compare with the current SoTA systems (all using Large version LM/GLM) of each specific data; meanwhile we implement a T5 (Base version) system, namely GEN-T5, using the same generative manner (based on prompt input, generating LHE as ours) for running each task individually.
We also retrofit the
GEN-T5 system by injecting into the external syntax parse trees via additional training on the syntax annotated corpus, including the dependency syntax ( +DepSyn ), constituency syntax ( +ConSyn ) and both two types syntax ( +Dep&ConSyn ), respectively.
In unified IE , we mainly make comparisons with the current UIE system [43].
Note that the default UIE model (marked as UIE ∗† ) in raw paper uses T5 Large and meanwhile takes additional supervised pre-training on the large-scale IE corpus (the version without supervised IE pre-training marked as UIE ∗ ).
To ensure a fair comparison, we re-implement their system with T5-Base parameters and without supervised IE pre-training, marked as UIE.
Same as to GEN-T5, we also retrofit the UIE model by integrating heterogeneous syntax parse trees in different combinations.
Following each of previous works, we use the F1 evaluation metrics.
For each task, we consider the end-to-end prediction.
For example, for the span extraction (NER), we measure if both the mention span and the mention attribute are correct.
For the pair(/hyper-pair) extraction, we measure if the span boundary & span attribute & relation & type are all correct simultaneously.
For all experiments, we report the average scores with unbiased standard deviations on five runs with different random seeds.
5.2 Main Results
We present the overall comparison results on various IE tasks in Table 1 and Table 2 under the fully-supervised and low-resource scenario, respectively.
As can be seen, our proposed LasUIE framework consistently outperforms [Table] [Table] M
[Table]
[Table]
M
[Table]
[Table]
[Table]
[Table]
[Table]
M +DepSyn 91.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M +ConSyn 92.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M +Dep&ConSyn 92.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M
[Table]
Ours) 93.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M w/o SB 92.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M w/
[Table]
SDR 93.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
M w/
[Table]
FS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
the baseline UIE and other SoTA models on all tasks in both two learning scenarios, under both the Large or Base T5 initiations.
This demonstrates the efficacy of our proposal.
Also we compare the counterparts between M2-M5 and M9-M12, where the only difference between these generative methods lies in the seprate or unified modeling of IE.
From the results we learn that the unified modeling of IE (i.e., UIE) is more effective than the traditional separate modeling of specific IE task.
This verifies that the universal modeling helps share the task-invariant IE features, coinciding with the findings in [43].
5.3 In-depth Analysis
To aid better understanding the strengths of our method, we further present in-depth analyses from varying angles, i.e., by asking four key questions concentrating on the structure-aware GLM for UIE.
Q1: Can fusing syntax structure knowledge into GLM contribute to UIE?
Let’s compare the results in Table 1: M2 vs. M3&M4&M5 in separate IE setup, and M9 vs. M10&M11&M12 in unified IE setup, where either in separate or unified IE setup, integrating additional linguistic syntax features into GLM evidently improves all end task performances.
Interestingly, different tasks can receive varying degree of improvements from the syntax features.
Importantly, we see in Table 2 that with the aids of structure knowledge, the performances of low-resource transfer can be promoted, especially in the combination with the unified modeling of IE tasks.
This proves that the syntactic structures in GLM can serve as IE task-invariant features, further contributing to UIE.
Q2: What are the differences to integrate the constituency and dependency syntactic structure?
We now observe the results of different tasks in both Table 1&2, and we can find that on the span extraction type IE (i.e., NER) the improvements from constituency syntax prevail, while the dependency type of structure features dominate the pair-wise tasks, i.e., (hyper-)pair extraction.
We further analyze the error rate on the predictions of two kernel elements of IE, i.e., boundary recognition and relation detection, respectively on various tasks.
We see from Fig. 4 that the constituency structure more tends to offer key clues for the boundary recognition; while the dependent trees are more apt to cope with the relation detection, solving long-range dependence issue.
This shows that two heterogeneous structures have complementary advantages to UIE.
Thus, when combining both of them together, all the end tasks receive the enhancements to the greatest extent [Table] [Figure]
Q3:
For UIE, is it more advanced for GLM to automatically learn latent structures than injecting external syntax parse trees?
First, the comparisons in the main results directly prove the advance of using latent structural features for UIE.
For example, under the fair comparison, our LasUIE beats the UIE +Dep&ConSyn model with average 1.6%(=76.4-74.8) F1 improvement.
Even comparing with the UIE ∗† that takes additional pre-training on large-scale supervised IE corpus, LasUIE keeps its superiority in almost all cases.
This evidently verifies that it is necessary for LMs to automatically learn latent structure information for better UIE.
The underlying reason of our model’s improvements could be that the dynamically learned richer structural knowledge in LasUIE largely avoids the noises that are introduced in external syntax parse annotations.
Besides, as shown in Fig.
4, LasUIE reduces the errors on predicting the mention boundaries and relational pairings more significantly than the baseline counterparts.
Further we step into our LasUIE system itself, and inspect the ablation models, M14-M15, as shown in Table 1.
We see that the proposed structural broadcaster module plays important role to the overall system, i.e., without SB, LasUIE is downgraded to the level of UIE +Dep&ConSyn .
Also the structure diversification regularization mechanism serves positive effect.
Q4:
Is it necessary to further fine-tune the structures in GLM for UIE?
According to the results of the ablation model, M16, in Table 1, we can directly claim the answer is positive.
Without performing structural fine-tuning, the results by LasUIE hurt clearly, with averaged 1.4%(=76.4-75.0) F1 drop.
We next dig into the structural fine-tuning mechanism, analyzing how the auto-induced structural features influence the UIE performances. [Figure]
[Figure] We first study the changing trajectories of the 1) the structure agreement rates Ω D/C and 2) the structure densities ∆ D/C of the structural forests F D/C , during fine-tuning.
Ω C or Ω D is defined as the percentage that gold spans correspond to the phrasal spans in the constituency forest F C , or the gold relational pairs coincide with the word-word edges in the dependency forest F D .
As plotted in Fig.
5, along with the fine-tuning process the task performance climbs gradually.
Meanwhile, both the agreement rate Ω D/C increases, which means that the structural fine-tuning indeed can effectively adjust the learned structural information towards task-specific.
Also, the structural densities of two forests change from dense to sparse, which depicts a structure pruning process in our system.
Finally, in Fig. 6 we present the distribution of the range of word-word dependency links in F D and distribution of the constituency phrasal span width in F C .
We can discover that different end tasks rely on subtly varying structural features or attributes.
For example, hyper-pair extraction tasks require longer-range dependency features for relation determination, comparing to the IE tasks of other prototypes.
In turn, this certifies that our system can correctly learn the peculiar structural bias for a specific IE task , thanks to the task-oriented structure fine-tuning mechanism.
Conclusion and Discussion
This work investigates a novel structure-aware generative language model (GLM) that learns rich heterogeneous syntactic structure representations for better unified information extraction (UIE).
First, a well pre-trained GLM is taken as backbone to reach the goal of UIE, feeding with label prompt-based texts and predicting linearized hierarchical expressions that describe the actual IE target.
During post-training, the proposed heterogeneous structure inductor automatically generates rich structure information without relying on any additional syntax annotation.
A structural broadcaster then compacts various trees into forests for enhancing the structural feature utility and guiding better context generation.
The learned structural knowledge is further fine-tuned on the in-house training data so as to adapt into the task-specific need.
Extensive experiments and in-depth analyses demonstrate the efficacy of our system on improving the UIE.
Potential impact and limitations of the work. Our proposed structure-aware GLM learns rich syntactic knowledge relies only on the plain texts with easy access, without any cost of syntax annotation, or large-scale human-labor IE annotations for pre-training. Our UIE system effectively enhances the development of IE community, i.e., training one single unified model for effectively solving various IE tasks, which especially addresses the issue of IE data annotation scarcity in the real-life applications. One biggest potential risk is that the GPU-based training of our language model will cost energy consumption and CO 2 emissions.
This research is supported by the Sea-NExT Joint Lab. We would also like to thank the anonymous reviewers for their valuable feedbacks. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]