CLEVE : C ontrastive P re-training for E vent E xtraction (2021)

Abstract: Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved modeling event characteristics, resulting in the developed EE models cannot take full advantage of large-scale unsupervised data. To this end, we propose CLEVE, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) obtained with automatic parsers. CLEVE contains a text encoder to learn event semantics and a graph encoder to learn event structures respectively. Specifically, the text encoder learns event semantic representations by self-supervised contrastive learning to represent the words of the same events closer than those unrelated words; the graph encoder learns event structure representations by graph contrastive pre-training on parsed event-related semantic structures. The two complementary representations then work together to improve both the conventional supervised EE and the unsupervised “liberal” EE, which requires jointly extracting events and discovering event schemata without any annotated data. Experiments on ACE 2005 and MAVEN datasets show that CLEVE achieves significant improvements, especially in the challenging unsupervised setting. The source code and pre-trained checkpoints can be obtained from https://github.com/THU-KEG/CLEVE .

## introduction event extraction (ee) is a long-standing crucial information extraction task, which aims at extracting event structures from unstructured text. as illustrated in figure 1 , it contains event detection task to identify event triggers (the word "attack") and classify event types (attack), as well as event argument extraction task to identify entities serving as event arguments ("today" and "netanya") and classify their argument roles (time-within and place) @xcite . by explicitly capturing the event structure in the text, ee can benefit various downstream tasks such as information retrieval @xcite and knowledge base population @xcite . existing ee methods mainly follow the supervised-learning paradigm to train advanced neural networks @xcite @xcite with humanannotated datasets and pre-defined event schemata. these methods work well in lots of public benchmarks such as ace 2005 @xcite and tac kbp @xcite ), yet they still suffer from data scarcity and limited generalizability. since annotating event data and defining event schemata are especially expensive and laborintensive, existing ee datasets typically only contain thousands of instances and cover limited event types. thus they are inadequate to train large neural models @xcite and develop methods that can generalize to continually-emerging new event types @xcite . inspired by the success of recent pre-trained language models (plms) for nlp tasks, some pio-neering work @xcite attempts to fine-tune general plms (e.g, bert @xcite ) for ee. benefiting from the strong general language understanding ability learnt from large-scale unsupervised data, these plm-based methods have achieved state-ofthe-art performance in various public benchmarks. although leveraging unsupervised data with pretraining has gradually become a consensus for ee and nlp community, there still lacks a pre-training method orienting event modeling to take full advantage of rich event knowledge lying in largescale unsupervised data. the key challenge here is to find reasonable self-supervised signals @xcite for the diverse semantics and complex structures of events. fortunately, previous work @xcite has suggested that sentence semantic structures, such as abstract meaning representation (amr) @xcite , contain broad and diverse semantic and structure information relating to events. as shown in figure 1 , the parsed amr structure covers not only the annotated event (attack) but also the event that is not defined in the ace 2005 schema (report). considering the fact that the amr structures of large-scale unsupervised data can be easily obtained with automatic parsers @xcite , we propose cleve, an event-oriented contrastive pre-training framework utilizing amr structures to build self-supervision signals. cleve consists of two components, including a text encoder to learn event semantics and a graph encoder to learn event structure information. specifically, to learn effective event semantic representations, we employ a plm as the text encoder and encourage the representations of the word pairs connected by the arg, time, location edges in amr structures to be closer in the semantic space than other unrelated words, since these pairs usually refer to the trigger-argument pairs of the same events (as shown in figure 1 ) @xcite . this is done by contrastive learning with the connected word pairs as positive samples and unrelated words as negative samples. moreover, considering event structures are also helpful in extracting events @xcite and generalizing to new event schemata @xcite , we need to learn transferable event structure representations. hence we further introduce a graph neural network (gnn) as the graph encoder to encode amr structures as structure representations. the graph encoder is contrastively pre-trained on the parsed amr structures of large unsupervised corpora with amr subgraph discrimination as the objective. by fine-tuning the two pre-trained models on downstream ee datasets and jointly using the two representations, cleve can benefit the conventional supervised ee suffering from data scarcity. meanwhile, the pre-trained representations can also directly help extract events and discover new event schemata without any known event schema or annotated instances, leading to better generalizability. this is a challenging unsupervised setting named "liberal event extraction" @xcite . experiments on the widely-used ace 2005 and the large maven datasets indicate that cleve can achieve significant improvements in both settings. event schema induction. supervised ee models cannot generalize to continually-emerging new event types and argument roles. to this end, chambers and jurafsky (2011) explore to induce event schemata from raw text by unsupervised clustering. following works introduce more features like coreference chains @xcite and entities @xcite . recently, @xcite move to the semisupervised setting allowing to use annotated data of known types. following @xcite , we evaluate the generalizability of cleve in the most challenging unsupervised "liberal" setting, which requires to induce event schemata and extract event instances only from raw text at the same time. ## methodology the overall cleve framework is illustrated in figure 2. as shown in the illustration, our contrastive pre-training framework cleve consists of two components: event semantic pre-training and event structure pre-training, of which details are introduced in section 3.2 and section 3.3, respectively. at the beginning of this section, we first introduce the required preprocessing in section 3.1, including the amr parsing and how we modify the parsed amr structures for our pre-training. ## experiment we evaluate our methods in both the supervised setting and unsupervised "liberal" setting of ee. ## conclusion and future work in this paper, we propose cleve, a contrastive pre-training framework for event extraction to utilize the rich event knowledge lying in large unsupervised data. experiments on two real-world datasets show that cleve can achieve significant improvements in both supervised and unsupervised "liberal" settings. in the future, we will (1) explore other kinds of semantic structures like the frame semantics and (2) attempt to overcome the noise in unsupervised data brought by the semantic parsers.