C lar ET : Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification (2022)

Abstract: Generating new events given context with correlated ones plays a crucial role in many event-centric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlation-aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of event-centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zero- and few-shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability.

## introduction an 'event', usually a text span composed of a predicate and its arguments @xcite , is a fine-grained semantic unit to describe the state of entities/things (e.g., he looks very worried) and how they act (e.g., i grab his arms). understanding events and modeling their correlations are fundamental to many reasoning tasks @xcite , e.g., abductive reasoning, story ending classification and generation, counterfactual reasoning, script reasoning. for instance, in the left example of figure 1 , to generate the missing event @xcite in the given context, it is essential to understand that there are four events ('it tries paragraph 𝑥: it tries the knob but it's locked, so the creature starts pounding on … an event mention 𝑒: figure 1: left: an example of abductive reasoning which aims to generate the missing event @xcite given correlated events (underlined) and connectives (w/ orange) in the context. right: a toy example (x, e, {ē} m @xmath0. see appendix a for real examples to pre-train our model. the knob', @xcite , 'the creature starts pounding on the door', and '(the creature) to break it down'), and then predict @xcite based on the other three events and its correlations to them (i.e., the contrast relation indicated by 'but' and the causal relation by 'so'). event-aware reasoning has gained much attention and achieved promising success in recent years @xcite . however, many algorithms are designed to solve only some specific tasks. for example, @xcite propose to improve unsupervised decoding for counterfactual and abductive reasoning; @xcite and @xcite advance story ending generation via incremental encoding and multi-level graph convolutional networks. although these works show effectiveness in corresponding applications, they are limited to specific scenarios, and cannot generalize well to a broad scope of reasoning. meanwhile, some pioneering works follow a recently arising paradigm to conduct event-based pretraining for those downstream reasoning tasks @xcite @xcite . however, these solutions have their own limitations: comet @xcite learns event correlations from a human-curated knowledge graph and thus limits its scalability. @xcite and @xcite only model temporal relations and cannot be expanded to other relations (e.g., causal, contrast). eventbert @xcite is proposed for event-based classifi-cations and is thus inapplicable to generation tasks. in this work, we propose a general pre-training framework for event-centric reasoning by learning a correlation-aware context-to-event transformer (claret) from an event-rich text corpus. we propose three novel self-supervised objectives, dubbed as whole event recovering (wer), contrastive event-correlation encoding and prompt-based event locating, respectively. the first one aims to capture event correlation by recovering a whole event from its masked context. the second one enhances the representation of the masked event in wer by contrasting it with the gold event against the negative ones. the last one is a simplified wer task by providing hints in its prompt and thus facilitates effective learning for wer. claret explicitly models event correlations and contributes to various scenarios. from one aspect, it covers a variety of correlation types (e.g., causal, temporal, contrast) attributed to correlation typeagnostic objectives. from another aspect, it is applicable to both generation and classification task formulations by its unified structure. lastly, it highlights event-level correlations and thus is more effective for diverse event-centric tasks, e.g., abductive, counterfactual and ending reasoning. to evaluate claret, we compare it with strong baselines on 9 diverse benchmarks. while claret is continually pre-trained from bart @xcite with very limited extra resources, i.e., training on a small subset of bart-used corpus (i.e., 200m out of 2.2t tokens) within 90 gpu hours (only 0.13% of 70,000h bart pre-training), it achieves state-of-the-art (sota) performance on all 5 generation benchmarks. it also outperforms all unified models on 4 classification benchmarks and achieves competitive, or even better, accuracy to strong discriminative baselines. we further exhibit that the claret provides a good initialization for downstream tasks by zero-and few-shot learning. ## experiments this section begins with descriptions of downstream datasets and experimental setups. ## conclusion we present a novel correlation-aware context-toevent transformer to self-supervisedly learn eventcorrelation knowledge from text corpus and benefit various event-centric reasoning scenarios. besides sota fine-tuning results on 5 generation and 4 classification tasks, we conduct zero-/few-shot learning and extensive ablation studies to exhibit our model's effectiveness. lastly, we find our model is competitive to a twice larger general-purpose model, reduces learning difficulty for event generation, and retains nlu ability from its basic model. although this work learns context-to-event knowledge, our self-supervised objectives are applicable to other semantically-meaningful text units besides events. for example, text units can be entities and concepts to learn relational and commonsense knowledge, which can benefit more downstream tasks.