U ni E vent: Unified Generative Model with Multi-Dimensional Prefix for Zero-Shot Event-Relational Reasoning (2023)

Abstract: Reasoning about events and their relations attracts surging research efforts since it is regarded as an indispensable ability to fulfill various event-centric or common-sense reasoning tasks. However, these tasks often suffer from limited data availability due to the labor-intensive nature of their annotations. Consequently, recent studies have explored knowledge transfer approaches within a multi-task learning framework to address this challenge. Although such methods have achieved acceptable results, such brute-force solutions struggle to effectively transfer event-relational knowledge due to the vast array of inter-event relations (e.g. temporal, causal, conditional) and reasoning formulations (e.g. discriminative, abductive, ending prediction). To enhance knowledge transfer and enable zero-shot generalization among various combinations, in this work we propose a novel unified framework, called UNIEVENT. Inspired by prefix-based multitask learning, our approach organizes event relational reasoning tasks into a coordinate system with multiple axes, representing inter-event relations and reasoning formulations. We then train a unified text-to-text generative model that utilizes coordinate-assigning prefixes for each task. By leveraging our adapted prefixes, our unified model achieves state-of-the-art or competitive performance on both zero-shot and supervised reasoning tasks, as demonstrated in extensive experiments

## introduction an 'event' is defined as a semantic molecule to explain the states or actions of a person, entity, or thing @xcite . in natural language literature, it is usually represented as a span in narrative text (e.g., sentences, paragraphs or documents), which is composed of an event trigger (e.g., predicate) and its arguments (e.g., subject, object, adverbial modifier). based on the semantic unit at the event level, a broad spectrum of eventrelational reasoning tasks was presented to learn various inter-event relations (e.g., temporal, causal, conditional) and thus enable commonsense or cognitive reasoning capabilities for advanced ai systems. the inherent event-relational reasoning logic has been formulated as tasks as relation extraction @xcite , question answering @xcite , intend prediction @xcite , summarization @xcite and knowledge base construc-tion @xcite . attributed to recently advanced language models (e.g., bert and gpt-3) pre-trained on raw corpora with billions of words in a self-supervised manner, data-driven methods via a pre-training & fine-tuning paradigm achieves acceptable performance on the event-relational reasoning tasks @xcite @xcite . nonetheless, its inherently complex intra-event semantics and intricate inter-event relations inevitably increase the labor intensity of human annotation processes (e.g., experts-required, timeconsuming, label-inconsistent). this limits the scale of human-labeled data for fine-tuning and thus affects the effectiveness of the data-driven methods on those tasks @xcite . for example, considering the event temporal question answering task, there are only 198 training instances in cider @xcite among all datasets. therefore, such a data-scarcity issue necessitates knowledge transfer to an event-relational reasoning task. besides task-specific heuristic pseudo labeling in a self-or semi-supervised framework to transfer from large-scale in-domain raw corpus, recent event-centric research works resort to supervised knowledge transfer due to its general learning methodology and superior fine-tuning performance. that is, transferring knowledge among supervised datasets under a variety of inter-event relations (e.g., temporal, causal) @xcite and reasoning formulations (e.g., event relation extraction, question answering) @xcite @xcite . despite their superior transfer performance, as shown in figure 1 , these works do not well consider knowledge transfer among a variety of both targeted relations and reasoning formulations in event-relational reasoning, and they usually fail to generalize to unseen event-relational reasoning tasks with distinct relations and/or formulations. for example, according to our empirical study shown in nt column of to enhance knowledge transfer and empower zero-shot generalization among event-relational reasoning tasks, in this work we propose a brandnew unified framework unievent for zero-shot event-relational reasoning tasks transferring. we first categorize all event-relational reasoning tasks according to their original formulation types and event relation. we then construct generative formats for each task and convert them into generation forms. we train on adapted tasks based on a pretrained generation model @xcite . based on that, the proposed unified model enables implicit transfer across event-relational reasoning tasks. however, without explicitly discriminating the categorical coordination of the data, straightforward multi-task training may suffer from negative transfer @xcite and intensive diversity of formulations and relations. therefore, inspired by recent success of prompt tuning @xcite @xcite where prompt instruction show great benefit in multi-task training @xcite @xcite , we propose to add prefix @xcite adapting to diversified formulations and relations. this multi-dimensional prefix additionally facilitates further transfer across tasks. we introduce to generate of these prefixes via the adaptive prefix generators which allow sharing of flexible features among distinct dimensions. we then perform a contrastive regularization @xcite to learn to discriminate various task formulations and relations and enhance the representation. we conduct extensive experiments on 16 datasets (3 for multi-task training, 13 for testing). experiment results demonstrate that our method shows significant transferability and outperforms baselines in both zero-shot and full data multi-task settings we summarize our contributions as: unievent "curfew was imposed to bring the situation under control and prevent the clashes from spreading to nearby areas , sources said . is it causal related?" encoding relation-wise apg encoding formulation-wise apg decoding relation-wise apg decoding formulation-wise apg adaptive prefix generator ! "# @xmath0 % "# @xmath1 "with the general election due this september , … what is the relation between election and come. options: before after simultaneous including during" temp re "hosseini however did not give details of any retaliation… question: what could iran do if the security council sanctions against tehran?" cond qa ca nli "after [eos]" temp re "blocking united nations inspections. [eos]" cond qa "causal[eos]" ca nli bottleneck ! "& @xmath2 figure 2: overview of unievent. we propose to unify event-relational reasoning tasks with constructed generative formats. we use the adaptive prefix generators to generate formulation-wise and relation-wise prefixes in both the encoder and decoder sides. in total, there are four adaptive prefix generators of the same architecture. we illustrate the architecture on the right. ## related work unified training to fulfill knowledge transfer, sorts of brute-force solutions known as multitask learning trains parameter-sharing neural models @xcite @xcite @xcite . however, learning out-of-domain and -formulation data could diminish the model efficacy on the targeted tasks, not to mention domain/formulation varying significantly in event-relational reasoning. built upon a multi-task learning framework recent works are dedicated to integrating knowledge by unifying massive tasks @xcite @xcite @xcite . via unified task formulations (e.g., text-to-text generation) and advanced training strategies, these works excel single task finetuning in conventional multi-task learning. yang et al. (2022a zhou et al. (2020, 2021 ); han et al. (2021b) learn from unsupervised or distant supervision. yang et al. (2020) asks for counterfactual statements. du et al. (2022) aims to choose correct cause or effect from choices. poria et al. (2021); han et al. (2021a); yang et al. (2022c) question about diversified event relations. among all methods, we are the first to study the unification across these relations and formulations. ## conclusion in this work, we propose unievent to transfer knowledge for unseen event-relational reasoning tasks. we first categorize these tasks. then we construct generative formats and then unify them with generated multi-dimensional prefixes. unievent outperforms all baselines in both zero-shot and fulldata settings.