Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation (2022)

Abstract: Human reading comprehension often requires reasoning of event semantic relations in narratives, represented by Event-centric Question-Answering (QA). To address event-centric QA, we propose a novel QA model with contrastive learning and invertible event transformation, call TranCLR. Our proposed model utilizes an invertible transformation matrix to project semantic vectors of events into a common event embedding space, trained with contrastive learning, and thus naturally inject event semantic knowledge into mainstream QA pipelines. The transformation matrix is fine-tuned with the annotated event relation types between events that occurred in questions and those in answers, using event-aware question vectors. Experimental results on the Event Semantic Relation Reasoning (ESTER) dataset show significant improvements in both generative and extractive settings compared to the existing strong baselines, achieving over 8.4% gain in the token-level F1 score and 3.0% gain in Exact Match (EM) score under the multi-answer setting. Qualitative analysis reveals the high quality of the generated answers by TranCLR, demonstrating the feasibility of injecting event knowledge into QA model learning. Our code and models can be found at https://github.com/LuJunru/TranCLR.

## introduction since 2019, many larger-scale pre-trained language models (plms) @xcite @xcite have been introduced to address the question-answering (qa) tasks, reaching performance on par with humans on entity-centric qa datasets such as squad @xcite , trivi-aqa @xcite , and newsqa @xcite , in which answers are often entities extracted from text. a raising challenge is to research and develop new plm-based frameworks tackling figure 1 : an event-centric qa example from the es-ter dataset @xcite . all event triggers are highlighted in bold and underlined in the paragraph. the question event trigger and answer event triggers are further highlighted in red colors with different shades. in-text answer is smeared with blue. intuitively, it is possible to inject event information through a multi-task learning framework where event-related tasks such as event relation type detection and event embedding learning could be potentially useful to guide the qa model to generate better answers. for example, event relation type detection aims to detect the desired event semantic relation given a question, while event embedding learning aims to push events holding the desired semantic relation closer in the new event embedding space. however, in a qa model, the answer generator is usually built on a plm in which the original event representations learned in the plm should be preserved. that is, we want to map event representations onto a new event embedding space in order to inherently capture their semantic relations specified by an input question, but at the same time, we need to keep the original event representations learned by the plm in order to generate coherent answers. to deal with this dilemma, we propose an invertible transformation operator, which makes it possible to learn new event embeddings without changing the mutual information of any given event pairs, making it effective in injecting event information for event-centric qa. more concretely, to leverage the event semantic knowledge into qa models, we propose a novel multi-task learning framework, named tranclr (fig. 2 ), combining a general-purpose qa model, with an event invertible transformation operator to encode event relations across questions and paragraphs. it builds on the unifiedqa @xcite ) model for answer generation, and employs an invertible event transformation operator to project the hidden representations from the unifiedqa encoder onto a new event embedding space. the transformed representations are then used for (i) contrasting learning and for (ii) event relation type classification. the contrastive learning mechanism is adopted to realign the event vectors, strengthening the relations between the events mentioned in questions and those candidate answer events in paragraphs and improving the generalization to out-of-distribution event relations. on the other hand, the event relation type classification is used to further fine-tune the transformation matrix through contextualized question representations. the combination of the transformation operator, along with contrastive learning and event relation type classification, leads the model to focus on the textual and relation features characterizing the event occurrences in text, and results in an overall boost in performance on event-centric qa tasks. our contributions can be summarized as follows: (1) we introduce a novel multi-task learning framework for event-centric qa, tranclr, in which we design an invertible event transformation operator and a contrastive learning mechanism, further combined with event relation type classification, to perform better reasoning on event semantic relations; this work is related to two lines of research: eventcentric qa, and contrastive learning. event-centric qa the growing interest into event understanding has recently led to the development of new resources for event-centric qa and event relation extraction. souza costa et al. (2020) proposed eventqa, an event-centric qa dataset to access semantic information stores in knowledge graphs. the questions are created via a random walking on the eventkg (gottschalk and demidova, 2019), then manually translated into natural language. ning et al. (2020a) modified and converted an event temporal relation extraction dataset -matres @xcite into a reading comprehension format focused on event temporal ordering questions, named torque. instead of solely focusing on simple arguments or temporal relations, the ester dataset @xcite was developed to highlight how events are semantically related in terms of five most common semantic relations: causal, conditional, counterfactual, sub-event, and coreference relations. aforementioned work built dataset baselines with popular entity-based plms, and thus leave significant performance gaps compared with human evaluation. asai and hajishirzi (2020), dua et al. (2021) and shang et al. (2021) leverage features of closely related questions to capture temporal difference to deal with certain types of event-centric questions. compared to the existing works, we target to various types of event-centric questions. therefore, we introduce an invertible event transformation to (i) model the event semantic relations through an auxiliary classification task, and to (ii) realign the event latent representations via contrastive learning in the space of the transformed events. contrastive learning approaches to contrastive learning for text focus on the generation of positive and negative training pairs from pretrained language models. for example, @xcite proposed a new pretraining framework named elec-tra, which defines a new generative training task, i.e., replacement token detection (rtd), with the aim of determining whether a token was originally replaced by the language model. based on electra, @xcite designed two new pretraining tasks: the correct language modeling (clm), aiming at restoring a corrupted sentence; and the a contrastive learning-based task, in which the positive pairs are made of recovered sentences and corresponding previously corrupted sentences. similarly, @xcite designed another contrastive learning framework erica for documentlevel text understanding, via specific entity discrimination pre-training task and relation discrimination pre-training task. @xcite proposed a two-stage framework, integrating answeraware span-based contrastive learning for crosslingual machine reading comprehension. @xcite and @xcite designed a framework similar to simclr @xcite to generate sentence representations by applying several data augmentation strategies to create contrastive pairs, such as word deleting and swapping, backtranslation and synonym replacement. yet, @xcite reported that simply using dropout masks twice within a plm can led to rather reliable positive pairs. our work adopts standard contrastive learning framework. the positive and negative pairs of events are composed directly from the different text sections: questions, paragraphs, and answers within the paragraph. ## methodology in this section, we first define the task of eventcentric qa and then present our proposed tranclr model. we build our model mainly based on the ester dataset @xcite . ## experiments in this section, we will first introduce the experimental setup including the dataset used and the hyperparameter setting, followed by the discussion of experimental results and ablation studies. ## conclusions in this paper, we have proposed a novel framework, called tranclr, to tackle the event-centric qa task on the ester dataset @xcite . the core idea of tranclr is to effectively explore the event knowledge in both questions and context through event-centric contrastive learning and the auxiliary task of event type classification. our experimental results show superior performance of tranclr on event-centric qa compared to the strong baseline, gaining 8.4% and 3% absolute improvements in f t 1 and em scores respectively. further zero-short inference and qualitative analysis verify the promising event semantic understanding and reasoning capability of our model.