Event Causality Identification via Generation of Important Context Words (2022)

Abstract: An important problem of Information Extraction involves Event Causality Identification (ECI) that seeks to identify causal relation between pairs of event mentions. Prior models for ECI have mainly solved the problem using the classification framework that does not explore prediction/generation of important context words from input sentences for causal recognition. In this work, we consider the words along the dependency path between the two event mentions in the dependency tree as the important context words for ECI. We introduce dependency path generation as a complementary task for ECI, which can be solved jointly with causal label prediction to improve the performance. To facilitate the multi-task learning, we cast ECI into a generation problem that aims to generate both causal relation and dependency path words from input sentence. In addition, we propose to use the REINFORCE algorithm to train our generative model where novel reward functions are designed to capture both causal prediction accuracy and generation quality. The experiments on two benchmark datasets demonstrate state-of-the-art performance of the proposed model for ECI.

## introduction in information extraction (ie), event causality identification (eci) aims to predict causal relation between a pair of events mentioned in text. for instance, in the sentence "massive fires cause major damages in the downtown area.", an eci system needs to realize the causal relation between the two events triggered by "fires" and "damages" (called event mentions), i.e., "fires" cause ---→ "damages". eci is an important problem with many applications in nlp @xcite . compared to the feature-based methods @xcite , recent deep learning models have demonstrated their state-of-the-art performance for eci @xcite @xcite . as such, prior work has mainly treated eci as a classification problem where the only output from the models is a label to indicate causal or non-causal relation between input events. a major issue with this classification formulation is that current eci models do not output important contexts for causal prediction of two event mentions. in this work, important contexts refer to the words in the input sentence that are critical to reveal the causal relation between two given event mentions (e.g., the words "caused" and "by" in our example). this limitation of current eci models is undesirable as we expect that including important context words as a part of the outputs for eci models can improve the training signals for the models. in particular, motivated by relation exaction models in ie @xcite , we use the words along the dependency path between the two event mentions in the dependency tree to represent important context words for eci. our intuition is that dependency path generation is a related/complementary task for causal label prediction in eci, and training a model to jointly generate causal labels and dependency path words (i.e., multi-task learning) can boost the performance. a potential challenge with this idea involves the varying number of dependency path words where the generation of a context word or causal label might need to condition on previously generated ones (e.g., dependencies at the output level). as the result, such dependencies make it difficult to extend existing classification-based eci models to perform multi-task learning with important context prediction. to address this issue, we propose to solve eci via a new generative formulation: given a pair of event mentions in an input sentence, our eci model aims to simultaneously generate causal label and the dependency path words between the two event mentions. in our model, causal label and dependency path words are combined into a single output sequence that will be generated by a generative model from the input sentences in an autoregressive fashion, thus facilitating the encoding of dependencies between output words in our multitask learning idea. finally, to solve the resulting sequence-to-sequence problem for eci, we leverage the generative pre-trained language model t5 @xcite . to our knowledge, this is the first work to use generative models to solve eci. ## model given a sentence w and two event mentions e s and e t in w , eci aims to predict whether e s and e t are involved in a causal relation in w . in this work, we depart from the traditional classification formulation @xcite to a generative approach for eci. our generative model follows the sequence-to-sequence setting where the input sequence should capture the input sentence w along with the two event mentions e s and e t . in contrast, the output sentence will include the causal label and the dependency path between e s and e t in the dependency tree of w to achieve multi-task learning with important context word generation. to this end, the input i for our generative eci model is obtained by combining w and a prompt p (e s , e t ) to specify the two input event mentions and the goal of eci, i.e., @xmath0: p (e s , e t ). in this work, we use a simple template for p (e s , e t ) in the form of "is there a causal relation between e s and e t ?". as such, the output sequence o is then formed using the concatenation: @xmath1, d(e s , e t ) (called golden output). here, l is either "yes" or "no" to indicate the existence of a causal relation between e s and e t (i.e., causal label) while d(e s , e t ) represents the dependency path between e s and e t in w . in our example, the input and output sequences are: i: massive fires cause major damages in the downtown area: is there a causal relation between fires and damages? o: yes, fires cause damages given the transformed input-output pair (i, o) for every example in the training data of eci, we adopt the pre-trained tranformer-based language model t5 @xcite to solve the resulting sequence-to-sequence problem. in particular, we train t5 on the transformed input-output pairs (i, o) from eci training data. at inference time, given an input sentence and two event mentions, we use the trained t5 model to generate the output sequence (with greedy decoding) from which the causal label can be extracted from the first token (i.e., l in o) to serve as the prediction. training: as presented in the introduction, to employ label accuracy as the direct training signal, we propose to leverage the reinforce algorithm @xcite to train our t5 model for eci where label accuracy will be used to form the reward function. in addition, the flexibility of rein-force allows us to include the similarity between the predicted output sequence, denoted by c, from t5 and the golden output o and input i as terms in the reward function to train our generative model. as such, we propose the following information for the reward function r(c) for reinforce: consequently, the overall reward function r(c) to train our t5 model for eci is: r(c) = α per r per (c) + α out r out (c) + α in r in (c) (α per , α out , and α in are trade-off parameters). in this way, we can explicitly make sure that label accuracy (i.e., our main performance goal) is well represented and not dominated by the generation rewards in the training. let p (c|i) be the distribution over generated sequences that t5 induces. in our model, reinforce trains t5 by minimizing the negative expected reward r(c) over the possible choices of c from t5: @xmath2. using policy gradient and one roll-out sample with the generated sequence c, the gradient of l can be estimated for training via: ∇@xmath3. here, we obtain the baseline b via: @xmath4, where |b| is the mini-batch size and c q is the generated sequence for the q-th sample. finally, before reinforce training, we first bootstrap t5 by training it over the transformed pairs (i, o) with maximum likelihood objective. this helps constrain the large action space with text generation to improve the learning for rein-force @xcite . 1 we have tried blue, meteror, and other variants of rouge; however, rouge-2 leads to the best performance. we tune the hyperparameters for geneci on the development data of eventstoryline; the chosen parameters are employed to train the models for both eventstoryline and causal-timebank. the selected hyperparameters from our tuning process involve: 5e-5 for the learning rate with the adam optimizer; 32 for the mini-batch size; and 1.0, 0.5 and 0.1 for the trade-off-parameters α per , α out and α in (respectively) in the overall reward function r(c). finally, we use the base version of t5 (raffel et al., 2020) for the generative model in this work. comparison: we compare our model with the state-of-the-art (sota) models for eci. for eventstoryline, we consider the following baselines: (1) lstm @xcite adopted from (cheng and miyao, 2017); (2) seq (gao et al., 2019) adopted from ( choubey and huang, 2017) for eci; and (3) lr+ and lip @xcite : document structure-based models for eci. for causal-timebank, we evaluate rb: a rule-based system in @xcite , and ml: a feature-based model for eci in @xcite . for both datasets, we also compare with the following bert-based models for eci: (i) bert: a bert-based baseline in @xcite ; (ii) knowdis @xcite : a model with distant supervision; (iii) know @xcite ): a model with conceptnet; (iv) richgcn (tran and nguyen, 2021): a graph convolutional network with rich information, and (v) learnda @xcite : a data augmentation method. richgcn has the best reported performance on eventstoryline while learnda is the current sota model for causal-timebank. finally, we also report the performance of t5 classify that is similar to the classification-based model bert @xcite , but replaces the bert encoder with the encoder from t5. ## related work in the early methods, eci has been mostly approached by feature-based models @xcite @xcite @xcite @xcite . recently, eci has been further solved by deep learning models @xcite where external knowledge and additional training data are leveraged to improve the performance @xcite @xcite . we are different from such prior work as we are the first to model eci via a generative model. using generative models for traditional classification-based problems has also been explored recently, e.g., for named entity recognition @xcite , sentiment analysis @xcite , and event extraction @xcite . however, none of such prior work considers generative models for eci. finally, we also note related work on extracting other types of relations between event triggers, including temporal relation @xcite @xcite , subevent relation @xcite @xcite , and coreference relation @xcite @xcite @xcite . ## conclusion we introduce a novel model for eci that solves the problem via a generation framework with the t5 model. our model explores multi-task learning that jointly generates the dependency paths between two event mentions for eci. we also introduce a training procedure based on reinforce and novel reward functions, which leads to the sota performance for eci. in the future, we plan to extend the model to other relation extraction tasks.