Does It Happen? Multi-hop Path Structures for Event Factuality Prediction with Graph Transformer Networks (2021)

Abstract: The goal of Event Factuality Prediction (EFP) is to determine the factual degree of an event mention, representing how likely the event mention has happened in text. Current deep learning models has demonstrated the importance of syntactic and semantic structures of the sentences to identify important context words for EFP. However, the major problem with these EFP models is that they only encode the one-hop paths between the words (i.e., the direct connections) to form the sentence structures. In this work, we show that the multi-hop paths between the words are also necessary to compute the sentence structures for EFP. To this end, we introduce a novel deep learning model for EFP that explicitly considers multi-hop paths with both syntax-based and semantic-based edges between the words to obtain sentence structures for representation learning in EFP. We demonstrate the effectiveness of the proposed model via the extensive experiments in this work.

## introduction in information extraction (ie), an event mention is represented via an anchor/trigger word that evokes an event in the input sentence. we study the problem of event factuality prediction (efp) that aims to identify the degrees of uncertainty/factuality for event mentions in text. among others, efp finds its applications in knowledge base construction to differentiate between factual and non-factual event mentions. in this work, we follow the recent regression formulation for efp that seeks to predict a real-valued score in the range of @xcite to indicate the occurrence possibility for a given event mention @xcite . for instance, in the sentence "he cannot go to the restaurant.", "go" is the trigger word for an event mention with the factuality score of -3 (i.e., certainly not happened). in order to predict the factuality scores for the event mentions, the efp models need to locate the important context words in the sentences (i.e., the cue words) and combine them appropriately to reveal the factuality for the event triggers. as the important context words might be distributed at different positions in the sentences, the current stateof-the-art deep learning models for efp have relied on the sentence structures to facilitate the identification of the cue words. in particular, the sentence structures in the efp models can be represented via the importance score matrices that involve cells to quantify the contribution of a context word for the representation vector computation of the current word for efp @xcite . the sentence structures would then be used to induce the representation vectors for the words to perform factuality prediction. both syntactic and semantic structures of the sentences have been exploited in the deep learning models for efp. as such, the syntactic structures are based on the direct connections between the words in the dependency parsing trees of the input sentences while the contextual similarities between the words are employed to form the semantic structures @xcite . despite their success, a major limitation of the current deep learning models for efp is their inability to capture the multi-hop paths between the words to produce the importance scores in the sentence structures for efp. in particular, the current deep learning models for efp have only focused on the direct connection/relation (i.e., one-hop path) between a pair of words to determine the importance score for the words in the structures. for example, the syntactic structures in @xcite involve an binary importance score matrix where a cell is only set to 1 if the two words corresponding to that cell are directly connected in the dependency tree. this is not desirable as based on our analysis, the multi-hop paths between the words are also important and should be considered to generate better importance scores for the structures for efp. consider the trigger word "involvement" in the following sentence as an example: it was confirmed that hagai, basir's brother, had been a key member of al-qaeda while their representatives constantly denied the involvement of any basir's family members in this terrorist group. the important context words to correctly predict the factuality score +3 (i.e., actually happened) for "involvement" in this case involve "any basir's family members", "basir's brother", "hagai", "a key member", and "confirmed". in the deep learning models for efp, these important words should be encoded into the representation vector for "involvement" to perform factuality prediction. using the dependency tree for this sentence, ones can use the one-hop paths (i.e., the edges) to link "involvement" with the information in 'any basir's family members". similarly, the direct semantic similarity between "involvement" and "member" can also be used to connect "a key member" to "involvement" for representation learning in this case. however, it is very challenging for the one-hop paths to directly connect "involvement" with the other important context words (i.e., "basir's brother", "hagai", and "confirmed") (either syntactically or semantically) due to the far distances/differences between them in the sentence. fortunately, by considering the multi-hop paths between the words, we can rely on "any basir's family members" to link "involvement" with "basir's brother" (i.e., with the semantic connection between "family members" and "brother") that can be further extended to "hagai", "member", and "confirmed" via the dependency connections for representation learning. besides the multi-hop nature, we also note that the edges along the multihop path in this example contains both syntactic and semantic connections (i.e., heterogeneous edge types) that are necessary to identify the important context words for efp. motivated by this limitation, in this work, we propose to learn the importance scores for the sentence structures, leveraging graph transformer networks (gtn) @xcite to facilitate the emergence of the effective multi-hop paths with heterogeneous edge types for efp. in particular, we propose to first generate the initial sentence structures for efp based on both the syntactic and semantic information. these initial sentence structures are then combined by the gtn model via the weighted sums, serving as the intermediate struc-tures that are able to capture both syntactic and semantic one-hop connections between the words for efp. afterward, the intermediate structures are multiplied to induce the final structures that enable the modeling of the multi-hop paths with heterogeneous edge types to compute the importance scores for the structures @xcite . as illustrated by our example, we expect that these multi-hop paths between the words can help to produce more effective representation vectors for the deep learning models to achieve better performance for efp. finally, in order to improve the generalization of the proposed model for efp, we propose a novel inductive bias for the gtn model based on the information bottleneck technique @xcite . in particular, the rich combined structures from the syntactic and semantic information might offer the proposed gtn model with the high capacity for representation learning to encode the detailed information in the input sentences. as the training datasets for efp are generally small, such high capacity might eventually lead to the overfitting of the gtn model where all the context information in the input sentences, including the irrelevant ones, is preserved in the induced representation vectors. to this end, we propose to promote the gtn model in this work as an information bottleneck so the gtnproduced representations are trained to not only have good factuality prediction performance but also maintain a minimal mutual information with the input sentences @xcite . the extensive experiments on four benchmark datasets demonstrate the benefits of the proposed model, yielding the state-of-the-art performance for efp in this work. ## related work various methods have been proposed to solve efp, including the early rule-based approaches @xcite @xcite , the feature-based machine learning approaches @xcite @xcite , and the hybrid methods @xcite . the recent work has featured deep learning as the state-of-the-art method for efp. in particular, @xcite presents a model based on generative adversarial networks (gan) while @xcite applies long-short term memory networks (lstm) over both the sequential order and the dependency tree of the input sen-tences for factuality prediction. the best performance for efp so far is reported by @xcite that linearly combines the syntactic and semantic structures for graph convolutional neural networks (gcn). we also employ syntactic and semantic structures for efp in this work; however, our model presents novel techniques with triggerbased structure customization, gtns to learn the sentence structures with the multi-hop path reasoning, and information bottleneck to improve the generalization for efp. model-wise, our work bears some similarity with other nlp models that leverage syntactic structures and gcns to encode input texts for different nlp tasks, including relation extraction @xcite , joint information extraction @xcite , metaphor detection (le et al., 2020), and rumor detection @xcite . finally, we also note some related tasks for efp that seek to classify event trigger words in texts, including event detection @xcite @xcite , event realis classification @xcite and uncertainty detection @xcite . ## model we formalize efp as a regression problem in this work. in particular, given an input sentence @xmath0, w 2 , . . . , w n of n words/tokens (i.e., w i is the i-th token) and an event mention with the trigger word located at the k-th position (i.e., w k ), we need to predict a real-valued score between -3 and +3 to indicate the factual degree for w k . in order to achieve a fair comparison with the prior work for efp @xcite , we first apply the bert base model in @xcite to obtain a pre-trained embedding vector x i for each word w @xmath1. in particular, we run the bert base model over the input sentence w and use the hidden vector for the first wordpiece of w i in the last layer of bert as the embedding vector x i (of 768 dimensions) for w i . this encoding step transforms w into a sequence of embedding vectors @xmath2, x 2 , . . . , x n (called the input vectors) for the neural computation in the next steps. the efp model in this work involves three major components: (i) structure generation, (iii) structure combination, and (iii) representation regularization. we will explain the details of these components in the following sections. ## conclusion we present a novel deep learning model for efp that combines the customized sentence structures (i.e., based on both syntactic and semantic information) to learn effective representation vectors. our model features gtns to infer rich sentence structures with multi-hop reasoning paths for the importance scores and information bottleneck to improve the generalization. we perform extensive experiments to demonstrate the effectiveness of the proposed model. in the future, we plan to extend the proposed model to the related tasks of efp.