Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations (2020)

Abstract: Representing, and reasoning over, long narratives requires models that can deal with complex event structures connected through multiple relationship types. This paper suggests to represent this type of information as a narrative graph and learn contextualized event representations over it using a relational graph neural network model. We train our model to capture event relations, derived from the Penn Discourse Tree Bank, on a huge corpus, and show that our multi-relational contextualized event representation can improve performance when learning script knowledge without direct supervision and provide a better representation for the implicit discourse sense classification task.

## introduction representing world knowledge, and reasoning over it, to help improve language understanding is one of the longest standing ai goals. structured knowledge representations such as scripts @xcite capture temporal relations between events to describe human-level representations of common scenarios. for example, the restaurant script captures the fact that food is first ordered and only then paid for. initial works relied on manual script construction, a labor-intensive task that is hard to scale to the number of possible scenarios. more recent works focus on extracting this knowledge directly from text, using symbolic event representations @xcite or more recently, exploiting representation learning advances and representing events using dense vectors, learned from data @xcite @xcite @xcite . while these works differ in the way the internal structure of the event is represented, broadly speaking, the resulting models resemble word-embedding approaches @xcite , representing event co-occurrence in a low-dimensional vector space, and as a result use vector similarity over their embedding to measure their relationship. in this paper, we follow the observation that many natural language understanding tasks require a more expressive representation that can capture the context in which events appear @xcite and consider multiple relations between events @xcite , and going beyond simple event similarity to represent relations. to help explain the intuition behind it, consider the following example, consisting of a short story and a multiple-choice narrative cloze (mcnc) question @xcite , the standard evaluation for such models. ## related work statistical script learning is an unsupervised learning problem addressing the probabilities of event co-occurrence. @xcite started the early work, using pairwise mutual information (pmi) -based models to calculate the conditional probability distribution. in recent years, neural-based learning frameworks emerged, leading to a wave of model evolution. @xcite combine skip-gram @xcite , a word embedding model, with neural networks for learning event representations. @xcite built a long short term memory (lstm) network to learn the next relation along coreferent event chains, modeling relationships over event sequences. @xcite constructed a highdimensional tensor-based neural network, inspired by computer vision models, to learn event representations. @xcite and @xcite showed that adding event features, such as entity animacy, sentiments, or event order information, help commonsense inference. @xcite started using graph structures, beyond pairwise or sequential models, to capture event context. lee and goldwasser (2019) in the literature, the definitions of events can be categorized in two ways: entity-centric or predicate-centric. early works @xcite operated on the entity-centric events, following coreference chains of a specific entity to model sequences of events. predicate-gr @xcite was a widely adopted event definition here, consisting of a pair of dependency type, such as subject or object, and predicate token, such as verbs. recent works @xcite @xcite moved to the predicate-centric events (also called multi-argument events). each event was anchored at a predicate and considered related entity mentions and modifiers as context to the event. other works focusing on event extraction @xcite or relation extraction @xcite also adopted this definition, as it tends to capture more comprehensive view of events' semantics, aggregating information from multiple entities. in this paper, we also choose this definition, since our goal is to utilize events' context to model event relationships. graph neural models are often applied in knowledge base completion. early works used randomwalk-based methods, aiming to build scalable neural models, such as deepwalk @xcite , node2vec @xcite , line @xcite and graphsage @xcite . graph convolution networks (gcn) introduced by @xcite provide an efficient way of aggregating features from neighboring nodes. several gcn variants followed, relational graph convolution networks (r-gcn) @xcite added relation type information to address the multi-relational knowledge bases. graph attention networks (gat) @xcite manipulated attention layers for aggregating neighboring messages. gated mechanisms @xcite were proposed for mitigating the impact of data noise. gcn were also used for nlp applications, to represent structure @xcite and social information @xcite . in this paper, we adopt r-gcn for ng, as modeling different types of relationships is crucial for event commonsense inference, as attested by @xcite . discourse relations are crucial aspects for completing language understanding. early works focused on identifying explicit and implicit discourse relations under supervised settings @xcite @xcite , while recent works mined discourse connectives to refine sentence representations unsupervisedly @xcite @xcite . our work learns discourse relations between events by leveraging the fact that some explicit connectives and their categories are relatively easy to identify. we build a simplified discourse annotator that can be used to extract discourse relations between events without suffering from high noise. ## evaluations our evaluation consists of two parts. the first part conducts intrinsic evaluation, evaluating the basic characteristics of the ng model. in the second part extrinsic evaluation is performed, by using the ng event embedding for a downstream task-implicit discourse relation sense classification @xcite , from conll 2016. the source code and models used in this paper are publicly available foot_2 . ## conclusions we propose the narrative graph embedding model to learn contextualized event representations for disambiguating discourse relations. we use weak supervision, provided by the predictions of off-theshelf nlp tools and a rule-based discourse annotator, to learn event representations capturing world knowledge useful for downstream tasks. our model considers multiple discourse relations types, such as "contrast" or "cause". we evaluate our model on three intrinsic tasks, including triplet classification and event/relation predictions, as well as an extrinsic task-discourse relation classification. our results show that the model can outperform competitive systems. in the future we intend to apply our model to discourse analysis tasks which require modeling long-range dependencies.