Event Representation with Sequential, Semi-Supervised Discrete Variables (2021)

Abstract: Within the context of event modeling and understanding, we propose a new method for neural sequence modeling that takes partially-observed sequences of discrete, external knowledge into account. We construct a sequential neural variational autoencoder, which uses Gumbel-Softmax reparametrization within a carefully defined encoder, to allow for successful backpropagation during training. The core idea is to allow semi-supervised external discrete knowledge to guide, but not restrict, the variational latent parameters during training. Our experiments indicate that our approach not only outperforms multiple baselines and the state-of-the-art in narrative script induction, but also converges more quickly.

## introduction event scripts are a classic way of summarizing events, participants, and other relevant information as a way of analyzing complex situations @xcite . to learn these scripts we must be able to group similar-events together, learn common patterns/sequences of events, and learn to represent an event's arguments @xcite . while continuous embeddings can be learned for events and their arguments @xcite , the direct inclusion of more structured, discrete knowledge is helpful in learning event representations @xcite . obtaining fully accurate structured knowledge can be difficult, so when the external knowledge is neither sufficiently reliable nor present, a natural question arises: how can our models use the knowledge that is present? generative probabilistic models provide a framework for doing so: external knowledge is a random variable, which can be observed or latent, and the data/observations are generated (explained) from it. knowledge that is discrete, sequential, or both-crush occurred at @xmath0 figure 1 : an overview of event modeling, where the observed events (black text) are generated via a sequence of semi-observed random variables. in this case, the random variables are directed to take on the meaning of semantic frames that can be helpful to explain the events. unobserved frames are in orange and observed frames are in blue. some connections are more important than others, indicated by the weighted arrows. in this paper, we provide a successful approach for incorporating partially-observed, discrete, sequential external knowledge in a neural generative model. we specifically examine event sequence modeling augmented by semantic frames. frames @xcite are a semantic representation designed to capture the common and general knowledge we have about events, situations, and things. they have been effective in providing crucial information for modeling and understanding the meaning of events @xcite @xcite . though we focus on semantic frames as our source of external knowledge, we argue this work is applicable to other similar types of knowledge. we examine the problem of modeling observed event tuples as a partially observed sequence of semantic frames. consider the following three events, preceded by their corresponding bracketed frames, from fig. 1: [event] crash occurred at station. we can see that even without knowing the [killing] frame, the [event] and @xcite frames can help predict the word killed in the third event; the frames summarize the events and can be used as guidance for the latent variables to represent the data. on the other hand, words like crash, station and killed from the first and third events play a role in predicting [impact] in the second event. overall, to successfully represent events, beyond capturing the event to event connections, we propose to consider all the information from the frames to events and frames to frames. in this work, we study the effect of tying discrete, sequential latent variables to partially-observable, noisy (imperfect) semantic frames. like @xcite , our semi-supervised model is a bidirectional auto-encoder, with a structured collection of latent variables separating the encoder and decoder, and attention mechanisms on both the encoder and decoder. rather than applying vector quantization, we adopt a gumbel-softmax @xcite ancestral sampling method to easily switch between the observed frames and latent ones, where we inject the observed frame information on the gumbel-softmax parameters before sampling. overall, our contributions are: our work builds on both event modeling and latent generative models. in this section, we outline relevant background and related work. ## method our focus in this paper is modeling sequential event structure. in this section, we describe our variational autoencoder model, and demonstrate how partially-observed external knowledge can be injected into the learning process. we provide an overview of our joint model in §3.1 and fig. 2 . our model operates on sequences of events: it consists of an encoder ( §3.2) that encodes the sequence of events as a new sequence of frames (higher-level, more abstract representations), and a decoder ( §3.3) that learns how to reconstruct the original sequence of events from the representation provided by the encoder. during training ( §3.4), the model can make use of partially-observed sequential knowledge to enrich the representations produced by the encoder. in §3.5 we summarize the novel aspects of our model. ## experimental results we test the performance of our model on a portion of the concretely annotated wikipedia dataset @xcite , which is a dump of english wikipedia that has been annotated with the outputs of more than a dozen nlp analytics; we use this as it has readily-available framenet annotations provided via semafor @xcite . our training data has 457k documents, our validation set has 16k documents, and our test set has 21k documents. more than 99% of the frames are concentrated in the first 500 most common frames, so we set @xmath1. nearly 15% of the events did not have any frame, many of which were due to auxiliary/modal verb structures; as a result, we did not include them. for all the experiments, the vocabulary size (v ) is set as 40k and the number of events (m ) is 5; this is to maintain comparability with haqae. for the documents that had more than 5 events, we extracted the first 5 events that had frames. for both the validation and test datasets, we have set i @xmath2; frames are only observed during training. documents are fed to the model as a sequence of events with verb, subj, object and modifier elements. the events are separated with a special @xmath3. in order to facilitate semi-supervised training and examine the impact of frame knowledge, we introduce a userset value : in each document, for event m, the true value of the frame is preserved in i m with probability , while with probability 1we set i @xmath4. this is set and fixed prior to each experiment. for ## conclusion we showed how to learn a semi-supervised vae with partially observed, sequential, discrete latent variables. we used gumbel-softmax and a modified attention to learn a highly effective event language model (low perplexity), predictor of how an initial event may progress (improved inverse narrative cloze), and a task-based classifier (outperforming fully supervised systems). we believe that future work could extend our method by incorporating other sources or types of knowledge (such as entity or "commonsense" knowledge), and by using other forms of a prior distribution, such as "plug-and-play" priors @xcite @xcite . so f m @xmath5,i ) 1/@xmath6. after sampling a frame simplex vector, we multiply it to the frame embeddings matrix e f . with an appropriate temperature for gumbel-softmax, the simplex would be approximately a one-hot vector and the multiplication maps the simplex vector to the embeddings space without any limitation on the norm.