What is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020)

Abstract: Given a video with aligned dialogue, people can often infer what is more likely to happen next. Making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of commonsense knowledge. In this work, we explore whether AI models are able to learn to make such multimodal commonsense next-event predictions. To support research in this direction, we collect a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips. In order to promote the collection of non-trivial challenging examples, we employ an adversarial human-and-model-in-the-loop data collection procedure. We also present a strong baseline incorporating information from video, dialogue, and commonsense knowledge. Experiments show that each type of information is useful for this challenging task, and that compared to the high human performance on VLEP, our model provides a good starting point but leaves large room for future work.

## introduction given a video clip (premise event), humans can often describe logical events that might happen next (future events), and interestingly people tend to agree on which future events are more likely to happen than others. making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of multimodal commonsense knowledge about the world. in figure 1 (top), 1 dataset, code are available at https://github. com/jayleicn/videolanguagefuturepred 00:21,320 --> 00:23,381 @xcite oh yeah! maybe a shake. a. the woman in the white shirt pours the slush into a cup. ## dataset the vlep dataset contains 28,726 examples from 10,234 tv show and youtube lifestyle vlog video clips. of these, 50% are collected directly from human annotators over two rounds: (1) round one: standard data collection; (2) round two: adversarial data collection. we collect human examples using amazon mechanical turk (amt), with an average cost of $1.10 per example. more detail about the annotators and quality checks are presented in appendix section a.2. the other 50% are obtained from human-annotated examples via adversarial matching @xcite . hence, overall we build our dataset with 3 collection methods (standard-human, adversarial-human, adversarialmatching), allowing a balance between easy and hard examples while reducing potential biases. ## conclusion we introduce a new task, video-and-language event prediction (vlep) -given a video with aligned dialogue, and two future events, machines is required to predict which event is more likely to happen. to support this task, vlep dataset is collected. we present a strong transformer-based baseline that incorporates information from video, dialogue, and commonsense knowledge, each of which is necessary for this challenging task. 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 negative event sentence length (#words) 0% 10% percentage of sentences figure 10: distribution of negative future event length. to ensure the quality of the collected examples, we conduct a strict filtering step in which each example is verified by three extra workers (verifiers) and we only accept examples where at least three out of four (one writer + three verifiers) reach an agreement, as @xcite ; @xcite .