D eep M aven: Deep Question Answering on Long-Distance Movie/ TV Show Videos with Multimedia Knowledge Extraction and Synthesis (2023)

Abstract: Long video content understanding poses a challenging set of research questions as it involves long-distance, cross-media reasoning and knowledge awareness. In this paper, we present a new benchmark for this problem domain, targeting the task of deep movie/TV question answering (QA) beyond previous work’s focus on simple plot summary and short video moment settings. We define several baselines based on direct retrieval of relevant context for long-distance movie QA. Observing that real-world QAs may require higher-order multi-hop inferences, we further propose a novel framework, called the DeepMaven, which extracts events, entities, and relations from the rich multimedia content in long videos to pre-construct movie knowledge graphs (movieKGs), and at the time of QA inference, complements general semantics with structured knowledge for more effective information retrieval and knowledge reasoning. We also introduce our recently collected DeepMovieQA dataset, including 1,000 long-form QA pairs from 41 hours of videos, to serve as a new and useful resource for future work. Empirical results show the DeepMaven performs competitively for both the new DeepMovieQA and the pre-existing MovieQA dataset.

## introduction our world tells an evolving story of people, objects, and their interactions. this storytelling may exist in various forms, from textual summaries and spoken dialogues, to accompanying images and videos. because of its dynamically evolving and multi-media nature, long-distance video question answering on movies/tv shows provides a useful setting for studying the computational understanding of interconnected stories and events that aligns closely with real-world application scenarios. yet, current intelligent systems still struggle with adequately processing the rich multimedia content in long videos and fail to answer many common inquiries that humans are interested in. if we ask a virtual assistant about the relationship between two main characters in a well-known movie/tv show, it likely defaults to some null response such as ªhmm... i don't know this one". the research challenge of long video content understanding and question answering frameworks stems from the need to support information probing on certain specific details over a large multimedia context space. propagating information across long-distance has been a well-known challenge due to the vanishing gradient and memory loss problem @xcite . long videos also can not fit its entire data in-memory typically for end-toend feature extraction and neural network training. meanwhile, retrieval-based methodologies, which first find sections in the video relevant to a query through semantic cues from the corresponding dialogues and/or frames, and then return the relevant textual dialogue and/or visual frame context for question answering, tend to narrowly isolate media instances semantically similar to the query inputs. query inputs are short and succinct in nature, so simply considering data points that match the query semantics will overlook the larger picture behind the selected media instances i.e. how they relate to each other as well as to other relevant media components that are initially missed out from retrieval as they involve additional reasoning. for example, a question about ªthe hobby interests of midge's husband" in tv show ªthe marvelous mrs. maisel" would likely be missed by text retrieval if the dialogue mostly uses the name of midge's husband -joel. a question about ªmidge and joel's marriage breakdown" not only needs to identify the point of confrontation between midge and joel, but also other past, concurrent, or future events connected to these two person entities, he was shot to death by a motel manager for attempting to molest a woman, but his death was controversial and involved conspiracy theories. midge and joel, relevant to their marriage dynamics, such as infidelity, pent-up frustration, and aftermaths. previous work tends to focus on very high level questions with answers coming only from text summary, such as whether a character is good/bad, or focus on very superficial questions such as what object is behind this person, whereas we want to enable more interesting qas such as from mimicking movieclub conversations. so in this work, we define a new task of deep movie question answering (deepmovieqa), which gauges the content understanding of long movie/tv show videos that align with generic human inquiries. in contrast to previous work, the deepmovieqa task consists of questions that naturally arise from watching the full length of a movie/tv show asset (as opposed to plot summary or short minute-long clips), and involve a more challenging set of reasoning summarized in to tackle deepmovieqa, we propose a novel framework, deepmaven , which extracts context from multimedia video asset semantically rele-vant to the queries and generates a coherent answer conditioned on the retrieved context using a transformer-based backbone, with added benefits of transparency and explainability through this two-stage process. observing that the new deep-movieqa task involves more challenging types of reasoning, we further leverage a structured approach to incorporate wider context and better handle multi-hop queries, such as bridging the connection between the mention of midge's husband in a query and the canonical joel entity. our key novel contributions can be summarized as follows: 1) we present a new research task of deepmovieqa in the long-distance multimedia video understanding domain that involves reasoning beyond surface-level understanding of short moments and summaries; 2) we present deep-maven, a novel multimedia approach that leverages complementary information from (local) dialogue passages and visual frames and (widercontext) moviekg subgraph to guide deeper content understanding and question answering, achieving 10+% absolute gain in answer extraction and relative gain in answer generation over single modality baselines; and finally, 3) we contribute a deepmovieqa dataset that includes 1000 qa annotations, to serve as a new resource for studying and evaluating this challenging but exciting task. ## deepmovieqa data collection pre-existing video/movie/tv show qa datasets fall into two extremes of either focusing on too narrow and minor scene-specific details such as an object on the couch that is irrelevant to the storyline development @xcite , or overly brief and simple details from summaries @xcite ) that align poorly with real-world needs such as the year in which hook's squad is sent to belfast, which is written directly in the short plot summary. in comparison, we aim for qas closer to the conversations in movie clubs or what we talk about when we step out of the movie theatre i.e. the detailed yet crucial information to main event developments. this means we want question annotations that involve longer context of video clips and answer length, compared to prior work as reported in with this motivation in mind, we collect our dataset, deepmovieqa, on 4 movies and 5 tv shows listed in ## conclusions and future work this work is the first to study long-distance movie question answering. through guidance from multimedia source context and structured knowledge retrieval, our proposed model, deepmaven, is shown to perform well for extractive question answering, as well as abstractive answer generation for the questions in pre-existing movieqa benchmarks. but improvements are needed for generating answers to the more challenging questions presented by our new deepmovieqa benchmark, which serve as a better reflection of real-world application settings with more reasoning involved. we aim to kickstart an extractive movie question answering interface for human users, and through this, naturally acquire genuine movie questions for more effectively expanding deepmovieqa annotation, and later extend this into interactive conversational ai settings.