TV - TREES : Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning (2024)

Abstract: It is challenging for models to understand complex, multimodal content such as television clips, and this is in part because video-language models often rely on single-modality reasoning and lack interpretability. To combat these issues we propose TV-TREES, the first multimodal entailment tree generator. TV-TREES serves as an approach to video understanding that promotes interpretable joint-modality reasoning by searching for trees of entailment relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs. We also introduce the task of multimodal entailment tree generation to evaluate reasoning quality. Our method’s performance on the challenging TVQA benchmark demonstrates interpretable, state-of-the-art zero-shot performance on full clips, illustrating that multimodal entailment tree generation can be a best-of-both-worlds alternative to black-box systems.

## introduction videos account for a large portion of content available and consumed online, but automated reasoning over semantically complex video-language data remains a challenging and under-explored problem. a popular task for assessing models' video understanding is narrative-centric video-language question-answering: given a natural language question, a video clip of a movie or tv show, and a corresponding dialogue transcript, the goal is to return a correct natural language answer to the question using the video-language data. methods tackling the video-language qa task @xcite @xcite frequently take the form of large, jointmodality transformer models. analyses suggest their ability to perform joint visual-language reasoning is limited, as they often rely on either text or visual content but not both @xcite . @xcite , and a multimodal entailment tree, recursively produced by our approach (top-down). trees are created by recursively retrieving atomic evidence from the transcript and video frames and decomposing the qa pair into compositionally equivalent hypotheses until each can be directly entailed by the retrieved evidence. assessing modality reliance and reasoning quality overall is difficult given their lack of interpretability: while llms now facilitate increasingly transparent explanation generation alongside outputs @xcite , video-language models generally lack this ability. entailment trees @xcite , or trees of entailment relationships between atomic premises and higher-level conclusions, have been shown to serve well as the structural basis for text-only qa tasks by systematically and transparently modeling logical reasoning chains @xcite . we embrace this approach: we develop (1) the first multimodal entailment tree generator, tv-trees (the transparent video-text reasoning with entailment system), and (2) the task of multimodal entailment tree generation to assess the reasoning ability of such systems. in contrast to existing black-box qa systems, tv-trees focuses on the search and manipulation of atomic "facts" retrieved from video clips to search for proofs for video-language questionanswer pairs. the approach jointly reasons over both modalities and, crucially, the resulting entailment trees provide human-interpretable evidence and natural language explanations for each logical operation, enabling direct analysis of the model's underlying reasoning. our entailment tree evaluation method builds on work in informal logic, adapting these ideas to the multimodal domain with an emphasis on reliable evaluation. we show that our multimodal reasoning system performs competitively on zero-shot videolanguage qa for the difficult tvqa dataset @xcite while simultaneously providing comprehensive and interpretable reasoning traces. further, tv-trees achieves state-of-the-art performance using full-length clips as input. in summary, our contributions are: ## multimodal entailment trees we now introduce the task of multimodal entailment tree generation for video-language qa. ## tv-trees we now introduce our multimodal entailment tree generator, pictured in figure 2 . ## evaluation methodology traditionally, qualitative natural text evaluations have been conducted using humans @xcite . recently, researchers have considered whether these human evaluations could be replaced by high-performing llms like gpt-4 @xcite . we detail how we implement the evaluation metrics described in section 3.2 with both human annotators and gpt-4. we report evaluation results for both methods in section 6. ## experiments we evaluate tv-trees using the tvqa dataset as input data. we compare its zero-shot qa performance against competing video-language qa approaches to illustrate its practical usage, evaluate its overall tree quality through our evaluation method described in section 5, and organize its reasoning error modes through a qualitative study. setup we instantiate tv-trees as it is described in section 4, allowing for trees with up to 3 levels (@xmath0. our experiments focus on the multiple-choice qa domain, and so we consider a question's correct answer to be the answer that results in a complete tree. in the case that the system does not successfully complete any tree for the five answer candidates, we consider the answer candidate with the "most complete" tree to be the correct answer, breaking ties by average entailment score. when complete trees are generated for multiple answers, we break ties in the same way. ## conclusion we introduce the first neuro-symbolic entailment tree generator for multimodal content to improve the robustness, reliability, and interpretability of video-language understanding systems. we propose the task of multimodal entailment tree generation for the assessment of generated tree reasoning quality, establishing an information-theoretic evaluation method grounded in informal logic theory. we show that our approach achieves state-ofthe-art results on the zero-shot tvqa benchmark with full video clips, illustrating the potential for generated reasoning traces to improve downstream video-language understanding task performance. we show that interpretable, neuro-symbolic approaches to video understanding like tv-trees are a strong alternative to existing methods, provide substantial new benefits, and highlight exciting directions for future research. this paper is presented as an initial exploration into multimodal neuro-symbolic systems, and so there are many exciting avenues for future development and research. individual components of this system could be improved for better performance on tvqa and related tasks -for instance, the vision querying system is fairly end-to-end, and semantically deconstructing the frames using a visual semantic role labeling model and using its outputs as evidence could result in a more sophisticated and transparent logical system. in future work, we also hope to explore the possibility of producing a collection of entailment trees pertaining to the same video clip and aggregating them to produce a comprehensive knowledge graph of the full video. we also hope to explore ways to improve computational efficiency and cost of using the system. finally, there is significant room for future work in decomposing natural language text for entailment tree generation, and for establishing entailment between premises and grounded multimodal evidence.