Video Discourse Parsing and Its Application to Multimodal Summarization: A Dataset and Baseline Approaches (2024)

Abstract: This paper tackles a new task: discourse parsing for videos, inspired by text discourse parsing based on Rhetorical Structure Theory (RST). The task aims to construct an RST tree for a video to represent its storyline and illustrate the event relationships. We first construct a benchmark dataset by identifying events with their time spans, providing corresponding captions, and constructing RST trees with events as leaves. We then evaluate baseline approaches to video RST parsing: the ‘parsing after captioning’ framework and parsing via visual features. The results show that a parser using gold captions performed the best, while parsers relying on generated captions performed the worst; a parser using visual features provided intermediate performance. However, we observed that parsing via visual features could be improved by pre-training it with video captioning designed to produce a coherent video story. Furthermore, we demonstrated that RST trees obtained from videos contribute to multimodal summarization consisting of keyframes with texts.

## introduction videos often consist of several parts, including an introduction, development, turn, and conclusion, which together form a coherent plot to effectively convey a story. 1 even shorter videos, such as consumer-generated videos lasting only a few minutes, possess similar story structures. the primitive units in the video that make up the story structure are called events, foot_1 which are used to develop and advance the story @xcite . the recent success of discourse parsing, particularly rhetorical structure theory (rst) style discourse parsing, e.g., @xcite , inspires us to introduce it to help us understand video stories. consider the significance of discourse parsing for video summarization tasks. for instance, the video in fig. 1 captures the high jump competition at summer universiade held in taipei. it focuses on three competitors, displaying their attempts at different event spans. then, it features the winner of the competition, which is the third competitor. the first competitor's event span is shown between seconds 9.6 and 21.5. the second competitor's event span is between seconds 21.6 and 27.9, and the third competitor's event span is between seconds 32.1 and 50.7. it is worth noting that the second competitor also joined the lap of honor. consequently, omitting the third competitor's span in a summary may mislead viewers into believing that the second competitor is the winner (see appendix a). this highlights the requirement that a summary must maintain the structure of the story in the original video. that is, we need to understand how events are related to each other in order to form a coherent storyline. discourse parsing is a promising solution to this issue and can be helpful in many tasks that depend on video comprehension, such as video summarization @xcite , video storytelling @xcite , video qa @xcite , and other related tasks. this paper focuses on discourse parsing for videos (hereinafter, video rst parsing), drawing inspiration from rst-style text discourse parsing approaches. since this is still a relatively new task, we begin by building a benchmark dataset. we constructed a reliable benchmark dataset for the task, video discourse treebank (vdtb), comprising 1,100 videos obtained from youtube. this dataset includes annotations of events with time spans and figure 1 : example of annotations for a video captions. 3 then, we compare 'parsing after captioning' using a given text for a video and 'direct video parsing' based on visual features to address the research question of which approach is better suited for video rst parsing. the results show that while the former approach is comparable to human performance when using gold captions, its performance is worse when using generated captions. the latter provided an intermediary performance between the above two caption-based parsers. we further observed significant improvement in performance by pre-training this parser with video captioning designed to generate the story of the video. these findings highlight the potential benefits of incorporating both visual and textual features in video rst parsing. furthermore, we provide annotations for multimodal summarization based on keyframes and text for the vdtb test set. then, we highlight that video rst trees enhance the performance of multimodal summarization. we will release our dataset (annotations for videos) at https://github.com/ titech-nlp/videoparsing_emnlp24 ## preliminary according to rst, a text is represented as a constituent tree whose leaves are an elementary discourse unit (edu), i.e., a clause-like unit, and whose intermediate nodes represent the nuclearity status (nucleus (n) or satellite (s)) of a text span consisting of edu(s). since an rst tree can be represented as a binary tree, a mono-nuclear relation (s-n or n-s), such as 'elaboration' and 'attribution,' or a multi-nuclear relation (n-n), such as 'list' and 'same-unit,' is given as an edge label between two sibling intermediate nodes. the set of rhetorical relations depends on the text domain. let us illustrate the annotation for discourse structures in a video, inspired by rst, by again using fig. 1 . time spans for events can be identified in the video, including the introduction, attempts by the three competitors, and the winner of the competition. the relations between the events are represented by a constituent tree according to the nature of rst. the event with the blue-colored time span, the approach of the third competitor from second 32.1 to 37.6, modifies the event with the red-colored time span, his jump from second 37.7 to 39.7, using the rhetorical relation 'cause.' the nuclearity statuses of the events of the blue-and red-colored time spans are satellite and nucleus, respectively. then, the event with the brown-colored time span, the third competitor's best record from second 48.3 to 50.7, modifies the event consisting of the blue-and red-colored time spans using the rhetorical relation 'supplement.' the nuclearity status of the event of the brown-colored time span is satellite, and that of the event consisting of the blue-and red-colored time spans is nucleus. ## related work action parsing is a popular method for identifying local relationships between atomic actions in a video. this is achieved by identifying temporal action segments and determining their corresponding action labels @xcite . most studies on action parsing focus on understanding the structure within events rather than the overall storyline for a video @xcite . for example, a complex action, such as 'hammer throw', may be broken down into smaller atomic actions, such as 'swing,' 'rotate the body,' and 'throw.' some stud-ies, however, delve deeper into the relationships among atomic actions. @xcite represented an event as a graph with nodes as atomic actions and edges as the relations between them. @xcite introduced hypergraphs to represent the relationships among events. such structures are useful for predicting intent @xcite , future actions @xcite , and answering questions @xcite . these studies are similar to ours in terms of analyzing the structure for videos; however, our study aims to represent the overall coherency of the storyline in a video, that involves long-distance relationships between events. @xcite proposed a method for rst parsing of news videos with the aim to capture the overall storyline in the video. their approach assumes shots as basic discourse units and constructs rst structures based on them. each discourse unit in a video corresponds to a transcript, which is utilized to parse the video's rst tree using manually designed rules. importantly, this method solely relies on text and does not incorporate any features from the videos themselves. in a recent preliminary study, akula and song-chun (2020) proposed a video discourse parsing approach based on a 'parsing after captioning' framework. this process involves generating a few captions for a given video, predicting an rst tree using these captions, and subsequently aligning the video frames and the captions to construct the rst tree. it's important to note that the primitive discourse units in their approach are video frames. that is, their approach presents a limitation in handling longer videos because the leaf nodes of their rst trees correspond to video frames rather than events. as a result, the dataset mainly includes very brief videos, with an average duration of mere 19 seconds, and it is accompanied by only a few captions. additionally, there are no annotations for event time spans. accordingly, due to the limited number of captions, constructing a meaningful rst tree seems impractical. although both studies tackled the issue of video discourse parsing, their focus leaned towards using text discourse parsing instead of purely video discourse parsing. nevertheless, they have major limitations, such as the general lack of high-quality transcripts or captions that can be used for text discourse parsing. ## constructing a dataset we construct video discourse treebank (vdtb), a dataset for video discourse parsing, as there are no existing datasets available for this purpose. from youtube, we selected 1,100 videosshowing human activities of less than a few minutes. we manually searched for channels on youtube that featured news, sports, cooking, diy, and other activity-oriented topics. we carefully selected videos with engaging stories that were free of sensitive or potentially harmful content. the domains of the videos included news (24%), activities (13%), sports (13%), instructions (15%), homemade videos (16%), and misc (19%). next, we asked two annotators, with backgrounds in natural language processing, to identify events. the annotators then generated a concise caption, preferably using simple sentences with generally one subject and one verb, for each event to form a story of the entire video. this guidance aimed to avoid detecting lengthy events consisting of several events. after that, they built an rst tree for each video based on the events by following the instructions for constructing rst discourse treebank (rst-dt) @xcite . note that 50 videos in vdtb, separated as the test set, have two different annotations by two different annotators to verify the inter-annotator agreement. that is, each of the 50 videos has two different sets of events, captions, and rst trees made by the different annotators. the detailed procedure is shown in appendix b. since the domains of vdtb and rst-dt are different from each other, we refined the rhetorical relations used in rst-dt for our purposes by introducing new rhetorical relations and omitting other relations. 4 we show our nine rhetorical relations assigned between two intermediate siblings in appendix c. when dealing with new video domains, we may need to adjust rhetorical relations by introducing new ones or modifying existing ones, similar to the process in text rst parsing. ## inter-annotator agreement in event detection and captioning vdtb provides annotations for both event spans and their corresponding captions. these annotations are similar to those in activitynet captions ## inter-annotator agreement in rst tree construction in this section, we compare vdtb and rst-dt for their rst tree qualities. ## baseline parsers we developed two parsers, the 'parsing after captioning' framework and parsing via visual features, with the same architecture, except for the feature extraction module. ## conclusion this paper introduced a new task, video discourse parsing, to build an rst tree whose leaf nodes correspond to events in a video. we constructed a dataset, vdtb, consisting of 1,100 videos with high-quality annotations for events, captions, and rst trees, as demonstrated by the comparison with activitynet captions and rst-dt. the evaluation of baseline parsers suggested that improving the parser with visual features and transferring textual knowledge to the vision encoder are promising approaches. finally, we demonstrated that video rst trees enhance multimodal summarization performance. by employing dfs or bfs to traverse the dependency format of the rst tree and extract keyframes and captions, we achieved better event agreement rate and rouge scores than other methods, even when using predicted trees. these results highlight the potential of leveraging discourse structures in videos for various downstream tasks that require video understanding.