Introducing Spatial Information and a Novel Evaluation Scheme for Open-Domain Live Commentary Generation (2024)

Abstract: This paper focuses on the task of open-domain live commentary generation. Compared to domain-specific work in this task, this setting proved particularly challenging due to the absence of domain-specific features. Aiming to bridge this gap, we integrate spatial information by proposing an utterance generation model with a novel spatial graph that is flexible to deal with the open-domain characteristics of the commentaries and significantly improves performance. Furthermore, we propose a novel evaluation scheme, more suitable for live commentary generation, that uses LLMs to automatically check whether generated utterances address essential aspects of the video via the answerability of questions extracted directly from the videos using LVLMs. Our results suggest that using a combination of our answerability score and a standard machine translation metric is likely a more reliable way to evaluate the performance in this task.

## introduction understanding videos has been a central task in developing machine learning, both in computer vision (cv) and natural language processing (nlp). a wide variety of vision-and-language (v&l) tasks has been developed over the years, including generative tasks like video captioning @xcite , predictive tasks like action classification @xcite , and localization @xcite , and grounding via temporal moment localization @xcite . this paper focuses on automatically generating live commentary for videos in an open-domain setting, a task proposed recently by @xcite . this task is very similar to video captioning, as comments may include descriptions of what is happening in videos, but it differs significantly because the timing and contents of utterances may vary significantly, as a commentator is free to choose what to say, when to say and how to say things @xcite . this setting proved particularly challenging due to the absence of domain-specific features. this fact was clearly shown by the stark contrast of bleu scores, which amounts to a difference of approximately one order of magnitude (2.38 v/s 24.01) @xcite . looking at v&l tasks, we see that approaches often focus on creating multi-modal contextualized embeddings. these embeddings generally rely on activity-centric features derived from image classification models, such as vit @xcite or activity classification models, such as i3d @xcite , and contextualized text embeddings @xcite . however, previous works also show that solving these tasks often requires reasoning about relationships between objects and subjects present in the video @xcite . we observe a similar trend in live commentary generation and hypothesize that due to the complexity of the task, which can be seen as a combination of more fundamental ones @xcite , incorporating the spatial dimension is critical to improve performance as it allows models to reason about objects and subjects in the video. therefore, we propose a spatial graph to contextualize features derived from an action classification model via a language-conditioned messagepassing algorithm utilizing information extracted from objects in the video. unlike previous work, our graph gives the model freedom to learn relationships between objects, as commentaries might cover aspects not centered on humans. these updated features are then used to generate live commentary via a transformer encoder-decoder model. furthermore, we note that evaluation for live commentary generation has so far relied on n-gram overlap metrics such as bleu @xcite . however, given the open-ended nature of the task, commentators may choose to talk about a given subject at different times and attend to different points. thus, we believe evaluation schemes based on text similarity are fundamentally limited, as they cannot correctly capture variations across commentaries for the same video. thus, we also propose a novel evaluation approach based on question answerability to complement text similarity-based evaluation. concretely, we use llms to evaluate if commentary utterances address essential aspects of the video by checking if key questions extracted directly from the video using lvlms can be answered. our results highlight the importance of considering spatial information for open-domain live commentary generation, substantially improving performance and allowing this setting to be competitive with in-domain instances of the task. critically, we show that not constraining the model to humancentered activities leads to the best performance. finally, our findings suggest that our proposed answerability metric, along with a standard machine translation metric, such as bleu, can be a more reliable way to evaluate the performance of models in this task, as it helps us identify instances where the generated utterances are not very informative. ## related work video captioning developing techniques that can automatically describe what happens in a video remains an open challenge. to the best of our knowledge, work by @xcite was the first to tackle the task of describing videos in an open-domain setting. this task was extended by @xcite , who proposed dense video captioning (dvc), where a model is required to detect multiple events that occur in a video and then describe each one using natural language. this task was initially tackled using a pipeline approach, where relevant segments in the input video are first identified. then, a captioning model generates the natural language descriptions of the identified zones @xcite . more recently, end-to-end approaches have also been proposed @xcite . ## proposed approach consider a video @xmath0. our goal is to generate a commentary ## experimental setting data we work with the dataset by @xcite , which, to the best of our knowledge, is the only one to tackle live commentary generation in the open-domain scenario. this dataset was built on top of the videos in the activitynet dataset @xcite , where human annotators were asked to record commentarylike narrations of the videos in english under two settings: (1) without prior knowledge and (2) after watching them once. these audios were then transcribed into text. it consists of 25k commentaries, covering a total of 6,771 videos. in our experiments, we considered only the annotations in setting (1), resulting in 6,854 commentaries for training and 6142 commentaries for validation. after an initial inspection, we noticed some quality problems in the data that prompted us to conduct an in-depth human assessment, which we detail in §a. this study indicated that a significant portion of the problems we identified were partially due to issues in the transcripts. therefore, we relied on whisper @xcite (openai/whisperlarge), a speech-to-text model based on a transformer encoder-decoder with 1,550 m parameters, to once again transcribe the recordings, which were kindly provided to us by the authors † . in section 5.1, we detail the impact of this step via several experiments and directly use our answerability metric to assess data quality. ## conclusions in this paper, we tackled the open-domain live commentary task and proposed improvements to two critical aspects of the task: (1) we integrated spatial information by proposing a novel spatial graph that is flexible in dealing with the open-domain charac-teristics of the commentaries; (2) we proposed a novel evaluation scheme for live commentary that automatically checks whether utterances addressed essential aspects of the video via answerability of questions extracted directly from the videos. our open graph significantly improved performance, making it competitive with in-domain instances of the task. our results highlight the importance of considering spatial information for opendomain live commentary generation, where not constraining the model to human-centered activities led to the best performance. moreover, results show that our proposed answerability score helps us identify instances where the generated utterances are not very informative. however, it is not very robust in terms of grammar issues. therefore, combining our answerability score and a standard machine translation metric might be more suitable for evaluating this task's performance. finally, our findings highlight potential future research. we express reservations about the current dataset, particularly when the answerability score indicates a likely poor commentary gold standard. it opens up the possibility of a new dataset iteration, where we could engage other annotators to enhance the quality of the commentary. additionally, we recognize the need to refine our proposed answerability metric, as it only measures the number of questions answered without differentiating the types of questions answered.