Open-domain Video Commentary Generation (2022)

Abstract: Live commentary plays an important role in sports broadcasts and video games, making spectators more excited and immersed. In this context, though approaches for automatically generating such commentary have been proposed in the past, they have been generally concerned with specific fields, where it is possible to leverage domain-specific information. In light of this, we propose the task of generating video commentary in an open-domain fashion. We detail the construction of a new large-scale dataset of transcribed commentary aligned with videos containing various human actions in a variety of domains, and propose approaches based on well-known neural architectures to tackle the task. To understand the strengths and limitations of current approaches, we present an in-depth empirical study based on our data. Our results suggest clear trade-offs between textual and visual inputs for the models and highlight the importance of relying on external knowledge in this open-domain setting, resulting in a set of robust baselines for our task.

## introduction when watching tv programs or online live streams we find that the visual content is often accompanied by objective statements or subjective remarks about the events in the video given by a commentator in real time, aiming to help the spectators understand events in the videos. it has been shown that such live commentaries play an important role in sports broadcasts and video game matches or streams, for example, making spectators more excited, more immersed, and better informed about the content they are viewing @xcite , thus enhancing the value of online and home videos @xcite . in view of this, the task of automatically generating such live commentaries on specific domains, such as sports (kim and choi, 2020) or video games @xcite has been proposed recently, with models often relying on field-specific information to aid the generation. live commentary, in this paper, refers to commentary that can be listened to as audio or read as subtitle together with the original visual content. in light of this, in this paper we depart from previous work on automatic live commentary generation by proposing an open-domain setting, where the goal is to enable models to generate live commentaries for videos containing actions in a variety of situations. compared with related work @xcite and in-domain versions of our task @xcite , ours is particularly challenging as it keeps us from using domain-specific features, which have proven essential for attaining good performance. our task can be seen as a combination of fundamental tasks in computer vision (cv) and natural language processing (nlp), like video captioning @xcite , dense video captioning (dvc) @xcite , and data-to-text @xcite . the task is similar to video captioning, for example in the context of instructional "cooking" videos as there are no overlaps between the relevant video sections with key events occurring sequentially. however, it is critically different from previous work because each utterance is only partially aligned with the events in the video, both in terms of their temporal location and of the correlation between their content and the objects in the video. this partial alignment, which is due to multiple factors such as the presence of speech in the original video, attempts of the commentator to make the content more amenable, among others, is the key factor defining our task. to benchmark progress on this newly-introduced task, we construct a new large-scale dataset of transcribed live commentaries aligned with videos containing various human actions in a variety of domains. our dataset contains a total of 6,771 videos sampled from activitynet @xcite , and annotated for our purposes using crowdsourcing, resulting in more than 25k timestamped commentary utterances containing information to help the spectators understand events in the videos. finally, we present a multi-modal approach to tackle our proposed task and accompanied by indepth empirical study based on our data. in the absence of field-specific information due to the opendomain nature of our task, we show how existing models can be successfully leveraged, highlighting the importance of access to external knowledge, in our case in the form of pre-trained models, in order to attain better performance. furthermore, our study sheds light on the temporal alignment issue that is fundamental to our task, suggesting clear trade-offs between the role of the textual and visual modalities, as well as limitations between on-line and off-line settings. we hope our work helps provide a concrete direction for further research on commentary generation tasks, both in the in-domain and our newlyproposed open-domain settings. our dataset and models are available at github.com/epochx/ live-commentary. ## related work developing machine learning techniques that can utilize language to describe what happens in a video remains an open challenge, which lies at the core of both cv and nlp. in this context, seminal work by @xcite was, to the best of our knowledge, the first to tackle the task of describing videos in an open-domain setting. in their proposed video captioning task (vc), a system had to generate a natural language description of the main activity on a given short video (<30 s). this video captioning task was later extended by @xcite who proposed the task of dense video captioning, where a model is required to detect multiple events that occur in a video, and then describe each one using natural language. current approaches generally focus on two separate sub-tasks. first, there is a temporal event proposal step, where relevant segments in the input video are first identified, after which a captioning model generates the natural language descriptions of the identified zones @xcite . however, end-to-end approaches based on the transformer @xcite have been also recently proposed @xcite . in the context of automatically generating commentary for sports matches, @xcite used play event data (i.e., structured data about the involved player, the location in the pitch, the type of play, etc.) to generate live soccer-match commentaries to be displayed on a web page, in a data-to-text generation setting. also, kim and choi (2020) recently proposed a system to automatically generate summarized commentary for baseball games, directly from match videos, which are to be displayed during short breaks between plays. our approach is therefore different from these systems as they aim to generate commentaries that are not supposed to be displayed alongside the video content. crucially, we also note they rely on domain-specific data for solving their respective tasks. while this is obviously the case for the former, we observe that the latter is based on several sub-components that aim at modeling specifics of the game from the input video -including a player detector and a pitching result recognizerin addition to relying on a domain ontology for the generation step. more recently, @xcite proposed the task of automatically generating commentaries for racing car videogame streams, releasing the first annotated dataset of gameplay videos aligned with transcribed spoken commentaries. in this context, their main contribution is the release of structured data, that is, telemetry data, including the positions and the speeds of cars, and the steering wheel angles as extracted directly from the video game engine, which they leverage on their proposed approach. their results show that utilizing these features, it leads to substantial performance improvements over models that only receive visual and textual inputs, again showing the importance of in-domain information for the task. finally, our work is also related to simultaneous translation @xcite . specifically, our task is similar to this in the real-time scenario (please see section 4 for details), as it also requires the model to account for the trade-off between waiting in order to have more information to caption, and providing a caption with a shorter delay. ## dataset to benchmark progress on our proposed task, we present a new large-scale dataset of transcribed live commentaries aligned with videos. since our aim is to cover a large variety of actions, we rely on data from the popular activitynet benchmark @xcite , which contains videos showing various human actions in a variety of domains. we sample videos from both the training and validation splits, limiting their duration to be between thirty seconds and four minutes, and making sure we only use videos that were currently available on youtube as of 2021. ## proposed approach in this section, we discuss our proposed models to tackle the task of open-domain commentary generation. in the following, we assume that a given video @xmath0. each video in v is annotated with a sequence of natural language utterances @xmath1. we propose two different settings: (1) off-line live commentary generation: automatic commentary generation of pre-recorded videos, where the model has access to the whole video during both training and inference, and (2) real-time live commentary generation: automatic commentary generation for a live video, where the models need to balance the trade-off between waiting for additional data and generating an utterance without a too long delay. this is a more challenging setting that can be regarded as multi-modal analogous to simultaneous translation. though in this paper we focus on the off-line setting, our empirical study is also concerned with challenges that we foresee will be critical for the on-line version, as we will show in section 5. previous work on related tasks @xcite and on in-domain versions of our task @xcite have traditionally divided the efforts into simpler sub-tasks that can be combined into a pipeline approach. we follow a similar approach, also presenting two subtasks. in the following sub-sections, we propose models to tackle these sub-tasks using the first round of annotations we obtained, leaving further analysis on the second round for future work. in the rest of the paper, for clarity we focus on a given input example v . in all cases, we assume the existence of video encoder that maps the f input video frames into a sequence of video features @xmath2, @xmath3, . . . , n , accompanied by a mapping function foot_3 that allows us to transform timestamps t into feature indices τ ∈ {1, . . . , n }. ## conclusions and future work in this paper, we have proposed the task of generating live commentaries in an open-domain fashion. this is a substantial extension to previous work, which so far as only focused on single domains, where it is possible to use specific information. to benchmark progress on our newly-introduced task, we also present a dataset of transcribed live commentaries aligned with more than 6k videos containing various human actions in a variety of domains. we present models to tackle our task alongside an in-depth experimental study, showing the strengths and limitations of current approaches. based on our results, for the future we are interested in developing in-domain multi-modal transformer models can also incorporate in-domain specific information such as the structured data by @xcite . for the open-domain task, we are interested in exploring end-to-end models recently proposed for dvc @xcite , as well as in tackling the on-line version of our task.