VIEWS : Entity-Aware News Video Captioning (2024)

Abstract: Existing popular video captioning benchmarks and models often produce generic captions for videos that lack specific identification of individuals, locations, or organizations (named entities). However, in the case of news videos, the setting is more demanding, requiring the inclusion of such named entities for meaningful summarization. Therefore, we introduce the task of directly summarizing news videos into captions that are entity-aware. To facilitate research in this area, we have collected a large-scale dataset named VIEWS (VIdeo NEWS). Within this task, we face challenges inherent to recognizing named entities and navigating diverse, dynamic contexts, all while relying solely on visual cues. To address these challenges, we propose a model-agnostic approach that enriches visual information extracted from videos with context sourced from external knowledge, enabling the generation of entity-aware captions. We validate the effectiveness of our approach across three video captioning models. Additionally, we conduct a critical analysis of our methodology to gain insights into the complexity of the task, the challenges it presents, and potential avenues for future research.

## introduction summarizing video content into a natural language description (i.e. 'video captioning') is a crucial aspect of video understanding. it typically requires understanding salient information and dynamics displayed in the video. existing datasets typically describe videos using largely generic captions: "a woman is giving a speech on stage" (msr-vtt @xcite ), "an elderly man is playing the piano in front of a crowd" (activitynet captions @xcite ), "pour in spicy sauce" (youcook2 @xcite ) etc. while these captions suffice for daily activity or cooking videos, they are rendered almost meaningless in the domain a man is meeting with another woman. of news videos. as shown in fig. 1 , a generic caption saying "a man meets another woman" provides little to no information about: 1) who is meeting whom? 2) where? 3) in what context? 4) in relation to which organization? in other words, such captions lack named entities. indeed, summarizing news videos into entityaware captions is a challenging task. there has been a line of recent works which additionally utilize the news article paired with the video to generate entity aware captions @xcite @xcite . however, this assumes that a paired news article will always be available for the video. we argue that this is an impractical assumption -there are a number of standalone news video content uploaded online, especially by individuals uninvolved with big news media corporations. in this paper, we introduce the task of summarizing news videos into entity-aware captions. to support research on this task, we present video news (views foot_0 ), comprising a large-scale collection of news videos accompanied by entity-rich captions, totaling 144k samples. unlike previous news video datasets, we rigorously verify and ensure high alignment between the captions and the corresponding videos. approximately 94% of samples in our training set exhibit high precision, with any remaining precision or alignment issues in the test set manually corrected. upon acceptance, we will publicly release the dataset. this task and dataset introduces numerous challenges. firstly, it requires models to explicitly use visual cues to recognize people, places, events, organizations etc. (named entities) depicted in the video. for instance, in fig. 1 , it is required to recognize the people involved in the meeting -george bush and ellen johnson. the landmarks and cultural clues need to be utilized to predict that the meeting is taking place in liberia. secondly, understanding the background context from the presented video is a non-trivial task. in fig. 1 , this would mean identifying that george bush is not just touring liberia, instead, this meeting is part of his african nations tour. further, context is incredibly hard to learn -it's almost unique for every new sample and even changes for the same entities. lastly, these entities and contexts need to be seamlessly incorporated in the final captioning process. we propose a novel approach to tackle these challenges. the approach first trains an entity-perceiver(ep) to recognize named entities directly from the video. then, we feed the detected entities to a knowledge-extractor (ke) to extract a relevant news data as context from external source. finally, we leverage recent transformer-based video cap- dataset domain #videos entities cap.-vid in cap. align ↑ msr-vtt (xu et al., 2016) open 7.18k ✗ ✓ actynet cap @xcite open 20k ✗ ✓ youcook ii @xcite cooking 2k ✗ ✓ tvr/tvc @xcite tv show 21.8k ✗* ✓ lsmdc @xcite movie 128k ✗* ✓ mm-avs @xcite al., 2021) news 2.17k ✓ ✗ mlask (krubiński and pecina, 2023) news 41.2k ✓ ✗ views (ours) news 144k ✓ ✓ ## related work video captioning. video captioning is a longstanding task that has been explored in various domains: daily activities @xcite @xcite , cooking @xcite and movies/tv shows @xcite @xcite . however, these established video captioning benchmarks do not cover named entities in the real world. this leads to a vacancy in model development as well: recent multimodal large language models (llms) @xcite demonstrate impressive performance on traditional loc, 11.1k cardinal, 40.8k time, 3.05k fac, 8.53k quantity, 976 ordinal, 7.29k money, 1.12k law, 673 percent, 1.28k product, 3.01k work_of_art, 2.62k language, 537 event, 4.9k person 147k norp 124k org 103k gpe 168k benchmarks, but tend to hallucinate named-entities @xcite . we address this gap through the proposed dataset, views. views contains captions that are rich in entities by focusing on the news domain, and encourages entitiesaware model development. news content understanding. recent works have attempted to specifically generate entityaware captions for videos @xcite @xcite and images @xcite @xcite . however, they require paired news articles or additional metadata (tags, dates, or named-entities) to do so. this makes these approaches unrealistic in real life, where paired articles or additional metadata may not be readily available. further, it's not easily scalable. we address these limitations via the novel task of summarizing news videos directly to entity-aware captions and establishing a large-scale dataset specifically for training and evaluation. external knowledge. to acquire information beyond the training set, many approaches leverage external knowledge sources such as wikimedia @xcite , google search (lin and byrne, 2022), wikidata @xcite , and so on. orthogonally, @xcite argue that using an llm to generate knowledge is better than retrieving it. zhang et al. (2023b) extend this idea to images. building on these works, we use llms as a knowledge source for videos. unlike previous approaches, we propose a two-stage pipeline that combines learning and generation for more factual knowledge. ## dataset in this section we describe our newly collected video news (views) dataset which contains news videos accompanied by captions and additional metadata such as the title, a paired news article and the news media source. we list further details below. dataset collection. we build our dataset on top of an existing news video-article dataset, m 2 e 2 r @xcite . m 2 e 2 r has a total of 220k videos-article pairs sourced from diverse news media outlets like associated press, bbc news, axios, npr etc. we leverage the articles paired with videos to generate caption for the videos. a straightforward approach would be to generate a summary from the article. however, articles often contain related historical stories, opinions, and predictions in addition to describing the video's content. including such information in the caption would render it unfaithful to the video, as this information is not visually depicted. to mitigate this issue, we exploit an interesting property of m 2 e 2 r -some articles contain an explicit event description of each camera shot in the video, apart from the regular article story. we leverage these event descriptions and title (for context) to generate a video caption using a llm (cf. figure 2 ). specifically, we use palm-2-l-it @xcite (more details in appendix). this results in captions that are highly aligned with video content. the filtering step to collect videos having event descriptions in article leads to a total data size of ∼144k. we split this data into 1.6k dev and test set each, leaving 141k for training. quality control and annotation. while the captions are already highly aligned with the video, we do further quality control, verification and rectification on dev/test set. we employ a human-in-theloop procedure for this task. first, we ask a llm rater to score the quality of captions according to the following criteria: 1) it should contain the name of important entities (people, place, organizations etc.); 2) it should not hallucinate ; 3) it should not leave out critical information from the event de- ## method summarizing news videos directly to entity-aware captions is a challenging task. it requires not just recognition of named entities from people, places, and landmarks, but also an understanding of the context in which the events in the video are un-folding. the straightforward way of directly training the model with supervision from entity-aware ground-truth captions may enable recognition of frequently occurring named entities. however, learning context this way is nearly impossible since context changes dynamically with every sample. to address this, we divide the learning into three stages: 1) entity perceiver (ep) is trained to detect high-frequency entities from videos. 2) knowledge extractor (ke) uses detected entities to extract relevant context from external sources, eliminating the need for learning contexts. the extracted context also often contains missing long-tailed entities. 3) captioning model incorporates video, entities and context into informative entity-aware captions. the method is depicted visually in fig. 5 . we detail each of the components below. ## conclusion in this work, we proposed the new task of generating entity-aware captions from videos only. we also introduced a dataset, views, to help spur research in this field. the task is challenging for existing captioning models as it requires recognizing named-entities in the video and also the context in which the event in video is happening. we proposed a modular approach to address these challenges, which we demonstrate improves over vanilla video-only model. still, the relatively modest numbers suggest the task remains challenging and exciting for further research. we provide insights into these future research directions through our rich experiments and ablation studies