A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot (2023)

Abstract: Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. There is a dearth of large annotated training datasets in the multimedia domain hindering the development of supervised learning models with satisfactory performance for real-world applications. On the other hand, the rise of large language models (LLMs) has witnessed remarkable zero-shot performance in various natural language processing (NLP) tasks, such as emotion classification, question answering, and topic classification. To leverage such advanced techniques to bridge this performance gap in multimedia understanding, we propose verbalizing long videos to generate their descriptions in natural language, followed by performing video-understanding tasks on the generated story as opposed to the original video. Through extensive experiments on fifteen video-understanding tasks, we demonstrate that our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. Furthermore, to alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.

## introduction recently, large video pre-trained models (lvms) like videomae @xcite , internvideo @xcite , and videoclip @xcite have proved to be powerful in enhancing reasoning skills on video data. for e.g., intern-video showed a performance increase in action classification and question answering tasks. nevertheless, these models have a few shortcomings that impair their performance on video understanding tasks: 1) lvms are mostly trained on short videos (<10s) consisting of majorly motion-centric actions, such as those present in kinetics @xcite and something something v2 @xcite ; and 2) they often require a significant amount of task-specific finetuning data to perform on videounderstanding tasks like summarization, questionanswering, and emotion classification. on the other hand, stories are often longer than 10 seconds and are typically much more complex than motioncentric videos. for example, we test our models on five datasets with average video lengths of 12.4 minutes (video story) and 3.5 minutes (other tasks). our videos contain dialogues, text overlaid on frames, fast-moving scenes, graphics like symbols and cartoon characters, and rhetoric elements like emotions and taglines, other than motions and generated story : a family had the perfect idea for a paw patrol-themed birthday party. the kids had a blast blowing bubbles in the backyard and playing in the pool with a mascot in a paw patrol costume. the birthday boy got to play with a pinata in the backyard and a hot air balloon flew over the house. the kids played with a toy bike and a paw patrol playhouse, and the birthday girl hugged a mascot. they also set up a lego train set and a dc comics mascot in the pool. the birthday party ended with a game of catch and a pin the badge on the mascot game. everyone had a great time, and the birthday cake was a paw patrol-themed delight. this vlog shows that family fun can be had at any birthday party with a little imagination. topic : celebrations emotion : cheerful persuasion strategy: social identity, social proof action : consider using your imagination to create a fun and memorable birthday party for your family or friends. reason : the reason is to create a uniquely fun-filled and memorable birthday party for your family or friends transcript : well hey zuma that's the wrong pinata …. how old are you …. my birthday cake is its zoo …. paw patrol you …. got as a present … thank you marshall…. are you ready to blow out the candle…. marshall let's push chase and the pool … wow all the surprises we got…. go first okay put on the blindfold ready …… for more fun click the red subscribe button figure 1 : an example of a story generated by the proposed pipeline along with the predicted outputs of the videounderstanding tasks on the generated story. the generated story captures information across scenes, characters, event sequences, dialogues, emotions, and the environment. this helps the downstream models to get adequate information about the video to reason about it correctly. the original video can be watched at https://youtu.be/_amwpjacoc8 . recently, zero-shot performance in the natural language processing (nlp) domain has increased substantially owing to the recent growth of generative large language models (llms). instead of fine-tuning llms, in-context learning has recently gained noticeable attention to exploring the reasoning ability of llm, where several input-output exemplars are provided for prompting @xcite @xcite . for example, the chain of thought prompting @xcite has been discovered to empower llms to perform complex reasoning by generating intermediate reasoning steps. owing to in-context learning, tasks like emotion recognition, named entity recognition @xcite , and even higher-order composite tasks like table understanding @xcite have shown performance improvements. however, the capability of llms on video reasoning tasks is still unexplored. there are several technical challenges preventing leveraging llms for video-based reasoning tasks. first, a video in the raw form can be quite long, spanning minutes and thus containing many frames. directly encoding all frames via pre-trained models could be computationally intractable and interfere with huge amounts of irrelevant information. second, for effective video understanding, we need information about multi-ple sources and modalities such as dialogue, text, characters, and scenes. all of them provide both intersecting and mutually exclusive information helpful to understand and reason about the video. as a first work, we propose to address these challenges by exploiting the power of llms in the video domain. the llms are used to decompose long videos into stories and then to reason about videos using the generated stories instead of the raw information. in this way, we can retain the relevant evidence and exclude the remaining irrelevant evidence from interfering with the decision. the main contributions of our paper are: 1) we propose converting long videos from the multimodal domain to "small" coherent textual stories by verbalizing keyframes, audio, and text-overlaid scenes with the help of a powerful llm and instructions (figs. 1 , 5 ). we experimentally test the story generation capability of our method and note that our method outperforms state-of-the-art story generation methods. gmflow sampling algo ocr module blip 2 automatic speech recognition [closed captions] prompt combining all information, input to the llm for generation youtube channel name & video name "hasbro" "littlest pet shop toys" ## method large language models (llms) have been demonstrated to perform well for downstream classifica-tion tasks in the text domain. this powerful ability has been widely verified on natural language tasks, including text classification, semantic parsing, mathematical reasoning, etc. inspired by these advances of llms, we aim to explore whether they could tackle reasoning tasks on multimodal data (i.e. videos). therefore, we propose a storytelling framework, which leverages the power of llms to verbalize videos in terms of a text-based story and then performs downstream video understanding tasks on the generated story instead of the original video. our pipeline can be used to verbalize videos and understand videos to perform complex downstream tasks such as emotion, topic, and persuasion strategy detection. ## conclusion in this study, we delve into the challenge of understanding long videos. these videos, including advertisements and documentaries, encompass a wide range of creative and multimodal elements, such as text, music, dialogues, visual scenery, emotions, and symbolism. however, the availability of annotated benchmark datasets for training models from scratch is limited, posing a significant obstacle. to overcome these challenges, we propose leveraging the advancements in large language models (llms) within the field of natural language processing (nlp). llms have shown remarkable zeroshot accuracy in various text-understanding tasks. thus, our approach involves verbalizing videos to generate stories and performing video understanding on these generated stories in a zero-shot manner. we utilize signals from different modalities, such as automatic speech recognition, visual scene descriptions, company names, and scene optical character recognition, to prompt the llms and generate coherent stories. subsequently, we employ these stories to understand video content by providing task explanations and options. our proposed method demonstrates its effectiveness across fifteen videounderstanding tasks getting state-of-the-art results across many of them. the entire pipeline operates in a zero-shot manner, eliminating the reliance on dataset size and annotation quality.