Personalized Video Comment Generation (2024)

Abstract: Generating personalized responses, particularly in the context of video, poses a unique challenge for language models. This paper introduces the novel task of Personalized Video Comment Generation (PVCG), aiming to predict user comments tailored to both the input video and the user’s comment history, where the user is unseen during the model training process. Unlike existing video captioning tasks that ignores the personalization in the text generation process, we introduce PerVidCom, a new dataset specifically collected for this novel task with diverse personalized comments from YouTube. Recognizing the limitations of existing captioning metrics for evaluating this task, we propose a new automatic metric based on Large Language Models (LLMs) with few-shot in-context learning, named FICL-Score, specifically measuring quality from the aspects of emotion, language style and content relevance. We verify the proposed metric with human evaluations. We establish baselines using prominent Multimodal LLMs (MLLMs), analyze their performance discrepancies through extensive evaluation, and identifies directions for future improvement on this important task. Our research opens up a new direction of personalizing MLLMs and paves the way for future research.

## introduction what if your ai assistant could not only help you with tasks but also share your unique sense of humor or express your excitement about your favorite sports team? while recent advancements in ai have led to impressive progress in foundational models @xcite and task-oriented assistants @xcite @xcite @xcite @xcite , one crucial aspect of human interaction remains largely unexplored: personalized comment history: "he could've done that to shaquille o'neal ", "this is what happens when no one will play with you. you develop the coolest skills on the block ", "i'm happy to see emmanuel finding his spot, getting more comfortable at college level playing like if it was high school congrats bro there is nowhere else to go but up ", " this is the real world grandson just teaching you how it's going to be ", …… target comment for this video: "bro, has it been that long is lebron that old? " ## dataset collection and statistics this section details the process of constructing our personalized video comment dataset and the statistics of the resulting pervidcom dataset. ## methods in this section, we first introduce all the baseline methods and then illustrate the proposed model. ## evaluation properly evaluating personalized video comments is challenging because of the rich variety in user's attention and emotion. conventional metrics are used for widely used for evaluating text generation and captioning tasks such as bleu-4 @xcite , cider @xcite , me-teor @xcite , and rouge-l @xcite are not really suitable for evaluating the quality of generated comments because their string-matching design. a more suitable automatic evaluation metric is imperative for the pvcg task. to understand how to design a more suitable automatic metric, we first perform human evaluation. ## experiment results in this section, we first report and analyze the comparison between the proposed ficl-score and conventional automatic evalution metrics. then we provide benchmark results of all the methods considered for this task. finally, we ablate and discuss important questions and design choices. ## data release plan to ensure responsible data sharing and comply with copyright regulations, we will not directly release the video data. instead, we will provide a comprehensive code repository so that the audience can download the videos using the provided list of video ids. similarly, to protect user privacy, we will only provide the list of private user ids that are are not visible publicly except to the users themselves. our code repository will also include the code we used to call youtube data api to download comments with comment text and private user ids. we encourage the audience to carefully read the fair use notice in the appendix if there are any questions about the usage of the dataset. ## conclusion we introduce a novel task of personalized video comment generation, along with pervidcom, the first benchmark dataset for generalization to unseen users. we propose ficl-score, a new automatic evaluation metric leveraging few-shot incontext learning with llms to measure the quality of generated comments, demonstrating strong alignment with human judgment. our experiments with prominent mllms and a fine-tuned pv-llm model highlight both the potential and limitations of current models. this work paves the way for future research on multimodal personalized content generation.