Multi-Criteria Evaluation Framework of Selecting Response-worthy Chats in Live Streaming (2024)

Abstract: Live streaming, a dynamic medium that merges real-time audiovisual content with interactive text-based chat, presents unique challenges for maintaining viewer engagement and ensuring streamers’ well-being. This study introduces a multi-criteria evaluation framework designed to identify response-worthy chats during live streaming. We proposed a system that evaluates chats based on sentiment polarity and intensity, contextual relevance, and topic uniqueness. We also constructed a dataset annotated by human reviewers who validates the framework, demonstrating a closer alignment with human preferences compared to single-criterion baselines. This framework not only supports the development of more responsive and engaging live streaming environments but also contributes to the broader field of dialog systems by highlighting the distinct needs of real-time, large-scale conversational contexts.

## introduction live streaming, which merges real-time audiovisual content with simple text-based chat, has seen a surge in popularity and is now influential in various sectors @xcite . live streaming is transforming how streamers and viewers interact online, creating a novel type of dialog system that can either facilitate human interaction or autonomously host the live streaming conversation @xcite . the challenge for these live streaming dialogue systems lies in boosting user engagement, prolonging viewing duration, and improving viewer satisfaction. @xcite . the main goal of introducing a system to selecting chats in live streaming is to address the challenges that human streamers face due to their limited time and capabilities. for instance, when dealing with large audiences, it's not feasible for streamers to sift through and reply to every chat during live interactions with potentially thousands of viewers. automation can support streamers by helping them identify important chats and craft responses. additionally, the demands of streaming for extended periods and frequently can take a toll on streamers' health, both physically and mentally. through the implementation of automation in live streaming, we can reduce the burden on streamers and contribute to their overall well-being @xcite . research into dialogue systems, both traditional and those tailored for live streaming, reveals distinct differences in their design and functionality. traditional dialogue systems are built for one-onone interactions, whereas those for live streaming must handle simultaneous real-time conversations with numerous users. this demands that the system quickly processes inputs from potentially thousands of participants @xcite . while traditional dialogue systems strive to offer a personalized experience, those on live streaming also need to personalize but prioritize delivering responses that are relevant to a wide audience @xcite . content moderation is a feature of traditional dialogue systems, but it is not as critical as it is for live streaming. here, dialogue systems require advanced monitoring and filtering tools to immediately address and eliminate any inappropriate content @xcite . this study proposed a novel pipeline in capturing the most interactively significant chats from the real-time interactions in live streaming. the key contributions of this study include: ## related work automated dialog systems for live streaming systems can be categorized into two types: those that partly assist human interaction and those that are fully automated, with an ai streamer taking the place of a human host. an example of the former is nightbot 1 , a tool used on platforms such as twitch, youtube and trovo. it helps manage live chats by filtering out spam and facilitating custom chat commands. the framework in our study incorporated a module for filtering that draws on strategies similar to nightbot. however, these assisted systems rely on a predefined set of keywords to filter or respond, which can limit their ability to adapt to the dynamic context of live streams. on the other hand, fully automated live streaming systems are often performed as vtubers, or 1 https://nightbot.tv/ virtual youtubers @xcite . these are streamers who utilize animated avatars. ai-hosted vtubers generate replies and animate their avatar's expressions and movements by feeding chats into a large language model. for instance, neuro-sama foot_0 is recognized for engaging in smooth dialogue with viewers. however, it was temporarily banned from twitch for generating hateful speech and has shown difficulty in grasping the context of conversations (seiji, 2023). ai streamers are also expected to not only chitchatting but also handling multimodal information. the open-source framework luna ai foot_1 equips ai streamers with tools for voice and singing synthesis, as well as image generation. meanwhile, goroundgame foot_2 presents an ai streaming project tailored for gaming broadcasts. ai streamers in goroundgame streams while playing mahjong against another ai streamer but struggles to strike a balance between commenting on the game and interacting with chats. we gathered chat data from a segment of the goroundgame live stream replays and included it in the evaluation dataset. ## framework figure 2 presents the proposed framework for evaluating chat from viewers in this study. the framework is designed to filter, evaluate, and finally identify the response-worthy chats. this process is structured into three distinct phases: chat filtering, chat rating, and chat selection. in chat rating phase, three criteria are employed: sentiment polarity and intensity, contextual relevance, and topic uniqueness. ## evaluation this chapter discusses the evaluation of the proposed multi-criteria framework for selecting response-worthy chats in live streaming. it involves the creation of a dataset from youtube live streams, annotated by human reviewers to reflect preferences. the framework's accuracy is compared to single-criterion baselines, showing improved alignment with human selections, and highlights differences between ai-hosted and human-hosted streams. ## conclusion in this study, we proposed a framework based on various criteria, including sentiment polarity and intensity, contextual relevance, and topic uniqueness-to evaluate view chats in live streaming. we also constructed a dataset reflecting human preferences to assess the performance of above framework. our findings suggested that a composite criteria better reflects human preferences than a single approach, and identified differences in interaction preferences between human-hosted and ai-hosted live streams. moving forward, we plan to improve our method by incorporating feedback from crowd-workers and train a chat-scoring model directly from the labels of human feedback. additionally, we intend to make this dataset publicly available to support further research in enhancing automated dialog systems for live streaming.