L ive C hat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming (2023)

Abstract: Open-domain dialogue systems have made promising progress in recent years. While the state-of-the-art dialogue agents are built upon large-scale social media data and large pre-trained models, there is no guarantee these agents could also perform well in fast-growing scenarios, such as live streaming, due to the bounded transferability of pre-trained models and biased distributions of public datasets from Reddit and Weibo, etc. To improve the essential capability of responding and establish a benchmark in the live open-domain scenario, we introduce the LiveChat dataset, composed of 1.33 million real-life Chinese dialogues with almost 3800 average sessions across 351 personas and fine-grained profiles for each persona. LiveChat is automatically constructed by processing numerous live videos on the Internet and naturally falls within the scope of multi-party conversations, where the issues of Who says What to Whom should be considered. Therefore, we target two critical tasks of response modeling and addressee recognition and propose retrieval-based baselines grounded on advanced techniques. Experimental results have validated the positive effects of leveraging persona profiles and larger average sessions per persona. In addition, we also benchmark the transferability of advanced generation-based models on LiveChat and pose some future directions for current challenges.

## introduction building dialogue systems to converse naturally with humans has been one of the longest-running goals in artificial intelligence @xcite . to usher that chatbot response properly in diverse scenarios, it is desirable to train a conversational agent based on massive large-scale datasets with multiple domains. current dialogue datasets mainly leverage online forum posts to build reply-to relationships between users, such as reddit @xcite and weibo @xcite . despite the scalability and diversity of current dialogue corpora, dialogue models pre-trained on these conversation datasets can not perform effectively when applied to a completely new domain, such as live streaming. the reason lies in the intrinsic domain gap between online-post constructed data and those required in downstream conversational tasks. even recent state-of-the-art (sota) dialogue models built upon large pre-trained language models (plms) like lamda @xcite and chatgpt foot_0 heavily rely on publicly available text-only data. these large pre-trained models' distributions remain different across domains @xcite and are distinct from those of models learning the information contained in other modalities, video as an example. video is also an important dialogue data source in the wild with great diversity. as a form of popular video-based conversations, streaming is a broadcasting scenario that transcribes and broadcasts at the same time, which involves entertainment, lifesharing, education and so on @xcite . such video-based conversations are one of the main ways human beings spread and exchange information efficiently in their daily lives and are naturally in line with the way people communicate. they are also the desired sources of dialogue datasets that are vitally significant in training large-scale dialogue models for homologous downstream virtual human scenarios, such as virtual youtubers, virtual employees, and virtual celebrities. nevertheless, works that extract data from online videos do not receive enough attention although video-sourced dialogues are more life-oriented and naturally abundant. current video-sourced spoken corpora can be separated into two main categories @xcite : scripted and unscripted. the former refers to planned dialogues such as movie and tv scripts @xcite . the latter means spontaneous conversations in real situations, for instance, the interview dataset of @xcite . however, these previous video-sourced dialogues can not meet the scale of training a satisfied chatbot, owing to the trouble of continuously obtaining and processing various kinds of videos, and troubles of extracting valid dialogue sessions from them. for example, it is challenging to build valid dialogue sessions automatically from movies without human annotators. thus, a large-scale video-sourced dialogue dataset in live streaming is essential for facilitating research in this area. the live broadcast is a typical one-to-many chat scene, which generally involves one streamer and multiple audiences. the challenge of building such a dataset lies in retrieving the reply-to relationships between the streamers and audiences. unlike post-based social media with clear links between posts and replies, the streamer's responses in the live scene have no explicit reply-to relationships with audiences' comments. to tackle the aforementioned problems, in this paper, we propose a novel and automatic videosourced dialogue-constructing method and build a large-scale personalized dialogue dataset from the live streaming domain, named livechat. it is a non-trivial work since this dataset originates from a video-based source, distinct from most previous text-sourced data. meanwhile, as far as we know, this is almost the only work that can effectively and endlessly extract dialogue sessions from videos. as illustrated in @xcite , one of the main challenges of existing open-domain chatbots is lacking a consistent personality as these agents are trained over different dialogues each with no or limited speaker information, while livechat naturally contains distinctive persona features (especially for streamers). to promote research in this field, we collect publicly available information for each streamer and add manual annotations to create the persona profiles, with individual information anonymized for privacy concerns. compared to the previous personalized dialogue datasets @xcite @xcite @xcite , our dataset provides more fine-grained persona profiles, and more importantly, the average session number of each speaker exceeds previous ones extraordinarily, as shown in moreover, live streaming is also a multi-party conversation (mpc) scene involving more than two interlocutors. an example of livechat is illustrated in figure 1 . during the streaming process, a streamer naturally has to recognize which audience to reply to. we collect public live videos and process the streamer's responses and all audiences' comments to form multiple sessions of dialogues where each session contains a streamer's response and multiple candidates of addressee comments. a reply-to-whom matching method is brought forward to accurately find the correct candidate for a streamer's response. in this way, we can leverage the reply-to-whom relationship to build datasets for two classical tasks: response modeling and addressee recognition. our proposed two classical dialogue tasks in livechat can help solve the mpc problem in a unified dataset, essential for building a practical dialogue agent in live streaming. to sum up, our main contributions are as follows: as for those spoken dialogue corpora @xcite @xcite , most are pre-scripted or manually transcribed, intrinsically difficult to scale up because of the restricted video-or audio-based sources where people can effortlessly extract valid dialogue sessions. personalized response modeling early works use explicit persona profiles from predefined information or implicit persona vectors from dialogue history to generate personality-coherent responses. explicit models use persona descriptions, attributes, or extracted profiles to learn personalized response modeling. @xcite leverages a persona knowledge base to extract predefined triples and entities in a retrieval-based dialogue system. @xcite propose an explicit persona model to generate personalized responses based on a prespecified user profile. @xcite propose a memory-augmented architecture to exploit persona information from context to generate diverse and sustainable conversations. on the other hand, implicit methods like @xcite generate consistent responses by maintaining certain features related to topics and personas, while @xcite encodes all the dialogue history of a speaker into the implicit persona. @xcite design a personality selecting module to obtain abundant and accurate persona information from the user dialogue history. in livechat, we leverage explicit persona information to maintain persona consistency. addressee recognition addressee recognition which is also named explicit addressee modeling aims at understanding who speaks to whom in a multi-party conversation. previous works mainly focus on predicting the targeting addressee of the last utterance in one conversation @xcite . later on, a who-to-whom model for predicting all the missing addressees to understand the whole conversation was introduced by @xcite . @xcite further leverages a pre-trained language model for learning this problem in a unified manner. we follow this learning paradigm, and furthermore, are able to investigate personalized addressee recognition in livechat attributed to the available persona profiles. ## experiments we train retrieval baselines for two tasks as described in section 4.1: response modeling and addressee recognition. we also investigate transfer learning of current popular generation-based models on livechat. experimental settings including training details and evaluation metrics can be found in section b. ## transfer learning to further investigate the performance of the pretrained dialogue model on our livechat, we finetune bart, chinese cdialgpt, and eva2.0 to study whether pre-trained dialogue corpora can contribute to the learning of our case. the latter two are trained on dialogue data from text-based as a supplement, we also have performed a series of experiments of in-context learning on different shots to study the influence of demonstrations. the rouge1 and bleu1 results are depicted in figure 4 . the performances keep growing as the shots gradually increase. however, when the number of demonstrations exceeds 8 shots, the performances of the llms slightly decrease due to the random manual selection of demonstrations. ## conclusion in this paper, we propose livechat, a chinese video-sourced and personalized dialogue dataset from the live streaming domain with detailed persona profiles. it maintains the largest average sessions per persona and is also the largest mpc dataset for addressee recognition since live streaming is a natural mpc scenario. this is achieved owing to the reply-to-whom matching method that enables automatically extracting dialogue sessions from live videos, while most video extraction methods can not. experimental results on two benchmark tasks show that the selected persona profiles and the larger number of average sessions per persona are advantageous in learning the speaker's personalized response and addressee decision. in addition, the comparisons between bart with other pre-trained dialogue models and llms have unveiled the distinctiveness of this video-sourced dialogue domain and we expect further research on parameters-efficient transfer learning of llms for livechat.