PRODIG y: a PRO file-based DI alogue Generation dataset (2024)

Abstract: Providing dialogue agents with a profile representation can improve their consistency and coherence, leading to better conversations. However, current profile-based dialogue datasets for training such agents contain either explicit profile representations that are simple and dialogue-specific, or implicit representations that are difficult to collect. In this work, we introduce the PRODIGy (PROfile-based DIalogue Generation) dataset, which brings diverse representations together, providing a more comprehensive profile dimension set for each speaker. This resource comprises more than 20k dialogues, sourced from movie scripts, aligned with speaker representations such as communication style, biography, personality and gender. Initial experiments with diverse baselines show that providing generative language models with these aspects of a profile, both separately and jointly, enhances models’ performance. This improvement holds true in both in-domain and cross-domain settings, for both fine-tuned and instruction-based LLMs.

## introduction dialogue agents capable of holding human-like interactions have drawn increasing interest in the fields of ai and nlp, becoming a key topic and challenge in both industry and academia. unlike task-oriented systems focusing on solving specific tasks, open-domain dialogue systems aim to discuss various topics, possibly maintaining a consistent profile in their responses @xcite . in this work, we investigate the role of profile information in open-domain dialogue systems. despite recent advancements in conversational agents, due to the continuous development of neural models @xcite @xcite @xcite , these agents often struggle to maintain coherence, resulting in inconsistent or uninformative responses. this issue adversely affects user engagement and trust @xcite . in this scenario, endowing dialogue systems with profile information is crucial for enhancing the models' ability to generate fluent, consistent, and informative responses @xcite @xcite @xcite . the concept of profile in a dialogue can refer to three aspects: personalisation, persona, and personality. personalisation refers to employing users' information to drive engagement and help them satisfy their needs @xcite . personality, on the other hand, is a psychological concept meant to capture how we behave and react to the world @xcite . the notion of persona can have diverse meanings in literature. in this work, we will stick to the definition provided by @xcite , according to which the persona is the character that an artificial agent plays during conversations and includes elements such as background facts, language, and interaction style. several approaches have been explored to integrate persona information into dialogue generation @xcite @xcite @xcite @xcite @xcite @xcite . however, these methods are typically sporadic and disjointed, addressing only one persona dimension at a time, either through an explicit representation (a few simple, dialogue-specific sentences about the user) or an implicit representation (a collection of the user's previous dialogues) that is challenging to obtain. consequently, these approaches fail to model the complex nature of human communication, which is influenced by the interaction of multiple aspects. in this paper, we investigate the impact of diverse profile representations in the development of dialogue systems by comparing and benchmarking them. to this end, we introduce a new dataset, named prodigy (profile-based dialogue generation) foot_0 , that combines existing profile representations (i.e., language style, gender, personality) with novel and more complex representations of the persona, such as biographies. prodigy is created starting from the cornell movie dialogs corpus @xcite , which includes movie script dialogues, and adopting the character ids and binary gender labels from the original corpus. this approach avoids privacy concerns related to employing real user data and simplifies the distribution. moreover, the dataset has been aligned with external resources containing characters' profiles, and it can be further expanded by adding new scripts or scripts in other languages. figure 1 illustrates an example from prodigy, in which the dialogue is aligned with the target speaker's profile representation. we validated prodigy by benchmarking it with diverse baselines. in particular, we employed either fine-tuning or instruction prompting, and tested a range of configurations varying the profile dimensions, both in-domain and cross-domain. evaluation involved both automatic metrics and human assessment. as for automatic metrics, in-domain experiments show that fine-tuning lms with diverse profile aspects significantly improves their predictive capabilities. additionally, instructing non-fine-tuned llms with profile information also improves their performance. in cross-domain settings, prodigy-based models show better generalisation than those trained on other persona-based resources. in human evaluations, evaluators had a tendency of favouring generic responses for broader applicability. however, when responses were consistent with both profile and dialogue they were clearly preferred. profile information proves beneficial especially in dialogues with limited context, and when disclosed to evaluators, profile-based responses are deemed more appropriate. we discuss three main topics relevant to our work: (i) theories on persona and personality (ii) available datasets for persona-based generation and (iii) persona and personality based models. persona and personality our communication style is closely related to social status, gender, and motivations, and offers insights into our psychological state @xcite . these aspects are closely related to the concepts of persona and personality, which fall under the more general concept of profile @xcite . persona can be defined as the character that an artificial agent acts during a conversation and it is a combination of identity factors, such as background facts, language use, and communication style @xcite . personality is a psychological concept grasping different behaviours, feelings and way of thinking @xcite . it can be formalised using theoretical frameworks called trait models, such as big five @xcite and the myers-briggs type indicator (mbti) @xcite . persona-based dialogue datasets several dialogical datasets contain a persona representation, many of which were collected starting from social media such as twitter, reddit, weibo or kialo. however, these datasets have various limitations. they may encounter challenges related to ephemer-ality @xcite ; they can include short conversations, thus failing to fully represent real dialogues @xcite ; they can rely only on users' dialogue history @xcite ; they may include only generic persona representations such as gender or age @xcite ; finally, they may not consider linguistic style, being based on controlled and redacted conversations @xcite . other resources were collected from television series transcripts @xcite , but are small and not sufficient to train open-domain dialogue models. one of the most widely used persona-based datasets is persona-chat @xcite , collected in a controlled crowd-sourcing environment. however, it provides a generic fact-based persona representation (e.g. "i just got my nails done") specific to single dialogues and leaving out complex aspects, such as linguistic style or biographical history. persona/personality based dialogue models several approaches have been investigated to condition the dialogue generation through the persona information. on the one hand, diverse studies were based on resources using users' past dialogues to represent the persona @xcite @xcite . on the other hand, a line of research has been built on persona-chat. various approaches employed this dataset to train persona-based models in under-resourced scenarios @xcite @xcite . other methodologies used persona-chat to test commonsense expansion @xcite , mutual perception persona @xcite , or enriching persona information through background stories @xcite . however, these studies present the same limitations of the resources they rely on. regarding the personalitydriven generation, few seminal studies have been conducted @xcite @xcite . however, they leave the interactions between personality and persona unexplored. ## construction of the prodigy dataset to build the prodigy dataset, we started from the cornell movie dialogs corpus, a dataset of dialogues from movie scripts that includes metadata about movie genre, release year and characters' gender @xcite . the dialogues in the cornell movie dialogs corpus are between two actors and have an average length of 4 turns. the reason for using this resource as a starting point is three-fold: (i) data persistency and accessibility: it eliminates privacy issues or ephemerality problems @xcite that would arise from collecting data from real users and, therefore, facilitates the distribution of prodigy to the research community; (ii) data enrichment: it is possible to enrich prodigy with the profile of movie characters through the alignment with external web resources containing information about characters and movie plots; (iii) data expansion: it leaves room for further development/extension; for example, it can be aligned with similar movie script resources in other languages or new movie scripts. below, we outline the profile representations and detail the methodology employed to annotate the characters within the dataset. ## automatic evaluation in this section, we describe the metrics and experiments for the validation of our resource. ## human evaluation besides the automatic evaluation, we also run an human evaluation study to validate prodigy. this evaluation involved six subjects, comprising four phd students in computer science and two msc students in data science. evaluators received 100 dialogues each, 50 with profile information disclosed and 50 without profile disclosure, so to enable an assessment of profile information's impact on judgements. we focused on output generated using top-p decoding by four models trained during inter-character experiments: the model trained on dialogues only and the models trained with one profile dimension. evaluators ranked five possible responses for each dialogue, including the gold response used as a control condition, on a scale from 1 (most likely) to 5 (least likely) based on perceived likelihood of being the target speaker's response. in total, we collected 3000 evaluations. subsequently, we conducted post-hoc qualitative interviews with the evaluators. ## conclusion in this paper we introduced prodigy, a new dataset of movie dialogues aligned with characters' profile information, i.e. personality type, gender, biography, and a collection of speakers' dialogues, useful for inferring their vocabulary and language style. derived from movie scripts, prodigy also mitigates privacy concerns associated with real user data. to validate this resource, we conducted several experiments using diverse baselines, both via fine-tuning and instruction prompting. results indicate that including profile information in both approaches improved models' performance. moreover, the cross-domain experiments showed that prodigy-based models exhibit better generalisation than those trained on similar resources. results from the human evaluation showed that, despite a preference for generic responses due to their broader applicability, responses consistent with both profile and dialogue are clearly favoured. moreover, the results highlight the value of incorporating profile information, especially when speaker's information provided within the dialogue context is limited.