PGT ask: Introducing the Task of Profile Generation from Dialogues (2023)

Abstract: Recent approaches have attempted to personalize dialogue systems by leveraging profile information into models. However, this knowledge is scarce and difficult to obtain, which makes the extraction/generation of profile information from dialogues a fundamental asset. To surpass this limitation, we introduce the Profile Generation Task (PGTask). We contribute with a new dataset for this problem, comprising profile sentences aligned with related utterances, extracted from a corpus of dialogues. Furthermore, using state-of-the-art methods, we provide a benchmark for profile generation on this novel dataset. Our experiments disclose the challenges of profile generation, and we hope that this introduces a new research direction.

## introduction building conversational systems that mimic human attributes has always been a long-term goal in natural language processing (nlp). various works have attempted to leverage speaker profile information to improve the consistency of dialogue generation models @xcite @xcite . by incorporating speaker-specific characteristics, such as age, gender, personality traits, and cultural background, into the conversational systems, it is possible to create more personalized and human-like interactions. however, for dialogue systems, this sort of information is scarce and requires annotation efforts that are expensive to obtain, so there is a need to build methods that automatically gather this knowledge from dialogues. @xcite introduced personachat, a dataset comprising a collection of profile sentences (persona) that reflect each speaker's individual characteristics and personal facts. these profiles serve as a knowledge base for promoting the consistency between utterances from speakers, and various recent dialogue models have incorporated this information using diverse techniques @xcite @xcite . few works have attempted to infer profile information from personachat dialogues. @xcite restructured personachat and built the persona detection task, where the goal was to retrieve the correct persona amongst a set of distractor personas. although introducing an interesting research path, this task is limited to a set of pre-defined personas, which is not suitable for extracting profile sentences from unseen conversational data. cao et al. (2022) also manipulate per-sonachat to incorporate model-agnostic personas into the dialogue generation task. nevertheless, for the profile generation task, personachat is structured in a profile-to-dialogue manner and lacks information about the corresponding profile sentence per turn, which may become a challenge when the task becomes extracting profile information from utterances. the goal is to generate profile sentences given speaker utterances. for this, we create a new dataset, the profile generation dataset (pgdataset), which relates utterances with the respective profile sentences upon the existing personachat corpus. in figure 1 , we can observe several examples of relations between profile sentences and the corresponding speaker's utterance. notice, however, that the task is more challenging than just finding, within the dialogues, utterances that highly relate to each profile sentence. for instance, the profile sentence "i like all genres of music." is probably at the origin of the utterance "yes, sometimes i also listen to classical music.", but we cannot extract that profile sentence from that single utterance (the goal of pgtask). we framed our problem as an entailment classification task and, after human feedback, we reached the final pgdataset. finally, we provide results from three state-of-the-art models trained and evaluated in the proposed dataset. ## building pgdataset in this section, we demonstrate how we formulated our task as an entailment detection problem and describe the utilization of human experts' feedback to build a consistent dataset. ## benchmarking the pgtask in this task, the goal is to generate a profile sentence conditioned on an utterance. transformer-based decoders have achieved substantial progress in various nlp tasks @xcite . we leverage these models and rely on a causal language modeling (clm) objective for our profile generation task. more precisely, considering a sentence @xmath0, ..., w n } composed of n words, in clm, the maximum likelihood objective over s is: for our task, we are only interested in calculating the loss for the words from the profile sentence conditioned on the utterance. considering an utterance @xmath1, ..., w u m } and a profile sentence @xmath2, ..., w p k }, we redefine the objective from equation 3 : as seen in equation 4 , the loss is only calculated for the generation of the profile sentences. in the model's input, we separate the utterance and profile sentences using a special @xmath3, as it can exist more than one profile sentence, we @xmath4. ## experiments in this section, we evaluate transformer decoders on the novel dataset and provide benchmark results for future research. additional experimental details are provided in appendix b.1. ## related work recent research has focused on building personalized dialogue systems using profile information. @xcite proposed a neural conversational model to capture background information and speaking style from interlocutors in dialogue. @xcite introduced a dataset composed of personas, which are essentially 3 to 5 profile sentences describing the speaker's profile. @xcite studied how to include profile information such as age, location, and interests by explicitly incorporating this knowledge into the sequence-tosequence framework. few works have attempted to identify profile knowledge from conversational data. @xcite introduced a framework for detecting the correct profile amongst a set of distractor profiles. nevertheless, the authors do not consider the correlation between utterances and profile sentences. @xcite proposed a data manipulation method to construct distilled and diversified dialogue data containing profile information and leverage it into the dialogue generation task. ## conclusion we propose the pgtask and contribute with pg-dataset, a dataset with more than 30 000 pairs of utterances and profile sentences built with the feedback of human annotators. in addition, we train state-of-the-art models and achieve promising results in the proposed task. we hope that this new line of research will help the task of personalizing dialogues, although the task of automatically extracting profiles from dialogues is valuable by itself.