P ersona CLR : Evaluation Model for Persona Characteristics via Contrastive Learning of Linguistic Style Representation (2024)

Abstract: Persona-aware dialogue systems can improve the consistency of the system’s responses, users’ trust and user enjoyment. Filtering nonpersona-like utterances is important for constructing persona-aware dialogue systems. This paper presents the PersonaCLR model for capturing a given utterance’s intensity of persona characteristics. We trained the model with contrastive learning based on the sameness of the utterances’ speaker. Contrastive learning enables PersonaCLR to evaluate the persona characteristics of a given utterance, even if the target persona is not included in training data. For training and evaluating our model, we also constructed a new dataset of 2,155 character utterances from 100 Japanese online novels. Experimental results indicated that our model outperforms existing methods and a strong baseline using a large language model. Our source code, pre-trained model, and dataset are available at https://github.com/1never/PersonaCLR.

## introduction persona-aware dialogue systems can improve the consistency of the system's responses @xcite , users' trust in the system @xcite , and user enjoyment @xcite . in constructing persona-aware dialogue systems, automatic estimation of persona characteristics' intensity is important in two ways. first, if we can detect low-intensity utterances of persona characteristics, inappropriate system responses can be prevented. second, the automatic measure helps construct the persona's sample utterance set. two methods for constructing persona-aware dialogue systems include the following: (1) using persona descriptions @xcite @xcite @xcite and (2) sample utterances @xcite @xcite . in the method using samples, if we can filter out samples not matching the persona, the system's performance will improve. this paper presents an evaluation model for persona characteristics via contrastive learning of linguistic style representation (personaclr), which can measure a given utterance's intensity of the target persona's characteristics. in this paper, the term persona indicates both real-life individuals and fictional characters. personaclr receives the evaluation target's utterance and a target persona's set of sample utterances and then returns a score indicating the target persona characteristics' intensity. the model is trained by contrastive learning based on the sameness of the utterances' speaker. contrastive learning enables personaclr to evaluate the persona characteristics of a given utterance, even if the target persona is not included in training data. to the best of our knowledge, there are no public datasets for the training and evaluation of models to assess the intensity of persona characteristics in utterances. we constructed and published two datasets: the naro utterance dataset (narou), containing 2,155 characters' utterances from 100 japanese online novels, and an evaluation dataset based on dialogue scenarios between a user and a character in narou. we use the dataset to train and evaluate personaclr in the experiments. the creation of these datasets contributes to the advancement of research on dialogue systems that mimic fictional characters. additionally, this dataset can be utilized for speaker identification tasks @xcite @xcite and for building persona-aware dialogue systems using sample utterances @xcite . the evaluation dataset for this task was also constructed and published. this study contributes the following: (1) a new model for assessing the intensity of target persona characteristics in a given utterance that does not require retraining or fine-tuning, even if the persona is not included in the training data; (2) a new open dataset including over 2000 character utterances from 100 japanese online novels and annotated human-character dialogue scenarios; and (3) a demonstration of the effectiveness of our model using a comparison with existing methods and a strong baseline involving chatgpt. ## personaclr our task is to estimate the intensity of the characteristics of a target persona c within a given utterance x. the existing sota model, psprob @xcite , is based on a multi-class classification model classifying which character uttered the given input utterance. therefore, when evaluating the persona characteristics of a new character, psprob must collect not only the reference utterance sets for the evaluation of the target persona as well as the utterance sets of non-target personas, then training the model from scratch. we propose personaclr, which does not require the retraining and utterance sets for non-target personas. personaclr uses contrastive learning to distinguish whether a given utterance and a set of utterances come from the same persona. note that personaclr does not evaluate the similarity of any given utterance to utterances in the training data; rather, it observes the similarity between the given utterance and a reference set of utterances. therefore, our model can evaluate utterances of characters that are not included in the training data without requiring retraining, and it requires only a small number (≥ 20) of references. we define two embedding vectors of the same speaker's utterances as positive pairs and two vectors by different speakers as negative pairs for contrastive learning. however, because utterances do not necessarily reflect persona characteristics, one of the pair's embedding vectors is obtained from a set of utterances rather than from a single utterance. in contrastive learning, the model distinguishes whether an utterance and a set of utterances are from the same persona. ## experiment figure 2 shows a summary of experimental procedure used to evaluate the effectiveness of per-sonaclr. in this experiment, we constructed and used two types of datasets: narou and the evaluation dataset. the narou dataset consists of utterances from novels. however, there is the concern that, compared to utterances in novels, utterances in a dialogue between a user and a persona-aware dialogue system differ in length and tendency. to address this concern, we created the evaluation dataset. ## ablation study we conducted experiments using ablation models. the following two models were compared: a model that randomly samples from a pool of utterances instead of using bm25 to construct a set of reference utterances (w/o bm25), and a model that uses a single utterance as a reference that is the most similar to the target utterance by bm25 instead of the utterance set (w/ single ref.). the results for each character in the ablation models using spearman's rank correlation coefficient are shown in ## visualization the embedding vector h of the utterance obtained by the transformer encoder in personaclr reflects persona characteristics, and the same speaker's utterances are closely placed in the vector space. we figure 3 shows the embedding results of the 17 characters, that is, all utterances encoded by personaclr and dimension reduction by uniform manifold approximation and projection (umap) @xcite . we found that the manner of speaking determines the position. characters who use polite language (e.g., rudeus, roxy, olivia, luxion, keith) are placed in the upper area, those who do not use polite language (e.g., paul, subaru, leon) in the lower, and those who use female language (e.g., myne, tuuli, effa, catarina) are placed on the left. we can also observe a tendency for characters in the same novel to be placed close. for example, myne, tuuli, and effa, as well as catarina and keith, are located near each other due to common use in their utterances of novel-specific terms and character names. in contrast, leon and luxion, who is from the same novel, are positioned far apart, indicating that they are embedded with more emphasis on the manner of speaking than on being from the same novel. ## conclusion we proposed a novel model for evaluating a given utterance's intensity of persona characteristics and constructed the naro utterance dataset (narou) for training our model. the proposed model employs contrastive learning, and experimental results show that our model outperforms existing methods. future work includes constructing personaaware dialogue systems by applying personaclr and evaluating its performance experimentally. we also plan to extend personaclr to be able to evaluate on a context-response basis rather than an utterance basis. this extension is expected to further improve the response performance of the system.