Fundamental Exploration of Evaluation Metrics for Persona Characteristics of Text Utterances (2021)

Abstract: To maintain utterance quality of a persona-aware dialog system, inappropriate utterances for the persona should be thoroughly filtered. When evaluating the appropriateness of a large number of arbitrary utterances to be registered in the utterance database of a retrieval-based dialog system, evaluation metrics that require a reference (or a “correct” utterance) for each evaluation target cannot be used. In addition, practical utterance filtering requires the ability to select utterances based on the intensity of persona characteristics. Therefore, we are developing metrics that can be used to capture the intensity of persona characteristics and can be computed without references tailored to the evaluation targets. To this end, we explore existing metrics and propose two new metrics: persona speaker probability and persona term salience. Experimental results show that our proposed metrics show weak to moderate correlations between scores of persona characteristics based on human judgments and outperform other metrics overall in filtering inappropriate utterances for particular personas.

## introduction maintaining utterance quality is important for commercial dialog systems. to achieve better quality, methods of filtering inappropriate utterances have been proposed from the perspectives of offensive language @xcite , grammar, topics @xcite , discourse relation @xcite , and so on. in addition to these perspectives, we need a filter for personas of dialog systems. persona-aware dialog systems are important in that having a consistent persona makes a dialog system believable @xcite and entertaining @xcite . throughout this paper, we use the term persona to indicate individuals such as real-life people and fictional characters. in ad- ## related work since the release of the persona-chat dataset @xcite , many more studies have been conducted on persona-aware utterance generation @xcite @xcite , including studies by the 23 teams that participated in the convai2 competition @xcite . the persona-chat dataset was created by crowdworkers who were asked to converse as the personas described in the given descriptions. each description consisted of five sentences on average, such as "i am a vegetarian," "i like swimming," "my father used to work for ford," "my favorite band is maroon5," and "i got a new job last month, which is about advertising design." in this manner, facts about the personas are described. however, the linguistic styles of the personas were not focused on. linguistic style is also an important aspect of persona-aware utterances. for example, big five personalities (mairesse and walker, 2007), gender, age, and area of residence @xcite can affect the linguistic styles of the utterances. in text style transfer, transfer success is often measured by transfer accuracy @xcite ## existing metrics this section introduces the existing evaluation metrics for persona-aware utterances that can be computed without the references being tailored to the evaluation targets. ## proposed metrics we propose a trained persona speaker probability (psprob) metric and an untrained persona term salience (ptsal) metric. ## conclusion we investigated the performances of existing metrics and new metrics (namely psprob and ptsal) to find metrics that we can use to capture the intensity of persona characteristics and we can compute without the references tailored to the evalua-