Synthesising Personality with Neural Speech Synthesis (2023)

Abstract: Matching the personality of conversational agent to the personality of the user can significantly improve the user experience, with many successful examples in text-based chatbots. It is also important for a voice-based system to be able to alter the personality of the speech as perceived by the users. In this pilot study, fifteen voices were rated using Big Five personality traits. Five content-neutral sentences were chosen for the listening tests. The audio data, together with two rated traits (Extroversion and Agreeableness), were used to train a neural speech synthesiser based on one male and one female voices. The effect of altering the personality trait features was evaluated by a second listening test. Both perceived extroversion and agreeableness in the synthetic voices were affected significantly. The controllable range was limited due to a lack of variance in the source audio data. The perceived personality traits correlated with each other and with the naturalness of the speech. Future work can be making a chatbot speak in a voice with a pre-defined or adaptive personality by using personality synthesis in speech together with text-based personality generation.

## discussion this pilot study shows that using the personality traits to control the perceived personality of a synthetic voice is feasible with a modern dnn / neural vocoder system. readers are invited to listen to sample natural and synthetic speech from https://cereproc.s3-eu-west-1 . two limitations have compromised the results of the study: 1) the corpus used as a basis for this experiment was comprised of voices originally selected for being extrovert and agreeable, which can be seen from figure 1 and figure 3 . with a machine learning approach this means when targets are set within outlying regions the system has to extrapolate the results which leads to unnatural results as they are not based on actual observations. this is shown for agreeableness where lower target scores (unseen in the data) generate stimuli rated lower for naturalness. in future work it will be important to source a corpus with a much wider variation in perceived big five personality traits. such collinearity means it is difficult to produce stable results. the confounding effect is possibly intensified by using an average voice built with a limited amount of source data. ## conclusion and future work to summarise our findings: 1) the prototype system showed a big five trait could be learned and controlled, though control may be limited in the