Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue (2024)

Abstract: To create conversational systems with human-like listener behavior, generating short feedback responses (e.g., “mhm”, “ah”, “wow”) appropriate for their context is crucial. These responses convey their communicative function through their lexical form and their prosodic realization. In this paper, we transplant the prosody of feedback responses from human-human U.S. English telephone conversations to a target speaker using two synthesis techniques (TTS and signal processing). Our evaluation focuses on perceived naturalness, contextual appropriateness and preservation of communicative function. Results indicate TTS-generated feedback were perceived as more natural than signal-processing-based feedback, with no significant difference in appropriateness. However, the TTS did not consistently convey the communicative function of the original feedback.

## introduction in dyadic human-human conversations, interlocutors often take turns listening and speaking. however, while one interlocutor speaks, the listener doesn't remain silent; instead, they give short feedback responses like "uh-huh", "yeah" and "wow". although these responses are known by different names (e.g., backchannels (yngve, 1970), continuers @xcite , assessments @xcite )), we follow @xcite in adopting the term feedback, since it encompasses the many communicative functions of these short responses. feedback responses are crucial for smooth turntaking and establishing common ground, i.e., people's mutual knowledge or beliefs @xcite . if the listener hasn't understood or heard what was said, they might say "huh?", "sorry?", or "what?", prompting the speaker to clarify. other responses, such as "mhm", can be used to unobtrusively signal the speaker to continue. the communicative functions of feedback are conveyed through both their lexical form and prosody, with prosody sometimes being the most important. for example, "yeah" can express agreement, disagreement or surprise depending on its prosodic realization. incorporating feedback in spoken dialogue systems for conversational agents is an active research area @xcite . many studies have focused on predicting the timing of backchannels @xcite , while others have focused on predicting their communicative function @xcite @xcite . previous studies have used signal processing to manipulate prosodic features to understand how these affect the perceived communicative functions of synthesized feedback @xcite @xcite . short feedback responses have been incorporated into unit selection text-to-speech (tts) synthesis systems by treating entire responses as units rather than concatenating diphones or phones @xcite . further, @xcite used statistical parametric speech synthesis for feedback responses. recently, @xcite introduced a tts system that can synthesize feedback without transcriptions. despite these efforts, there has been little focus on predicting the prosodic features of feedback or evaluating their contextual appropriateness. @xcite predicted prosodic features of discourse markers, which are lexically similar to many feedback responses, at the token level, but suggested future work should focus on the frame level. when it comes to evaluation, most studies (on tts in general) have primarily focused on whether the speech sounds natural, and less on whether the intended communicative function is conveyed. in this paper, we investigate to what extent feedback responses can be synthesized, using existing synthesis methods, so that they sound natu-ral and appropriate in their context, while at the same time conveying their intended communicative function. thus, our research question is not how to predict the prosodic and lexical features of feedback responses, but whether it is possible to synthesize them, given that we could make those predictions. to investigate this, we re-synthesize feedback responses in human-human u.s. english telephone conversations by transplanting their original prosody. we use two synthesis methods: (1) signal processing and (2) text-to-speech, which both have different advantages. signal processing allows for more fine-grained control of prosody (compared to the tts used here) but can degrade audio quality and introduce artifacts, while tts tends to sound more natural. in our listening tests, we let participants listen to these synthesized feedback responses in their dialogue context, and ask participants to rate their naturalness and appropriateness, as well as to assign the most likely communicative function. for comparison, we also let them rate the original feedback responses, as well as a re-synthesized monotone version, where signal processing is used to flatten the pitch and thus to remove intonation. to the best of our knowledge, this is the first work evaluating the appropriateness of the prosody of synthesized feedback responses in context. ## method to manipulate the prosodic features of feedback responses, we use two synthesis methods: a signal processing and a tts approach. we transplant the prosody of feedback responses from "listeners" in the u.s. english switchboard corpus @xcite ) -referred to as our reference speakers -onto our target voice, a female voice talent. ## conclusion and future work this paper investigated to what extent existing synthesis methods (signal processing and tts) can produce feedback that sound natural and appropriate, while at the same time conveying the various communicative functions of feedback responses. we found that the tts produced the most natural sounding feedback, but that both synthesis methods produced feedback that were deemed to be equally appropriate, given the context. however, we find that the tts method fails to convey the intended communicative function, beyond the lexical form, while the signal processing method does provide additional prosodic information, most likely due to the more fine-grained prosodic control. the implication of these findings are that, if we were to build a model that predicts the prosodic features of feedback, it may be beneficial to predict these features at the frame-level because the framelevel signal processing best preserves the intended communicative function. such a prediction model could for example extend @xcite , in which an external predictor was trained to predict intonation.