Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning (2023)

Abstract: Produced in the form of small injections such as “Yeah!” or “Uh-Huh” by listeners in a conversation, supportive verbal feedback (i.e., backchanneling) is essential for natural dialogue. Highlighting its tight relation to speaker intent and utterance type, we propose a multi-task learning approach that learns textual representations for the task of backchannel prediction in tandem with dialogue act classification. We demonstrate the effectiveness of our approach by improving the prediction of specific backchannels like “Yeah” or “Really?” by up to 2.0% in F1. Additionally, whereas previous models relied on well-established methods to extract audio features, we further pre-train the audio encoder in a self-supervised fashion using voice activity projection. This leads to additional gains of 1.4% in weighted F1.

## introduction the development of a topic constitutes an integral part of any conversation. at any time, some participant, the speaker, is said to be in control of the topic flow. simultaneously, other participants attend the talk with various degrees of verbal feedback. such feedback may take the form of simple vocalic sounds with little to no meaning, e.g., "uh-huh", or be extended to more stereotypical phrases, e.g., "wow!", "seriously?" @xcite . in the absence of such so-called backchannels, speakers have been found to tell qualitatively worse stories that they cut off abruptly and half-heartedly @xcite . not only do backchannels shape the narrative, but they also help build a feeling of understanding and trust between conversation partners @xcite . therefore, it is unsurprising that continuous research efforts are put into equipping dialogue agents with the all-so-important ability of backchannel prediction. backchannel prediction * corresponding author. the phenomenon of backchanneling is tightly linked to dynamics of turn-taking, i.e., a fundamental human ability to orderly manage speaking opportunities. listeners do not intend to claim the turn when producing a backchannel. on the contrary, they use them to signal their continued willingness to remain in the listener position @xcite . this willingness certainly depends on what the speaker has said. if the previous statement is descriptive or a personal narrative that the listener is just learning about, they might be more likely to encourage the speaker to continue through general backchannels, e.g., "uh-huh". in comparison, if the previous statement holds an opinion or some alleged general fact, the listener could agree using a specific backchannel, e.g., "that's right!", but they might as well have reason to dispute and may want to present their own view. first evidence that such a relation between dialogue acts and backchanneling exists was provided by @xcite in a preliminary theoretical investigation. building on the theoretical observations above, we propose a multi-task model, bcda, to simultaneously learn the main task of backchannel (bc) prediction and the sub-task of dialogue act (da) classification. a model trained on the task of da classification quickly learns to identify utterance types. we allow the bc prediction model to "eavesdrop on" this knowledge and use it as additional guidance to extract more relevant textual features. we demonstrate the effectiveness of this approach and compare it to the previously suggested sub-task of sentiment classification. lastly, we verify that bc prediction models benefit from self-supervised voice activity projection pre-training @xcite even in the presence of large quantities of labeled task-specific backchannel data. ## related work regarding the task of backchannel prediction, it has been a long-standing observation that audio features are superior to lexical features @xcite , with some early models avoiding the inclusion of the latter entirely @xcite . in the process, mel frequency cepstral coefficients (mfccs) have established themselves as a simple yet powerful measure of waveform representation @xcite that is only slowly superseded by hidden representations from pre-trained speech recognition models @xcite . when later models started to incorporate lexical features, they first needed to handle issues of automatic speech recognition quality @xcite and the ramifications of possible time delays in fast-paced real-time dialogues @xcite . gradually, additional gains obtainable through lexical features convinced several authors to incorporate naive word embeddings @xcite or, more recently, pre-trained language models @xcite . reflecting the multi-faceted nature of dialogue, backchannel prediction has been performed successfully in tandem with other tasks, such as sentiment classification @xcite or turntaking prediction @xcite . while results from @xcite suggest that a model trained jointly on the tasks of turn-taking, filler, and backchannel prediction outperforms its single-task variant, @xcite find that it is not necessarily turn-taking cues itself that are useful, but instead, the participants' turnmanagement willingness (e.g., willingness to listen, willingness to speak). so far, no multi-tasking approach employed dialogue acts directly. ## results we compare our results to the models of @xcite and @xcite . @xcite perform backchannel prediction in tandem with sentiment classification. initially proposed for private korean counseling data, we reimplement their approach using labels from a pre-trained english sentiment classifier. surprisingly, we ob-served no significant gains when the sub-task of sentiment classification was introduced. notably, our model outperforms all baselines by at least a margin of 3% in weighted f1. this performance increase comes at the cost of a comparably small increase in model size of about 4.5m parameters compared to bpm. in comparison, bpm replaced the traditional word2vec embedding employed by ortega with the powerful bert model. despite an additional 109m parameters, they outperformed ortega by no more than 4.6% in weighted f1. to provide insight into the origins of our improvements, we perform an extensive ablation study (see additionally, we performed significance tests for each pair of rows in ## discussion when applied to counseling data, the sub-task of sentiment classification has been previously shown to yield considerable improvements of as much as 3% in weighted f1 @xcite . the discrepancy between those findings and ours could be explained by a difference in the nature of the data. while counseling data is emotion-loaded and the counselor committed to validating those, casual conversations between strangers are quite the opposite. in our data, more than 72.6% of all backchannels have been produced in response to a completely neutral utterance. the proposed sub-task of dialogue act classification constitutes a promising extension and alternative, especially in such casual conversations where the sub-task of sentiment classification fails. our dialogue actaided multi-task model achieves an accuracy of 75.12% at the task of dialogue act classification. it is especially good at the differentiation between statement-opinion (sv) and statement-non-opinion (sd). @xcite describe sv as viewpoints that the listener could conceivably have their own (possibly differing) opinion about, while sd are non-disputable descriptives or narratives. they further state that this distinction was introduced to reflect "the different kinds of responses [they] saw to opinions (which are often countered or disagreed with via further opinions) and to statements (which more often get [...] backchannels)" (jurafsky et al., 1997, section 5.1.1). while they express doubts about whether or not this distinction is actually fruitful, we have shown that it provides indeed useful hints for the prediction of certain types of backchannels. ## conclusion we proposed a multi-task learning scheme that effectively incorporates dialogue act annotations to enhance backchannel prediction without establishing any requirement for such annotations to be available at inference time. we demonstrated its superiority over the previously suggested subtask of sentiment classification in casual settings void of strong emotions. especially, the prediction of specific backchannels could be improved by as much as 2%. moreover, we have shown that selfsupervised pre-training using voice activity projection benefits the performance of backchannel prediction models. overall, we outperform previous baselines by 3%. all experiment code is made publicly available foot_0 .