Improving Backchannel Prediction Leveraging Sequential and Attentive Context Awareness (2024)

Abstract: Backchannels, which refer to short and often affirmative or empathetic responses from a listener during a conversation, play a crucial role in effective communication. In this paper, we introduce CABP(Context-Aware Backchannel Prediction), a sequential and attentive context approach aimed at enhancing backchannel prediction performance. Additionally, CABP leverages the pretrained wav2vec model for encoding audio signal. Experimental results show that CABP performs better than context-free models, with performance improvements of 1.3% and 1.8% in Korean and English datasets, respectively. Furthermore, when utilizing the pretrained wav2vec model, CABP consistently demonstrates the best performance, achieving performance improvements of 4.4% and 3.1% in Korean and English datasets.

## introduction backchanneling is a conversational technique that involves providing short responses, such as "wow" or "uh-huh," to display attention and engagement with the speaker's utterances @xcite . @xcite has shown that timely backchanneling can enhance the speaker's storytelling ability and prolong their speaking time. therefore, it is crucial to understand the speaker's intentions and emotions and use appropriate backchannels. backchannel prediction is the task of predicting which backchannel category a competent listener will use during the current speaker's utterance. backchannels can be categorized into two main types: generic and specific @xcite . generic backchannels, including phrases such as "mm-hm" or "uh-huh," do not carry a specific meaning and instead encourage the speaker to continue their utterance. hence, generic backchannels can be employed irrespective of the conversational context. in contrast, specific backchannels encompass reactions that express empathy or agreement with the speaker's utterance, as seen in phrases like "really?" or "i see." therefore, an accurate understanding of the speaker's utterance is necessary to engage in specific backchanneling. since a conversation is a continuous interactive process, grasping the context of the entire conversation is crucial. backchannel prediction models usually use both text and audio data. however, when dealing with textual information, past models relied solely on fixed-length text inputs @xcite , which posed limitations in understanding possible contextual implications. to enhance the understanding of the current utterance, we aim to incorporate information from previous utterances. moreover, while mel frequency cepstral coefficients (mfcc) have established themselves as a near default form of audio embedding in the domain of backchannel prediction, they have long been superseded by more powerful approaches in other audio processing tasks. thus, we intend to leverage one such approach, namely wav2vec @xcite , to enhance the audio information extraction capabilities of our model. our contributions can be summarized as follows: (1) we introduce context-aware backchannel prediction (cabp), a model that considers both sequential context embeddings and attentive context embeddings to improve backchannel prediction. (2) we use the pre-trained wav2vec @xcite model to encode audio information. (3) we conduct experiments on both korean and english backchannel datasets, demonstrating performance improvements across both datasets. ## related works audio has played a crucial role since the early days of backchannel prediction. it has been modeled using various methods from simple characteristics like pitch, power and pause length @xcite to more complex spectrogram encodings like mel frequency cepstral coefficients (mfcc) @xcite . recently, even pre-trained deep convolutional neural networks have been applied @xcite . @xcite found audio features to be superior to text features while also showing that additional gains were possible when combining both. subsequently, studies have used word embeddings to encode text @xcite . later, with the appearance of pre-trained models, jang et al. (2021) adopted bert for this task. the text input length encoded using those methods varies across publications. while a few authors tie text and audio, extracting word transcriptions and acoustic features from the same time window @xcite , e.g. 1500ms, most extract text from a (much) larger window. employed units of text input include whole inter pausal units @xcite or a fixed number of words ranging from 5 to 20 @xcite . however, existing research has limited their definition of context to the most recent speaker utterance, i.e. the current utterance. ## models the proposed model architecture for context-aware backchannel prediction (cabp) is illustrated in figure 1 . cabp leverages not only the audio and current utterance but also previous utterances. it has four modules to produce the current utterance embedding (u t ), sequential context embedding (c seq ), attentive context embedding (c at t ), and acoustic embedding (a e ). these embeddings are concatenated and passed to a classifier. ## conclusion in this paper, we proposed context-aware backchannel prediction (cabp). cabp employs sequential context, summarized using a gru, and attentive context, summarized selectively using attention. experimental results show that cabp outperforms a context-unaware baseline by margins of 1.3% and 1.8% in korean and english, respectively. notably, significant performance enhancements are observed in specific backchannel categories, where the model must accurately comprehend the speaker's utterances. even greater margins could be observed when introducing the pretrained wave2vec model for audio encoding.