Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection (2025)

Abstract: In human conversations, short backchannel utterances such as “yeah” and “oh” play a crucial role in facilitating smooth and engaging dialogue.These backchannels signal attentiveness and understanding without interrupting the speaker, making their accurate prediction essential for creating more natural conversational agents.This paper proposes a novel method for real-time, continuous backchannel prediction using a fine-tuned Voice Activity Projection (VAP) model.While existing approaches have relied on turn-based or artificially balanced datasets, our approach predicts both the timing and type of backchannels in a continuous and frame-wise manner on unbalanced, real-world datasets.We first pre-train the VAP model on a general dialogue corpus to capture conversational dynamics and then fine-tune it on a specialized dataset focused on backchannel behavior.Experimental results demonstrate that our model outperforms baseline methods in both timing and type prediction tasks, achieving robust performance in real-time environments.This research offers a promising step toward more responsive and human-like dialogue systems, with implications for interactive spoken dialogue applications such as virtual assistants and robots.

## introduction in natural human conversations, short backchannels, such as "yeah" and "right," play a crucial role in facilitating smooth and engaging interactions @xcite @xcite . they function as feedback mechanisms, signaling attentiveness, understanding, and agreement without interrupting the speaker. accurate prediction and generation of backchannels in spoken dialogue systems are essential for creating more natural and human-like interactions @xcite @xcite . although some definitions of backchannels include longer and more linguistic tokens such as "i see," this work focuses on short tokens that are frequently and dynamically used by listeners. backchannel modeling remains a significant challenge due to their subtle and context-dependent characteristics. given the dynamic nature of backchannels, it is essential to predict them on a frame-by-frame basis and in real-time for live spoken dialogue systems, as depicted in figure 1 . however, most previous studies have focused on utterance-based systems, or, in the case of framebased systems, have artificially balanced the test data by reducing non-backchannel samples. this data manipulation introduces a discrepancy between the training models and real-world systems. consequently, for practical applications, it is necessary to develop models capable of real-time, continuous frame-wise prediction and evaluate them using unbalanced, real-world datasets. transformer-based architectures have emerged as powerful tools for a broad range of sequential prediction tasks, such as language modeling and speech recognition. among these, the voice activity projection (vap) model, a transformerbased architecture, has shown its efficacy in predicting future voice activity within dialogues @xcite . since voice activity is closely intertwined with turn-taking dynam-ics and backchannel behaviors, the vap model seems to have the potential to enable more accurate backchannel prediction. furthermore, as a previous study implemented the real-time vap model @xcite , a vap-based backchannel prediction model could serve as a promising candidate for real-time backchannel generation systems. in its initial application, the vap model was employed for backchannel prediction in a zero-shot manner. however, this approach exhibited a sensitivity to threshold selection and depended on a balanced dataset. these findings suggest that while the vap model shows promise for backchannel prediction, further research is necessary to refine its training methodology for improved performance. in this study, we propose a novel approach to backchannel prediction by utilizing the vap model as a foundational framework. we first train the vap model on a large corpus of general dialogue data to capture the fundamental patterns of conversational dynamics. subsequently, we fine-tune the model on a specialized dataset focused on backchannel prediction. this two-stage training process is analogous to the pre-training and fine-tuning paradigm employed by models like bert, aiming to demonstrate the vap model's versatility as a generalpurpose base model. moreover, to the best of our knowledge, our model is the first to predict both the timing and type of backchannels in a continuous and real-time manner. the contributions of this paper are twofold. note that the source codes and trained models are publicly available foot_0 . ## related work effective backchannel generation necessitates accurate prediction of three key elements: temporal placement (timing), linguistic form (type), and prosodic patterns. the majority of prior studies have focused exclusively on predicting the timing and types of backchannels. the definition and functions of backchannel types have been explored in conversational analysis and linguistic studies @xcite @xcite . despite the critical role of prosody in entrainment, existing research on this remains scarce @xcite . this review primarily summarizes the current state of research on predicting the timing and form of backchannels. before recent advances in machine learning technologies, backchannel prediction models were primarily based on hand-crafted features with heuristic rules or simpler models @xcite @xcite @xcite @xcite . with the advent of dataset creation paradigms and machine learning advancements, deep learning models began to be employed for backchannel prediction. initial models were built using long-short-term memory (lstm) networks @xcite @xcite , while more recent approaches leverage transformer-based models @xcite . most previous studies focused on timing prediction, with some also addressing type prediction. the most conventional approach involves framing the prediction task as a three-class classification problem: non-backchannel, continuer, and assessment @xcite , as described in section 4.2. the present work adopts this three-class classification scheme for both timing and type prediction. other research has explored a four-type classification system, encompassing non-backchannel, continuer, understanding, and empathy backchannels @xcite . another study introduced a five-type classification for single continuer, double continuers (e.g., "yeah yeah"), triple continuers, assessment, and non-backchannel @xcite . furthermore, a different approach proposed a two-step classification method where the first model predicts the timing, followed by a second model that determines the type @xcite . in terms of prediction unit, utterance-based or continuous, utterance-based models tend to incorporate linguistic features such as word embeddings @xcite . conversely, previous continuous models were generally restricted to using prosodic features @xcite ## dataset we employ two types of datasets: one specialized for backchannel prediction and the other for pretraining the proposed model. note that all the dialogue datasets were in japanese. ## proposed method in this section, we begin by explaining the voice activity projection (vap) model, which serves as the foundational model and is trained to predict future voice activities by using the largest spoken dialogue dataset explained in section 3.2. following that, we discuss how to adapt a pre-trained vap model for use in the current backchannel prediction task, using the data introduced in section 3.1. ## experiment to evaluate the effectiveness and applicability of the proposed method, we conducted the four experiments described below. ## integration with a cg agent based on the results from the previous experiment, we have developed a vap-based real-time backchannel prediction system and implemented it with a conversational cg agent foot_4 . figure 10 illustrates the system in operation with the agent, as well as its real-time gui visualization tool. note that since this backchannel generation system operates independently of the interface, it can be applied to various other interfaces, including physically embodied robots. we plan to conduct a user dialogue experiment with this dialogue system to evaluate the naturalness and effectiveness of the backchannel generation system. ## conclusion this paper presents a method for real-time, continuous backchannel prediction using a fine-tuned voice activity projection (vap) model. our approach combines pre-training on a larger dialogue dataset with fine-tuning on a specialized backchannel dataset, leveraging the vap architecture's generalizability. experimental results showed that our two-stage and multi-task training process improves the model's ability to predict both the timing and type of backchannels, demonstrating its adaptability to real-world, unbalanced data. we also validated the model for real-time use, confirming its effectiveness in live systems without compromising accuracy, especially for continuer backchannels. the results also highlight the need for longer input contexts for accurate assessment backchannel predictions. this study represents a step forward in enhancing conversational agents' interactivity by providing a more human-like and responsive backchanneling system. future research will concentrate on evaluating the effectiveness of the backchannel generation system through user dialogue experiments, as well as further refining backchannel prediction for more complex conversational contexts.