Transition Relevance Point Detection for Spoken Dialogue Systems with Self-Attention Transformer (2025)

Abstract: Most conventional spoken dialogue systems determine when to respond based on the elapsed time of silence following user speech utterances. This approach often results in failures of turn-taking, disrupting smooth communications with users. This study addresses the detection of when it is acceptable for the dialogue system to start speaking. Specifically, we aim to detect transition relevant points (TRPs) rather than predict whether the dialogue participants will actually start speaking. To achieve this, we employ a self-supervised speech representation using contrastive predictive coding and a self-attention transformer. The proposed model, TRPDformer, was trained and evaluated on the corpus of everyday Japanese conversation. TRPDformer outperformed a baseline model based on the elapsed time of silence. Furthermore, third-party listeners rated the timing of system responses determined using the proposed model as superior to that of the baseline in a preference test.

## introduction turn-taking is a fundamental aspect of speech communication, yet spoken dialogue systems often fail to take turns at the right moments. this awkward behavior arises from conventional systems that determine when to speak based solely on the elapsed time of silence after user utterances. consequently, smooth communications between humans and machines become challenging @xcite . in this study, we aim to detect when a spoken dialogue system is permitted to start speaking, in contrast to conventional models that predict whether speaker shift will actually occur. specifically, we investigate the transition relevant point (trp) detection during a user utterance. to achieve this, we employ a self-supervised speech representation and a self-attention transformer. the proposed model receives a single-channel audio stream of user speech and infers whether it is preferable for the spoken dialogue system to take a turn at each time frame. most prior studies on turn-taking focused on batch inference conducted at the end of speech. additionally, previous research commonly relied on linguistic features extracted from transcriptions. the proposed model is notable for performing consecutive inference without relying on linguistic features. we avoid transcription to eliminate any additional latency introduced by automatic speech recognition (asr) and perform consecutive inference in the frame unit. these characteristics differentiate our approach from earlier studies on turntaking @xcite @xcite @xcite @xcite @xcite @xcite . notably, trp detection differs from voice activity projection (vap) @xcite . trp detection infers whether a listener is allowed to start speaking without specifying who will actually take the turn. conversely, vap predicts whether each interlocutor will speak next, thereby identifying the likely next speaker. hence, trp detection and vap address distinct tasks. importantly, a trp does not necessarily guarantee a speaker change; the present speaker may keep talking, or a listener may simply produce a backchannel response. moreover, vap does not distinguish whether the subsequent speech is a backchannel or turn. for the effective use of a vap model for turn-taking, another model is required to predict whether it is appropriate to make a backchannel or take a turn. from the data perspective, a vap model learns from whether each interlocutor actually started speaking in the conversational speech data, rendering it self-supervised. conversely, a trp detection model requires spoken dialogue data with trp labels for training. similar to vap, most previous studies on turn-taking focused on predicting who would actually take the turn @xcite @xcite @xcite @xcite @xcite . prior studies have used the term "trp" to refer to different tasks. @xcite analyzed the effect of pause duration on the probability of a speaker shift, assuming that the temporal position of the turn end is known. gervits et al. (2020) focused on predicting the completion point of the current speech utterance from its partial transcription. @xcite used a trp detection model as the first step in a two-step approach to actual turn-taking prediction. the definition of trps in @xcite aligns with ours; however, their inference was performed in batch and relied on linguistic features, in contrast to our approach. ## method we introduce fundamental elements and formalize trp detection. an inter-pausal unit (ipu) is a continuous speech utterance without an intermediate pause. furthermore, a pause (i.e., a period of time during which the speaker is not talking) may occur between adjacent ipus. ipus and pauses formulate a turn construction unit (tcu), i.e., a unit of speech that makes up a turn in the conversation, as illustrated in figure 1 . a trp is located at the end of a tcu, where a turn shift (i.e., speaker change) is acceptable. conversely, the end of an ipu in the middle of a tcu is referred to as a non-trp. ## model this study proposes a trp detection model with a self-attention transformer (trpdformer). tr-pdformer consists of a pre-trained contrastive predictive coding (cpc) model @xcite , a causal self-attention transformer, and a detection head, as illustrated in figure 3 . the cpc model yields speech representations at a sampling rate of 100 hz. a subsequent convolutional layer downsamples the representations to 20 hz before passing them to the transformer. a causal attention mask is applied in the transformer. the detection head is a time-distributed dense layer with a logit function. the entire model yields the transition relevance score in intervals of 20 ms. the cross-entropy loss is employed for training. this model structure is identical to that of the stereo vap model, except for the inevitable modifications that arise from changes in inputs and outputs. specifically, the input of the proposed model is single-channel audio, whereas the vap model receives stereo audio. consequently, the proposed models do not include cross-attention transformers between channels. the head is altered to yield a single score, namely r(t), instead of stereo voice activities. notably, the proposed and vap models address different tasks: trp detection and voice activity prediction, respectively. therefore, the vap model is not well-suited for trp detection (see also section 4.2). we employ a model based on voice activity detection (vad) as a baseline. this model detects a trp when the pause duration exceeds a certain period of time after the ipu ends. ## data this study utilizes a large-scale spontaneous conversational speech corpus known as the corpus of everyday japanese conversation (cejc) @xcite . cejc consists of 200 hours of speech data recorded during daily conversations involving 862 distinct speakers. the annotations contain long utterance units (luus). an luu is a basic unit for spoken dialogue that is determined considering syntactic, pragmatic, and prosodic aspects @xcite . notably, an luu coincides with a tcu in most cases. in fact, turn-taking frequently occurs at luu boundaries @xcite . even in the case where a turn-switch does not occur, the luu endpoint can mostly be considered as a trp @xcite . therefore, considering the completion points of luus as surrogate labels for trps is reasonably valid. this approach is widely accepted in prior studies @xcite @xcite . a few exceptions to this identification of luus and tcus were noted by @xcite ; @xcite . a key distinction between luus and tcus lies in their use of backchannels. an luu can function as a backchannel and therefore may not constitute a full turn. however, the luu labels provide advantages for incorporating a trp detection model into spoken dialogue systems; this is because the systems need to determine when to speak, regardless of whether the current user's speech is a turn or a backchannel. hence, we treat the end of an luu as a point to be detected and refer to it as a "trp." ## experiments we conducted model training and evaluation in ipu units. for each ipu, all precedent ipus within the same luu were included in the data unit as the ipu reception period fixed ipu reception period variable figure 5: reception period. a fixed reception period ends when a certain interval s elapses after the ipu offset, whereas a variable reception period ends at the subsequent ipu onset. ## conclusion this study addressed the continual trp detection from acoustic features to determine for spoken dialogue systems. the proposed model, trpdformer, outperformed the vad-based model when trained and evaluated on cejc in units of ipus. furthermore, a preference test conducted with third-party listeners revealed that the superiority of the proposed model was perceptible to humans. finally, we discuss future direction. the computational delay of the proposed model in stream processing and its robustness against noise and reverberation should be evaluated in a manner similar to previous studies on vap @xcite . the proposed model should also be investigated in different domains other than everyday conversation, as well as in languages other than japanese. creating a new dataset of human-system spoken dialogues in quiet and anechoic conditions with trp labels is a promising direction, though it would require considerable investment. finally, evaluating a spoken dialogue system incorporating the proposed model remains an important direction for future research.