Large Language Models Know What To Say But Not When To Speak (2024)

Abstract: Turn-taking is a fundamental mechanism in human communication that ensures smooth and coherent verbal interactions. Recent advances in Large Language Models (LLMs) have motivated their use in improving the turn-taking capabilities of Spoken Dialogue Systems (SDS), such as their ability to respond at appropriate times. However, existing models often struggle to predict opportunities for speaking — called Transition Relevance Places (TRPs) — in natural, unscripted conversations, focusing only on turn-final TRPs and not within-turn TRPs. To address these limitations, we introduce a novel dataset of participant-labeled within-turn TRPs and use it to evaluate the performance of state-of-the-art LLMs in predicting opportunities for speaking. Our experiments reveal the current limitations of LLMs in modeling unscripted spoken interactions, highlighting areas for improvement and paving the way for more naturalistic dialogue systems.

## introduction when humans interact verbally, they avoid speaking simultaneously and take turns to speak and listen, a process essential for mutual understanding and smooth communication @xcite . unlike in formal settings with pre-assigned roles, participants in everyday conversations decide when to speak or listen on a perturn basis @xcite . this local management system hinges on conversationalists' ability to recognize and anticipate so-called transition relevance places (trps), which are points in a speaker's utterance that signal appropriate opportunities for the listener to speak. in other words, at a trp, listeners have the opportunity, but are not obligated, to speak. importantly, interlocutors anticipate and recognize trps using various lexico-syntactic, contextual, and intonational cues @xcite . the ability to predict trps is therefore crucial for artificial conversational agents, as it enables them to take turns and provide verbal feedback signals with socially appropriate timing. recent advances in large language models (llms) have generated interest in improving turn-taking capabilities in spoken dialogue systems (sds) using these models @xcite . specifically, approaches like turngpt and rc-turngpt introduce probabilistic models to predict trps using contextual and speaker-identity information @xcite . however, most methods struggle to handle unscripted spoken interactions, often resulting in long silences or poorly timed feedback @xcite . there are two critical issues with the current approaches. first is the optimistic assumption that llms trained predominantly on written-first language can learn the complex dynamics of spokenfirst language @xcite @xcite , which are distinct in structure and use (e.g. drieman 1962; @xcite . a second, more fundamental issue is that, while trps at speaker switches can be identified unambiguously, it is challenging to clearly identify trps within turns. this means that we have no 'ground truth' data about these 'silent' trps, where a listener could have responded but chose not to. in this work, we address these issues by first developing a novel and unique empirical dataset 1 based on human responses that allows us to identify within-turn trps in natural conversation. second, we use this dataset to establish the baseline performance of state-of-the-art llms to predict within-turn trps. this ability is vital for future dialogue systems to appropriately time their turns, use strategically timed silences to convey social cues, and maintain conversational flow. ## theoretical background when humans verbally interact with each other, they avoid speaking at the same time and instead take turns speaking and listening @xcite . this allows them to respond sequentially to each other's utterances and facilitates mutual comprehensibility @xcite . however, the alternation between speaker and listener roles in natural conversation is not predetermined (e.g., allotted time slots in court proceedings). rather, it is locally managed by speakers themselves on a per-turn basis @xcite . but how can participants in conversations manage to avoid speaking at the same time, or having long silences in which they are waiting for one another? conversationalists follow rules imposed by a universal turn-taking model, proposed by @xcite (see also @xcite . this system crucially depends on the notion of the transition relevance place (trp), which is an opportunity in the current speaker's utterance at which a listener can, but is not obligated to, take over the role of speaker. even short feedback-like turns, such as 'hmm', known as backchannels (yngve, 1970) or continuers @xcite , are precisely timed to occur at trps. importantly, trps are not a function of a speaker's intentions but a consequence of the turn-taking mechanism itself. this distinguishes speech at trps from interruptions or barge-ins, which can occur at any point in a speaker's utterance and are noticeable precisely because their timing does not meet normative expectations. in the turn-taking literature, a crucial distinction is made between a turn, which is the entire contribution by one speaker, and a turn construction unit (tcu), which ends at a trp. since a listener is not required to speak at every trp, a turn can consist of multiple tcus, with potentially multiple trps occurring within a turn. this implies that, by definition, turn-switches can only occur at a trp. while it is relatively straightforward to identify turn-final trps -where the listener takes over -it is challenging to reliably locate turn-medial trps (where the speaker continues) due to the absence of observable cues suggesting the presence of a trp. to function, the turn-taking system requires listeners to not only recognize but also anticipate the occurrence of a trp in the current speaker's contribution @xcite . listeners process various turn-taking cues -primarily lexico-syntactic @xcite , but also contextual and intonational cues @xcite -incrementally to predict upcoming trps, ensuring that their responses are normatively timed. this anticipatory ability is essential not only for successful human interactions but also for designing artificial conversational agents capable of a) taking over the floor at the right moment and b) providing verbal feedback with correct timing. beyond the basic mechanisms, cultural variations also play a role in how turn-taking unfolds. while the fundamental mechanisms of turn-taking, such as the cues for taking or passing on turns @xcite , are largely universal, cultural norms can shape the timing and style of these transitions @xcite . therefore, understanding both the culture-invariant and culture-specific components of turn-taking is crucial for developing dialogue systems that are not only responsive but also adaptable across diverse cultural contexts. ## related work efforts to improve turn-taking in spoken dialogue systems (sds) have increasingly leveraged the linguistic capabilities of llms, driven by the need for these systems to manage natural unscripted interactions, particularly in multi-party settings @xcite . one notable approach is turngpt, which introduces a probabilistic model to predict transition relevance places (trps) using turn-shift tokens based on both contextual and speaker-identity information @xcite ). an extension of this approach is rc-turngpt, which incorporates the predicted responses of interlocutors, conditioning predictions on upcoming linguistic content @xcite . however, these methods so far do not generalize well to corpora of unscripted dialogue. as a consequence, current dialogue systems still tend to produce long silences and ill-timed feedback @xcite . ablation studies on these llms suggest that previous linguistic content is generally sufficient for accurate prediction of turn-ends, and that the relative gain in accuracy diminishes with larger context windows-i.e., trps can often be predicted effectively using the local linguistic content of a turn. additionally, although several approaches attempt to predict turn-ends using acoustic signals (e.g., ek-stedt and @xcite , our work is grounded in human turn-taking literature, where linguistic cues are recognized as both necessary and sufficient for anticipating trps @xcite ). ## our approach there are two common methods for identifying trps in recorded conversation corpora (e.g, @xcite @xcite . the first is to locate speaker changes, which are directly observable, and infer that these changes occur only at trps. the second is to have experts in conversation analysis manually annotate trps. while these methods are widely used, both have limitations. speaker changes only account for a subset of all trps, as there are opportunities where a listener could respond but chooses not to i.e., within-turn trps @xcite . expert annotations, meanwhile, are subjective and do not align with the task faced by participants in real-time dialogue, who must predict trps instinctively 'on-the-fly'. in contrast, annotators analyze conversations retroactively, without engaging in the same anticipatory processing as active participants -leading to low ecological validity (see @xcite ). to address these limitations, we designed an experiment that engaged participants in natural conversations, asking them to produce auditory responses at any location they felt it was possible to respond-not necessarily where they would have responded in everyday interactions. by having multiple participants repeat this task, we gathered a wide range of responses for the same turns. while individual responses varied, the aggregated distribution (with a sufficiently large sample size) provides a reliable indicator of within-turn trps, reflecting how humans identify these opportunities in real conversations. ## evaluation metrics classification metrics we can evaluate the performance of a model for the within-turn trp prediction task (see definition 4.1) by comparing its predictions t p redicted r,s against the participants' indications of trps t p articipants r,s . it is important to note the imbalance inherent in the data i.e., intervals that contain trps are much less frequent those that do not. in this case, we cannot use accuracy since a model that simply predicts the majority class for all intervals will have achieve a high value. instead, the f1 score i.e., the harmonic mean of precision and recall, is well suited since it emphasizes models that perform well in identifying intervals that contain trps (t @xmath0, which are the vast minority of intervals. ## experiments and results we employed several state-of-the-art llms to perform the within-turn trp prediction task described in section 4.2. our focus was on models pretrained on diverse datasets, some of which were explicitly designed with capabilities for spoken interaction. to adapt llms for downstream tasks, two main strategies are commonly used: fine-tuning and incontext learning (icl). fine-tuning involves updating the weights of a pre-trained model to specialize it for a particular task, resulting in a single model tailored for that task. this approach is advantageous because it can accommodate training sets of any size, often leading to significant performance improvements. however, most state-of-theart llms are not available for direct fine-tuning due to restricted open-source access and instead are accessible only through public apis (liesenfeld and dingemanse, 2024). given these limitations, we employed in-context learning (icl) as our adaptation strategy. unlike fine-tuning, icl does not require modifying model weights. instead, it adapts the model to a specific task by using task demonstrations provided through prompts. however, it is crucial to note that icl is highly sensitive to the formulation of these prompts, and optimizing prompts requires careful consideration and specific strategies @xcite . we tested each model under two prompting conditions: expert and participant. in the expert condition, the model was provided with theoretical background on trps, similar to what an expert annotator might know. in the participant condition, the model was given a version of the instructions that the human participants received. these two prompting conditions explore how the level of provided information affects the model's performance on the trp prediction task. overall, the performance of the best performing model reveals significant shortcomings. first, the model exhibits low precision (0.137) and recall (0.169), leading to a low f1 score (0.151), indicating frequent false positives and missed trps. while the kappa statistic across all intervals (k all f @xmath1.876) suggests good general agreement, the much lower kappa for participant-agreed trp intervals (k there are also differences between the participant and expert conditions. the expert condition yielded higher precision (0.147) compared to the participant condition (0.126), indicating more accurate identification of trps. the expert condition also achieved higher recall (0.185 vs. 0.153), suggesting a better ability to detect intervals in which trps occur. the f1 score, balancing precision and recall, was slightly higher in the expert condition (0.164) than in the participant condition (0.138). kappa statistics also showed variability: k all f ree was higher for participants (0.891 vs. 0.860), reflecting stronger overall agreement, while k true f ree was higher for participants (0.325 vs. 0.201), indicating better performance in correctly identifying participantagreed trps. error metrics further demonstrated that the expert condition had lower nmae (0.253 vs. 0.286) but higher nmse (5.36 vs. 3.135) and significantly greater density-adjusted nmae (nmae @xmath2.56 vs. 11.28). these results suggest that while the expert prompts provided more theoretical accuracy, the participant prompts offered more practical relevance and alignment with true trps. ## discussion half a century of research on turn-taking has demonstrated that humans rely on various cues to achieve rapid and seamless turn-transitions in natural conversation by accurately predicting upcoming trps. this ability is crucial for minimizing response delays and avoiding overlapping speech, both of which are interactionally significant @xcite @xcite . poorly timed turn-taking can negatively affect how utterances are interpreted; for example, longer response delays often signal reluctance or hesitation to deliver a dispreferred response @xcite . current spoken dialogue systems (sds), however, struggle to replicate human-like turn timing, resulting in reduced user satisfaction and diminished communicative effectiveness @xcite . state-of-the-art llms, pre-trained on large and diverse datasets, are well-suited for leveraging linguistic information-which has been shown to be sufficient for predicting opportunities for speech in humans-and increasingly, multimodal information, to perform a range of spoken language tasks @xcite . however, contrary to expectations, we find that the llms we tested underperform across multiple measures on a simple binary prediction task to identify within-turn trps when using in-context learning (icl) as the adaptation strategy. this holds true even when providing essential background context through various prompts (expert versus participant). these findings point to a major issue: llms are currently unable to effectively utilize their extensive linguistic knowledge for unscripted turn-taking in spoken interaction. this limits their application in dialogue systems by preventing these systems from accurately anticipating opportunities for speaker transitions. our work attempts to advance the performance of llms for turn-taking in spoken interaction. first, by empirically demonstrating that current llms struggle with trp prediction despite their extensive pre-training, we expose a critical bottleneck that needs to be addressed. second, we provide evidence that high performance on writtenlanguage benchmarks does not necessarily translate to high performance on spoken language tasks, emphasizing the need for specialized evaluation in conversational settings. third, we contribute a specialized dataset containing empirical, on-the-fly human judgments on where trps occur in natural conversation. this dataset is a valuable resource for the nlp research community, offering opportunities for targeted fine-tuning and evaluation of llms, and enabling the development of models that more closely replicate human conversational behavior. ## conclusion even though large language models show impressive performance on a range of challenging language-related tasks, it is as yet unclear whether they can be employed for determining when they can start producing their turn in spoken dialogue at a socially appropriate time. this would require them to have human-level ability to predict transition relevance places, locations in speaker's contribution where they may take over the turn and start speaking. to test this ability in state-of-the-art llms, we collected data from humans that perform this task on-the-fly, and compared the performance of the llms with that of the human participants. it turned out that the performance of selected llms on this task was far below the level of that of the human participants. apparently, the pretraining of llms on vast amounts of written data was not sufficient to generalize to this particular task. possible causes for the disappointing performance could be that we haven't found the optimal prompts, and/or that the models would either need more spoken dialogue input during pre-training, or explicit fine-tuning on spoken dialogue data. either way, the dataset that we have developed will allow researchers in the area of human-machine turn-taking to explore ways to improve the models' performance on this crucial task.