The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems (2023)

Abstract: Speech recognition systems are a key intermediary in voice-driven human-computer interaction. Although speech recognition works well for pristine monologic audio, real-life use cases in open-ended interactive settings still present many challenges. We argue that timing is mission-critical for dialogue systems, and evaluate 5 major commercial ASR systems for their conversational and multilingual support. We find that word error rates for natural conversational data in 6 languages remain abysmal, and that overlap remains a key challenge (study 1). This impacts especially the recognition of conversational words (study 2), and in turn has dire consequences for downstream intent recognition (study 3). Our findings help to evaluate the current state of conversational ASR, contribute towards multidimensional error analysis and evaluation, and identify phenomena that need most attention on the way to build robust interactive speech technologies.

## introduction speech recognition (asr) is a key technology in voice-driven human-computer interaction. although by some measures speech-to-text systems approach human transcription performance for pristine audio @xcite , real-life use cases of asr in open-ended interactive settings still present many challenges and opportunities @xcite . the most widely used metric for comparison is word error rate, whose main attraction -simplicity-is also its most important pitfall. here we build on prior work calling for error analysis beyond wer @xcite and extend it by looking at multiple languages and considering aspects of timing, confidence, conversational words, and dialog acts. as voice-based interactive technologies increasingly become part of everyday life, weaknesses in speech-to-text systems are rapidly becoming a key bottleneck @xcite . while speech scientists have long pointed out challenges in diarization and recognition @xcite , the current ubiquity of speech technology means new markets of users expecting to be able to rely on speech-to-text systems for conversational ai, and a new crop of commercial offerings claiming to offer exactly this. here we put some of these systems to the test in a bid to contribute to richer forms of performance evaluation. ## aims and scope a central question relevant at every moment of human interaction is why that now? @xcite , referring to the importance of position and composition in how people ascribe intent to communicative actions. for speech-to-text systems, in order to even approach this question, a key prerequisite is to detect who says what when. this means that diarization, content recognition and precise timing are all highly consequential and best considered in tandem. here we address this challenge by presenting a multipronged approach that lays some of the empirical groundwork for improving evaluation methods and measures. using principles of black-box testing @xcite , we evaluate major commercial asr engines for their claimed conversational and multilingual capabilities. we do so by presenting case studies at three levels of analysis. study 1 considers word error rates and treatment of overlaps. study 2 looks into what goes missing and why. study 3 looks into the repercussions for intent ascription and dialog state tracking. we show that across these areas, timing is both a mission-critical challenge and an ingredient for ways forward. ## discussion the ubiquity of voice interfaces coupled with reports of human parity in speech recognition might make robust voice-driven interaction seem within easy reach. indeed, all major vendors now advertise speech-to-text pipelines that claim both multilingual ability and conversational utility. here we put five such systems to the test and find that the results are bleak: word error rates are nowhere near the oft-claimed human parity; performance drops dramatically for languages other than english; precise timing and diarization is hard to come by; overlap is systematically ignored; conversational words go missing; and as a result, intent ascription and dialog state tracking are severely hampered. commercial speech-to-text systems are frequently exposed to conversational settings, whether it is in home use, business meetings, or customer service interactions. our results imply that these systems are likely to fall short of several of their intended applications. word error rate does not sufficiently reflect the performance of speech-to-text systems in most real-life contexts. the erasure of conversational elements and inability to deal with overlap renders these systems effectively oblivious to important aspects of user feedback. differences in diarization and turn allocation across systems also have strong effects on dialog act classification, with the implication that switching vendors might have untold consequences for dialog state tracking and intent ascription. our results show that current speech recognition systems privilege what is said over when it is said; and that even systems claiming conversational utility appear to treat the problem as fundamentally one of turning a rich tapestry of turns into running text. these text-first design choices become visible when exposed to the rapid turn-taking patterns of natural conversation -not only to analysts in case studies like this, but inevitably also to users, where they cause friction, interactional turbulence, and user dissatisfaction. the results are in line with recent arguments that the current language technology landscape is fundamentally built around monologic text instead of dialogical talk @xcite . the rise of conversational interfaces motivates a course correction if not a refurbishing of the foundations. here we hope to have shown that data from human interaction can inform such work. ## conclusion when you're a voice-driven conversational agent, life comes at you fast, and talk comes at you faster. we have presented evidence and arguments to support our contention that timing is more than a niceto-have for any truly conversational system: it is mission critical and despite decades of attention from speech scientists remains largely unsolved today. but rather than despair we take our findings as an opportunity to identify areas where novel work can make big differences. while diarization remains hard in real-life settings, representing overlap instead of erasing it is likely to offer meaningful improvements. while overlap-vulnerable elements will always remain acoustically at risk, exposing asrs to more ecologically valid training data and abandoning text-based sanitizing techniques will likely improve the recognition of short conversational elements. and while intent ascription will always be hampered by missing data, taking timing into account will enable new gains. dealing with conversational words computationally is hard: not only are their forms short and prone to overlap, their meanings are cognitively demanding and interactionally subtle. a focus on information and sentence structure over interaction and sequential organization has long enabled us to look away from these elements. as conversational words are backgrounded as 'backchannels' and the artful interweaving of turns is classified as mere 'overlap' if not 'noise', it becomes easy to lose sight of the intricacies of human interaction. one way to see this paper is as contributing to a reframing that is underway in the language sciences at large: a reframing that foregrounds talk over text, that attends to interaction alongside information, and that recognizes the key role of timing. timing is the secret sauce that can turn text into talk, chat into conversation, and perhaps, one day, clunky bots into fluid interactive tools.