A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI (2020)

Abstract: Automatic Speech Recognition (ASR) systems are increasingly powerful and more accurate, but also more numerous with several options existing currently as a service (e.g. Google, IBM, and Microsoft). Currently the most stringent standards for such systems are set within the context of their use in, and for, Conversational AI technology. These systems are expected to operate incrementally in real-time, be responsive, stable, and robust to the pervasive yet peculiar characteristics of conversational speech such as disfluencies and overlaps. In this paper we evaluate the most popular of such systems with metrics and experiments designed with these standards in mind. We also evaluate the speaker diarization (SD) capabilities of the same systems which will be particularly important for dialogue systems designed to handle multi-party interaction. We found that Microsoft has the leading incremental ASR system which preserves disfluent materials and IBM has the leading incremental SD system in addition to the ASR that is most robust to speech overlaps. Google strikes a balance between the two but none of these systems are yet suitable to reliably handle natural spontaneous conversations in real-time.

## introduction automatic speech recognition (asr) is one of the first important milestones of deep learning @xcite and the standard evaluation metric for asr systems, the word error rate (wer), has been consistently improving to impressive levels @xcite @xcite . but with a few notable exceptions discussed below, the suitability of these state of the art asr systems for conversational ai (henceforth sds for spoken dialogue system 1 .) is rarely evaluated. to be truly natural and responsive, sds architectures have had to be redesigned @xcite @xcite to take into account the inherently time-linear and incremental nature of human language processing (see @xcite ; @xcite among many others). these incremental architectures impose additional evaluation criteria and further constraints on the output required of asr systems: first, asr hypotheses should be produced incrementally, word by word, with minimal latency. this means that downstream processing (e.g. for natural language understanding (nlu)) can begin before a turn is complete. this has been shown to lead to dialogue systems which feel more natural @xcite and more satisfying to interact with @xcite . asr hypotheses should furthermore be stable through time (minimising 'jitter' in the output) @xcite @xcite . second, incremental detection and processing of disfluent speech is crucial for robust natural language understanding @xcite @xcite . and people are not only capable of processing disfluencies, but they exploit them in the understanding that results @xcite . @xcite observe that disfluency structure and its constituent tokens such as edit terms (e.g. "sorry", "i mean") are often removed or rewritten by the asr system. furthermore, in multi-party settings involving an sds, people often interact directly with each other without using the simpler and more fluent language they tend to use when interacting with a computer @xcite . another crucial challenge in multi-party settings is that conversational overlaps impact the asrs performance as speech inputs combine. speaker diarization (sd), partitioning an input audio stream into speaker-labeled segments, can be combined with asr @xcite but must also work incrementally if we are to produce naturally interactive sdss. in this paper, we evaluate three of the most well-known asr services, namely ibm watson speech to text foot_1 @xcite , google speech to text foot_2 , and microsoft azure speech to text foot_3 @xcite . the incremental performance criteria and metrics we use were first introduced in @xcite @xcite , used by @xcite , and detailed below in sec. 2. for this evaluation, we have used the switchboard corpus of dyadic telephone conversations @xcite because it is open-domain (with speech overlaps) and contains high quality annotations of disfluency structure @xcite . both ibm watson and google's platforms include sd components that we also evaluate on the switchboard corpus in this paper. as the switchboard corpus is strictly dyadic however, we additionally evaluate these sd systems on a second corpus, called avdiar @xcite , containing multi-party conversations with up to four speakers. results show that microsoft has the most responsive, stable, and accurate incremental asr system which also preserves disfluent materials. ibm and google both filter filled pauses but even though all systems are impacted by speech overlaps, ibm is the most robust to them. ibm also has the leading incremental sd system which suggests it's fitting for multi-party settings. google is a neat balance between the two others but all three of the systems are not yet adequate to reliably handle natural spontaneous conversations in real-time. to aid experiment reproducability and future research in this area, we will release our incremental asr code for audio processing foot_4 and framework for evaluation foot_5 . alongside this framework, we will also release the switchboard transcripts we have generated as part of the speech disfluencies experiment detailed in sec. 4.2. ## the switchboard corpus (swb) the switchboard corpus (swb; originally: @xcite ) is a large corpus of human-human dyadic conversations in english over the phone on a wide range of topics. the corpus can therefore be described as open-domain. for evaluation of asr systems, swb has become the de facto standard through its nist 2000 evaluation test set/subcorpus foot_7 -see @xcite among many others. here we use switchboard-1 foot_8 , together with its more recent dialogue act and disfluency annotations @xcite foot_9 to evaluate both asr and sd systems' incremental performance for use in spoken dialogue systems (sdss). we choose swb for two main reasons: (1) more recent versions of the corpus contain accurate and fine-grained annotations of disfluency material @xcite enabling our disfluency fidelity experiments below in sec. 4.2; and (2) existence of separate channels for each speaker allowing us to study the robustness of asr systems to speech overlaps. the disfluency annotations in swb contain tags for our phenomena of interest in this paper: edit terms e/ , filled pauses f / , as well as the detailed structure of self-corrections. (1) shows an example of how self-corrections are annotated in swb: here, 'uh' is a filled pause, 'i mean' is an edit term; 'without' is the repair signalling substitution of 'with', which is the reparandum. in self-corrections in general, the reparandum can be deleted, substituted by the repairing phrase, or simply repeated as in (2): as noted above in sec. 2.3, these annotations allow us to create controlled experimental gold test sets where edit terms or filled paused are filtered out; or where self-corrections are rewritten to their 'clean', complete constituent form; e.g. (1) would be rewritten simply to "without any strings attached". switchboard's audio files are stereo with each channel representing one of the two speakers. for our system comparison in sec. 4.1, we report overall incremental wer on the audio with the channels merged together. this more closely resembles the common dyadic dialogue setting as the speech inputs are not separate in a dyadic conversation or a multi-party setting in which one interactant is an sds. in order to run the experiments in sec. 4.2 and 4.3, we separated switchboards stereo files into their individual channels that each represent one speaker and therefore do not contain speech overlaps. these channels do have audio channel bleeding however so both speakers can sometimes be heard in a single channel. the isolated wers are increased due to a mismatch between the single speaker gold transcript and the system's output containing both speakers. this is crucial to note but does not impact the findings of our experiments as explained within each experiment description. ## asr system evaluations for this paper we experimented with three of the most well-known, production-level asr systems with the ability to transcribe audio incrementally. these were mentioned in sec. 1 and from now on we will refer to them by their company names -namely: microsoft, ibm, and google. we used the most up to date systems as of may 2020. the underlying system architectures are undisclosed but, if access was granted, it would be useful to tie them to our metric findings. microsoft azure speech to text: microsoft supports continuous speech recognition which, as the name suggests, is their incremental asr. it processes an audio stream continuously and can output its current hypothesis upon request. we requested hypotheses every 0.05s and there were often differences in hypotheses at that frequency. ibm watson speech to text: ibm has a http interface but using their websocket interface is required in order to receive interim results. ibm notes that this feature returns intermediate hypotheses that are for interactive applications and real-time transcription. these intermediate results are returned by the system as soon as they are generated and they are notably less frequent than the other systems, usually returning increments spanning more than one word. this is an important characteristic of the ibm system in explaining our latency and stability results to which we return below. google speech to text: google returns incremental speech recognition requests via their 'streaming recognize' cloud api method when interim results are requested in the config. as google provides an option of four standard asr models, we selected and used the 'phone call' model as the switchboard corpus is a collection of dyadic conversations over the phone. similarly to ibm, interim results are returned as they are generated. unlike ibm however, these are returned much more frequently. ## speaker diarization (sd) system evaluations here we present our evaluation of the google and ibm sd systems -microsoft does not currently include a sd service. we evaluate both systems within a real-time (i.e. incremental) multi-party setting. for both systems, we therefore only take into account the interim results via a streaming method instead of the final results, although we acknowledge that the final result usually performs better than the interim ones. google cloud sd: in the past three years or so, google have presented several sd models. of these, @xcite achieves the lowest diarization error rate (der) of 7.6% on the nist speaker recognition evaluation benchmark (ldc2001s97) compared to their previous models @xcite @xcite 12 . for our experiments below, we cannot ascertain which model is currently deployed on the cloud service. of the different models within the google cloud services to support different types of audio stream, we evaluate the 'video' model only below. ibm watson speaker diariztion: @xcite present the first online, real-time, production level sd system within the ibm cloud speech-to-text (stt). the deployment proceeds in two stages: (1) an unsupervised audio segmentation to the homogeneous segments per speaker; and (2) speaker labelling. since extracting acoustic representations from audio within non-speech sessions may affect sd performance, @xcite move the asr module forward in their architecture, where asr results are used as input to the segmenter to separate different speaker chunks, and to improve the overall sd performance without dropping the asr quality. it is this sd system which we evaluate below, noting that it operates together with asr. ## conclusion looking at only each system's incremental wers, which are all very similar, it is impossible to distinguish which system is best for a certain use-case. using the additional metrics and further experiments detailed in this paper however, we have carried out a more sophisticated analysis. microsoft has the most responsive, stable and accurate incremental asr system, while preserving disfluent material but does not have a real-time sd system. ibm and google both filter filled pauses but even though all systems are impacted by speech overlaps, ibm is the most robust to them. ibm also has the leading real-time sd system which suggests it's a better fit for multi-party settings. google is a neat balance between the two others. all of the systems report impressive wers on the switchboard corpus using non-incremental (bidirectional) models but this does not mean that this challenge is complete. none of these systems are yet adequate to reliably handle natural spontaneous conversations in real-time. we hope that by releasing our evaluation framework, we will encourage future assessment of such systems and promote the design of speech recognition and diarization systems with the above metrics in mind.