How Might We Create Better Benchmarks for Speech Recognition? (2021)

Abstract: The applications of automatic speech recognition (ASR) systems are proliferating, in part due to recent significant quality improvements. However, as recent work indicates, even state-of-the-art speech recognition systems – some which deliver impressive benchmark results, struggle to generalize across use cases. We review relevant work, and, hoping to inform future benchmark development, outline a taxonomy of speech recognition use cases, proposed for the next generation of ASR benchmarks. We also survey work on metrics, in addition to the de facto standard Word Error Rate (WER) metric, and we introduce a versatile framework designed to describe interactions between linguistic variation and ASR performance metrics.

## introduction the applications of asr systems are many and varied; conversational virtual assistants on smartphones and smart-home devices, automatic captioning for videos, text dictation, and phone chat bots for customer support, to name a few. this proliferation has been enabled by significant gains in asr quality. asr quality is typically measured by word error rate (wer), or, informally, the levenshtein distance between the target transcript and the machine-generated transcript @xcite -see section 3. current state-of-the-art accuracy is now in low-single-digits for the widely used librispeech benchmark set @xcite , with e.g. @xcite achieving a wer of 1.4%. however, as @xcite have pointed out, overall, our current asr benchmarks leave much to be desired when it comes to evaluating performance across multiple real-world applications. typical benchmark sets beyond librispeech include timit @xcite , switchboard @xcite , wsj @xcite , callhome @xcite , and fisher @xcite . 1 however, with many asr systems benchmarking in the low single digits, small improvements have become increasingly difficult to interpret, and any remaining errors may be concentrated. for example, for switchboard, a considerable portion of the remaining errors involve filler words, hesitations and non-verbal backchannel cues @xcite . furthermore, achieving state-of-the-art results on one of these sets does not necessarily mean that an asr system will generalize successfully when faced with input from a wide range of domains at inference time: as @xcite show, "no single validation or test set from public datasets is sufficient to measure transfer to other public datasets or to real-world audio data". in one extreme example, @xcite show that modern asr architectures may even start emitting repetitive, nonsensical transcriptions when faced with audio from a domain that was not covered at training time-even in cases where it would have achieved perfectly acceptable librispeech evaluation numbers. inspired by goodhart's law, which states that any measure that becomes a target ceases to be a good measure, we argue that as a field, it behooves us to think more about better benchmarks in order to gain a well-rounded view of the performance of asr systems across domains. in this paper, we make three contributions. first, we provide a taxonomy of relevant domains, based on our experience developing asr systems for use in many different products, with the goal of helping make nextgeneration benchmarks as representative as possible @xcite . second, we argue that optimizing only for wer, as most current benchmarks imply, does not reflect considerations that are ubiquitous in real-world deployments of asr technology: for example, prohttps://github.com/syhw/wer_are_we . additionally, fair recently released the casual conversations dataset intended for ai fairness measurements @xcite . ## asr use cases with asr use cases spanning many applications and tasks, ideally asr systems would be robust to various classes of variation in speech input. for example, an asr system which provides automatic captions for video meetings would recognize words from many different semantic fields, adaptable to the topic of the meeting. speech characteristics may also vary across domains: for example, the speech style used when dictating text messages differs from the style of a group conversation, where speakers may occasionally talk over each other. an ideal benchmark set would include what we will call 'horizontal' and 'vertical' variation. horizontal challenges refer to a wide variety of scenarios where asr may be used, while vertical challenges involve e.g. diversity in topics, encoding formats, and others. ## metrics: wer and beyond assume, for the sake of argument, that an impressive selection of test sets has been collected in order to create our imagined ideal next-generation benchmark for asr, covering many use cases, technical challenges, and so on. the performance of an asr system could now be measured simply by computing a single, overall wer across all the utterances in this collection of test sets-and a system that yields lower wer on this benchmark could be said to be 'better' than a system with higher wer. however, in a real-world deployment setting, the question of which system is 'best' typically relies on an analysis of many metrics. for example, imagine a system with a wer of 1.5% but an average transcription latency of 2500 milliseconds, and another system that achieves 1.6% wer but a latency of only 1250 milliseconds: in many settings, the second system could still be more suitable for deployment, despite achieving worse wer results. of course, 'latency' itself is not a well-defined term: sometimes the measurement is reported as the average delay between the end of each spoken word and the time it is emitted by the asr system, while in other cases the measure is based only on the first or the last word in an utterance. neither is well-defined in presence of recognition errors. yet another kind of latency is end-to-end latency, involving everything between the microphone activity and the final projection of results, including network overhead and optional post-processing like capitalization, punctuation etc. a "pure" asr latency metric ignores those and focuses on the processing time of the recognizer, while latency in the context of voice assistant commands may consider the delay before successful recognition of a command, which might sometimes precede the actual end of utterance. in this section, we describe how, much like latency, even wer itself has many nuances, and we point to other metrics, beyond wer and latency, that can be considered account when measuring asr systems. ## demographically informed quality as previously discussed, the ideal benchmark for asr systems would cover as many horizontals and verticals as possible, and would involve various kinds of metrics beyond just wer. another important dimension, however, would be the availability of demographic characteristics, and analyzing the metrics based on such characteristics. such demographic characteristics may correlate with linguistic variation-for example, non-native speakers of english may have an accent showing traces of their native language-which may in turn impact asr performance. having demographic characteristics can help produce analyses like the one reported by @xcite , who analyzed differences in recognition performance for different accents, age ranges, and gender within an asr system. the ideal benchmark set, then, should include sufficient metadata to run similar analyses, enabling developers to understand how their system behaves when processing various accents or dialects; to see whether factors like gender and age influence recognition performance in their system. linguistic variation may take many different shapes, including: even where demographic information is already present in asr evaluation sets, it can be a valuable to conduct an analysis of the target user base for a deployed asr system in order to ensure that all relevant tags are available. for example, if a data set has labels for four distinct accents, but the target user base is known from sociolinguistic research to use six distinct accents, this gap will not necessarily be evident when running an analysis of any possible differences among the four accents for which tags are available. it is important to understand the sociolinguistic characteristics of the target user base, and to cover as many of these properties as possible. given that language has almost infinite variation as you zoom in-in the extreme, everyone has a slightly different voice-this is a task that requires careful sociolinguistic judgement and analysis, calling for interdisciplinary collaboration between linguists and developers of asr systems. even when a rich set of tags is available, it can be difficult to interpret the results. we describe a simple, metric-independent population-weighted visualization framework designed to evaluate asr systems based on such demographic metadata. our approach supports the different language variations outlined above, and we propose this analyses as a valuable addition to future benchmarks. ## conclusion the ultimate goal of benchmarking should be the ability to predict how well an asr system is going to generalize to new and unseen data. in the previous sections we have argued that a single aggregate statistic like the average wer can be too coarsegrained for describing the accuracy in a real-world deployment that targets multiple sociolinguistic slices of the population. ideally, the insights generated by the proposed analysis would be actionable, from the composition of the training data to fine-grained twiddling with a clear objective function. before we conclude, we should point out that any benchmark that implemented even a fraction of the metrics outlined above would yield rich amounts of information-which will likely pose challenges in terms of organizing, presenting, and understanding all this material. model report cards, as outlined by @xcite , may be a natural way to capture this information for an asr system-although we would suggest calling them system report cards instead, given that most asr systems do not consist solely of a single monolithic model. given the sheer amount of variation in the ways in which people speak, and a large number of technical factors, measuring asr systems is a complicated task. today's benchmarks clearly leave room for improvement, whether it is through covering more horizontal domains (different kinds of speech), measuring the impact of cross-cutting vertical issues (e.g. factors like background noise), using more metrics than just wer (e.g. latency), and including demographic characteristics. we hope that our survey of these areas, and the simple population-weighted visualization framework we introduced, can help improve future benchmarks-not just for english, but also for the thousands of other languages spoken in our world today. this will clearly be a long-term journey, but it will be very important for the field as a whole to find ways to measure asr systems better as speech recognition research continues to advance.