Benchmarking ASR Systems Based on Post-Editing Effort and Error Analysis (2021)

Abstract: This paper offers a comparative evaluation of four commercial ASR systems which are evaluated according to the post-editing effort required to reach “publishable” quality and according to the number of errors they produce. For the error annotation task, an original error typology for transcription errors is proposed. This study also seeks to examine whether there is a difference in the performance of these systems between native and non-native English speakers. The experimental results suggest that among the four systems, Trint obtains the best scores. It is also observed that most systems perform noticeably better with native speakers and that all systems are most prone to fluency errors.

## introduction the rapid technological progress in the field of automatic speech recognition (asr) has lead to claims that speech-to-text systems can achieve up to 90% accuracy @xcite . the aim of this paper is to shed some light on the impact that this progress has on the productivity of end users. until now, the evaluation of asr systems relied exclusively on word error rate (wer) and similar metrics. calculating these metrics is usually an expensive and time-consuming task as manual transcriptions are used for reference. in addition, these traditional approaches do not provide information on the cognitive effort required to reach "publishable" quality. in this paper, the aim is to address the aforementioned issues by proposing a way to depart from the traditional methods of asr evaluation. the key idea is to deploy the post-editing (pe) method in the evaluation process. to bridge the gap of the underrepresented aspect of cognitive effort, four asr systems: amazon foot_0 , microsoft 4 , trint 5 and otter 6 were evaluated based on post-editing effort. to this end, the pet tool @xcite was employed to compute the post-editing (pe) effort in terms of pe time and pe distance at a sentence level. the objective of the pe process was to rank all systems based on their overall score. in the attempt to provide a qualitative analysis, a secondary objective seeks to investigate the types of errors that these systems produce. to accomplish this, a new error typology for transcription errors was developed following the taus dqf-mqm @xcite main error categories. in this novel typology, which is essential for this study, the subcategories are tailored to suit transcription errors. to the best of our knowledge, this is the first study that seeks to investigate which types of errors the asr systems are most prone to. the comparison between the error annotation results and the post-editing results will lead to new insights of their correlation. another goal of this study is to examine the role of the speaker's accent. it investigates whether the performance of the systems is affected by the speaker's accent. to address this question, results from native and non-native english speakers are compared. the rest of the paper is structured as follows: section 2 contextualises the current study by discussing related work. section 3 outlines the data used, section 4 presents the experimental setup. section 5 discusses the results of the experiments conducted. finally, section 6 provides the conclusions of this study. ## related work the rapid development of state-of-the-art automatic speech recognition systems led to the need for these systems to be evaluated. a recent study @xcite benchmarked commercial asr systems by comparing their results against human quality and evaluating them using the wer metric. this research pays specific attention to named entity recognition. related research includes @xcite , where a tool was designed to perform comparisons between commercial and open-source asr systems using the wer metric. a recent systematic review @xcite discusses the problems of benchmarking asr systems, which were presented in various studies and expresses skepticism for the very low wer results reported. they demonstrated that the wer rate was considerably higher than the best results reported in those studies. a further study @xcite also benchmarked three commercial asr systems, but they reported results using three metrics: wer, hper and rper. a qualitative analysis @xcite on asr systems was performed aiming to evaluate the accuracy of the language model adaptation; in order to do so, the wer metric was applied only to relevant words. it is worth noting that none of the existing approaches appears to overcome the limitations of the wer metric. there is therefore a need for a new evaluation approach. two new performance metrics: mer (match error rate) and wil (word information lost) were proposed in @xcite . furthermore, with the aim to represent human perception of asr accuracy, hpa (human perceived accuracy) was developed @xcite . another metric was introduced in @xcite seeking to achieve a better correlation to human evaluation. finally, an extension of the wer metric was proposed in @xcite , where weighted penalties were applied by implementing word embeddings. to the best of our knowledge, studies that employ post-editing in order to evaluate asr performance are scarce. post-editing was explored in @xcite , where users browsed and corrected automatic transcriptions of lectures in a web-based interactive interface. this study aimed to compare wer rates with comprehensibility improvements after transcripts were post-edited. as detailed in @xcite , an asr system was developed for polish, which introduced the novel idea of applying automatic post-editing in the asr output. finally, two crowdsourcing studies were compared in @xcite with the objective to investigate whether it is preferable to transcribe from scratch or to perform post-editing on asr output. they concluded that post-editing is preferable only when wer accuracy is lower than 30%. however, effort indicators of the post-editing task were not examined in this study. the above discussion provides compelling evidence that there is a pressing need for an alternative approach to account for the cognitive effort required to post-edit raw asr outputs. to the best of our knowledge, this study constitutes the first analysis of the evaluation of post-editing effort in this field. the added value of this paper is also highlighted by the qualitative analysis on the transcription errors, which remains unexplored in the literature. with this aim in mind, this paper puts forward an error typology for asr transcription errors. the suggested error typology is the first of its kind to be specifically designed for the use case where the asr output is post-edited by humans to reach "publishable quality". ## data description for the purpose of this study, the video data were obtained from the research seminar series "specialised seminar: technologies for translation and interpreting: challenges and latest developments" @xcite , hosted by prof r mitkov at the university of wolverhampton. more specifically, the videos were recordings of talks given by invited speakers on topics related to translation and interpreting technologies, which were held online via zoom. thus, all data have the same register and belong to the same domain. it should be mentioned that for microsoft and amazon asr systems the video files were converted to audio files (.wav), as these systems operate exclusively on this file format. two videos were used as input data: one with a native american english speaker and one with a non-native american english speaker. the mother tongue of the non-native english speaker is russian. the videos were trimmed in order to have the same length-approximately 15 minutes per video. each asr system produced a transcription of approximately 2,000 words per video, thus the size of the post-editing and error annotation tasks for all four systems consisted of approximately 16,000 words. ## results as seen in secondly, a weak correlation between accuracy errors and pe time is noted. a high rate of accuracy errors, contrary to the popular belief, does not require extra pe time. for instance, trint reported the highest accuracy rate; however, it was ranked second based on pe time. in this case, the low correlation could be justified by the high number of omission and addition errors, which are easily detectable and require less cognitive effort, combined with the low number of mistranscription errors, which require more cognitive effort. finally, it should be highlighted that there is a big performance difference in pe time between native and non-native speakers for amazon. this difference can be explained by the high number of filler word, mistranscription, segmentation and terminology errors of the non-native speaker transcription. ## conclusions in this study, outputs from commercial asr systems were post-edited and then the errors were annotated. the asr systems were ranked based on the postediting effort required to reach "publishable" quality and the number of errors they produced. in accordance with the pe and error results presented above, it can be concluded that with the data used in this experiment, trint is the best performing system in terms of pe distance and total number of errors, while microsoft is the best performing system in terms of pe time. moreover, the number of errors does not always correlate with the pe effort. it is also evident that there is a general tendency towards fluency errors, which are assumed to be the most time-consuming errors. the experiments point to the conclusion that most asr systems perform better with a native speaker. the constraints of this study include its limited scope and the involvement of only one post-editor and annotator; larger-scale study results may be different. while the size of the data was another constraint, the results reported remain insightful. in particular, this study will pave the way for further research in the field of asr evaluation, post-editing and error analysis. future work could explore the correlation between the suggested approach and the traditional wer metric.