ABSTRACT: The great potential of speech recognition systems in freeing users' hands while interacting with computers has inspired a variety of promising applications. However, given the performance of the state-of-the-art speech recognition technology today, widespread acceptance of speech recognition technology would not be realistic without designing and developing new approaches to detecting and correcting recognition errors effectively. In seeking solutions to the above problem, identifying cues to error detection (CERD) is central. Our survey of the extant literature on the detection and correction of speech recognition errors reveals that the systeminitiated, data-driven approach is dominant, but that heuristics from human users have been largely overlooked. This may have hindered the advance of speech technology. In this research, we propose a user-centered approach to discovering CERD. User studies are carried out to implement the approach. Content analysis of the collected verbal protocols lends itself to a taxonomy of CERD. The CERD discovered in this study can improve our knowledge on CERD by not only validating CERD from a user's perspective but also suggesting promising new CERD for detecting speech recognition errors. Moreover, the analysis of CERD in relation to error types and other CERD provides new insights into the context where specific CERD are effective. The findings of this study can be used to not only improve speech recognition output but also to provide context-aware support for error detection. This will help break the barrier for mainstream adoption of speech technology in a variety of information systems and applications. KEY WORDS AND PHRASES: cues to error detection, speech recognition, taxonomy, verbal protocol analysis. SPEECH RECOGNITION IS ONE OF THE MAIN information technologies that provide users with hands-free interaction with computers. The power of speech control promises to help many individuals who might otherwise not be able to interact with a computer through the conventional mouse or keyboard interface. Particularly, users who are physically challenged, visually impaired, or mobility impaired are offered new opportunities by speech recognition-enabled applications. Although many interesting applications have emerged or enhanced with the advance of speech technology, the current use of such a technology reflects "only a tip of the iceberg of the full power that the technology could potentially offer" (cf. [8, p. 69]). However, some fundamental and practical limitations of the technology result in unsatisfactory performance of speech recognition systems. The convenience and efficiency promised by speech technology in interacting with computers is seriously compromised by the laborious efforts and frustration experienced in detecting and correcting recognition errors [40] . Widespread acceptance of speech as a primary input modality for computers will not be possible unless the underlying recognition technology can produce sufficiently robust and low-error output [8] . To bridge the gap between what people expect from speech recognition and what the technology can achieve, it is desirable to find effective ways to detect and even DISCOVERING CUES TO ERROR DETECTION IN SPEECH RECOGNITION OUTPUT 239 correct recognition errors. Error detection is the premise for error correction. No matter whether it is systems or users who are responsible for error detection, they rely on cues to error detection (CERD). In the system-initiated approach, CERD are extracted automatically from internal parameters or the output of speech recognition, and then used to build machine learning models using the training data [5, 47, 50] . The machine learning models are then used to detect possible recognition errors and even to improve speech recognition output (e.g., [21, 47, 49] ). Thus, this approach is characterized as data-driven because CERD are selected primarily based on their impact on the training data and their automation potentials. This type of approach is efficient; however, its effectiveness may be undermined by overlooking other significant CERD based on user experience and heuristics knowledge, which are neither automatically generated nor predefined. According to our survey of the related literature, the data-driven approach is dominant in identifying CERD. The ultimate goal of speech recognition is to approximate native speakers' recognition capability. Given the unsatisfactory performance of systems-initiated error detection and the impact of human factors on information technology implementation [23] , there is a strong need to learn CERD from users in order to improve the usefulness of speech recognition systems. However, the extant studies have failed to consider users as an important source for CERD. Therefore, the research question that we aim to address in this study is: What types of CERD do users apply in detecting speech recognition errors? We propose a knowledge-driven, user-centered approach to discovering CERD in this paper. Instead of relying on trial-and-error, as shown in the systems-initiated, data-driven approach, the proposed method uncovers important CERD from users' knowledge and experience. To the best of our knowledge, this is the first study to discover CERD via a user-centered approach. An empirical study was designed and conducted to elicit verbal explanations from participants about strategies and cues used to detect speech recognition errors, which were then interpreted and encoded via verbal protocol analysis. This type of analysis lends itself to a taxonomy of CERD, consisting of linguistic, hypotheses-based, and other information. Moreover, the association between CERD and three types of speech recognition errors-insertion, deletion, and substitution errors-was analyzed to gain insights into the effectiveness of CERD for specific types of errors. Furthermore, the result of correlation analysis between different CERD suggests that CERD strongly associated with one another can be used together to enhance the performance of error detection. The developed CERD taxonomy and related findings of this research can make multifold contributions to the detection and correction of speech recognition errors. First, the taxonomy of CERD is the first of its kind. It helps advance our knowledge on CERD and allows future expansion and refinement. Second, it complements CERD discovered by the traditional data-driven approach. Third, the analysis of CERD, in relation to error types, allows us to contextualize CERD to maximize their utility. Fourth, the findings enable the development of supportive and context-sensitive environments to facilitate users in their detection of errors. They enhance the performance of error detection systems with additional knowledge.
Zhou et al. REF conducted user studies to find cues to error detection that could be used to improve the error correction capabilities of speech recognition systems.
Discovering Cues to Error Detection in Speech Recognition Output: A User-Centered Approach
Journal of Management Information Systems
Computer Science