Towards Detection and Remediation of Phonemic Confusion (2021)

Abstract: Reducing communication breakdown is critical to success in interactive NLP applications, such as dialogue systems. To this end, we propose a confusion-mitigation framework for the detection and remediation of communication breakdown. In this work, as a first step towards implementing this framework, we focus on detecting phonemic sources of confusion. As a proof-of-concept, we evaluate two neural architectures in predicting the probability that a listener will misunderstand phonemes in an utterance. We show that both neural models outperform a weighted n-gram baseline, showing early promise for the broader framework.

## introduction ensuring that interactive nlp applications, such as dialogue systems, communicate clearly and effectively is critical to their long-term success and viability, especially in high-stakes domains, such as healthcare. successful systems should thus seek to reduce communication breakdown. one aspect of successful communication is the degree to which each party understands the other. for example, properly diagnosing a patient may necessitate asking logically complex questions, but these questions should be phrased as clearly as possible to promote understanding and mitigate confusion. to reduce confusion-related communication breakdown, we propose that generative nlp systems integrate a novel confusion-mitigation framework into their natural language generation (nlg) processes. in brief, this framework ensures that such systems avoid transmitting utterances with high predicted probabilities of confusion. in the simplest and most decoupled formulation, an existing nlg component simply produces alternatives to any rejected utterances without additional guiding information. in more advanced and coupled formulations, the nlg and confusion prediction components can be closely integrated to better determine precisely how to avoid confusion. this process can also be conditioned on models of the current listener or task to achieve personalized or context-dependent results. figure 1 shows the simplest variant of the framework. as a first step towards implementing this framework, we work towards developing its central confusion prediction component, which predicts the confusion probability of an utterance. in this work, we specifically target phonemic confusion, that is, the misidentification of heard phonemes by a listener. we consider two potential neural architectures for this purpose: a fixed-context, feed-forward network and a residual, bidirectional lstm network. we train these models using a novel proxy data set derived from audiobook recordings, and compare their performance to that of a weighted n-gram baseline. ## background and related work prior work focused on identifying confusion in natural language, rather than proactively altering it to help reduce communication breakdown, as our framework proposes. for example, @xcite showed that certain features of recorded speech (e.g., repetition, hyperarticulation, strong emphasis) can be used to identify communication breakdown. the authors relied primarily on prosodic properties of recorded phrases, rather than the underlying phonemes, words, or semantics, for identifying communication breakdown. on the other hand, conversational repair is a turn-level process in which conversational partners first identify and then remediate communication breakdown as part of a trouble-source repair (tsr) sequence @xcite . using this approach, @xcite identified differences in tsr patterns amongst people with no, early-stage, and middlestage alzheimer's, highlighting the usefulness of communication breakdown detection. however, such work does not directly address the issue of proactive confusion mitigation and remediation, which the more advanced formulation of our framework aims to address through listener and task conditioning. our focus is on the simpler formulation in this preliminary work. @xcite identified four types of noise that may cause confusion: physical noise (e.g., a loud highway), physiological noise (e.g., hearing impairment), psychological noise (e.g., attentiveness of listener), and semantic noise (e.g., word choice). we postulate that mitigating confusion resulting from each type of noise may be possible, at least to some extent, given sufficient context to make an informed compensatory decision. for example, given a particularly physically noisy environment, speaking loudly would seem appropriate. unfortunately, such contextual information is often lacking from existing data sets. in particular, the physiological and psychological states of listeners is rarely recorded. even when such information is recorded (e.g., in alzheimer's speech studies @xcite , the information is very coarse (e.g, broad alzheimer's categories such as none, early-stage, and middle-stage). we leave these non-trivial data gathering challenges as future work, instead focusing on phonemic confusion, which is significantly easier to operationalize. in practice, confusion at the phonemelevel may arise from any category of rothwell noise. it may also arise from the natural similarities between phonemes (discussed next). while many of these will not be represented in the text-based phonemic transcriptions data set used in this preliminary work, our approach can be extended to include them. researchers in speech processing have studied the prediction of phonemic confusion but, to our knowledge, this work has not been adapted to utterance generation. instead, tasks such as preventing of sound-alike medication errors (i.e., naming medications so that two medications do not sound identical) are common @xcite . @xcite showed that the potential confusability of a word can be estimated by calculating the levenshtein distance @xcite of its phonemic transcription to that of all others in the vocabulary. we take inspiration from @xcite and employ a phoneme-level levenshtein distance approach in this work. in the basic definition of the levenshtein distance, all errors are equally weighted. in practice, however, words that share many similar or identical phonemes are more likely to be confused for one another. given this, @xcite developed a weighted phoneme-level levenshtein distance, where weights are determined by a human expert or a learned model, such as a hidden markov model. unfortunately, while these weights are meant to represent phonemic similarity, selecting an appropriate distance metric in phoneme space is non-trivial. the classical results of @xcite and @xcite group phonemes experimentally based on the noise level at which they become indiscernible. the authors identify voicing, nasality, affrication, duration, and place of articulation as sub-phoneme features that predict a phoneme's sensitivity to distortion, and therefore measure its proximity to others. unfortunately, later work showed that these controlled conditions do not map cleanly to the real world @xcite . in addition, @xcite found alternative phonemic distance features that could be adapted into a distance metric. while this prior research sought to directly define a distance metric between phonemes based on sub-phoneme features, since no method has emerged as clearly superior, researchers now favour direct, empirical measures of confusability @xcite . likewise, our work assumes that these classical feature-engineering approaches to predicting phoneme confusability can be improved upon with neural approaches, just as automatic speech recognition (asr) systems have been improved through the use of similar methods (e.g., @xcite @xcite . in addition, these classical approaches do not account for context (i.e., other phonemes surrounding the phoneme of interest), whereas our approach conditions on such context to refine the confusion estimate. ## model architectures and baseline with recent advances, various neural architectures have been applied to nlp tasks. early work includes n-gram-based, fully-connected architectures for language modeling tasks @xcite . recurrent neural network (rnn) architectures were then shown to be successful for applications such as language modeling, speech recognition, and phoneme recognition @xcite . rnn architectures such as the lstm @xcite and gru @xcite variants had been successful in many nlp applications, such as machine language translation and phoneme classification @xcite @xcite . recently, the transformer architecture @xcite , which uses attention instead of recurrence to form dependencies between inputs, has shown state-of-the-art results in many areas of nlp, including syllable-based tasks (e.g., @xcite . in this work, we propose a fixed-context-window architecture and a residual bi-lstm architecture for the central component of our confusionmitigation framework. while similar architectures have already been applied to phoneme-based applications, such as phoneme recognition and classification @xcite @xcite , to our knowledge, our study is the first to apply these architectures to identify phonemes related to confusion for listeners. in our opinion, these architectures strike an acceptable balance between compute and capability for this current work, unlike the more advanced transformer architectures, which require significantly more resources to train. 2 since the data set is imbalanced (see section 3.1), without sample weighting, early experiments showed that both architectures never identified any phonemes as likely to be mis-transcribed (i.e., high specificity, low sensitivity). accordingly, since the imbalance ratio is approximately 1:10, transcription errors are given 10-times more weight than properly-transcribed phonemes in our binary cross-entropy loss function. ## conclusion reducing communication breakdown is critical to successful interaction in dialogue systems and other generative nlp systems. in this work, we proposed a novel confusion-mitigation framework that such systems could employ to help minimize the probability of human confusion during an interaction. as a first step towards implementing this framework, we evaluated two potential neu-ral architectures-a fixed-context network and an lstm network-for its central component, which predicts the confusion probability of a candidate utterance. these neural architectures outperformed a weighted n-gram baseline (with the fixed-context network performing best overall) when trained using a proxy data set derived from audiobook recordings. in addition, qualitative analyses suggest that the fixed-context model has uncovered some of the more intuitive causes of phonemic confusion, including stop words, homonyms, near-homophones, and familiarity with the vocabulary. these preliminary results show the promise of our confusionmitigation framework. given this early success, further investigation and refinement is warranted.