The Post-Stroke Speech Transcription ( PSST ) Challenge (2022)

Abstract: We present the outcome of the Post-Stroke Speech Transcription (PSST) challenge. For the challenge, we prepared a new data resource of responses to two confrontation naming tests found in AphasiaBank, extracting audio and adding new phonemic transcripts for each response. The challenge consisted of two tasks. Task A asked challengers to build an automatic speech recognizer (ASR) for phonemic transcription of the PSST samples, evaluated in terms of phoneme error rate (PER) as well as a finer-grained metric derived from phonological feature theory, feature error rate (FER). The best model had a 9.9% FER / 20.0% PER, improving on our baseline by a relative 18% and 24%, respectively. Task B approximated a downstream assessment task, asking challengers to identify whether each recording contained a correctly pronounced target word. Challengers were unable to improve on the baseline algorithm; however, using this algorithm with the improved transcripts from Task A resulted in 92.8% accuracy / 0.921 F1, a relative improvement of 2.8% and 3.3%, respectively.

## introduction anomia, or word-finding difficulty, is the primary feature of aphasia @xcite , an acquired neurogenic language disorder that affects 2.5-4 million people in the us @xcite . the primary cause of aphasia is stroke, and 21%-40% of acute stroke patients are diagnosed with anomia by the time they are discharged. anomia is believed to be indicative of disruption in accessing a semantic description of the target concept, and/or retrieving a fully phonologically specified representation @xcite . specifically, paraphasias, which are unintended word production errors, typically result from reduced or insufficiently persistent activation of target representations relative to competing non-target representations and/or noise in the system @xcite . in some cases, people with aphasia produce real word errors. for example, reduced activation of lexical-semantic representations may result in semantic errors (e.g., "dog" for the target "cat") or unrelated errors, sharing no obvious semantic or phonological features with the target word ("chair" for "cat"). activation of inappropriate phoneme representations may sometimes result in real word errors (e.g., "dog" for the target "log"). however, breakdowns in phonological processing may also lead to non-word productions known as neologisms that may or may not be phonologically related to the target (e.g., "tat" for the target "cat" and "blat" for the target "dog", respectively). given the prevalence of anomia in the aphasic population and its tendency to persist even when other symptoms of aphasia remit @xcite , professionals typically assess anomia using confrontation naming tests @xcite @xcite , during which a patient is presented with pictures of simple ob-jects and they are asked to name them. the overall accuracy on such tests is an important clinical metric that has been found to be a good indicator of overall aphasia severity @xcite and is predictive of the ability to convey information during discourse production @xcite . furthermore, improvement in naming accuracy has been linked to improvement in overall communicative skills @xcite . further, in research settings, professionals develop individualized profiles based on the different types of errors elicited through confrontation naming tests (e.g., phonological, semantic, non-word errors, etc.) and then use these profiles to characterize patients' cognitive-linguistic deficits. such individualized error profiles have informed theoretical accounts of the cognitive machinery underlying word production @xcite @xcite ; lesion-symptom mapping @xcite @xcite ; personalization of treatments @xcite ; treatment efficacy studies @xcite @xcite ; the understanding of cross-linguistic treatment generalization @xcite ; and cortical reorganization investigations after a stroke @xcite error profiles also have the potential to be highly informative in clinical settings for developing individualized intervention plans @xcite . however, currently, developing a patient's profile is prohibitively time-and labor-intensive because it requires phonemic transcriptions for determining response accuracy and the nature of the errors. for naming tests with dozens or hundreds of items, this is rarely feasible in fast-paced clinical settings. as such, there is much interest in the clinical community in automating this process. to this end, we introduce the post-stroke speech transcription challenge (psst). the challenge is a shared task consisting of two sub-tasks, one for phonemic transcription (task a), one for binary classification @xcite . the goals of the psst challenge are threefold: first, to produce an accessible dataset relating to these clinical tasks for use by the machine learning community; second, to establish benchmarks for these tasks; and third, to lay the groundwork for a community of practice for machine learning researchers interested in aphasia and other similar disorders. ## preparing the psst corpus funding for the dataset preparation and baseline model development activities, and for the shared task itself, originated from the national institutes of health's office of data science strategy, under the "administrative supplements to support collaborations to improve the ai/ml-readiness of nih-supported data" program (not-od-21-094). the goal of this funding mechanism was to support efforts to promote and facilitate the use of existing biomedical datasets by the ai/ml community. the psst corpus is comprised of short speech segments from english aphasiabank @xcite , specifically responses to the boston naming test short form (bnt-sf) @xcite and verb naming test (vnt) @xcite portions of the protocol. participants included 107 individuals with aphasia who completed both bnt-sf and vnt as retrieved from english aphasiabank on september 1, 2021. we defined aphasia as an aphasia quotient (aq) @xmath0.8 on the western aphasia battery -revised @xcite @xmath1. participants were right-handed, predominantly englishspeaking, with a history of a single, left-hemispheric stroke, adequate hearing and vision, and no significant comorbid neurological or psychiatric illness. individuals with concomitant motor speech disorders were also included. the extracted segments averaged 3.9 seconds in length, include 3291 utterances from 107 speakers, and total approximately 3.5 hours of audio. ground truth phonemic transcriptions for the bnt-sf and vnt were derived from two previous studies and adapted for the purposes of this asr project. naming attempts were originally identified and phonemically transcribed by trained research assistants and disagreements resolved by a licensed speech-language pathologist. transcriptions were entered and time-aligned to audiovisual recordings using elan (max planck institute for psycholingustics, 2022). using the time alignments, we automatically extracted audio from the full aphasiabank videos, applying filters for loudness and clarity (see appendix b.2). a trained research assistant and licensed speech-language pathologist updated transcriptions to reflect the present study's conventions, then the transcripts were normalized and mapped to arpabet for asr purposes (see appendix b.3). correctness labels were assigned to all responses by a licensed speech-language pathologist, followed by an audit and resolution process by consensus. we defined correctness as the presence of the target word anywhere within the segmented response. pronunciation variations of the target word that could be explained by an individual's dialectal pattern and/or typical patterns of coarticulation were scored as correct. train validation test hours 2.59 (73%) 0.36 (10%) 0.59 (17%) segments 2173 (70%) 325 (10%) 624 (20%) speakers 74 (69%) 11 (10%) 22 (21%) data splits targeted train, valid, and test proportions of 70%, 10%, and 20% respectively, with quantities measured as hours of audio. as shown in ## task a: asr for aphasic speech task a asked participants to automatically transcribe the phonemes in each segment of recorded audio. we provided arpabet transcripts for the train and validation splits described in §3. we provided code to compute fer and per for these splits, and we made available our baseline model's source code and pretrained weights. shortly before the deadline, we released the audio from the test split with the transcripts withheld. challengers submitted the transcripts their models produced for the test set, and we used the same scripts to compute final metrics. we received entries from two challengers @xcite , who submitted transcripts for 7 models apiece. ## task b: correctness in task b, we asked participants to perform a simple example of a downstream task, namely, determining whether a recording contained a target word pronounced correctly. since the bnt-sf and vnt are confrontation naming tests, they are intended to elicit specific nouns and verbs (respectively) in response. for the challenge, we used the same audio samples as task a, with true/false labels provided by our annotators (see §3). we also provided a set of acceptable phoneme sequences for each stimulus, including all variations in conjugation, dialect, etc. that we found during data preparation. this allowed us to focus on the question of how to identify and preserve sufficient acoustic-phonetic information from a speech signal to improve on a downstream classification task. like task a, we provided scripts for the classification metrics for the train and valid splits, and we provided the source code for the baseline model. we received a submission from one challenger @xcite for task b. our baseline model relies on asr transcripts from task a, so we also experimented with the task a transcripts submitted by challengers. ## conclusion as we hoped, the psst participants improved on our baseline approach. the asr metrics fer and per were improved by a relative 18% and 24%, respectively. those improvements alone improved the f1 of the correctness task by a relative 3.3%. these ideas warrant further experimentation, and we expect progress will continue as a result of expanding the psst data and refining this work. to this end, as a next step we will investigate which linguistic and clinical characteristics pose the greatest challenge across the asr systems. further, we will assess how fer/per relate to the performance of downstream tasks; and, explore how different approaches to fer computation could improve its utility. at the same, we intent to continue expanding the psst dataset using aphasiabank data while also refining our evaluation methods. given the opportunity to hold another psst challenge, we see ample opportunity to raise the bar with the downstream tasks: introducing tasks like phonological and morphological similarity assessment, or leaning in to the complexities associated with accents and dialect.