Examining Prosody in Spoken Navigation Instructions for People with Disabilities (2024)

Abstract: The introduction of conversational systems have made synthesized speech technologies common tools for daily activities. However, not all synthetic speech systems are designed with the needs of people with disabilities in mind. This paper describes a study in which 198 people – 80 participants with self-reported disabilities and 118 participants without – were recruited to listen to navigation instructions from a spoken dialogue system with different prosodic features. Results showed that slowing down speech rate aids in participants’ number recall, but not in noun recall. From our results, we provide suggestions for developers for building accessible synthetic speech systems.

## introduction the introduction of conversational systems such as apple's siri and amazon's alexa have made synthesized speech technologies common tools for daily activities. however, people with disabilities still struggle to interact with synthetic speech systems @xcite . on the nlp side, current work in accessibility focuses on enhancing system or model functionality. examples include finding appropriate data to train deep learning models that can be tailored to people with disabilities @xcite , or examining negative language model biases against disabilities @xcite ). yet, user evaluation of these models deployed in systems are limited. on the other hand, past work in hci and speech systems have noted the importance of using appropriate prosody to achieve better user understanding and recall of the system outputs. @xcite @xcite @xcite . to tie together perspectives from both nlp and hci, we conducted a study in which user recall of instructions from an existing spoken dialogue system was evaluated in order to ( 1 ) determine what speech features are most beneficial for user recall of information, and (2) decide on future features to implement in the system. we recruited 198 people (with and without disabilities) and asked them to recall information under speech conditions which had either (1) slowed down speech or (2) pauses before keywords in the instruction. furthermore, we grounded our study to the realm of understanding navigation instructions, which is a challenging setting because it requires users to remember exact numeric entities (e.g., departure times, building numbers) and noun entities (e.g., unfamiliar street names) in order to navigate effectively. our results showed that across all participants (with and without disabilities), slowing down speech rate aided in recalling numeric entities, even as the number of numeric entities in an instruction increased, but was less effective for noun entity recall. furthermore, in speech conditions where breaks were inserted before numeric and noun entities, participants in general had lower performance in information recall. our findings suggest that developers may be able to make adjustments that promote overall accessibly. in addition, appropriate cues for different types of information (i.e., numeric vs. noun) should be considered. ## related work appropriate prosody in synthetic speech systems is important since synthetic speech often lacks natural cues and pacing found in human voices. previous works have shown that inappropriate or absent prosodic cues led to large performance gaps in recall tasks from natural and synthetic speech @xcite , and that human voices were preferred 1 over synthetic voices as task complexity increases @xcite . on the other hand, appropriate prosodic cues can improve peoples' abilities to disambiguate information -for example, distinguishing between similar spoken mathematical expressions @xcite . furthermore, adding appropriate prosodic cues can decrease performance gaps between different age groups in understanding speech @xcite @xcite @xcite . some existing nlp systems have taken this information into account. for instance, cmu getgoing @xcite is a trip-planning dialogue that introduced attention-grabbing prefixes and allowed "barge-in" to provide easier interaction for seniors. however, many modern spoken dialogue systems still fail to capture nuances of interaction for people with disabilities @xcite . while previous work such as @xcite showed that synthetic speech comprehension among people with disabilities were influenced by the complexity of task and acoustic-phonetic features, our study differs from previous work in that we grounded our study in a trip-planning task. this task is challenging to users (i.e., recalling unfamiliar street names), yet is important for independent daily travel, especially for people who are dependent on public transit. ## overview to analyze the effects of speech prosody on information retention, we recruited participants with 108 and without self-reported disabilities to listen to 109 audio clips containing navigation instructions. the outline of our paper is as follows: in section 4 we describe the audio clip collection process for the study. in section 5 we explain the study design and participant recruitment. finally, in sections 6, 7 and 8 we provide study results and further discussion. ## audio clips collection audio clips were collected from getgoing, a spoken dialog system that provides step-by-step navigation instructions to aid senior users in trip planning. getgoing was previously deployed in southwestern pennsylvania and could be accessed by users through telephone. it follows a traditional pipeline dialog system architecture with natural language understanding and speech modules. notably, the system uses google maps api foot_0 to curate sets of directions between two locations. furthermore, the system uses vonage's text-to-speech api foot_1 to output synthesized speech to the user. the audio clips collection process involved three stages. in the first stage, getgoing was used to generate navigation instructions using random start and end destination points around southwestern pennsylvania. in the second stage, a subset of the generated instructions was selected according to two information parameters defined in section 4.1 to ensure our instruction set was well-balanced. this resulted in 48 unique instructions. in the third stage, speech conditions (described in section 4.2) were created by either slowing down the speech or inserting breaks before keywords using the vonage api. we recorded the speech output of getgoing for each of the 48 instructions under each condition, and ended up with a total of 192 audio clips. ## study design in this section, we describe the study design, procedure, and participants. all participants were recruited on prolific.co. from there, they were directed to a qualtrics page to consent to the study. after consenting, participants were redirected to a website to do the tasks. an example of the task interface is provided in figure 4 in the appendix. each participant listened to 24 audio clips and entered what they heard in a text box on the website. participants listened to six clips from each of the four conditions. each of these six clips were randomly sampled from instructions from different parameter groups. all clips were presented to the participant in random order, and the website prevented participants from listening to the clip more than once in order to test recall. once the participants finished the tasks, they were redirected back the the qualtrics page where they answered a short questionnaire, which is included in ## results we grouped participants in two categories: those who had self-reported disabilities and those who did not, and analyzed recall of noun and numeric entities across the four speech conditions. while it is possible to examine the subgroups of selfreported disabilities individually (e.g., all participants with a mobility disability), we avoided this since the subgroups are imbalanced with respect to the number of participants. in addition many participants reported more than one disability. ## discussion there are several implications that can be drawn from our findings. first, from our experiments in section 6.2, we observed that speech conditions did not lead to large differences on noun retention between (1) individuals with and without self-reported disabilities, and (2) across age groups. in addition, speech conditions had little effect on noun retention even as noun length increased across all participants as highlighted in section 7.1.2. this suggests that system designers may have some flexibility when presenting nouns to users, as long as the nouns (i.e., names, streets, places) are not long. however, additional care may be taken into consideration when some nouns are more important to remember than others, and this is left as a direction for future research next, with respect to number retention, in section 6.1 we observed that having a slower overall speech rate was helpful for number recall, while inserting pauses before numbers had negative effects. furthermore, results in section 7.1.1 showed that a slower overall speech rate also aided in number retention as length of numbers (i.e., number of digits in a number) increased. however, a follow up question is why participants in general performed worse in break-short and break-long conditions compared to default-slow and default conditions with respect to number retention. a possible explanation may be that added pauses before every key piece of information caused participants to focus on too many cues. this can overload the participants with information, and cause forgetting. further investigation on where to place appropriate pauses is beneficial. overall, these findings suggest that system developers should take into account speech prosody when communicating numbers, regardless of whether the user has a disability. interestingly, our study showed little difference in noun and number recall between participants with and without self-reported disabilities in general. a possible explanation is that participants did our study online, and therefore had their environment and computing device set up for good listening. for example, people with hearing loss may have turned up the volume on their speakers, worn headphones, or enabled a bluetooth connection to hearing aids. hence, future research should explore use in less optimal conditions (e.g., using a phone from a city street). based on our work, we have several recommended directions for future research. one point to consider is limiting prosodic cues based on information priority. for instance, rather than inserting pauses before every noun and number, only inserting pauses before long numbers and uncommon nouns may lead to a positive effect on recall. furthermore, we acknowledge that our study is limited to noun and number recall, and that our analysis considered these parameters to be independent of each other. in realistic settings, other important pieces of information may also be present in navigation directions. for instance, instructing the user to "turn right on frew st and go up the ramp to the bus station" adds additional load to remember specific remember actions they must take. also, some pieces of information are more important for the user to understand and recall than others. for example, a user may prioritize recalling bus arrival times over the name of their destination stop street. finally, finding effective prosodic features for information retention could be explored as a problem of personalization (e.g., blind user preferences for screenreaders). for example, allowing users to select voice styles alongside information presentation styles can allow for easier usage for individuals with different disabilities. finding successful ways to achieve this goal also requires further investigation on how people with disabilities interact with spoken dialogue systems. ## conclusion in this study, we recruited people with and without disabilities and evaluated their information retention in different speech conditions. we found that having an overall slow speech rate was useful for number retention across all participants (with and without self-reported disabilities), but was less effective in improving noun retention. we also showed that inserting breaks before nouns and numbers did not improve in information retention. thus, finding appropriate prosodic cues for different pieces of spoken information is an interesting direction to explore.