Generating Signed Language Instructions in Large-Scale Dialogue Systems (2024)

Abstract: We introduce a goal-oriented conversational AI system enhanced with American Sign Language (ASL) instructions, presenting the first implementation of such a system on a worldwide multimodal conversational AI platform. Accessible through a touch-based interface, our system receives input from users and seamlessly generates ASL instructions by leveraging retrieval methods and cognitively based gloss translations. Central to our design is a sign translation module powered by Large Language Models, alongside a token-based video retrieval system for delivering instructional content from recipes and wikiHow guides. Our development process is deeply rooted in a commitment to community engagement, incorporating insights from the Deaf and Hard-of-Hearing community, as well as experts in cognitive and ASL learning sciences. The effectiveness of our signing instructions is validated by user feedback, achieving ratings on par with those of the system in its non-signing variant. Additionally, our system demonstrates exceptional performance in retrieval accuracy and text-generation quality, measured by metrics such as BERTScore. We have made our codebase and datasets publicly accessible at https://github.com/Merterm/signed-dialogue, and a demo of our signed instruction video retrieval system is available at https://huggingface.co/spaces/merterm/signed-instructions.

## introduction conversational systems have become increasingly integrated into our everyday lives, yet their accessibility to the deaf and hard-of-hearing (dhh) community, who predominantly communicate through signed languages, remains limited @xcite @xcite . despite growing advocacy for more inclusive interactive technologies from dhh users @xcite @xcite , a figure 1 : an overview of our multimodal dialogue system, capable of giving signed instructions to deaf or hard-of-hearing users in asl. we first translate task instructions to an intermediate textual representation called glosses using large language models; then, we fetch token-level sign videos to display on the screens of amazon alexa echo show. many signers prefer to use asl instead of text due to grammatical and linguistic differences between spoken and signed languages @xcite . yet currently, systems claiming to be accessible resort to textbased communication. as an alternative, videos or avatars of signers are options, yet these technologies are underutilized. in this paper, we show that deploying these signed systems on a large scale is, in fact, possible without much production cost and makes the system accessible to dhh users. further, prior linguistics research has shown that dhh community members can experience higher cognitive loads while reading compared to signing @xcite @xcite . in this paper, we investigate effective strategies of multimodal information presentation for the dhh to reduce cognitive load. with repeated consultations with cognitive scientists, we design the layout of our system's user interface specifically around the cognitive load of signers (see figure 2 ). we focus on creating a framework that is applicable to a large-scale global platform (in our case, amazon alexa), making it impossible at this time to access camera footage. we investigate ways of receiving input with other modalities instead of voice commands and without camera access. this leads us to focus on the task of instruction generation and delivery rather than recognizing signs produced by the user. we receive input from the user via touchscreen controls of amazon alexa echo show devices so that signers can interact without using voice commands (see figure 2 for the touch screen user interfaces where the user can interact via buttons to select tasks and navigate instructions). to address all of the aforementioned points, in the following sections, we introduce the components of our framework. our detailed contributions are as follows: ## related work with the rise of voice assistant devices, the dhh community has been mostly left behind. yet, there have been multiple lines of work to make them more accessible. accessibility of personal assistant devices to the deaf and hard of hearing community has been assessed multiple times before by @xcite ; @xcite . in addition, design approaches incorporating the dhh community have been proposed by @xcite ; @xcite . we build on these in our system design. most of the current work in interactive system design focuses on sign recognition with the help of cameras. for instance, in @xcite wizard-of-oz studies have been done where alexa is combined with a camera to detect signs. in signs project 1 , alexa recognizes specific gestures for simple task completion (such as getting the weather forecast with a specific gesture), and huang et al. recognized signs for a healing robot. even though these systems provide a means for recognizing signs, they fall short in generating signs, which we focus on in this paper. there has been some line of work by @xcite in generating avatars for 6-monthold babies to learn asl. also, @xcite deployed a kiosk with sign recognition and generation capabilities for czech sign language. however, these have not resulted in a widely available system. on the other hand, sign language processing has been widely studied under controlled conditions. even though sign language generation and translation tasks are still open problems, transformerbased models in @xcite read (2020); yin et al. (2021); moryossef et al. (2021); inan et al. (2022); müller et al. (2023); lin et al. (2023); viegas et al. (2023) have shown that it is possible to automate them better. as a core contribution, we present a framework to apply any of these models in largescale interactive environments. in order to make our system useful for signers, we need to mitigate their cognitive load interpreting instructions from multimodal devices. models 1 https://projectsigns.org/ for the cognitive aptitudes and cognitive loads of sign language interpreters have been studied before by @xcite ; du toit (2017); tiselius (2018); chambers (2020). these models help guide the design principles of our system, as the user will need to focus on multiple modalities simultaneously through the visual modality, which increases cognitive load. ## a goal-oriented dialogue system with signed instructions we design a multimodal goal-oriented dialogue system as part of the alexa prize taskbot challenge 2 @xcite and incorporate signed instructions. the main dialogue system that we develop follows a typical modular design: natural language understanding (nlu), dialogue manager (dm), and natural language generation (nlg). in this setting, we embed signed instructions into the multimodal nlg module (figure 3 ). due to privacy regulations, alexa does not allow third parties to process user gestures and videos. hence, to increase accessibility for signers, we choose to generate signed instructions instead of recognizing signs. to support users who cannotor prefer not to-provide voice input, our system has a scrollable touchscreen with buttons. this enables us to have a full dialogue system for signers while complying with regulations. ## our signed instruction framework we employ the framework shown in figure 1 to generate signed instructions. we first retrieve instructions for a given task, and then we convert each step into gloss tokens, which are intermediary textual representations using rule-based sign language translation algorithms and llms. afterward, we segment each instruction into separate gloss tokens, retrieve sign videos for each, and stitch them backto-back to create a continuous video sequence. for each step, we display this sequence of videos and a picture of the step. the picture for each step generally shows the result of the action as described in the sign instructions. this approach is summarized in algorithm 1. ## system evaluation we evaluate our system both quantitatively and qualitatively. because this is the first deployment of a task-oriented signed multimodal dialogue system, we chiefly compare the system with the non-signed portion of our task-oriented dialogue system. we first evaluate the performance of our llm text-togloss translation and discuss the trade-offs of using an llm for translation. then, we evaluate our algorithm using traditional information retrieval metrics. finally, we compare user ratings and provide detailed qualitative analyses by an expert who is fluent in asl. ## conclusion in this work, we discussed a multimodal, taskoriented dialogue system designed to generate asl instructions on a platform with global reach. emphasizing the critical importance of deaf and hard-of-hearing (dhh) community engagement throughout the development cycle, our approach integrates extensive feedback from both the signing community and experts in the field. our system not only marks a significant technological milestone but also enriches the dialogue on how video-based asl instruction delivery can be effectively scaled internationally. we observed a nuanced preference among signers for avatar-based instructionsa finding underscored by our expert analysis. our system has improved the landscape of conversational ai, making it accessible and responsive to the unique needs of the dhh community. we make the code available for our pipeline and encourage future researchers to incorporate it into their own task-oriented systems to increase accessibility. we hope that this system is a step towards developing dialogue systems that can understand and generate signs for all signed languages. we encourage everybody to interact with signed tasks by visiting https://huggingface. co/spaces/merterm/signed-instructions .