R ead A long Studio Web Interface for Digital Interactive Storytelling (2023)

Abstract: We develop an interactive web-based user interface for performing textspeech alignment and creating digital interactive “read-along audio books that highlight words as they are spoken and allow users to replay individual words when clicked. We build on an existing Python library for zero-shot multilingual textspeech alignment (Littell et al., 2022), extend it by exposing its functionality through a RESTful API, and rewrite the underlying speech recognition engine to run in the browser. The ReadAlong Studio Web App is open-source, user-friendly, prioritizes privacy and data sovereignty, allows for a variety of standard export formats, and is designed to work for the majority of the world’s languages.

## introduction a "read-along", as seen in figure 1 , is an interactive language tool that highlights words as they are spoken and allows users to replay certain words when clicked @xcite . language learners are able to interact with these multimodal text/audio documents by repeating the pronunciation of specific words, pausing at a specific place in the document, and following along visually as the text is spoken. this tool promotes reading and listening skills in language learners, which are target skills that are underrepresented in languagelearning technology @xcite . while the readalong studio web app is compatible with many languages ( §2.1), it was designed specifically to support learners in an indigenous language revitalization context, where listening comprehension is often a key priority @xcite . in addition to being more beneficial for comprehension than reading or listening alone @xcite , reading while listening can also help promote listeningbased skills such as auditory discrimination (i.e., the ability to discriminate between sounds) (chang, 2009). furthermore, read-alongs could be used to promote speaking skills by using them in conjunction with speaking tasks, such as shadowing (i.e., reading/speaking along with fluent speech while trying to match pace) @xcite . building read-alongs, however, can be challenging. aligning text and speech manually requires a considerable amount of time, and requires some expertise in using audio software. on the other hand, while text-speech alignment can be automated (e.g., @xcite @xcite , these systems require non-trivial expertise in speech technology and machine learning to train and deploy a model for a new language (mackenzie and turton, 2020). the zero-shot text-speech aligner described in @xcite partially addresses this issue; it can align speech and text in a new language without having seen any prior data in that language. however, it is still a command-line tool that, for full use of its capabilities, requires some familiarity with text and xml formats; it is still not something the average language teacher could use without significant training. thus, we decided to develop a web-based graphical user interface on top of that system. in the process, we ported a significant amount of the system to javascript so that audio processing could happen entirely in the browser. this allows users to create their own read-alongs without requiring them to write code or install anything on their computers, and without sending any audio data to a third party. the software is free and open source, and has a data privacy policy designed to affirm community data sovereignty. ## discussion & usage we held two 90 minute workshop sessions titled "watch me speak! interactive storytelling using readalong studio" at the 8 th international conference on language documentation & conservation to walk potential users through the readalong studio web app in march 2023. the final twenty minutes of each workshop were dedicated to a "language party", inspired by the aikuma project's initiative of the same name: 17 participants were invited to create a read-along for their language and share it with the group. participants created and shared read-alongs in crow (siouan), halkomelem and nsyilxc@n (salishan), michif, gitksan (tsimshianic), quechua, korean, nuu-chah-nulth (wakashan), paiwan (austronesian), sáliba (piaroa-saliban), takelma, and western armenian (indo-european). following the workshops, a participant shared that they had been nervous to create and share their read-along during the language party session of the workshop because they were worried they might not spell things correctly (since they did not have the language-specific keyboard installed), and potentially cause the system to break or not function properly. they were pleasantly surprised when the words of their text became highlighted with their voice. while we built readalong studio web app to be language agnostic, it is also agnostic to dialect and orthographic variations, which are very common for a variety of reasons in indigenous language revitalization contexts in canada (see §5 of @xcite . many tools that are created for a language revitalization context do not offer such generalized support for different dialects or writing systems, potentially systematically excluding certain users. by contrast, the readalong studio web app's tolerance for variations in pronunciation and spelling shows potential for fostering a non-judgemental environment for learners to practise speaking their language. according to plausible analytics, of the 525 unique visitors from 23 countries to visit the readalong studio web app in the first two months after the launch on february 26, 2023, 122 users created 396 read-alongs, with 65 users going on the tour and 56 users downloading their read-alongs. among the 56 unique users that downloaded readalongs, they downloaded read-alongs 174 times in the various available formats. the most popular format (which is also the default) was the offline html version. while the total number of people creating read-alongs is still modest, we are encour-17 https://www.languageparty.org/ aged by the amount those users are interacting with the tool and we believe that these statistics demonstrate achievement of our goal for users to be able to create read-alongs for many languages without requiring technological expertise. ## conclusion in this paper, we detailed the motivation for, and design decisions of, web-based software for creating read-alongs titled readalong studio web app. as key design considerations, we highlighted support for many languages ( §2.1), portability and longevity ( §2.2), avoiding vendor lock-in ( §2.4), and affirming privacy and data sovereignty concerns ( §2.5). future work involves improving the workflow for correcting and adjusting alignments, increasing language support, and refactoring the g2p engine to javascript for complete client-side processing.