Speakerly: A Voice-based Writing Assistant for Text Composition (2023)

Abstract: We present Speakerly, a new real-time voice-based writing assistance system that helps users with text composition across various use cases such as emails, instant messages, and notes. The user can interact with the system through instructions or dictation, and the system generates a well-formatted and coherent document. We describe the system architecture and detail how we address the various challenges while building and deploying such a system at scale. More specifically, our system uses a combination of small, task-specific models as well as pre-trained language models for fast and effective text composition while supporting a variety of input modes for better usability.

## introduction writing is a multi-step process that involves planning (ideation), translation (composition), and reviewing (revision) @xcite . in the ideation phase, the writer gathers information and organizes their thoughts. the composition step involves articulating the ideas effectively through the use of the right words and arranging them cogently in a draft. during revision, the focus is on grammatical correctness, logical flow of ideas, coherent document structure, and style. most current writing assistants have been limited in their ability to provide seamless writing assistance across all the stages, take into account the user context, and be robust to work on diverse real-world use cases at scale @xcite . in this work, we introduce speakerly ™ , a voicebased end-to-end writing assistance system that works across the different stages of writing, helping users become more efficient with their communication. the user uses the voice interface to articulate their thoughts in natural speech. our system then creates a polished and ready-to-send first draft while addressing all the intermediate issues, figure 1: an illustrative example of speakerly ™ for email composition on mobile. a user presses the microphone button at the bottom of their email application and starts speaking naturally (no templatization or structural tailoring of the speech input is needed). once they stop speaking, speakerly ™ converts the speech into structured, well-formatted, and polished compositions. we use voice as it is a natural and efficient input modality, allowing users to compose their thoughts quickly and even use the system in eyes-free scenarios while performing tasks such as walking and driving @xcite @xcite . moreover, with the increased ubiquity of voice-based assistants, such as alexa and siri, voice-based interactions have become more common and intuitive for users @xcite . however, using voice has some challenges. first, during the ideation stage, the user typically only has a rough idea of what they want to write. thus, if the system is unable to handle a lack of structure and slight incoherence in the input, users will end up spending a significant amount of time on fixing the output. second, different writers can have varied needs requiring the system to handle the demands and constraints of different use cases. for example, short vs. long inputs, instructional vs. dictation inputs, open vs closed-ended inputs, and specific structures and formatting for emails, instant messages, and notes ( speakerly ™ is composed of multiple stages (section 3) that progressively refine the relatively noisy and unstructured speech from the user and address the aforementioned challenges and requirements. in the remainder of this paper, we describe the technical system architecture and our approaches to address challenges related to modeling, evaluation, inference, and sensitivity. ## related work most research in the past has been limited to either a single use case for composition or one particular stage of the writing process. for example, previous works have focused on email writing @xcite , science writing @xcite , story writing @xcite , slogan and metaphor writing @xcite , poetry writing @xcite , and support comments @xcite , to name a few. our system, in contrast, can handle various use cases ranging from short instant messages to long notes to open-ended instructions to closedended and information-dense dictations. on the other hand, some writing assistancefocused works disproportionately emphasize specific stages of writing, such as editing and revision @xcite @xcite @xcite rather than end-to-end writing assistance. again, in contrast, our system is much more extensive as it takes in noisy and unstructured speech input and iteratively refines it to produce a final well-formatted output rather than focusing on a single-shot, structured text-to-text transformation. voice-based input has been known to optimize people's interaction and has been studied in the past @xcite and is well-integrated in virtual assistants (such as siri and alexa). it has been used for various tasks such as voice notes @xcite , data capturing @xcite , information querying @xcite and data exploration @xcite . such systems can have speech recognition errors that are difficult to recover from and restrict the user's natural speaking behavior @xcite . to tackle these problems, recent works have looked at voice-based text editing @xcite . ## system description our system takes natural speech from the user as input and generates a coherent and well-formatted text output. as shown in fig. 2 , the input progressively gets refined and enhanced as it traverses the pipeline, consisting of multiple task-specific models. each stage can have its own errors. hence, models across the pipeline are designed with complementary, sometimes overlapping capabilities, which allows them to recover from errors collectively and improve robustness to variation and noise in the input. the pipeline has three main components: automatic speech recognition (asr), normalization, and comprehension. the asr module takes raw speech and converts it to text. then, the normalization module cleans up speech disfluencies, adds punctuation, and applies grammatical error corrections (gec). finally, the comprehension module cleans the text of remaining issues, such as incoherent document structure, word choice, formatting, formality, and style, and composes the final output text, handling instruction or dictation, or any other mode of input for a variety of use cases. we now explain these three components in more detail: ## conclusion in this paper, we presented speakerly ™ , a real-time voice-based writing assistant for text composition. it provides a low barrier to entry into the writing process, where a user can interact naturally, either using dictation, instructions, or unstructured thoughts. in turn, it generates a high-quality first draft with low latency, thus, providing them with a simple and efficient way to articulate their thoughts into ready-to-send emails, messages, or notes. we present comprehensive technical details of the different stages of the pipeline and experiments which guided our decisions while deploying the system to our users.