Toward Interactive Dictation (2023)

Abstract: Voice dictation is an increasingly important text input modality. Existing systems that allow both dictation and editing-by-voice restrict their command language to flat templates invoked by trigger words. In this work, we study the feasibility of allowing users to interrupt their dictation with spoken editing commands in open-ended natural language. We introduce a new task and dataset, TERTiUS, to experiment with such systems. To support this flexibility in real-time, a system must incrementally segment and classify spans of speech as either dictation or command, and interpret the spans that are commands. We experiment with using large pre-trained language models to predict the edited text, or alternatively, to predict a small text-editing program. Experiments show a natural trade-off between model accuracy and latency: a smaller model achieves 30% end-state accuracy with 1.3 seconds of latency, while a larger model achieves 55% end-state accuracy with 7 seconds of latency.

## introduction speech can be preferable for text entry, especially on mobile devices or while the user's hands are occupied, and for some users for whom typing is always slow or impossible. while fast and accurate automatic speech recognition (asr) is now ubiquitous (kumar et al., 2012; xiong et al., 2016; chiu et al., 2018; radford et al., 2022), asr itself only transcribes speech. in practice, users may also wish to edit transcribed text. the asr output might be incorrect; the user might have misspoken; or they might change their mind about what to say or how to phrase it, perhaps after seeing or hearing their previous version. azenkot and lee (2013) found that users with visual impairment spent 80% of time editing text vs. 20% dictating it. figure 1 : a user writes an email using speech input, interleaving dictation (a,c) and commanding (b,d). top shows the continuous user utterance, while bottom shows the document state at each point of the utterance. dictations are transcribed verbatim, while commands are interpreted and executed. our system supports openended commanding (i.e., b,d both invoke a replace operation but use vastly different phrasing). in this work, we study the task of interactive dictation, in which users can both perform verbatim dictation and utter open-ended commands in order to edit the existing text, in a single uninterrupted speech stream. see figure 1 for an example. unlike commercial systems like dragon (dns; nuance, 1997, 2022) and dictation for word (microsoft, 2022) that require reserved trigger words for commanding, the commands in our data are invoked using unrestricted natural language (nl). for example, in figure 1 , both (b) and (d) invoke replace commands, but (d) uses nested syntax to specify both an edit action and location, while (b) is implicit (as natural speech repairs often are). in interactive dictation, users do not need to memorize a list of specific trigger words or templates in order to invoke their desired functionality. a dictation system should be as intuitive as dic-tating to a human assistant-a situation in which people quite naturally and successfully intersperse speech repairs and commands with their dictation. beyond eliminating the learning curve, letting users speak naturally should also allow them to focus on what they want to say, without being repeatedly distracted by the frustrating separate task of getting those words into the computer. because we accept unrestricted nl for commands, both segmentation and interpretation become nontrivial for a system to perform. @xcite segmentation requires capturing (sometimes subtle) changes in intent, and is especially difficult in cases where command boundaries do not align with asr boundaries. @xcite we collect a dataset of 1320 documents dictated in an interactive environment with live, incremental asr transcription and wizard-of-oz-style interpretation of user commands. annotators were not told a set of editing features they were allowed to use, but simply instructed to make their commands understandable and executable by a hypothetical human helper. collection required designing a novel data collection interface. both the interface and dataset will be publicly released to help unlock further work in this area. @xcite finally, we experiment with two strategies for implementing the proposed system: one that uses a pre-trained language model to directly predict the edited text given unedited text and a command, and another that interprets the command as a program specifying how to edit. predicting intermediate programs reduces latency because the programs are short, at the expense of accuracy. this strategy also requires additional work to design and implement a set of editing functions and annotate commands with programs that use these functions. for each strategy, we also experimented with two choices of pre-trained language model: a small finetuned t5 model and a large prompted gpt3 model. using the smaller model significantly improves latency, though again at the cost of accuracy. in summary, our contributions are: (1) a novel @xcite in template-based systems, by contrast, commands can be detected and parsed using regular expressions. an utterance is considered a command if and only if it matches one of these regular expressions. @xcite in figure 1 , for example, we must segment the first sentence into two parts, a dictation ("just wanted to ask about the event on the 23rd") and a command ("on friday the 23rd"). asr can also overpredict boundaries when speakers pause in the middle of a sentence. for example, in our data "change elude mansion to elude mentioned." was misrecognized by mss as "change. elude mansion to elude mentioned." ## background & related work many modern speech input tools only support direct speech-to-text (e.g., radford et al., 2022). occasionally, these models also perform disfluency correction, which includes removing filler words (e.g., um), repeated words, false starts, etc. prior work has collected datasets and built systems specifically for speech repair (heeman and allen, 1994, 1999; johnson and charniak, 2004). additionally, asr systems themselves make errors that humans may like to correct post-hoc; there has been work on correcting asr errors through respeaking misdetected transcriptions (mcnair and waibel, 1994; ghosh et al., 2020; vertanen and kristensson, 2009; sperber et al., 2013). beyond disfluencies that were not automatically repaired but were transcribed literally, humans must fix many other mistakes while dictating. they often change their mind about what to say-the human writing process is rarely linear-and asr itself commonly introduces transcription errors. most systems require the user to manually fix these errors through keyboard-and-mouse or touchscreen editing (e.g., kumar et al., 2012), which can be inconvenient for someone who already relies on voice for dictation. furthermore, most commercial systems that support editing through speech (dns, word) require templated commands. thus, while speech input is often used to write short-form, imprecise text (e.g., search queries or text messages), it is not as popular as it might be, and it is used less when writing longer and more precise documents. in our work, we study making edits through spoken natural language commands. interpreting flexible natural language commands is a wellstudied problem within nlp, with work in semantic parsing (zelle and mooney, 1993; zettlemoyer and collins, 2009; artzi and zettlemoyer, 2013), instruction-following (chen and mooney, 2011; branavan et al., 2009; tellex et al., 2011; anderson et al., 2018; misra et al., 2017), and task-oriented dialogue (budzianowski et al., 2018). virtual assistants like siri (apple, 2011), alexa (amazon, 2014), and google assistant (google, 2016) have been built to support a wide range of functionalities, including interacting with smart devices, querying search engines, scheduling events, etc. due to advances in language technologies, modern-day assistants can support flexible linguistic expressions for invoking commands, accept feedback and perform reinterpretation (semantic machines et al., 2020), and work in an online and incremental manner (zhou et al., 2022). our work falls in this realm but: (1) in a novel interactive dictation setting, (2) with unrestricted commanding, and (3) where predicting boundaries between dictations and commands is part of the task. recently, a line of work has emerged examining how large language models (llms) can serve as collaborative writing/coding assistants. because of their remarkable ability to generate coherent texts over a wide range of domains and topics, llms have proven surprisingly effective for editing, elaboration, infilling, etc., across a wide range of domains (malmi et al., 2022; bavarian et al., 2022; donahue et al., 2020). though our system also makes use of llms, it supports a different mode of editing than these prior works. some works use edit models for other types of sequence-tosequence tasks (e.g. summarization, text simplification, style transfer) (malmi et al., 2019; dong et al., 2019; reid and zhong, 2021), while others use much coarser-grained editing commands than we do, expecting the llm to (sometimes) generate new text (bavarian et al., 2022; zhang et al., 2023). in addition to these differences, our editing commands may be misrecognized because they are spoken, and may be misdetected/missegmented because they are provided through the same channel as text entry. ## task framework we now formalize our interactive dictation setting. a user who is editing a document speaks to a system that both transcribes user dictation and responds to user commands. this process results in a interactive dictation trajectory-a sequence of timestamped events: the user keeps speaking, several trained modules keep making predictions, and the document keeps being updated. supervision could be provided to the predictive modules in various ways, ranging from direct supervision to delayed indirect reward signals. in this paper, we collect supervision that can be used to bootstrap an initial system. we collect gold trajectories in which every prediction is correct-except for asr predictions, where we preserve the errors since part of our motivation is to allow the user to fix dictation errors. @xcite all predictions along the trajectory are provided in the dataset. our dataset is not completely generic, since it assumes that certain predictive modules will exist and interact in particular ways, although it is agnostic to how they make their predictions. it is specifically intended to train a system that is a pipeline of the following modules (figure 2 ): (a) asr as the user speaks, the asr module proposes transcripts for spans of the audio stream. due to asr system latency, each asr result normally arrives some time after the end of the span it describes. the asr results are transcripts of successive disjoint spans of the audio, and we refer to their concatenation as the current transcript (u in figure 2(a) ). (b) segmentation when the current transcript changes, the system can update its segmentation. it does so by partitioning the current transcript u into a sequence of segments u i , labeling each as being either a dictation or a command. (c) normalization (optional) each segment u i can be passed through a normalization module, which transforms it from a literal transcript into clean text that should be inserted or interpreted. this involves speech repair as well as text normalization to handle orthographic conventions such as acronyms, punctuation, and numerals. while the module (a) may already attempt some version of these transformations, an off-the-shelf asr module does not have access to the document state or history. it may do an incomplete job and there may be no way to tune it on gold normalized results. this normalization module can be trained to finish the job. including it also ensures that our gold trajectories include the intended normalized text of the commands. attached are the espeak events. please review. ## dataset creation to our knowledge, no public dataset exists for the task of interactive dictation. as our task is distinct from prior work in a number of fundamental ways ( §2), we create a new dataset, tertius. @xcite our data collection involves two stages. first, a human demonstrator speaks to the system and provides the gold segmentations, as well as demonstrating the normalizations and document state updates for the command segments. later, for each command segment, an annotator fills in a gold program that would yield its gold state update. for a command segments, we update the document during demonstration using the demonstrated state updates-that is, they do double duty as gold and actual state updates. thus, we follow a gold trajectory, as if the demonstrator is using an oracle system that perfectly segments their speech into dictations (though these may have asr errors) versus commands, and then perfectly interprets the commands. a future data collection effort could instead update the document using the imperfect system that we later built ( §5), in which case the demonstrator would have to react to cascading errors. ## modeling & training the overall system we build for interactive dictation follows our pipeline from figure 2 and §3: 1. a segmentation model m seg takes the current transcript u, and predicts a segmentation u 1 , . . . , u n , simultaneously predicting whether each u i corresponds to a dictation or command segment. below we describe the specific models we use. ## results we evaluate the segmentation model in isolation, and the normalization and interpretation steps together. (appendices d.2 and d.3 evaluate the normalization and interpretation steps in isolation.) for simplicity, we evaluate the models only on current transcripts u that end in final asr results (though at training time and in actual usage, they also process transcripts that end in partial ones). @xcite 6.1 segmentation metrics exact match (em) returns 0 or 1 according to whether the entire labeled segmentation of the final transcript u is correct. we also evaluate macro-averaged labeled f1, which considers how many of the gold labeled segments appear in the model's output segmentation and vice versa. two labeled segments are considered to be the same if they have the same start and end points in u and the same label (dictation or command). results segmentation results on an evaluation dataset of transcripts u (see appendix d.1) are shown in the top section of ## conclusion most current speech input systems do not support voice editing. those that do usually only support a narrow set of commands specified through a fixed vocabulary. we introduce a new task for flexible invocation of commands through natural language, which may be interleaved with dictation. solving this task requires both segmenting and interpreting commands. we introduce a novel data collection framework that allows us to collect a pilot dataset, tertius, for this task. we explore tradeoffs between model accuracy and efficiency. future work can examine techniques to push out the pareto frontier, such as model distillation to improve speed and training on larger datasets to improve accuracy. future work can also look at domains outside of (work) emails, integrate other types of text transformation commands (e.g., formatting), and may allow the system to respond to the user in ways beyond updating the document.