Accelerating Text Communication via Abbreviated Sentence Input (2021)

Abstract: Typing every character in a text message may require more time or effort than strictly necessary. Skipping spaces or other characters may be able to speed input and reduce a user’s physical input effort. This can be particularly important for people with motor impairments. In a large crowdsourced study, we found workers frequently abbreviated text by omitting mid-word vowels. We designed a recognizer optimized for expanding noisy abbreviated input where users often omit spaces and mid-word vowels. We show using neural language models for selecting conversational-style training text and for rescoring the recognizer’s n-best sentences improved accuracy. On noisy touchscreen data collected from hundreds of users, we found accurate abbreviated input was possible even if a third of characters was omitted. Finally, in a study where users had to dwell for a second on each key, sentence abbreviated input was competitive with a conventional keyboard with word predictions. After practice, users wrote abbreviated sentences at 9.6 words-per-minute versus word input at 9.9 words-per-minute.

## introduction experienced desktop and touchscreen typists can often achieve fast and accurate text input by simply typing all the characters in their desired text. however, for some users, such quick and precise input is difficult due to a motor disability. such users may use a virtual touchscreen keyboard, but their touch locations may be slow and inaccurate, e.g. people with cerebral palsy. other users may need to click keys by pointing at them with a head-or eyetracker and dwelling for a fixed time, e.g. people with amyotrophic lateral sclerosis (als). when a person's typing is slow or inaccurate, word completions may provide more efficient input. word completions predict the most probable words based on the current typed prefix. however, monitoring predictions carries a cognitive cost and may not always improve performance @xcite . further, monitoring predictions can be difficult without visual feedback. eyes-free text input can be slow for users who are visually-impaired @xcite , and even slower for users who are motor-and visually-impaired @xcite . finally, eyes-free text input may be needed in future augmented reality (ar) interfaces where visual feedback is limited or non-existent (e.g. due to lighting or device limitations). in audio-only ar, it is still possible to type on an invisible virtual keyboard @xcite . all these cases motivate our interest in exploring alternatives to conventional word completion. here we investigate accelerating input by allowing users to skip typing spaces and mid-word vowels. we decided to abbreviate in this manner based on past results on touchscreen text input without spaces @xcite , and a study we present here in which 200 people abbreviated email messages. our interaction approach of abbreviation is similar to features in commercial assistive interfaces (e.g. grid 3, nuvoice, lightwriter). our whole utterance prediction approach is similar to features in touchscreen phone keyboards and in commercial assistive interfaces (e.g. dwell-free sentence input in tobii communicator 5). we modified a probabilistic recognizer to accurately expand abbreviated input by 1) improving our language models by selecting well-matched training data via a neural network, 2) modifying the search to model the insertion of mid-word vowels, and 3) adding a neural language model to the search. we validate our method in computational experiments on over six thousand sentences typed on touchscreen devices. we found that even when 28% of letters were omitted, we recognized sentences with no errors 70% of the time. selecting from the top three sentences, user could obtain their intended sentence 80% of the time. finally, we compare word completion and abbreviated sentence input in a user study. in this study, users had to dwell for one second to trigger a tap. we found sentence input was slightly slower than using word completions, but still saved substantial time compared to typing all the characters. users obtained their desired sentence 68% of the time. shieber and nelken (2007) allowed users to drop non-initial vowels and repeated consonants. this deleted 26% of the total characters. using an ngram word language model and a spelling transducer for each word, they expanded abbreviated text at an error rate of 3.3%. our work differs in that we: 1) removed spaces between words, 2) did not remove consecutive consonants, 3) used a character language model with no fixed vocabulary. @xcite explored japanese text input with digits. they used an hmm to expand a sequence of digits into characters. users saved 35% of keystrokes typing on a mobile phone. @xcite also used an hmm to expand abbreviations learned from a corpus of java code. their approach did not require memorizing abbreviations and provided incremental feedback while typing. in two studies with 31 users, @xcite identified common abbreviation behaviors such as vowel deletion, phonetic replacement, and word truncation. they did not release their data and it was on a relatively small number of people. based on their work, we conducted an abbreviation study with 200 users and also share our data. data selection. mismatch between the training and target text domains can lead to sub-optimal language models. a variety of methods have been developed to address this problem. @xcite , @xcite used language modeling and in-domain perplexity to select training data. in this approach, a language model is trained on a small in-domain dataset. training instances from an out-of-domain dataset are selected if they are below some perplexity threshold. other work has investigated data selection using cross-entropy or cross-entry difference between inand out-of-domain datasets @xcite @xcite @xcite . in this approach, an indomain and out-of-domain language models are first trained. sentences are selected based on a cross-entropy threshold or cross entropy difference calculated from the two language models. @xcite and @xcite applied information retrieval based techniques to select data. other method include selecting based on infrequent n-gram occurrences @xcite , or levenshtein distance and word vectors @xcite . @xcite employed the data selection method of @xcite , which builds upon moore and lewis (2010)'s approach. the main distinction is that they used neural language models for selection rather than n-gram models. chen and huang (2016), @xcite , and @xcite selected based on convolutional and bidirectional long short-term memory neural networks. bidirectional neural models like bert @xcite has proven effective in many natural language tasks. @xcite used bert for domain-discriminative data selection. @xcite used bert for domain adaptation and instance selection for disease classification. our selection method is similar to these methods but focuses on selecting conversational-style sentences. decoding noisy input. text entry interfaces often use a probabilistic decoder to infer a user's text from time sequence data @xcite @xcite . typically, a keyboard likelihood model and a language model prior are used to infer a user's text from input with incorrect, missing, or extra characters. to date, these approaches have mostly used n-gram language models. @xcite corrected typos in tweets to a low character error rate of 2.4% by using a character convolutional neural network, an encoder with gated recurrent units, and a decoder with attention. the twitter typo data contained sequences with a similar number of characters to the target. in our work, we show acceptable character error rate can be achieved on input not only with typos, but also with missing spaces and mid-word vowels. we show the advantage of using a recurrent neural network language model (rnnlm) directly in the decoder's search or to rescore hypotheses. ## free-form abbreviation study to better understand how people do free-form abbreviation, we conducted a study on amazon mechanical turk. as a pilot, we had 26 workers abbreviate an email from the enron mobile data set @xcite . we designed our instructions based on @xcite . workers abbreviated the same email three times. each time the worker was asked to abbreviate in three ways: heavily, as little as possible, or as they saw fit. in our pilot, we found workers abbreviated similarly regardless of instructions. thus, we designed a single set of instructions for our main study that asked workers to imagine they were using artificially intelligent (ai) software that was good at guessing their intended text from an abbreviated form. they were told to shorten words by removing or changing letters, but they should avoid shortening words that might be hard for the system to guess and that they should not omit words entirely. see the appendix for our instructions. our supplementary data contains all the data from the study. we recruited 200 workers who each abbreviated ten emails. in our analysis, we used 1,308 of the 2,000 emails. we filtered out emails that did not have the same number of words as their original emails. this filtering helped us to align the sentences by word. punctuation was removed except apostrophes and at signs. we lowercased the text. ## conversational language modeling we think abbreviated input may most benefit users with slow input. from this point on, we focus on optimizing our system for use by augmentative and alternative communication (aac) users. aac users may not be able to speak due to a condition such as als. aac users slow input rate make taking part in conversations difficult @xcite . sentence abbreviation may be particularly useful for short phrases with predictable language. our search-based approach to abbreviation expansion relies crucially on a well-trained language model. for a language model to work well it needs to be trained on data that is suited to the target domain. ideally we would train our language models on large amounts of conversational communications written by aac users. for privacy and ethical reasons, it is difficult to find large amounts of such data. therefore, in this section, we explore selecting training data from an out-of-domain dataset using a small amount of in-domain aac-like data. ## user study thus far, we tested abbreviated sentence input only in offline experiments. to see if our method offers competitive performance in practice, we conducted a user study using a touchscreen web application. ## discussion we set out to show we could accelerated the writing of short and reasonably predictable phrases by combining sentence-at-a-time recognition with aggressive abbreviation. in our final user study, we found our method did not quite beat a conventional keyboard with word predictions. however, users in our study likely had substantial experience doing word-at-a-time input on their phones. it appears users got faster at abbreviated sentence input even during the brief study session. by the last eight phrases, users were only 3% relative slower using sentence abbreviated input compared to word-ata-time input with word completions. when users provided abbreviated input consisting of all the correct letters except mid-word vowels, 90% of these phrases were expanded correctly. we observed the abbreviation behaviour of a large number of non-aac users and designed a system supporting the most common behaviours. while we could have tried to learn abbreviation behaviors from actual aac user data, this presents a number of issues. first, actual text from aac users is difficult to obtain for ethical and privacy reasons. while it may be possible to obtain such text via donations from aac users and from online sources, such sources lack visibility into how the user actually produced the text (e.g. did they use word completions?). further, the propensity to abbreviate may be influenced by the particular aac interface used. second, even if we could source aac abbreviated text, we would have no reliable way to determine the unabbreviated text. we could have asked aac users to complete our abbreviation study, but this would have introduced more noise (incorrect key presses) that would have complicated our first study's goal of discovering natural abbreviation behaviors. it would have also limited the number of people we learned behaviors from. in this phase our goal was to discover what letters humans think are the most information carrying in a passage of text. while we suspect abbreviation strategies of aac users would be similar, this would benefit from validation with aac users. we tested our method on touchscreen data recorded in previous studies on phones and watches, and in a web-based crowdsourced user study. we think our method mainly would benefit users who have a slow input rate; fast typists may only be slowed by the cognitive overheads of deciding what letters to omit or by disrupting their muscle memory for typing familiar words. this led us to limiting the input rate in our study by requiring users dwell for one second. while this study allowed us to confirm our abbreviation method is competitive with a conventional keyboard with word predictions, this needs validation with users with actual input rate limits. aac user interaction may feature more imprecise key presses, more accidental key presses, and may introduce complications related to attending to word predictions (e.g. the "midas touch" problem in eye tracking). further, we only tested one input rate, it is possible our method may be better or worse at different input speeds. we think our approach may also offer advantages for eyes-free text input, but this also needs comparison against conventional eyes-free input approaches (e.g. iphone's voiceover feature). we investigated abbreviation by omitting midword vowels. we did not investigate other forms of abbreviation such as phonetic replacement (e.g. "you" → "u") or removal of consonants. our model may benefit from more sophisticated modeling on how and when vowels are inserted (e.g. penalizing repeated vowel insertions). ideally improved models would be based on data collected by users engaged in actual abbreviated input. as our results show, correctly inferring the intended sentences was challenging even when we asked users to obey a few simple behaviours, namely removing spaces and mid-word vowels. while an ideal system would support a wide-range of abbreviation behaviors and even adapt to individuals, we suspect this may be challenging given our current lack of training data on this task. in our initial study, participants abbreviated email text that was displayed visually. an alternative approach would be to play audio of the text. while this might be a more realistic abbreviation task, it also presents practical challenges to participants such as remembering the text and spelling any difficult words. perhaps an even more externally valid approach would be to have workers compose novel abbreviated sentences. this would require another step to obtain the unabbreviated compositions @xcite . given we now have a competent initial system, it would be interesting to undertake such a data collection effort. our results suggest a simple correction interface based on selecting from the top sentences would often, but not always work. designing an efficient and easy-to-use interface for correcting a few words within such sentence results would be interesting future work. this might be especially challenging to design for users with diverse motor abilities. we used language models trained on only 100 m characters of text. while this allowed us to compare the efficacy of the language model types and decoder configurations, substantially more training data is available along with neural architectures that scale to large training sets, e.g. @xcite . we suspect further recognition accuracy gains are possible for abbreviated, noisy input by incorporating such models. further, we could likely obtain additional improvements from the n-gram model by training on more data and then pruning the model to reduce its size. we avoided doing this in this work to fairly compare the ngram and rnn language models when trained on the same amount of text. our language model training data was drawn from common crawl. we used a corpus of aaclike crowdsourced messages to select training sentences from common crawl. other sources of training data such as twitter or reddit are likely more conversational in style. it would be interesting to investigate whether data selecting from a more targeted large-scale training source provides additional improvements in language modeling. we did not specifically investigate how our method would support text containing difficult words such as acronyms or proper names. users can often anticipate and alter their input behavior to avoid auto-correct errors, e.g. by force @xcite , by long pressing a key @xcite , or by switching to a precise input mode @xcite . similarly, our abbreviated input method needs a way to specify words that should not be expanded or auto-corrected. at the onset, we did not know that our proposed abbreviation technique would be competitive to conventional word completion. the results from our user study tell us we need to make further improvements to our recognition, better train users to abbreviate in supported ways, and conduct a longitudinal evaluation. further, testing an abbreviated input prototype with aac users will undoubtedly lead to new insights. this paper is a first step in producing a viable prototype for testing with users with rate-limited input abilities. ## conclusion we explored accelerating text communication by abbreviated sentence input. we conducted a user study to learn how users abbreviate. we showed the efficacy of a neural classifier to select conversational-style training instances from a large text corpus. we found that dropping spaces and mid-word vowels can provide compression of sentences from 28% to 38%. such abbreviated and noisy input can often be expanded correctly 59% to 72% of the time. we also showed how the accuracy of a statistical virtual keyboard decoder can be improved by using a neural language model to re-rank the top recognition results. finally, after practice, users wrote only slightly slower using sentence abbreviated input at 9.6 words-per-minute compared to a conventional keyboard with word predictions at 9.9 words-per-minute. if a phrase was abbreviated by removing spaces and mid-word vowels, our system expanded the abbreviated input to the intended phrase 90% of the time.