Context-Aware Abbreviation Expansion Using Large Language Models (2022)

Abstract: Motivated by the need for accelerating text entry in augmentative and alternative communication (AAC) for people with severe motor impairments, we propose a paradigm in which phrases are abbreviated aggressively as primarily word-initial letters. Our approach is to expand the abbreviations into full-phrase options by leveraging conversation context with the power of pretrained large language models (LLMs). Through zero-shot, few-shot, and fine-tuning experiments on four public conversation datasets, we show that for replies to the initial turn of a dialog, an LLM with 64B parameters is able to exactly expand over 70% of phrases with abbreviation length up to 10, leading to an effective keystroke saving rate of up to about 77% on these exact expansions. Including a small amount of context in the form of a single conversation turn more than doubles abbreviation expansion accuracies compared to having no context, an effect that is more pronounced for longer phrases. Additionally, the robustness of models against typo noise can be enhanced through fine-tuning on noisy data.

## introduction the prevalent paradigm of text entry on computing devices is sequential typing of characters. word completion and prediction can theoretically save up to 40-50% keystrokes when 3-5 predictions are provided @xcite . this reduces the motor and cognitive demand of entering text, especially on devices where typing is difficult, e.g., phones. in aac use cases such as eye-gaze keyboards for severely motor-impaired individuals, the cost per keystroke is so high that there is a desire to save as many keystrokes as possible. gaze-typing requires the user to precisely control the direction and timing of gaze for each keystroke, resulting in an extremely low text-entry speed of 8-10 words per minute and figure 1 : our approach to abbreviation expansion based on an llm with context compared to one without. the conversation context (e.g., a previous turn of conversation) along with the abbreviation of the intended phrase form the llm's input. sampled continuations from the model are filtered to discard those that do not match the abbreviation. top-5 options after sorting by frequency are presented. one potential paradigm is "sms language", a spontaneously-evolved system for saving keystrokes in which each word is abbreviated as a single letter, such as in the well-known abbreviations sg for sounds good and ttyl for talk to you later @xcite . sms language features a high ksr (75-80%), but is limited by its small closed set of common phrases of mostly six words or shorter. its abbreviation scheme is not applied to longer or less frequent phrases because such abbreviations would be hard for the recipient to decipher. for example, the abbreviation iipitb is highly ambiguous and may represent many possible phrases, e.g., it is pouring in the bay and it is pretty in the backyard (see figure 1 for more examples). some existing aac systems support abbreviation expansion (e.g., tobii), but are limited by hardcoded, closed phrase sets. the current study is based on the insight that although decoding open-set phrases from abbreviations is hard without context due to ambiguity, providing conversational context significantly constrains the space of likely phrases as shown by the example in fig. 1 (it is playing in the backyard). hence we propose a high-ksr abbreviation scheme that focuses on conversational scenarios. we apply this scheme to three existing dialog datasets and create datasets for abbreviation @xcite . this allows us to study whether llms, trained on web text including conversational data, can enable ae and benefit from added context. we take a 64b parameter llm and compare zero-shot, fewshot, and fine-tuning performance on the ae task. additionally, we simulate typing noise to study tolerance of the approach to typos. the main contributions of our work are: large language model prompting and finetuning. our approach builds on prior work on llms including few-shot prompting, fine-tuning, and conversation models @xcite @xcite . we focus primarily on fewshot prompting @xcite and finetuning @xcite . few-shot prompting uses a text description of a task along with a small number of examples for the task in the input text in order to elicit desired task responses from an llm. in the zero-shot scenario, no examples are provided. prompting involves no updates to the model parameters. model fine-tuning requires more data compared to prompting, but often leads to higher task accuracy than prompt engineering (e.g., @xcite . for our ae task, data for fine-tuning can be synthesized from existing conversation datasets based on an abbreviation scheme (sec. 3). thus, we explore both prompting and fine-tuning and compare their performance. assisting text entry with context. textual contexts have been exploited to aid email writing @xcite . for text entry in aac, @xcite demonstrated that providing noun phrases from a conversation partner's speech as selection options increases text-entry speed by 36.7%. @xcite concluded that with currently-attainable accuracy of asr, partner speech can be valuable in improving language modeling for aac text entry. @xcite used a fine-tuned gpt-2 model @xcite to expand bags of keywords into full phrases in conversational contexts based on the convai2 dataset @xcite and reported a ksr of 77% at a word error rate threshold of 0.65. our current study differs from the previous studies in the following aspects. first, we provide an abbreviation scheme to allow greater user control over the exact phrase structure and wording. second, we performed detailed quantitative analysis of the combined predictive power of state-of-the-art llms and context awareness. ## results we present the main results comparing the models on all datasets in the accuracy of llms at expanding wordinitial abbreviations is enhanced by fine-tuning. it's noteworthy that the basellm * , which has seen just four examples in its prompt (unlike the other models), shows performance that exceeds the look-up table (lut) baseline in many cases, 4 appendix tab. 8 reports performance on dev split of the tdc (tdc-dev) which was used for hyperparameter tuning. fine-tuning and tolerance to noise. for conditions that involve simulated typo noise in the abbreviation input, ftnoise-llm shows superior performance compared to other models (see the column "tdc-test + noise" in context is critical for ae accuracy figure 3 show how the ae accuracy of ft-llm varies when different amounts of context from previous turns of the conversation are provided. compared to having no context (dash-dotted curve), including just one previous turn of context (dashed curve) approximately doubles accuracy. using the full context (all dialog turns from the 1 st to the (n-1) th , solid curve) leads to further improvements indicating that prior turns carry useful information for the ae task. compared to the 1 st turn, ae under no context on subsequent turns (2 nd -6 th ) shows significantly worse accuracy. this is due to the fact that the first turn consists of conversation starters that are easier to predict without context. overall, irrespective of context, the accuracy of ae decreases as the number conversation turns increases, indicating increasing difficulty in predicting the full phrases from the abbreviation as the dialogs progress. however, including full context during inference still achieves accurate expansions for 60%-70% of the cases on the later turns. effect of context is more pronounced on longer abbreviations. when performance is sliced by the abbreviation length (figure 4 ), accuracy without context decreases sharply and nearly monotonically with increasing abbreviation length, regardless of whether it's the opening turn or the 2nd turn. with context however, the accuracy remains higher and decreases more slowly with abbreviation length, extending the approximately 80% or higher accuracy into longer phrase lengths. the variability and usefulness of few-shot prompts decreases after model tuning. here we focus on how much the llm benefits from the "no instr." setting is attractive due to its simplicity (no need to search for or hand-engineer a prompt) and reduced latency (due to shorter input prefix lengths). given these results, we use the "no instr." as the default setting and for all other experiments on ft-llm and ftnoise-llm. increasing number of decoded samples improves accuracy at the cost of latency. latency is important for interactive text-entry applications. during sampled decoding, the llms generate 128 continuations of length 16 tokens for a batch of prefix length 256 with a median latency of 0.568 s (interquartile range: 0.16 s). this latency is close to typical dwell time of eye-gaze keyboards @xcite and hence could be acceptable for the eye-gaze typing use cases. figure 5 shows the effect of increasing the number of continuations sampled from the llms. as expected, increasing sample count from 128 to 2048 improves top-5 accuracy for both basellm* (with 4-shot prompts) and ft-llm (no instr.). improved accuracy comes at the cost of increased latency. 5 basellm benefits significantly more from increasing sample count than ft-llm. comparison of model sizes figure 5 also compares fine-tuned models of different sizes (4b, 8b, 27b, and 64b). with model fine-tuning, the accuracy increases monotonically with increasing number of parameters. interestingly, even with the fewest samples (128), fine-tuned models of all sizes outperform the larger (64b) model under few-shot learning. amongst the encoder-decoder t5 models ( keystroke saving rates. ksr can be considered as a proxy measure of usability of the approach for aac use-cases. ksr success values are in the range of 73-77% for the 1st and 2nd turns of dialogs in the tdc and ddc datasets ( fine-tuning with noise improves typo tolerance. figure 6 compares the ae accuracies of llms fine-tuned with and without noise (ftnoise-llm and ft-llm). while both models show decreasing ae accuracies with increasing amounts of typos, ftnoise-llm is much more robust showing lesser drop in performance. further, on noise-free inputs (σ=0), ftnoise-llm shows only slight accuracy deterioration compared to ft-llm. we also find that typo tolerance, for both ft-llm and ftnoise-llm, is more pronounced with context than without. ## discussion qualitative analysis of ae failures. as indicated by the relatively high bleu scores in the benefit of ae relative to sequential text entry. word completion and prediction incur scanning cost: users scan the options in order to determine whether any of them match their intention, which has a detrimental effect on speed that needs to be overcome by the high quality of the options @xcite . although the speed of aebased text entry remains to be quantified in future studies, we point out that: (1) ae removes overhead of scanning for options in between keystrokes, (2) there are fewer characters to examine or correct when typing, both of which may offer speed-ups in addition to the higher ksr afforded by ae. although the current study is motivated by and focuses on the aac use case, our paradigm of abbreviated text entry may be applicable to text input on touch screens as well. the ae approach of the current study can be regarded as a variation of contextual prediction of user text @xcite that affords greater flexibility in message content at the trade-off of requiring specification of the message with a small number of keystrokes. future directions. we found fine-tuning to be significantly better than prompting in terms of (a) accuracy (for both scenarios with and without typonoise) and also (b) exhibit lower latency as we achieve better results with fewer samples. future work should investigate the differences in latencies between the encoder-decoder architecture and decoder-only models. for training efficiency, in-stead of fine-tuning, it will also be worth investigating strategies such as prompt tuning @xcite that continue to keep the model frozen, but learn some additional parameters for the task. even in the best case scenario models can fail to find accurate expansions foot_1 among the top-5 options. recovering from such failures is important for aac use cases. future studies should consider options for partial specifications of one or more words or selection of some words from the available options. once the recovery from failure is proven in offline analysis, user studies are required to validate and quantify the actual benefit of the ae text-entry paradigm in lab and real-life settings. integration with ui approaches is also an essential direction, e.g., speeding up eye-gaze typing such as cascading dwell time and dwell-free paradigms @xcite . ## conclusion in this work we proposed a high-ksr form of abbreviation expansion to dramatically save keystrokes for severely-disabled users. we use it to synthesize three datasets for the ae task. based on extensive experiments using few-shot prompting and model tuning we demonstrate that across the datasets, fine-tuned llms can accurately predict expansions for 48-77% of phrases that are replies to initial turns of dialogs and exhibit ksrs in the range of 73-77% for the correctly predicted expansions, thus pointing at a promising direction for future user studies of contextual and abbreviated text entry based on llms. models evaluated with conversation context show significantly higher accuracy than without, thus supporting our hypothesis that context is the key to effective abbreviated text entry in conversational settings. furthermore, finetuning with simulated typos substantially improves tolerance to noise in abbreviation.