Adapting Large Language Models for Character-based Augmentative and Alternative Communication (2025)

Abstract: Users of Augmentative and Alternative Communication (AAC) may write letter-by-letter via an interface that uses a character language model. However, most state-of-the-art large pretrained language models predict subword tokens of variable length. We investigate how to practically use such models to make accurate and efficient character predictions. Our algorithm for producing character predictions from a subword large language model (LLM) provides more accurate predictions than using a classification layer, a byte-level LLM, or an n-gram model. Additionally, we investigate a domain adaptation procedure based on a large dataset of sentences we curated based on scoring how useful each sentence might be for spoken or written AAC communication. We find our procedure further improves model performance on simple, conversational text.

## introduction augmentative and alternative communication (aac) devices allow non-speaking individuals to communicate in face-to-face conversations and asynchronously via messaging systems such as email. depending on their disability, aac users may be unable to use a conventional computer keyboard and may instead rely on alternative input methods such as eye tracking or triggering a single switch (e.g. by twitching a muscle or puffing air). unfortunately, such input methods can make writing slow and tiring. the writing speed of nonspeaking aac users varies, but is frequently less than 10 words per minute (wpm), with an average closer to 5 wpm @xcite . while some aac users may engage in long conversations, communications are often shorter and more isolated in nature due to the input rate limitation. aac users with severe motor impairments may write letter-by-letter using an interface such * these authors contributed equally to this work. as @xcite , the rsvp keyboard @xcite , or nomon @xcite . these aac interfaces all require the probability distribution over the next character based on a user's previous text. we wanted to know if recent advances in pretrained large language models (llms) really help with the "simple" case of predicting the next letter in a sentence with only that sentence as context. numerous papers have shown that ever-larger models such as llama 3 @xcite , mistral 7b @xcite , and @xcite provide gains on more "complex" natural language processing tasks such as question answering, sentiment analysis, and machine translation. many large transformer models do not operate on a character level. rather, they use a fixed vocabulary of subword tokens when making their predictions. these subword tokens can range anywhere from a single character to a full word in length. these tokens are learned during the training process and stem from frequently seen sequences of characters. as we detail later, this subword tokenization makes it difficult to obtain single character predictions from many llms. in this work we make four contributions: (1) an algorithm that produces a probability distribution over single characters from a subword token language model, (2) offline, novel test sets of aaclike text that can be used to evaluate llms without fear of their inclusion in the models' training data, (3) a classification of existing large (87 b words) training corpora based on their similarity to aaclike text, and (4) an empirical evaluation and comparison of different methodologies for obtaining single-character predictions using llms. we have released our algorithm's source code, our classified data sets, our classification model, and our best language models as part of this work. 1 this paper is structured as follows: first, we motivate why character predictions are important for aac text entry (section 2) and describe our algorithm for making such predictions using a subword tokenized llm (section 3). next, we find the best starting model, testing performance on a private collection of aac-like text (section 4). we then explore adapting the llm using small in-domain sets and using text classified from much larger outof-domain sets (section 5). next, we compare our search algorithm against other possible approaches (section 6). finally, we show our results on five test sets (section 7) before wrapping up with discussion (section 8) and conclusions (section 9). word completions and predictions. word completions attempt to predict a user's intended current word before they finish typing it, while word predictions attempt to predict subsequent words based on the likelihood of each word in the context of the user's sentence. prior work by @xcite showed that word predictions can increase or decrease a user's entry rate depending on how and when predictions are displayed and the user's typing strategy. @xcite used llms to generate word predictions for aac devices and showed entry rates up to 30.4 wpm. sentence and phrase retrieval. retrieval techniques allow users to store sentences or phrases that they frequently use @xcite . these messages can then be retrieved by the user by entering specific keywords @xcite or matching context tags @xcite . the drawback to retrieval is utterances must be prestored by the user in their device, so they are not as adaptable to different conversational situations. ## character prediction algorithm as we discussed previously, many modern large transformer models use subword tokens instead of single characters or full words. the first step in using a subword token language model is to convert the context into a sequence of tokens via tokenization. each model comes with its own tokenizer, which uses a greedy algorithm, such as byte pair encoding @xcite , to segment any given text into tokens in its model's vocabulary. this subword tokenization complicates the process of obtaining a distribution of character probabilities since adding another character to the end of a section of text could (1) begin a new token containing only the new character, (2) extend the existing most recent token with the new character, or (3) rearrange the optimal tokenization of the most recent word. for the last case, take for example writing the word "yesterday". if the user has written "yeste" thus far, the most likely tokenization may be into "y-este" (where -denotes token boundaries). adding an "r" could result in the most likely tokenization changing to "yes-ter". to properly consider each of these possibilities, we must remove one or more tokens from the end of the context to allow the model the option of extending it, as opposed to only adding a new token. removing more tokens allows the model more flex-ibility to rearrange the optimal tokenization and in our experience leads to better predictions, but at the cost of a higher prediction time. in this work, we remove all tokens in the current in-progress word to allow maximal flexibility in tokenizing the partial word. going further back would provide no benefit because, in all models we used, the space character only appears at the start of tokens and thus tokens cannot span word boundaries. we then ask the model to estimate the likelihood of the next token and evaluate any token that matches our context. for efficiency, we only track a certain number of hypotheses at a time, known as the beam width. we continue expanding each hypothesis until it has regenerated the removed context tokens plus at least one character. we then store the likelihood for each final prediction in a list based on the character that directly follows the context. we continue this process until there are no more hypotheses to extend, or, for efficiency, the number of completed hypotheses exceeds a max completed parameter. then we sum the likelihoods stored for each character in our symbol set and normalize to sum to one, arriving at a distribution over our symbol set. see appendix c for our pseudocode search algorithm or our supplementary materials for a python implementation. to give a concrete example of the benefit of removing tokens from the end of the context, we further examine the "yeste" example. if we do not remove any context, and simply marginalize the probabilities over the first character in each token, the probability of the next letter being r is 0.0047. using our search algorithm with context removal, this jumps to 0.9999. we suspect this is due to the training process of the model; during training it doesn't frequently encounter fragmented words and is allowed to tokenize the full word in the most optimal way. by not removing any context, we force the model to infer text with suboptimal tokenizations, causing it to struggle. ## pretrained model performance how do pretrained transformer language models perform on aac-like text? in this section, we focus on llms using subword tokenization. ## optimizing language models for aac the llms in the previous section were all trained on general text from the web. as with previous work on n-gram language models @xcite , we may be able to improve predictions on aac-like text by focusing the model on this style of text. we selected the opt-350m model for further refinement as it provided the majority of perplexity gains while being small enough to plausibly be used on an enduser's device. for privacy and latency reasons, users may prefer to use a local model. our results in ## other prediction approaches with opt-350m, our search algorithm (section 3) takes on average 79 ms to predict the next character on an a100 gpu. while this may be fast enough for many purposes, interfaces such as dasher @xcite or less capable devices may require more efficient prediction. we explored several other methods for obtaining faster predictions. ## final evaluation at this point, we have extensively used our private dev sets to guide our process. we conducted a final evaluation with our unseen private test sets and three additional test sets. firstly, we created a test set based on the text of an aac user. this set consisted of 3.6 k sentences (65 k words) written by a user of dasher @xcite . the text was from public and private presentations given by the user. we also added two public test sets: voice banked messages of people with als (costello, 2014), and the comm2 set of responses to conversational situations @xcite . as shown in as shown in to demonstrate the potential impact in a realworld aac interface, we simulated using each language model in a onscreen keyboard with five word predictions. we simulated the keystrokes a hypothetical perfect user would save compared to typing every character. keystroke savings is calculated as: as shown in ## discussion most of our trained models had higher perplexities on the aac test set than on any of the other test sets. the only exception was the base opt-350m model, which performed worst on the written test set. we believe this stems from the nature of the test sets. the written and spoken sets were created by crowdsourced workers writing about topics such as the weather or travel. as described in appendix a.4, workers were instructed to invent questions or statements they might include in a conversation with another person. the als bank and comm2 sets contain text of a similar nature. this aligns closely with the in-domain conversational datasets we used in our adaptation process. the aac test set was derived from prepared presentations by an aac user with als discussing their life experiences. it covers more complex topics and uses more advanced language. presentations are also not conversational in nature, which is why we believe our curriculum adaptation process was not effective at improving the model's performance on the aac test set. we also note that aac users are diverse in nature and our domain adaptation may be advantageous to other users or even to this user in other communication situations. it may also be advantageous to consider adapting the language model more directly to a particular user in lieu of, or in addition to, domain adaptation. we obtained the lowest perplexities using a subword tokenized llm and our search algorithm. while we investigated a multi-step curriculum to adapt the opt-350m model using sentences classified from large out-of-domain datasets, a simple fine-tuning using in-domain datasets worked better. the main drawback of the search algorithm is its inference time. if faster predictions are required for a particular interface or device, a byte or classification model may be preferred. however, currently subword llms are much more popular than byte llms. further, the search approach can be used directly on an llm without task-specific training. adding a classification layer to the domainadapted opt-350m model performed worse than any of the other llm-based approaches, including the base opt-350m model before domain adaptation. we have a few theories as to why this may have been the case. our classification training examples chose a random location in each sentence, so many of the sentences included in these further training steps were likely fragmented, which may have af-fected earlier layers in the model. to investigate this theory, we could freeze the lower model layers during the classification training. this might help to preserve the domain adaptation and prior model while also tuning the new classification layer. it is also possible that the subword nature of the model prevented it from fully learning the character classification task. as discussed in section 3, adding a new character can rearrange a word's optimal tokenization. because the model was originally trained to predict the next token in a sequence, it may not have considered all the different possible tokenizations as our search algorithm does. ## conclusions we investigated how to leverage large transformer language models to aid non-speaking individuals using augmentative and alternative communication (aac) devices. in particular, we focused on how to generate the character predictions needed by interfaces where users write one letter-at-a-time. we presented an algorithm for obtaining character probabilities from llms using subword tokenization, and compared our algorithm to using a bytelevel llm or adding a classification layer on top of a subword llm. we detail how we classified sentences in large corpora of web and movies subtitles text based on how similar sentences were to written and spoken communications. we investigated a multi-step fine-tuning process using our scored sentences as well as our smaller in-domain corpora to adapt llms to our target domain. we found that we were able to produce the lowest per-character perplexities using our search algorithm with the subword opt-350m model. we found that our domain adaptation curriculum was effective at improving the model performance for simple, conversational text, but did not improve performance for more complex prepared presentations from a single aac user. though it offered the best perplexity, our subword to character algorithm took almost seven times longer to produce character predictions than a native byte-level llm. to aid future research, we have made our unique datasets and models available. 5 this includes the model for classifying sentences as written or spoken, scored c4 and subtitle sentences, n-gram models trained on the classified sentences, fine-tuned opt-350m model, fine-tuned bygpt5 model, and letter classification model.