C olor C ode: A B ayesian Approach to Augmentative and Alternative Communication with Two Buttons (2022)

Abstract: Many people with severely limited muscle control can only communicate through augmentative and alternative communication (AAC) systems with a small number of buttons. In this paper, we present the design for ColorCode, which is an AAC system with two buttons that uses Bayesian inference to determine what the user wishes to communicate. Our information-theoretic analysis of ColorCode simulations shows that it is efficient in extracting information from the user, even in the presence of errors, achieving nearly optimal error correction. ColorCode is provided as open source software.

## introduction people with limited muscle control, such as those affected by amyotrophic lateral sclerosis (als), can have trouble communicating through conventional means. augmentative and alternative communication (aac) systems can help these people communicate effectively @xcite . these aac systems can range from lowtech solutions, such as pointing to messages on a piece of paper @xcite , to high-tech solutions, such as eye-tracking software that allows someone to select keys on a keyboard with their gaze @xcite . the more limited the muscle control, the less information that can be input into an aac system. in this paper, we refer to any discrete input into an aac system as a button, and a signal sent through a button as a click. clicking a button (sometimes referred to as a switch in aac contexts) can take many forms, such as twitching a particular muscle or looking to the left or right. people in the late stages of als may only be able to reliably click two different buttons. aac systems for people with such limited muscle control must efficiently extract information from a small number of buttons to allow them to communicate effectively. an effective aac system for users with severely limited muscle control needs to be designed to allow the individual to use a small number of buttons to choose from a large number of options (like letters on a keyboard). a successful design must achieve this objective while also being easy to use and resilient to errors in the user's input. designing a system that satisfies these properties is challenging. there are many aac software systems for a small number of buttons (e.g. grid 3 foot_0 and acat foot_1 ). most systems for two buttons involve a scanning keyboard where one button is used to scan through keys and the other button is used to select the chosen key @xcite . this method benefits from a simple interface, but it can take many clicks and therefore a lot of effort to communicate. previous research has used probabilistic reasoning and information-theoretic approaches to design effective aac systems for few buttons @xcite @xcite . common themes in this research include leveraging statistical language models to improve text entry and using concepts from information theory to analyze performance. in this work, we describe a new system whose design builds upon these existing ideas. we present a new aac system, named col-orcode, which allows users to communicate efficiently with only two buttons. colorcode's interface is a virtual keyboard with characters the user can select to write sentences and communicate (see figure 1a ). each key on the keyboard is assigned one of two colors, red or blue, and each of the two buttons is associated with one of the colors. colorcode repeatedly assigns colors to keys while observing the colors of the buttons the user clicks. the system uses bayesian inference to update its belief over the user's chosen key given its observations of button clicks. an accurate language model and the intelligent assignment of colors to keys allow colorcode to efficiently infer what the user wishes to type. additionally, colorcode adaptively corrects for errors in the user's input. we simulate colorcode on aac-like text and record the average clicks per character. we analyze the results from an information-theoretic perspective to show that colorcode efficiently extracts information from the user. we also simulate the input to colorcode as a binary symmetric channel to empirically show that it is close to optimally resilient to errors in the user's input. ## related work this section gives a brief overview of previous research that used probabilistic reasoning and information-theoretic approaches to build aac software systems for a small number of buttons. ## method to type a message in colorcode, the user first identifies the color of the key they wish to select (e.g. the letter a) and clicks the button for that color. after the user clicks a button, the system reassigns colors to all of the keys on the keyboard. the user repeats this process until the system selects the key they wanted and types the corresponding character in the display (or deletes a character if key was "undo"). colorcode also plays an audible "click" sound to notify the user when a key is selected. see figure 2 for a diagram demonstrating this process. colorcode maintains a belief over the user's desired key and uses baye's rule to update its belief after observing the user's button click. the belief is a probability distribution over the possible keys (see figure 1b ). when the probability of a particular key reaches a certain threshold, the system selects that key. this probability threshold is set to 0.95 in colorcode. if p (k) is the probability the user wishes to select key k, and c is the color of the button the user clicks, we can compute the belief update using baye's rule with: in this update, the two key components are the probability distributions p (k) and p (c | k), which are known as the prior and the likelihood, respectively. ## results and analysis the effectiveness of colorcode can be measured by the average number of clicks it takes the user to select a character. low clicks per character (cpc) indicate that a system is efficient in its ability to extract information from the user. another important metric used to evaluate aac systems is the text entry rate (ter). measuring the text entry rate of a system requires a user study, which has not yet been performed using colorcode (see section 6). we simulated colorcode on a test set of aaclike text, presented in @xcite , to calculate the average cpc. the simulator uses the undo key to correct any incorrect key selections, and these extra clicks are counted toward the cpc. through these simulations, colorcode achieved an average of 2.07 cpc. we can analyze this result from an informationtheoretic perspective by considering the theoretical lower bound on cpc given the language model colorcode uses. we can view the color clicks as binary symbols (bits) in a variable-length encoding of the characters the user wishes to type. the source coding theorem for symbol codes states that the entropy of a character distribution is the lower bound on the expected number of bits required to encode a character @xcite . however, the system uses a language model for the encoding because it does not know the true character distribution. so instead, the cross-entropy between the true distribution and the model distribution can be used as a lower bound on the expected number of bits per character in a coding scheme that uses the model distribution to encode characters that come from the true distribution @xcite . the cross-entropy is defined as: ## conclusions this paper presents the design of colorcode, a new and powerful aac system for two buttons. colorcode combines a powerful language model, bayesian learning of click errors, an informed undo operation, and intelligent color assignments into a bayesian belief framework that uses a simple interface to efficiently extract information from the user. our results demonstrate this efficiency by showing colorcode requires an average of only 2.07 clicks to select a character, which is within one half of a click of the theoretical lower bound which is 1.73 clicks. colorcode remains efficient in extracting information when there are errors in the input, and our results show that colorcode handles errors with nearly optimal efficiency. these results show that colorcode's design has the potential to help people who cannot communicate easily.