Design principles of an open-source language modeling microservice package for AAC text-entry applications (2022)

Abstract: We present MozoLM, an open-source language model microservice package intended for use in AAC text-entry applications, with a particular focus on the design principles of the library. The intent of the library is to allow the ensembling of multiple diverse language models without requiring the clients (user interface designers, system users or speech-language pathologists) to attend to the formats of the models. Issues around privacy, security, dynamic versus static models, and methods of model combination are explored and specific design choices motivated. Some simulation experiments demonstrating the benefits of personalized language model ensembling via the library are presented.

## introduction designing and building text-entry systems for individuals with severe motor impairments is a key challenge in the field of augmentative and alternative communication (aac). in successful cases this typically involves people with many diverse areas of expertise, including speech-language pathologists, those with human-computer interaction (hci) or natural language processing (nlp) expertise, and, of course, users of the technology themselves. given this diversity, few individuals will have the breadth of expertise to address all of the issues at play. for example, those focused on interface design or customization for a specific individual's needs may not have the nlp expertise to assemble effective predictive models to help drive the interface; and those who are building state-of-the-art language models (lms) often lack hci or aac experience and are not optimizing their models with text-entry scenarios in mind. as we will illustrate later in the paper, choices of how a system employs lms can make a big difference in the quality of the resulting predictions, thus effective lm services are critical for these systems. this paper presents the mozolm open-source software library foot_0 for building services that allow user interfaces (uis) to request probabilities from a collection of diverse lms without having to match their requests to the formats of the models. this frees those working on interfaces and user configuration optimization from having to focus on specific lm details, and frees those working on lms from necessarily focusing on ui or text-entry scenario specifics. this work was initially inspired by our interest in dasher @xcite , a text-entry system that in its standard implementation foot_1 is closely tied to certain dynamic language modeling methods (see section 2 for details). 3 the dynamic nature of the models in dasher have the virtue of automatically adapting to user input in an open-vocabulary manner, thus learning the idiosyncrasies of the individual over time; however the tight coupling between the modeling choice and interface has some serious drawbacks. first, dasher could make use of large static general background lms in addition to user-specific dynamic lms, to provide extra predictive power for novices with little personalized text but also for more advanced users. this was demonstrated in @xcite , who included a static word-based 𝑛-gram model in dasher; we also provide evidence of the benefit of using ensembled static and dynamic lms later in the paper. second, dasher may not be the only text-entry system that an individual makes use of, yet the personalized models maintained by the dasher system are not straightforwardly accessible to these other applications. finally, the tight coupling between modeling and the interface requires ui designers to take into account the specifics of the lm and those interested in improving the lms must attend to the interface. ideally, a text-entry application should be able to plug in any and all given lms to derive whatever useful information they can, even when all of the models are trained completely independently. the design of the library was motivated by several considerations. first, large, possibly remote, general purpose lms and small(er), local personalized models can profitably work in tandem to support open-vocabulary applications, and this is a potentially complex coordination that likely falls outside what many of those contributing to such an application design are interested in developing the expertise to perform. the library should allow for easy-to-configure support of these best practices. section 3.1 presents the language model design criteria for the library. second, personalized models must remain secure due to privacy concerns, so such services must include adequate security and privacy functionality. further, multiple applications could potentially share the same microservice -either multiple text-entry applications on the same local device (hence possibly sharing dynamic models) or many clients for remotely running hubs. finally, separation of the language modeling functionality into a completely separate component allows for independent development and testing. such general architectural considerations are presented in section 3.2. ## personalization experiment we have motivated this work in part with the idea that lms with different characteristics can be prof-itably ensembled to provide better estimates, and in this section we present a small experiment to demonstrate this. this experiment was run using the mozolm library with differently configured lm hubs and implementations of several common lm sub-classes. 7 we evaluated lm performance using data from the enron personalization validation set foot_6 @xcite . that data collects emails written by 89 individuals, each in their own separate file, 45 of which are available for dev and 44 for test. here we use the text from the 45 dev individuals, found in files dev??.message.text.tsv, up to a maximum of 140,000 characters per individual, in aggregate over 720k words and 3.9m characters. language models can be used for any number of applications -including text entry, the focus of this software library -but evaluation of language model quality is often performed intrinsically, by examining the probabilities assigned by models to attested text from the domain. operationally, one measures the log probability of the validation corpus; and for ease of comparison at the character level, normalizes by the number of characters. if the log is base 2, this provides the number of bits per character (bpc), and lower values correspond to higher probabilities, i.e., better models. since each file in the dev set consists of text written by a single individual, dynamic models that update counts as the text file is processed will personalize the model to predict frequent patterns of that particular user. we score the cumulative bpc of the model, which, at each character, shows us how well the model has predicted the text that was typed up to that point. lower bpc corresponds to higher probability assigned to the actual characters that were typed, i.e., better predictions of what the user will type. in dasher, for example, this would correspond to larger regions being allocated to actual target characters within its arithmetic coding approach. since we measure cumulative bpc at each character position, we can track the learning of any dynamic models that are included in the ensemble. figure 5 presents cumulative bpc aggregated over all 45 individuals in the dev partition, reporting the aggregate bits divided by aggregate characters as we synchronously step through each text (c) and (d) bayesian-mixed ensembles of the two static models and a dynamic ppm 6-gram for each individual, using history lengths 6 and 1, respectively, to determine mixing coefficients. figure 5 presents cumulative bpc for four conditions. first, we combined two large static lms trained on 20m sentences of english wikipedia text (426m words). the first is a trigram word model with a closed vocabulary of 414,715 words; all other words are mapped to an out-of-vocabulary (oov) token. this model was built using the open-grm ngram library @xcite and pruned to contain a total of 100m 𝑛-grams. the second static model is an unpruned ppm model with a maximum 𝑛-gram length of 6, trained on the same data. since this is a character-level model, the ensembling of these two models provides a fully open-vocabulary model over the characters found in the wikipedia training data, and there are no oov characters in the evaluation text. condition (a) in figure 5 shows the performance of an ensemble of the two static models, mixed uniformly. note that this yields lower bpc (i.e., better performance) than using either model independently, conditions which are not shown for clarity. the other three conditions mix the above static models with a dynamic ppm model (also maxi-mum 6-gram) that is only trained on previously typed strings in the dev set itself. this dynamic model on its own performs far worse than any of the ensembles shown (results are omitted for clarity). the difference between these three dynamic model conditions is in the ensembling method. in condition (b), all three models are mixed uniformly, i.e., each contributes 1/3 of the probability mass. one can see from figure 5 that condition (b) ends up improving substantially over condition (a), but at early character positions (b) has significantly higher bpc, since the dynamic model needs many observations before it can begin to produce useful probabilities. the other two conditions use a generalization of bayesian interpolation (see appendix a.2) to establish the ensemble mixing coefficients, which, among other things, reduces reliance on the dynamic model at earlier positions. choosing the mixing coefficients based on just the previously typed character (condition d) outperforms using the previous 6 typed characters to calculate the coefficients (condition c). this experiment is simply intended to motivate the ensembling of multiple diverse models, as we advocate for in the design of this library, as well as demonstrating the software in action. of course, actual optimal model configuration will depend on the user and on the specific text-entry system being used. we can at least say that different models can have complementary characteristics, so that combining them, even in simple ways, can yield better models. ## conclusion and future work we have presented the rationale for many choices made in designing an open-source microservice package for language modeling in aac text-entry applications. the code presented is available opensource, and the experiments run in section 4 were performed using the library. future work will include adding more lm sub-classes, including commonly used neural lms.