Application-Agnostic Language Modeling for On-Device ASR (2023)

Abstract: On-device automatic speech recognition systems face several challenges compared to server-based systems. They have to meet stricter constraints in terms of speed, disk size and memory while maintaining the same accuracy. Often they have to serve several ap- plications with different distributions at once, such as communicating with a virtual assistant and speech-to-text. The simplest solution to serve multiple applications is to build application-specific (language) models, but this leads to an increase in memory. Therefore, we explore different data- and architecture-driven language modeling approaches to build a single application-agnostic model. We propose two novel feed-forward architectures that find an optimal trade off between different on-device constraints. In comparison to the application-specific solution, one of our novel approaches reduces the disk size by half, while maintaining speed and accuracy of the original model.

## introduction on-device automatic speech recognition (asr) is subject to several constraints: it should return accurate results in a reasonable time frame without consuming too much memory and disk space. state-of-the-art research often is accuracy focused, while resource-constrained applications also need to take care of performance and size. finding an architecture that reaches all constraints is not trivial. another challenge is that asr systems often serve a large variety of requests. asr systems can serve an on-device virtual assistant (va) but also allow dictated messages, notes, e-mails, etc.we refer to the latter application as speech-to-text (stt). typical va requests are knowledge-driven questions such as "how old is barack obama?" or commands, e.g. "play some lady gaga music". stt requests are longer and of a different nature than typical va requests. the solution that yields the best accuracy for both va and stt is to train separate models for each application, but additional model size is prohibitive. we aim to develop a single model instead. in this paper, we focus on a neural network language model (nnlm) in the asr system. our baseline is a fixed-size ordinally-forgetting encoding (fofe) feed-forward nnlm @xcite . in asr, the search space can easily increase so we have to limit the context length used in decoding to reach an acceptable latency and lower memory. given this short context length, we find that the fofe feed-forward lm is competitive to the transformer @xcite in terms of accuracy and better in terms of latency. @xcite has also shown that transformers are less robust to short context lengths. to build a single application-agnostic (aa) nnlm, we developed a method to optimally sample training data. we sample data from different sources, e.g. anonymized and randomly sampled user requests from opted-in users for va and stt and artificial requests spanning many different domains that focus on improving the tail of the distribution. the data-driven approach tries to find the optimal balance between the application-specific data sources by creating a balanced development set and distributing the sampling weights based on the importance of each data source and each application on that development set. training a single fofe nnlm on the combined dataset can lead to accuracy degradations, even with a larger model or longer training. we explore two extensions to the baseline fofe nnlm: firstly, a mixture fofe nnlm @xcite which is composed of an ensemble of parallel sub-networks and a mixture subnetwork generating normalized probabilities across all sub-networks. these mixture weights are used to compute a weighted average of the ensemble before the softmax output. the second extension is an application-dependent (ad) fofe nnlm that has different sub-networks for each application. at training time, data and gradients are (back-)propagated only through the corresponding subnetwork belonging to an application. at inference time, the way the user invokes asr tells us which application is needed (wake @xmath0, microphone @xmath1. both approaches are able to match or outperform the application-specific model. while the accuracy of the mixture nnlm is slightly better than the ad-nnlm the situation is reversed in terms of speed. the contributions of this paper are as follows: ## related work we start by discussing related work on modeling several domains/tasks at once. many pattern recognition tasks are imbalanced since data from different categories do not occur at the same frequency. therefore, the less frequent categories are not well represented in the training data @xcite , which results in a sub-optimal model. data-driven approaches to deal with the data imbalance include under-and over-sampling @xcite . refinements of these methods select data more intelligently @xcite @xcite . others approaches modify the training and/or model architecture. curriculum learning @xcite emphasizes data by fine-tuning towards the corpus consumed by the end of training. smith et al. (2020) experiment with multi-task learning, data augmentation and a classifier combined with single-task models to appropriately model several skills in a conversation agent. balancing through interleaved sampling of different corpora was investigated in @xcite as well as model-based approaches like multi-task and weighted learning, which allows the model to self-control the impact of different corpora. other ways to increase the modeling power are using a mixture of experts @xcite or ensemble networks @xcite @xcite . the choice of architecture for language modeling has also been a recurrent topic of research. early neural lms use feed-forward layers @xcite . @xcite introduced recurrent neural lms that can in principle use unlimited history. these networks are trained with backpropagation through time which 'unrolls' the network in time for gradient computation, but this leads to vanishing gradients @xcite , essentially limiting the history that can be learned from. gated recurrent architectures @xcite mitigate this problem. recent extensions of the feed-forward architecture have been proposed that alleviate different disadvantages. @xcite proposed a fofe, which represents a sequence of words as a vector with fixed length that captures the word order. they show that feed-forward networks with fofe encoding outperform recurrent models in language modeling. the most widely-used architecture in recent years, is the transformer @xcite that combines feed-forward layers with multi-head attention, residual connections, and layer normalization @xcite . it has been successfully applied to asr, see e.g. @xcite . in this paper, we compare fofe feed-forward lms with transformer lms and two extensions of the base fofe feed-forward lms. ## data balancing the asr system in this paper serves two applications, va and stt, for which we observe very different linguistic patterns. to demonstrate these differences, we calculate statistics on two english development sets. each data set contains 23k anonymized queries and is randomly sampled from real user data similarly to the test sets described in section 5. 20 40 60 80 100 120 140 10 0 10 1 10 2 10 0 2,000 4,000 6,000 8,000 10,000 call wake2 wake1 the is to text what for a i you and . it that va stt figure 2: counts of the union of the 10 most frequent words in both english va and stt dev sets. "wake2" and "wake1" refer to "<@xmath2. va requests are typically shorter than stt requests. in figure 1 , we plot the number of queries (on a logarithmic scale) that have x number of words for both data sets. for example, in the va dev set there are 9968 requests with only two words (238 requests consist of only "<@xmath3, while the stt test set contains 1327 requests with two words. if we define a request of 30 or more words as a 'long' request, we see that the stt test has 2030 long requests while va has only 21 long requests. secondly, the content and style of the requests varies between the two applications. figure 2 plots the union of the top 10 most frequent words in each data set -ordered by the frequency in the va dev set. notice that we allow the user to also dictate punctuation marks, hence the presence of the dot in the list of words. it is clear from this distribution that va queries are often questions (what) or commands (call, text) while stt queries are often messages from the perspective of the user (i, you) who wants to make their message more readable with punctuation marks. because of the different linguistic nature of these two applications, balancing the nnlm training data has a large impact on the quality of the model. a common strategy to determine nnlm sampling weights for each application is to train individual n-gram lms on each data source and choose relevance weights based on the optimal linear interpolation weights on a development set @xcite . in our setup, the sampling weights for the application-specific text sources are derived from the count merging weights @xcite @xcite instead of a linear combination. we propose a balancing scheme to derive sampling weights for i text sources that benefit both applications. we create a balanced development set containing approximately the same amount of va and stt data. let α 1 , . . . , α @xmath4, 1] be the sampling weights such that i @xmath5, a} indicating if the text source belongs to stt or va. the redistribution probability masses β d and β a for stt and va respectively are calculated to serve the joint application. these probability masses are determined by the optimal weights that minimize the perplexity of the linear application-specific (as) language model combination on the balanced development set. the application-specific probability mass allocated by each application can be formalized as: now consider the ratio between the redistribution and application-specific probability mass: these ratios determine the scaling of the original sampling weights to achieve balancing. balanced sampling weights are then determined by a re-normalization of the scaled sampling weights: the heldout and training set for nnlm training is then randomly sampled from the text sources according to the balanced sampling weights. ## application-agnostic and application-dependent fofe nnlms in this section three different types of nnlm architectures are introduced for on-device asr. in the following let w n 1 := w 1 , . . . , w n be a word sequence. all nnlm architectures considered here follow a similar scheme. in each architecture a word embedding is followed by a fofe layer @xcite . let α > 0 be the forgetting factor of the fofe layer and e m be the word embedding of word w m then z m := z m-1 + α • e m generates the fofe encoding. afterwards an ngram context of the fofe encoding is generated by concatenating n subsequent fofe encodings for each position: z m-n+1 , . . . , z m . next, this context is flattened and passed to the hidden layers. the baseline fofe nnlm shown in figure 3a applies a stack of feed-forward layers to the flattened fofe n-gram context. the output of the last feed-forward layer is fed to a projection layer for dimension reduction before the final softmax layer. this architecture is used for the as-nnlm, where each application has its own nnlm, as well as for the application-agnostic (aa) nnlm, which is trained on balanced data for both applications. figure 3b shows the mixture nnlm, which has m parallel sub-networks and a mixture subnetwork. each sub-network is a stack of feedforward layers. the mixture sub-network is also a stack of feed-forward layers which finish with a softmax output of dimension m to produce mixture weights for each of the parallel sub-networks, similarly to @xcite @xcite except that the mixture combines fofe networks. the subsequent layer averages the output of all parallel sub-networks scaled by the corresponding weights of the mixture sub-network softmax output. figure 3c shows the application-dependent nnlm (ad-nnlm). this architecture uses the application information to train a nnlm in a multitask style. this nnlm has a separate sub-network and softmax output biases for each application. for training we follow a multi-task approach. the information of the application for each data sample is known and used to select the sub-network and softmax output bias corresponding to the application and only back-propagate through a part of the nnlm. at inference time, data are forwarded through the corresponding sub-network and softmax output bias belonging to the active application. a word-level nnlm holds the majority of parameters in the embedding. therefore, the disk size for the mixture and ad-nnlm should increase slightly compared to the baseline architecture. also the ad-nnlm speed should not increase since it is equivalent to the baseline architecture at inference time. ## experimental setup the training data of our lms consists of different data sources: anonymized and randomly sampled user requests from both va and stt that are manually or automatically transcribed, along with synthetic tail-focused datasets. for the latter, we sample from domain-dependent templates and lists of entities that can fill those slots, both of which are derived from real user data. as mentioned in the introduction, we train nnlms for three languages: us english, german and mandarin chinese. for our nnlms, we obtain weights according to the method described in section 3. for the as models we sample 6b words while for the aa and ad models we sample 12b words. we run bayesian hyperparameter optimization and select the final values based on optimal size-accuracy trade off. as a result, the models have a slightly different number of parameters, but we show in section 6 that this does not impact results noticeably. all models have 4 feed-forward layers and an embedding size of 256 -we tie the input and output embedding weights to reduce disk size @xcite . the hidden size is 768 for the base fofe model, 512 for the ad fofe and mixture fofe and 256 for the transformer. the transformer has the same configuration as @xcite and uses 4 attention heads of size 256. we use the top 100k most frequent words as vocabulary. to speed up training, we use noise contrastive estimation (nce) (liza and grzes, 2017) which is replaced by softmax during inference. we train our nnlms with block momentum stochastic gradient descent @xcite with an initial learning rate of 0.256 for as, aa and ad fofe and 1.024 for aa mixture fofe. for as models the optimization converges after 64 epochs while for aa and ad models the optimum is delayed to 128 epochs. we keep the initial learning rate fixed for 16 epochs for as and 64 epochs for the other models and apply a learning rate decay of 0.7 if the heldout perplexity increases for 4 epochs. to stabilize the training a clip norm of 6.0 is applied and the number of nce samples is set to 4096. for evaluation, we test on three types of test sets: (1) va and (2) stt, which consist of user requests sampled according to the distribution that we observe in our va/stt and thus contain many head queries, and (3) tail, which is designed to focus on queries with tail entities. since these do not occur often in our user data, tail consists of synthetic requests sampled from the same templates and entity lists that generate the synthetic training data. the requests cover a wide variety of domains such as music, sports and home automation and the audio is generated using text-to-speech. we evaluate the accuracy of our models using word error rate (wer) and latency using p95 realtime factor (rtf). if y is the duration of the audio signal and x the time it takes to decode y, rtf is defined as x/y. p95 refers to the 95th percentile and thus captures the latency of the most difficult queries. we run each test three times and average the rtf numbers to capture outliers. the asr system uses a deep convolutional neural network acoustic model (am) as described in @xcite . for the as models, we decode the va and tail test sets with a va-specific nnlm and the stt test sets with a stt-specific nnlm. during decoding, the context length of the nnlms is limited to 8 words to meet the memory and latency contraints of ondevice asr. we perform a single decoding pass, combining the am scores with the nnlm scores using optimized weights. we can achieve better wers by interpolating the nnlm with an n-gram lm trained on tail data and by adding a rescoring pass, but since we want to compare the impact of using different neural architectures, we remove any factors that might obscure that comparison. ## results we first evaluate the accuracy of the different neural architectures. we first observe that moving from as to aa fofe and thus reducing the number of parameters by half gives in some cases 1.5-3.8% wer degradation. secondly, even though the transformer architectures have been optimized using bayesian optimization similar to the fofe-based models, they give mixed results. for english va and stt we observe wer improvements while for all other setups we see large degradations. on va for all languages, while the aa mixture fofe gives the best accuracy on stt, but the differences between the two architectures are small. they outperform the baseline as/aa fofe and transformer models in almost all cases. the only exception are the english and german tail test sets: the as fofe models still achieve the best accuracy, probably because infrequent queries benefit the most from doubling the number of parameters. as explained in section 5, we choose hyperparameters based on the optimal accuracy-size trade off. as a result, the number of parameters of the models at the top of ## conclusion we aim to develop a single nnlm that can serve both va and stt requests with the same accuracy and speed as application-specific nnlms, while reducing the disk size approximately by half. we develop a method to optimally balance the data of the va and stt applications, and propose two novel fofe feed-forward architectures. the application-agnostic mixture fofe and the application-dependent fofe both outperform the baseline fofe and transformer models in terms of accuracy, and the latter is also competitive in terms of latency.