S o DA : On-device Conversational Slot Extraction (2021)

Abstract: We propose a novel on-device neural sequence labeling model which uses embedding-free projections and character information to construct compact word representations to learn a sequence model using a combination of bidirectional LSTM with self-attention and CRF. Unlike typical dialog models that rely on huge, complex neural network architectures and large-scale pre-trained Transformers to achieve state-of-the-art results, our method achieves comparable results to BERT and even outperforms its smaller variant DistilBERT on conversational slot extraction tasks. Our method is faster than BERT models while achieving significant model size reduction–our model requires 135x and 81x fewer model parameters than BERT and DistilBERT, respectively. We conduct experiments on multiple conversational datasets and show significant improvements over existing methods including recent on-device models. Experimental results and ablation studies also show that our neural models preserve tiny memory footprint necessary to operate on smart devices, while still maintaining high performance.

## introduction in today's world, people rely on their digital devices like mobile phones, smartwatches, home assistants like google and alexa to alleviate mundane tasks like play favorite songs, recommend food recipes among others. a big part of the language understanding capabilities of such assistive devices happens on cloud, where the relevant slots, entities and intents are extracted in order for the request to be fulfilled. however, is it not always safe to send data to cloud, or when we travel it is not always possible to have internet connectivity, yet we want to enjoy the same capabilities. these challenges can be solved by building ondevice neural models that can perform inference on device and extract the slot (entity) information needed for language understanding. the model will operate entirely on the device chip and will not send or request any external information. such ondevice models should have low latency, small memory and model sizes to fit on memory-constrained devices like mobile phones, watches and iot. recently, there has been a lot of interest and novel research in developing on-device models. large body of work focuses on wake word detection @xcite ), text classification like intent recognition @xcite , news and product reviews @xcite @xcite . in this paper, we propose a novel on-device neural sequence tagging model called soda . our novel approach uses embedding-free projections and character-level information to construct compact word representations and learns a sequence model on top of the projected representations using a combination of bidirectional lstm with selfattention and crf model. we conduct exhaustive evaluation on different conversational slot extraction datasets. the main contributions of our work are as follows: in this section, we describe the components of our soda architecture as shown in figure 1 . ## soda training and parameters we now describe details for training the on-device soda neural network. we implement the model using tensorflow. for each sequence labeling task, we train the parameters of the model on the corresponding dataset, then apply the same steps in order for inference and evaluate the decoded tag sequence output against the gold label sequence. ## results for conversational slot extraction this section presents results from the conversational slot extraction task on the atis and snips datasets. tables 1 and 2 show the obtained results from our on-device soda approach, which outperformed prior state-of-the-art on-device slot extractors based on single and multi-task convolution including the compressed convolution models @xcite . our on-device soda even outperformed prior non-on-device state-of-the-art neural models like capsule-nlu, stackpropagation, rnn, cnn, gated full attention, joint intent-slot modeling and even bert models on atis and snips datasets. ## soda performance analysis next, we show various ablation studies that evaluate the performance of different soda components. ## conclusion we introduced a novel on-device conversational slot extraction model called soda which uses embedding-free projections and character information to construct compact word representations, and then learn a sequence model using a combination of bidirectional lstm with self-attention and crf. we evaluate our approach on multiple slot extraction datasets. our on-device model soda achieves state-of-the-art results and also improved over nonon-device models like capsule-nlu @xcite , stackpropagation @xcite , interrelated sf-first with @xcite , joint bil-stm @xcite , attention rnn @xcite , gated attention @xcite and even bert models @xcite . our on-device soda model also significantly outperforms state-of-the-art on-device slot extraction models of @xcite , which are based on convolution and are further compressed with structured pruning and distillation. as shown in the evaluation and ablation studies, unlike existing large neural networks that rely on additional information such as pre-trained embeddings, intent information and knowledge bases, soda does not use any external resources, and yet it achieves good performance, while maintaining compact size.