A character representation enhanced on-device Intent Classification (2020)

Abstract: Intent classification is an important task in natural language understanding systems. Existing approaches have achieved perfect scores on the benchmark datasets. However they are not suitable for deployment on low-resource devices like mobiles, tablets, etc. due to their massive model size. Therefore, in this paper, we present a novel light-weight architecture for intent classification that can run efficiently on a device. We use character features to enrich the word representation. Our experiments prove that our proposed model outperforms existing approaches and achieves state-of-the-art results on benchmark datasets. We also report that our model has tiny memory footprint of ~5 MB and low inference time of ~2 milliseconds, which proves its efficiency in a resource-constrained environment.

## introduction in a time where consumers and businesses alike are constantly adopting new technologies in hope of increasing efficiency and convenience, the intelligent virtual assistant (iva) has been an immediate success 1 . for a high-quality iva, it is very crucial to understand the intentions behind customer queries, emails, chat conversations, and more in order to automate processes and get insights from customer interactions. thus research interest in intent detection is on the rise. intent classification is an important task in natural language understanding (nlu) systems which is the task of assigning a categorical intent label to an input utterance. most ivas use cloudbased solutions and mainly focus on accuracy 1 https://www.statista.com/topics/5572/virtual-assistants/ rather than model size. but due to factors like privacy and personalization, there is a need for deploying models on device and thus on-device intent classification is significant. the current state-of-the-art models are highly accurate on benchmark datasets. however, most of these models have a huge number of parameters and use complex operations. due to these reasons, they are not suitable for deployment on low-resource devices like mobiles, tablets, etc. gartner 2 predicts that by 2020, 80% of the smartphones shipped will have on-device ai capabilities. for this, there is a need for lightweight, fast and accurate models that can run efficiently in a resource-constrained environment. thus, in this paper, we propose an on-device intent classification model. in our proposed model, we use character features along with word embeddings to get enriched word representations. we use long short term memory recurrent neural network (lstm-rnn) @xcite to obtain the context vector for the input utterance. further, we benchmark our model against publicly available atis and snips datasets. our experiments show that the use of character features has resulted in improved accuracy on the benchmark datasets. the major contributions of this paper are given below. ## related work intent classification is the task of predicting an intent label for the given input text. it is a wellresearched task. early research include maximum entropy markov models (memm) by @xcite . @xcite and @xcite have approached this task using support vector machines (svm). another popular model used were crf based methods. @xcite first proposed conditional (crf) to build probabilistic models for segmentation and labelling sequence data which was proved to perform better over memms. following this, triangular-chain conditional random fields was proposed by @xcite which is used to jointly represent the sequence and meta-sequence labels in a single graphical structure. this method outperformed the base model. @xcite have demonstrated the effectiveness of using knowledge-guided patterns in short-text intent classification. @xcite have proposed the use of semantic hashing for intent classification for small datasets. recently joint models for intent classification and slot filling have been developed. taking inspiration from tricrf, @xcite proposed a cnn-based tricrf for joint intent and slot filling. this was a neural network version of tricrf which outperformed the base model by 1% for both intent and slot. a joint model using gated recurrent unit (gru) and max pooling for intent detection and slot filling was developed by @xcite . following this, @xcite and @xcite also developed a joint model using recurrent neural networks. to model the relationship between the intent and slots, goo et al. (2018 although above mentioned joint models achieve impressive results on benchmark datasets, they are inefficient for the applications where only intent information is sufficient. also, their heavy architecture and large model size make their ondevice deployment difficult. most of these models use multiple layers of operations that result in higher ram usage and inference time. our proposed model is light-weight, fast, and accurate, which makes it highly efficient for deployment on low-resource devices. ## approach in this section, we discuss our approach in detail. we use a light-weight architecture that can be efficiently deployed on device. we use characterlevel representation along with word embedding to enrich word-level representation. ## experimental results in this section, we share the details of the benchmark and custom datasets, describe the training set-up, present experimental results, and compare our model with existing baselines. ## conclusion in this paper, we present an on-device intent classification architecture that uses character level features to enrich the word representation. our experiments prove the effectiveness of our model as it achieves state-of-the-art results on benchmark datasets. system centric metrics like ram usage and inference time shows that our model is fast and light-weight to be deployed on low-resource devices. for future work, we want to extend this approach for slot filling and experiment with a joint model for on-device intent detection and slot filling.