On-Device detection of sentence completion for voice assistants with low-memory footprint (2020)

Abstract: Sentence completion detection (SCD) is an important task for various downstream Natural Language Processing (NLP) based applications. For NLP based applications, which use the Automatic Speech Recognition (ASR) from third parties as a service, SCD is essential to prevent unnecessary processing. Conventional approaches for SCD operate within the confines of sentence boundary detection using language models or sentence end detection using speech and text features. These have limitations in terms of relevant available data for training, performance within the memory and latency constraints, and the generalizability across voice assistant domains. In this paper, we propose a novel sentence completion detection method with low memory footprint for On-Device applications. We explore various sequence-level and sentence-level experiments using state-of-the-art Bi-LSTM and BERT based models for English language.

## introduction voice-intelligence enabled devices have tremendous potential in providing near natural human behavior based product experience to the users @xcite . such a potential has primarily stemmed from artificial intelligence based mimicking of speech recognition, question answering, dialogues, conversations, and command-induced actions. in order to meet user expectations and cater towards product usage satisfaction, knowing when the question, reply or command is complete, i.e., sentence completion detection, is a crucial task. voice assistants like google assistant, alexa, cortana, siri and bixby etc. are becoming very popular in modern world. these systems rely on the text, predicted by a streaming speech recognition (asr) system. streaming asr produces text continuously. it is computationally efficient to execute downstream nlp tasks only when a complete sentence is found in the text. this makes scd a crucial need in voice assistant system. for a partial text received from asr, the application can wait relatively longer than a complete sentence. various downstream nlp applications such as unsupervised dependency parsing, asr transcript readability, accurate information retrieval, and summarization can benefit from scd. detection of sentence completion, has been widely attempted on speech @xcite ) and text @xcite using sentence boundary detection (sbd) @xcite , end-of-utterance detection @xcite , sentence end detection (sed) @xcite models. the techniques evolved from rule-based using handcrafted heuristics @xcite , to machine learning and more recently, deep learning @xcite based methods. state-of-the-art deep learning architectures reported for sbd include bi-lstm crf @xcite , bert @xcite etc. techniques. on various test datasets, we found their performance highly promising. upon exploring at further depth, we found that these models have certain limitations with regards to their size, on device platform compatibility and system coupling ## related work recent scd and sbd works are primarily useful for legal text, long lectures, pdf documents etc. consequently, the datasets used for relevant work included clean texts such as wsj corpus and brown corpus @xcite , noisy unstructured texts generated from pdfs @xcite , @xcite , lecture (hasan, 2014) and asr transcripts @xcite , @xcite . for training a model that is suitable for the multi-domain voice assistant, we could not find broad-spectrum domain data focused on commands. further, we felt that, essentially, a shift of emphasis from formal, edited text towards more 1 https://www.tensorflow.org/lite/models/bert_qa/overview spontaneous language samples which represent asr output is required conventional language models are trained on long structured sentences leading to large memory footprints that cannot be supported for fast on-device applications. various techniques have been reported for downsizing, such as quantization, modifications in vocabulary, truncating input etc. we expand our work based on modifications in state-of-the-art architectures and extensive custom training with custom loss on multi-domain conversation data. we also experimented upon tensorflow lite post training quantization. we primarily looked at bi-lstm and bert architectures as described below. ## model we defined our scd models in two categories: ## results and discussion in the following, we present the results of our performance assessment on various models, i.e., both token-level and sentence-level inferencing using bi-lstm and bert techniques on the three test datasets.