Towards Efficient Audio-Text Keyword Spotting: Quantization and Multi-Scale Linear Attention with Foundation Models (2024)

Abstract: Open Vocabulary Keyword Spotting is essential in numerous applications, from virtual assistants to security systems, as it allows systems to identify specific words or phrases in continuous speech. In this paper, we propose a novel end-to-end method for detecting user-defined open vocabulary keywords by leveraging linguistic patterns for the correlation between audio and text modalities. Our approach utilizes quantized pre-trained foundation models for robust audio embeddings and a unique lightweight Multi-Scale Linear Attention (MSLA) network that aligns speech and text representations for effective cross-modal agreement. We evaluate our method on two distinct datasets, comparing its performance against other baselines. The results highlight the effectiveness of our approach, achieving significant improvements over the Cross-Modality Correspondence Detector (CMCD) method, with a 16.08% increase in AUC and a 17.2% reduction in EER metrics on the Google Speech Commands dataset. These findings demonstrate the potential of our method to advance keyword spotting across various real-world applications.

## introduction keyword spotting (kws) @xcite is essential for voice-driven interactions on edge devices, particularly with the growing demand for personalized voice assistants. traditional kws @xcite relies on pre-defined keywords, whereas user defined keyword spotting (udkws) @xcite ) enables recognition of user-specific keywords not seen during training, providing more flexibility. many udkws approaches utilize query by example (qbye) methods @xcite @xcite , which are limited by variations in speakers and environments. in contrast, text-based keyword enrollment @xcite methods have gained popularity, as they avoid these issues by focusing on linguistic features which are less prone to variability rather than acoustic characteristics. end-to-end text enrollment kws methods @xcite aim to map audio and text representations to a shared space, but challenges remain in distinguishing similar pronunciations. self-supervised learning speech models (ssls) @xcite like wav2vec 2.0 @xcite , hubert @xcite , wavlm @xcite , and whisper (openai, 2022) offer robust representations but can be computationally expensive for resource-limited devices. model quantization @xcite helps reduce the size and memory requirements of these models, enabling efficient deployment without sacrificing performance. the attention mechanisms of the transformer architecture @xcite enables models to attend to relevant portions of input sequences. recent research has explored cross-modal matching techniques, such as attention-based models @xcite , for aligning representations from different modalities such as speech and text effectively. efficientvit @xcite , introduces a multiscale linear attention (msla) mechanism for vision transformers to reduce the computational complexity of traditional softmax based self-attention. by attending to multiple scales and approximating attention scores, msla enables aggregation of finegrained and coarse-grained features through linear scaling while maintaining low memory usage, making it ideal for deployment on edge devices. in a nutshell, our contributions include: ## proposed method this section presents our proposed method, a multiscale linear cross-attention that computes the similarity between the speech and text by utilizing the capabilities of quantized foundation models. the fundamental blocks of this method are the encoders, a processor and an analyzer. figure 1 , illustrates the general architecture. ## results and discussion we trained our proposed approach on the google speech commands dataset and evaluated it on the test sets of google speech commands (g) and the qualcomm keyword speech (q), which included out-of-vocabulary keywords. the evaluation metrics used were equal error rate (eer) and area under the curve (auc) ( our approach significantly outperformed baseline methods, even when the training and test datasets differed. this highlights not only the strong generalization capabilities of the underlying foundation models but also the effectiveness of the msla mechanism, making it well-suited for open-vocabulary keyword spotting tasks.