Compact Residual Learning with Frequency-Based Non-Square Kernels for Small Footprint Keyword Spotting (2022)

Abstract: Enabling voice assistants on small embedded devices requires a keyword spotter with a smaller model size and adequate accuracy. It becomes difficult to achieve a reasonable trade-off between a small footprint and high accuracy. Recent studies have demonstrated that convolution neural networks are also effective in the audio domain. In this paper, taking into account the nature of spectrograms, we propose a compact ResNet architecture that uses frequency-based non-square kernels to extract the maximum number of timbral features for keyword spotting. The proposed architecture is approximately three-and-a-half times smaller than a comparable architecture with conventional square kernels. On the Google’s speech command dataset v1, it outperforms both Google’s convolution neural networks and the equivalent ResNet architecture with square kernels. By implementing non-square kernels for spectrogram-related data, we can achieve a significant increase in accuracy with relatively few parameters, as compared to the conventional square kernels that are the default choice for every problem.

## introduction keyword detection systems (kws) are implemented on embedded or mobile devices to detect predefined keywords in an audio stream. these words can function as wake words or trigger words for intelligent voice assistants (e.g., hey siri, alexa, or okay google) or as simple speech commands @xcite . due to the nature of deployment, these systems must have a reasonable compromise between a small footprint and high accuracy. however, implementing a fast, compact, and highly accurate kws model that can be deployed on embedded or mobile devices with limited hardware and computation is a significant challenge. recent studies have demonstrated that convolution neural networks (cnns) perform well in the audio domain as well. cnns are predominantly utilized for image-related tasks. cnn's lowest layers typically learn to detect edges. these edges can be oriented in any way. however, one cannot predict which kernel will acquire a given feature. also, since filter dimensions have a spatial meaning, square kernels are widely used to preserve symmetry. it is common practice in the audio domain to transform an audio stream into a spectrogram in which the x-axis represents time and the y-axis represents frequency. then, these spectrograms are fed to two-dimensional cnns as input. spectrograms are a visual representation of the audio intensity over time at various frequencies present in a specific waveform. on spectrograms, the x-axis represents time and the y-axis represents frequency. the images are then sent to cnn, which performs the feature extractions. the composition of spectrograms is known in advance, i.e., time is represented on the x-axis and frequency on the y-axis. in order to capture frequency-based and time-based characteristics, customized kernel designs can be implemented. a number of attempts have been made to use custom rectangular kernels for speech emotion detection and music rhythm classification @xcite . in this paper, we propose a compact resnet architecture, i.e., cnns with residual learning, that uses frequency-based non-square kernels to capture the maximum number of timbral features for keyword detection. typically, the timbral features are taller than they are wide. also, because a 3 × 1 matrix has fewer parameters than a 3 × 3 matrix, we experimented with non-square kernels for the kws use-case, with the goal of reducing the number of parameters while maintaining decent accuracy. as a result, it is worthwhile to employ non-square kernels. figure 1 depicts an example non-square frequency-based kernel. our architecture is a modification of the res8 model (tang and lin, 2018) with non-square kernels, and it is approximately 3.5 times more compact than the res8. in terms of number of parameters, our architecture also outperforms google's best cnn @xcite . compared to the res8 model, the proposed architecture achieves higher evaluation performance and has a smaller footprint. the remainder of the article is structured as follows. brief summaries of recent and pertinent keyword spotting research are presented in section 2. the functional details of our suggested approach for the keyword spotting use-case are presented in section 3. the experimental setup and results are presented in section 4. it also provides a comparative analysis of our proposed approach with respect to the two sota models, @xcite and @xcite , for keyword spotting use-case. finally, the paper concludes with a discussion of future research directions in section 5. ## related works traditionally, hidden markov models (hmms) based approaches were used for kws @xcite . these models were challenging to train, were computationally expensive, and had relatively long latency during inference. some other techniques used recurrent neural networks (rnns) like in @xcite , but they suffered from high latency. after that, @xcite proposed deep neural networks (dnns) with rectified linear unit (relu) activation functions that outperformed hmm-based models with low latency. but the drawback with dnns is that they ignores the audio's local temporal and spectral correlations. to capture these correlations, variations of cnn-based kws were introduced . they achieved better results with reduced footprints. combining the strength of cnns and rnns, arik et al. (2017) experimented with the convolution recurrent neural network-based kws systems. in august 2017, the google's speech command dataset by @xcite was released as a benchmarking dataset for evaluating kws systems. warden (2018) also released a baseline model based on the convolution architecture of @xcite , achieving 85.4% accuracy in v1 version of the dataset. the related kaggle competition was also organized, where the winner achieved 91% accuracy on the v1 version at that time. publication of the google's speech command dataset led to an acceleration in the research. here, we will discuss the most relevant work as per the paper's objective. @xcite (resnets) significantly advance deep learning, and hence they have also been adopted in various audio tasks like automatic speech recognition @xcite and speaker identification @xcite . tang and lin (2018) further experimented with compact residual architecture using dialated convolution to enlarge the size of the receptive field exponentially with the depth of the network, resulting in improved accuracy. they also experimented with model width by decreasing number of filters. @xcite proposed several enhancements to the inception network @xcite . they replaced several convolutions with lower dimension convolutions to decrease the parameters. for example, a 7x7 convolution was replaced with four layers by using 1x7 and 7x1 convolutions twice. to reduce model footprint, a number of recent works like time delay neural network (tdnn), attention mechanism, and temporal convolutional network (tcn) are done @xcite @xcite . there have been some attempts to use rectangular kernels for speech emotion detection and music rhythm classification @xcite . the authors of @xcite have used hexagonal kernels that utilized symmetry equivariance and in-variance of images. our paper uses 2d convolution with non-square kernels (m × 1) convoluted in the frequency domain to capture the maximum amount of timbral features, reducing the computation and number of operations. this study focuses on the family of cnn mod-els, as they continue to serve as the benchmark for kws systems. for kws systems, we desire an analysis between square and non-square kernels. since cnns have a simple architecture, we have conducted our experiments with cnn and residual blocks. in order to perform a more accurate benchmarking, we have referred to the results that were published in @xcite . ## proposed resnet architecture with non-square kernel in this section, the functional details of our proposed resnet architecture with non-square kernels are described. figure 2 provides a visual representation of the architectural layout of the proposed resnet with non-square kernel. the input audio stream is converted into melscaled spectrograms with a length of f f t window as 2048 and 512 samples between successive frames. the spectrogram is then converted to decibels (db), with the highest db being 80. we use the librosa python library to perform the conversion. spectrogram data is then normalized, standardized, and converted to three dimensions by repeating the matrix along all three axes [x, x, x]. ## experimental setup and results this section describes the experimental design and results. it also provides a comparative analysis of our proposed method with two sota models, @xcite and @xcite , for keyword spotting use-case. ## conclusion and future work in this paper, we have presented a fast, small footprint model for real-time kws with non-square kernels (m × 1) that may be useful for small embedded devices. depending on the characteristics of the spectrogram, non-square kernels may be a suitable alternative to square kernels, provided that non-square kernels have fewer trainable parameters than square kernels. experiments indicate that non-square kernels reduce model size by reducing the number of required parameters without degrading performance. the frequency-based kernels have few parameters, and the proposed architecture demonstrates satisfactory performance during the evaluation phase. our proposed work may inspire other researchers and developers to experiment with network architecture based on the dataset and use-case. after comprehending the structure of our data, i.e., spectrogram, we have utilized non-square kernels in the audio domain for kws. it might be worthwhile to consider conducting additional research on benchmarking different architectures using non-square kernels in the future.