Voice Activity Detection (VAD) is an important preprocessing step in any state-of-the-art speech recognition system. Choosing the right set of features and model architecture can be challenging and is an active area of research. In this paper we propose a novel approach to VAD to tackle both feature and model selection jointly. The proposed method is based on a CLDNN (Convolutional, Long Short-Term Memory, Deep Neural Networks) architecture fed directly with the raw waveform. We show that using the raw waveform allows the neural network to learn features directly for the task at hand, which is more powerful than using log-mel features, specially for noisy environments. In addition, using a CLDNN, which takes advantage of both frequency modeling with the CNN and temporal modeling with LSTM, is a much better model for VAD compared to the DNN. The proposed system achieves over 78% relative improvement in False Alarms (FA) at the operating point of 2% False Rejects (FR) on both clean and noisy conditions compared to a DNN of comparable size trained with log-mel features. In addition, we study the impact of the model size and the learned features to provide a better understanding of the proposed architecture. Voice Activity Detection (VAD) refers to the process of identifying segments of speech in an audio utterance [1] . This task is often a pre-processing stage of an automatic speech recognition (ASR) system to both reduce computation and to guide the user interface. A typical VAD system uses a frame-level classifier with acoustic features to make speech/non-speech decisions for each audio frame (every 10ms) [2] . Significant research has been devoted to finding the optimal features for this task [3, 4, 5] , as well as the best classifier or model to use [6, 7, 8] . Deep Neural Networks (DNNs) are a commonly used model for VAD [6] . However, inspired by the advancements in acoustic modeling for speech, we explore alternative deep learning architectures for VAD. Convolutional Neural Networks (CNNs) [9] and Long Short-Term Memory (LSTM) recurrent neural networks [10] are popular choices since they have shown improvements over DNNs for several speech recognition tasks [11, 12] . The modeling capabilities of these different architectures are complementary. CNNs are good at reducing frequency variations, LSTMs are good at sequence modeling, and DNNs mapping features into a more separable space. To exploit the complementary traits of these systems, [13] introduced "Convolutional, Long Short-Term Memory, Fully Con-* The author performed the work as an intern at Google, Mountain View nected Deep Neural Networks" (CLDNNs), obtaining better performance than any of those architectures individually. Since VAD is a sequence task, we believe that architectures which model the temporal structure, such as a CLDNN, are better than a DNN for this task. Even though the mel-scale was designed to mimic the critical bands of hearing in the human ear and has been successful in several speech related tasks, the problem of VAD and noise detection slightly different than speech processing. Thus, since neural networks are good at learning features, we explore if learning directly from the raw waveform has benefit for VAD. In this work, we propose using raw waveform CLDNNs for VAD [14] . We compare the performance of this system against a standard DNN system, an LSTM system, and a CLDNN system all trained with log-mel features. We demonstrate that this approach achieves over a 78% relative improvement (in terms of FA when fixing FR at 2%) on both clean and noisy conditions when compared to a standard DNN trained using log-mel filterbank energies as input. Furthermore, we analyze these results and show the benefit of temporal modeling for VAD with the CLDNN, as well as the importance of learning features for the task at hand with the raw waveform. The rest of the paper is as follows. In Section 2 we describe the CLDNN and raw waveform architectures proposed, as well as the LSTM and DNN reference architectures. The experimental setup is described in Section 3. Section 4 is devoted to presenting and analyzing the results and, finally, Section 5 concludes the paper. In this section we describe the neural network architectures we compare for VAD, namely DNNs, LSTMs, CLDNNs and raw waveform CLDNNs. For each architecture we explore the impact of model size and select their configurations to obtain models of size ∼30k, ∼100k, and ∼200k parameters. DNNs have been shown to give good performance for VAD [6] . The baseline DNN model used in this paper is a standard feedforward fully connected neural network with k hidden layers and n hidden units per layer. For each hidden layer, a rectified linear unit (ReLU) function is used. The output layer of our DNN model is a softmax with 2 units to predict speech and nonspeech. The input into this model is a 40-dimensional log-mel feature, surrounded by a context of 5 past frames and 5 future frames. We experimented with larger input context windows, but it did not affect performance significantly. More details on this topology for different model sizes can be seen in Table 1 .
REF proposed a end-to-end framework for A-VAD that combines convolutional neural network (CNN) and RNN.
Feature Learning with Raw-Waveform CLDNNs for Voice Activity Detection
INTERSPEECH
Computer Science