Original: Robust Speech Command Recognition using Label-Driven Time-Frequency Masking
Revised: Robust Speech Command Recognition using Label-Driven Time-Frequency Masking

Original: Anonymous Author
Revised: Anonymous Author(s)

Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Speech enhancement driven robust Automatic Speech Recognition (ASR) systems typically require a parallel corpus with noisy and clean speech utterances for training.
Revised: Speech enhancement driven robust Automatic Speech Recognition (ASR) systemstypically require a parallel corpus with noisy and clean speech utterances fortraining.

Original: Moreover, many studies have reported that such front-ends, even though improve speech quality, do not always improve the recognition performance.
Revised: Moreover, many studies have reported that such front-ends, even thoughimprove speech quality, do not always improve the recognition performance.

Original: On the other hand, multi-condition training of ASR systems provides little visualization or interpretability capabilities of how these systems achieve robustness.
Revised: On theother hand, multi-condition training of ASR systems provides little visualization orinterpretability capabilities of how these systems achieve robustness.

Original: In this paper, we propose a novel neural architecture with uniﬁed enhancement and sequence classiﬁcation block, that is trained in an end-to-end manner only using noisy speech without having information of clean speech.
Revised: In this paper,we propose a novel neural architecture with uniﬁed enhancement and sequenceclassiﬁcation block, that is trained in an end-to-end manner only using noisy speechwithout having information of clean speech.

Original: The enhancement block is a fully convolutional network that is designed to perform Time Frequency (T-F) masking like operation, followed by an LSTM sequence classiﬁcation block.
Revised: The enhancement block is a fullyconvolutional network that is designed to perform Time Frequency (T-F) maskinglike operation, followed by an LSTM sequence classiﬁcation block.

Original: The T-F masking formulation enables visualization of learned mask and helps us to visualize the T-F points important for classiﬁcation of a speech command.
Revised: The T-Fmasking formulation enables visualization of learned mask and helps us to visualizethe T-F points important for classiﬁcation of a speech command.

Original: Experiments performed on Google Speech Command dataset show that our proposed network achieves better results than the baseline model without an enhancement front-end.
Revised: Experimentsperformed on Google Speech Command dataset show that our proposed networkachieves better results than the baseline model without an enhancement front-end.

Original:  Introduction
Revised:  Introduction

Original: Performance degradation of Deep Neural Network (DNN) based Automatic Speech Recognition (ASR) systems in the presence of channel distortions, reverberation, and additive noise is still a well known issue [1].
Revised: Performance degradation of Deep Neural Network (DNN) based Automatic Speech Recognition(ASR) systems in the presence of channel distortions, reverberation, and additive noise is stilla well known issue [1].

Original: There are two major paradigms to achieve robustness to various noise conditions, (a) use of model adaptation techniques to achieve robustness against various degradation conditions [1, 2, 3, 4], (b) use of enhancement front-end to map noisy speech features to clean features [5, 6, 7, 8, 9, 10, 11].
Revised: There are two major paradigms to achieve robustness to various noiseconditions, (a) use of model adaptation techniques to achieve robustness against various degradationconditions [1, 2, 3, 4], (b) use of enhancement front-end to map noisy speech features to cleanfeatures [5, 6, 7, 8, 9, 10, 11].

Original: Model adaptation techniques majorly use representation power of
Revised: Model adaptation techniques majorly use representation power of

Original: DNNs to train the model with various degradation conditions.
Revised: DNNs to train the model with various degradation conditions.

Original: This approach is reported to work well in wide range of degradation conditions [2, 4], without using information of clean speech.
Revised: This approach is reported to work wellin wide range of degradation conditions [2, 4], without using information of clean speech.

Original: However, they do not give much insights regarding their inner workings.
Revised: However,they do not give much insights regarding their inner workings.

Original: Other popular approach to achieve robustness is to employ an enhancement front-end using De-noising Autoencoder (DAE) based on various DNN architectures such as DNN-DAE [12], Time-Delay Neural Network (TDNN)-DAE [8], Recurrent Neural Network (RNN)-based DAE [6, 7], or Time-Frequency (T-F) masking-based approaches to enhance the noisy signal [10, 9].
Revised: Other popular approach to achieverobustness is to employ an enhancement front-end using De-noising Autoencoder (DAE) based onvarious DNN architectures such as DNN-DAE [12], Time-Delay Neural Network (TDNN)-DAE[8], Recurrent Neural Network (RNN)-based DAE [6, 7], or Time-Frequency (T-F) masking-basedapproaches to enhance the noisy signal [10, 9].

Original: To train such front-ends, a parallel corpus containing noisy and clean speech pairs is required.
Revised: To train such front-ends, a parallel corpus containingnoisy and clean speech pairs is required.

Original: However, it is reported that such front-ends do not always yield improvement in performance in unseen noise conditions [12].
Revised: However, it is reported that such front-ends do not alwaysyield improvement in performance in unseen noise conditions [12].

Original: In this paper, we propose a novel neural network architecture that can leverage advantages of both these approaches.
Revised: In this paper, we propose a novel neural network architecture that can leverage advantages of boththese approaches.

Original: We propose a network with an enhancement front-end block that has a T-F masking like formulation in such that it learns feature detectors to locate T-F regions important for classiﬁcation.
Revised: We propose a network with an enhancement front-end block that has a T-F maskinglike formulation in such that it learns feature detectors to locate T-F regions important for classiﬁcation.

Original: The output of this enhancement block is given to an LSTM-based sequence classiﬁcation block.
Revised: The output of this enhancement block is given to an LSTM-based sequence classiﬁcation block.

Revised: [Figure]

Original:  The entire network is jointly trained in an end-to-end manner using only noisy data with random parameter initialization and without providing the network any information about clean speech data.
Revised: block The entire network is jointly trained in an end-to-end manner using only noisy data with randomparameter initialization and without providing the network any information about clean speech data.

Original: The proposed architecture enables an easy visualization of the enhancement process by inspecting the T-F mask applied by the enhancement block and activation maps of convolution ﬁlters in the ﬁrst [Figure] BaselineModel [Figure]
Revised: The proposed architecture enables an easy visualization of the enhancement process by inspectingthe T-F mask applied by the enhancement block and activation maps of convolution ﬁlters in the ﬁrstlayer of the enhancement block.

Original: CNN enhancement block layer of the enhancement block.

Original: This visualization gives insights on what T-F regions are important for classiﬁcation of a speech command.
Revised: This visualization gives insights on what T-F regions are importantfor classiﬁcation of a speech command.

Original: Experiments on Google Speech Command dataset [13] demonstrate the effectiveness of the proposed model and its visualization capabilities.
Revised: Experiments on Google Speech Command dataset [13]demonstrate the effectiveness of the proposed model and its visualization capabilities.

Original:  Proposed model architecture
Revised:  Proposed model architecture

Original: Our proposed model based on label-driven T-F masking is shown in Figure 1.
Revised: Our proposed model based on label-driven T-F masking is shown in Figure 1.

Original: It consist of two fully convolutional layers in the enhancement block.
Revised: It consist of two fullyconvolutional layers in the enhancement block.

Original: The output of convolutional block is then applied to the input T-F representation by treating the output of the convolutional block as a T-F mask.
Revised: The output of convolutional block is then appliedto the input T-F representation by treating the output of the convolutional block as a T-F mask.

Original: We constrain the mask values between 0 − 1 by applying the sigmoid activation function at the output of the convolutional block.
Revised: Weconstrain the mask values between 0 − 1 by applying the sigmoid activation function at the outputof the convolutional block.

Original: Here, the input is log-magnitude domain T-F representation such as log Mel-Filterbank Energies (FBEs).
Revised: Here, the input is log-magnitude domain T-F representation such aslog Mel-Filterbank Energies (FBEs).

Original: Mathematically, operations of the enhancement block can be summarized as follows:
Revised: Mathematically, operations of the enhancement block can besummarized as follows:

Original: [Equation]
Revised: [Equation]

Original: where X ( t, f ) is the input T-F representation (e.g. FBEs), M ( t, f ) is the T-F mask taken at the output of the enhancement block, and Y ( t, f ) is the enhanced T-F representation.
Revised: where X ( t, f ) is the input T-F representation (e.g. FBEs), M ( t, f ) is the T-F mask taken at the outputof the enhancement block, and Y ( t, f ) is the enhanced T-F representation.

Original: An LSTM layer (referred as the baseline model) takes the enhanced T-F representation Y ( t, f ) as an input and the ﬁnal hidden state of the LSTM cell is then propagated to fully connected and softmax classiﬁcation layer.
Revised: An LSTM layer (referred as the baseline model) takes the enhanced T-F representation Y ( t, f ) as aninput and the ﬁnal hidden state of the LSTM cell is then propagated to fully connected and softmaxclassiﬁcation layer.

Original: The parameters of classiﬁcation and enhancement block are optimized using the ﬁnal hidden state of the LSTM layer.
Revised: The parameters of classiﬁcation and enhancement block are optimized using theﬁnal hidden state of the LSTM layer.

Original: Hence, the enhancement block tries to enhance the entire T-F sequence, in contrast with the frame-level enhancement.
Revised: Hence, the enhancement block tries to enhance the entire T-Fsequence, in contrast with the frame-level enhancement.

Original: The enhancement block, along with the baseline model is trained to maximize the target class probability.
Revised: The enhancement block, along with thebaseline model is trained to maximize the target class probability.

Original: Hence, the T-F mask is learned in a manner that will increase correct classiﬁcation probability.
Revised: Hence, the T-F mask is learned in amanner that will increase correct classiﬁcation probability.

Original: We train one more model to compare results with T-F masking based formulation.
Revised: We train one more model to compare results with T-F masking based formulation.

Original: In this model we use the same enhancement block as used earlier.
Revised: In this modelwe use the same enhancement block as used earlier.

Original: However, rather than treating output of the enhancement block as T-F mask, we treat the output of the enhancement block as an enhanced T-F representation and directly feed it to the baseline model. 
Revised: However, rather than treating output of theenhancement block as T-F mask, we treat the output of the enhancement block as an enhanced T-Frepresentation and directly feed it to the baseline model. [Table] [Table]

Original:  Experiments and Results 3.1 Database description
Revised:  Experiments and Results3.1 Database description

Original: We use Google Speech Command dataset for our experiments [13].
Revised: We use Google Speech Command dataset for our experiments [13].

Original: The database consists of 64, audio ﬁles, each of 1 second duration, and consisting of one spoken command.
Revised: The database consists of 64,727audio ﬁles, each of 1 second duration [Table] and consisting of one spoken command.

Original: Each utterance is labelled with one of the possible 30 commands.
Revised: Each utterance islabelled with one of the possible 30 commands.

Original: The splits for train (80%), validation (10%), and test
Revised: The splits for train (80%), validation (10%)

Original: [Table]
Revised: [Table]

Original: [Table] (%) datasets are provided in the database.
Revised: and test(%) datasets are provided in the database.

Original: The dataset also provides background noise audio ﬁles with six types of noise.
Revised: The dataset also provides background noise audio ﬁles withsix types of noise.

Original: In the initial observation we found that the audio ﬁles were already containing little noise.
Revised: In the initial observation we found that the audio ﬁles were already containing littlenoise.

Original: To evaluate the robustness of the proposed model and visualization of enhancement process, we add the provided noises at 15 dB, 10 dB and 5 dB SNR to test utterances.
Revised: To evaluate the robustness of the proposed model and visualization of enhancement process,we add the provided noises at 15 dB, 10 dB and 5 dB SNR to test utterances.

Original: We do not add any noise in training data.

Original: 3.2 Model architectures and results
Revised: 3.2 Model architectures and results

Original: In our proposed model, the ﬁrst convolution layer in the enhancement block had 60 convolutional ﬁlters of size 15 × 7 followed by ReLU activation.
Revised: In our proposed model, the ﬁrst convolution layer in the enhancement block had 60 convolutionalﬁlters of size 15 × 7 followed by ReLU activation.

Original: The second convolution layer had one convolutional ﬁlter of size 7 × 7 × 60 followed by sigmoid activation.
Revised: The second convolution layer had one convolutionalﬁlter of size 7 × 7 × 60 followed by sigmoid activation.

Original: Theﬁlter number of ﬁlters and ﬁlter dimensions were optimized on validation set.
Revised: The number of ﬁlters and ﬁlter dimensionswere optimized on validation set.

Original: The model was trained to jointly optimize the parameters of enhancement block and the baseline model.
Revised: The model was trained to jointly optimize the parameters ofenhancement block and the baseline model.

Original: Results of this model are tabulated under the label “T-F masking enhancement".
Revised: Results of this model are tabulated under the label “T-Fmasking enhancement".

Original: The baseline model shown in Figure 1 has an LSTM layer with 128 units with ReLU activation.
Revised: The baseline model shown in Figure 1 has an LSTM layer with 128 units and ReLU activation.

Original: The
Revised: The

Original: LSTM layer was followed by a fully connected layers with 128 units with ReLU activation.
Revised: LSTM layer was followed by a fully connected layers with 128 units with ReLU activation.

Original: The output layer had 30 softmax units for 30 class classiﬁcation.
Revised: Theoutput layer had 30 softmax units for 30 class classiﬁcation.

Original: Input to our models were 40 dimensional
Revised: Input to our models were 40 dimensional

Original: FBEs of 1 second utterance extracted by taking the frames of 25 ms with 10 ms overlap.
Revised: FBEs of 1 second utterance extracted by taking the frames of 25 ms with 10 ms overlap.

Original: The model was trained using cross-entropy objective and ADAM optimizer with learning rate of 0.001 forepochs and model that gives the best accuracy on validation dataset was used for testing.
Revised: The modelwas trained using cross-entropy objective and ADAM optimizer with learning rate of 0.001 for 10epochs and model that gives the best accuracy on validation dataset was used for testing.

Original: The model parameters and training scheme for the model with the  enhancement blockwithout T-F masking were same as the model with T-F masking enhancement.
Revised: The model parameters and training scheme for the model with the direct enhancement block same asthe model with T-F masking enhancement.

Original: In this case we used linear activation instead of sigmoid activation.
Revised: In this case we used linear activation instead of sigmoidactivation.

Original: Results of this model are tabulated under the label “Direct enhancement".
Revised: Results of this model are tabulated under the label “Direct enhancement".

Original: Results for all the models are shown in Table 1.
Revised: Results for all the models are shown in Table 1.

Original: The baseline results are better than the CNN and
Revised: The baseline results are better than the CNN and

Original: Capsule Network models trained on the same database with same training/testing condition [14].
Revised: Capsule Network models trained on the same database with same training/testing condition [14].

Original: On 20 commands evaluation our LSTM baseline (91.12 %) performed better than CNN (77.9 %) and
Revised: On20 commands evaluation our LSTM baseline (91.12 %) performed better than CNN (77.9 %) and

Original: CapsNet (87.3 %) for 20-class decoding [14].
Revised: CapsNet (87.3 %) for 20-class decoding [14].

Original: Model with direct enhancement gave improvement over baseline system in both validation (91.5%) and test(91.47%) dataset.
Revised: Model with direct enhancement gave improvement overbaseline system in both validation (91.5%) and test(91.47%) dataset.

Original: While our proposed model with T-F masking enhancement gave the best results on the original validation (92.92%) and test (92.9%) dataset.
Revised: While the proposed model gavethe best results on the original validation (92.92%) and test (92.9%) dataset.

Original: Table 2 shows the results of evaluating the trained models on noisy dataset.
Revised: Table 2 shows the resultsof evaluating the trained models on noisy dataset.

Original: Performance of LSTM baseline degraded greatly in the presence of noise.
Revised: Performance of LSTM baseline degraded greatly inthe presence of noise.

Original: By employing CNN enhancement block, the results on noisy database improved signiﬁcantly.
Revised: By employing CNN enhancement block, the results on noisy database improvedsigniﬁcantly.

Original: In caseof noisy test data also proposed model with label-driven T-F masking enhancement gave the best results.
Revised: While in this case also proposed model with label-driven T-F masking enhancementgave the best results.

Original: [Figure]
Revised: [Figure]

Original: II
Revised: II

Original: 3.3 Visualizing the enhancement process
Revised: 3.3 Visualizing the enhancement process

Original: To visualize the enhancement process we show input, output [Figure] and T-F mask applied by enhancement block in Figure 2-I for an original test utterance as well as noisy versions of it.
Revised: To visualize the enhancement process we show input, output, and T-F mask applied by enhancementblock in Figure 2-I for an original test utterance as well as noisy versions of it.

Original: It can be observed that the original utterance shown in Figure 2-I (a) taken from test dataset already has some noise.
Revised: It can be observedthat the original utterance shown in Figure 2-I (a) taken from test dataset already has some noise.

Original: The enhancement block gives the output shown in Figure 2-I (b).
Revised: Theenhancement block gives the output shown in Figure 2-I (b).

Original: T-F mask learned by the enhancement block is shown in ﬁgure 2-I (c).
Revised: T-F mask learned by the enhancementblock is shown in ﬁgure 2-I (c).

Original: Figure 2-I (d)-(f) and (g)-(i) show the similar plots for the same utterance with 15 dB and 10 dB additive noise of type running_tap, respectively.
Revised: Figure 2-I (d)-(f) and (g)-(i) show the similar plots for the sameutterance with 15 dB and 10 dB additive noise of type running_tap, respectively.

Original: Visual inspection of the enhanced representation and T-F mask suggests that the enhancement block tries to achieve two things : (1) ﬁnding out the important T-F points in the input T-F representation where acoustic information about spoken word is present, (2) ﬁnding out the boundary between T-F regions where acoustic information about spoken word is present and silence regions.
Revised: Visual inspection of the enhanced representation and T-F mask suggests that the enhancement blocktries to achieve two things : (1) ﬁnding out the important T-F points in the input T-F representationwhere acoustic information about spoken word is present, (2) ﬁnding out the boundary between T-Fregions where acoustic information about spoken word is present and silence regions.

Original: Additionally, the T-F masks for original as well as noisy utterances are fairly similar except for some T-F regions with very less SNR.
Revised: Additionally,the T-F masks for original as well as noisy utterances are fairly similar except for some T-F regionswith very less SNR.

Original: Figure 2-II shows the activation maps of 9 selected ﬁlters (out of 60 ﬁlters) for the utterance in Figure 2-I (a) as input.
Revised: Figure 2-II shows the activation maps of 9 selected ﬁlters (out of 60 ﬁlters) for the utterance in Figure2-I (a) as input.

Original: Figure 2-II (a)-(c) suggest that the underlying ﬁlters tries to locate the T-F regions where spoken command is present.
Revised: Figure 2-II (a)-(c) suggest that the underlying ﬁlters tries to locate the T-F regionswhere spoken command is present.

Original: Figure 2-II (d)-(f) show that the underlying ﬁlters are locating the boundaries between acoustic information of spoken command and non speech T-F regions.
Revised: Figure 2-II (d)-(f) show that the underlying ﬁlters are locating theboundaries between acoustic information of spoken command and non speech T-F regions.

Original: While
Revised: While

Original: Figure 2-II (g)-(h) show the activation of ﬁlters that are ﬁnding out the important T-F points in the area where acoustic information of spoken command is present.
Revised: Figure 2-II (g)-(h) show the activation of ﬁlters that are ﬁnding out the important T-F points in thearea where acoustic information of spoken command is present.

Original: Figure 2-II (i) shows the activation of ﬁlter that is not signiﬁcant for classifying the utterance.
Revised: Figure 2-II (i) shows the activationof ﬁlter that is not signiﬁcant for classifying the utterance.

Original: We found that majority of the activation maps resembled Figure 2-II (a)-(c), i.e. trying to locate the T-F regions where spoken command is present.
Revised: We found that majority of the activationmaps resembled Figure 2-II (a)-(c), i.e. trying to locate the T-F regions where spoken command ispresent.

Original: Other signiﬁcant number of ﬁlters resembled the ﬁlters shown in Figure 2-II (d)-(f).
Revised: Other signiﬁcant number of ﬁlters resembled the ﬁlters shown in Figure 2-II (d)-(f).

Original: These visualizations suggest that enhancement for robust speech classiﬁcation is different than traditional enhancement.
Revised: These visualizations suggest that enhancement for robust speech classiﬁcation is different thantraditional enhancement.

Original: While traditional enhancement front-ends try to remove or suppress noise, the label driven enhancement approach focuses on ﬁnding out important regions in T-F representation that are signiﬁcant to increase the correct classiﬁcation probability.
Revised: While traditional enhancement front-ends try to remove or suppress noise,the label driven enhancement approach focuses on ﬁnding out important regions in T-F representationthat are signiﬁcant to increase the correct classiﬁcation probability.

Original:  Summary and Conclusions
Revised:  Summary and Conclusions

Original: In this paper, we propose a novel neural network architecture with a fully convolutional enhancement block and LSTM-based classiﬁcation block for robust speech command recognition.
Revised: In this paper, we propose a novel neural network architecture with a fully convolutional enhancementblock and LSTM-based classiﬁcation block for robust speech command recognition.

Original: We trained our model directly on noisy data to jointly train the enhancement and classiﬁcation block.
Revised: We trainedour model directly on noisy data to jointly train the enhancement and classiﬁcation block.

Original: The enhancement block had T-F masking like formulation for enhancement purpose.
Revised: Theenhancement block had T-F masking like formulation for enhancement purpose.

Original: Our proposed model gave signiﬁcantly better classiﬁcation accuracy for both original and noisy test set.
Revised: Our proposed modelgave signiﬁcantly better classiﬁcation accuracy for both original and noisy test set.

Original: The visualization of enhancement process for improving classiﬁcation accuracy gave signiﬁcant insights on the working of the proposed network.
Revised: The visualizationof enhancement process for improving classiﬁcation accuracy gave signiﬁcant insights on the workingof the proposed network.

Original: We observed that instead of removing or suppressing noise present in the noisy T-F representation, the enhancement block locates the important regions in the input T-F representation.
Revised: We observed that instead of removing or suppressing noise present inthe noisy T-F representation, the enhancement block locates the important regions in the input T-Frepresentation.