Original: Cycle-Consistent GAN Front-end to Improve ASR Robustness to Perturbed Speech
Revised: Cycle-Consistent GAN Front-end to Improve ASR Robustness to Perturbed Speech

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Automatic Speech Recognition (ASR) systems, which perform well on regularspeech, are found to be vulnerable to adversarial examples generated by smallperturbations in the audio signal.
Revised: Automatic Speech Recognition (ASR) systems, which perform well on regularspeech, are found to be vulnerable to adversarial examples generated by smallperturbations in the audio signal.

Original: Even naturally introduced perturbations in audiosignal, caused by emotional and physical states of the speaker, can signiﬁcantlydegrade the ASR performance.
Revised: Even naturally introduced perturbations in audiosignal, caused by emotional and physical states of the speaker, can signiﬁcantlydegrade the ASR performance.

Original: In this paper, we propose a front-end based on
Revised: In this paper, we propose a front-end based on

Original: Cycle-Consistent Generative Adversarial Network (CycleGAN) to reduce theperturbations, and hence add robustness to ASR performance.
Revised: Cycle-Consistent Generative Adversarial Network (CycleGAN) to reduce theperturbations, and hence add robustness to ASR performance.

Original: CycleGAN istrained using non-parallel examples of perturbed and normal speech.
Revised: CycleGAN istrained using non-parallel examples of perturbed and normal speech.

Original: Experimentson spontaneously generated laughter-speech and creaky voice datasets tested with
Revised: Experimentson spontaneously generated laughter-speech and creaky voice datasets tested with

Original: Google cloud ASR show absolute improvements in WER of 14.9% and 11%,respectively, on speech converted using the CycleGAN based front-end as comparedto the original perturbed speech.
Revised: Google cloud ASR show absolute improvements in WER of 14.9% and 11%,respectively, on speech converted using the CycleGAN based front-end as comparedto the original perturbed speech.

Original:  Introduction
Revised:  Introduction

Original: The advent of powerful deep learning techniques in the recent past have resulted in signiﬁcantimprovements in the performance of Automatic Speech Recognition (ASR) systems [1, 2, 3].
Revised: The advent of powerful deep learning techniques in the recent past have resulted in signiﬁcantimprovements in the performance of Automatic Speech Recognition (ASR) systems [1, 2, 3].

Original: Insome scenarios, ASR systems are performing in par with the human-level accuracy.
Revised: Insome scenarios, ASR systems are performing in par with the human-level accuracy.

Original: Further, theadvent of voice assistants such as Google Home, Amazon Echo etc., have led to the wide use of ASRsystems in various day-to-day applications such as voice-based search [4], home automation [5] andelderly care [6].
Revised: Further, theadvent of voice assistants such as Google Home, Amazon Echo etc., have led to the wide use of ASRsystems in various day-to-day applications such as voice-based search [4], home automation [5] andelderly care [6].

Original: However, recent studies have shown that the adversarial examples, generated by either adding asmall amount of noise or by modifying a few bits of the audio signal can be used to attack the
Revised: However, recent studies have shown that the adversarial examples, generated by either adding asmall amount of noise or by modifying a few bits of the audio signal can be used to attack the

Original: ASR systems to generate a completely different output [7, 8, 9], even though the changes in theaudio signal cannot be perceived by humans.
Revised: ASR systems to generate a completely different output [7, 8, 9], even though the changes in theaudio signal cannot be perceived by humans.

Original: If small artiﬁcial perturbations in the audio signal canaffect the performance of ASR systems so signiﬁcantly [10], natural perturbations in human speechmay also have an adverse effect.
Revised: If small artiﬁcial perturbations in the audio signal canaffect the performance of ASR systems so signiﬁcantly [10], natural perturbations in human speechmay also have an adverse effect.

Original: Naturally perturbed speech can arise due to the psychological andphysical state of the speaker.
Revised: Naturally perturbed speech can arise due to the psychological andphysical state of the speaker.

Original: For instance, expressive speech such as excited, frustrated etc., andspeech generated with different voice qualities such as creaky, breathy, etc.
Revised: For instance, expressive speech such as excited, frustrated etc., andspeech generated with different voice qualities such as creaky, breathy, etc.

Original: In this paper, we show that the performance of the state-of-the-art deep neural network based ASRsystems can signiﬁcantly degrade for speech colored by either emotion or voice-quality.
Revised: In this paper, we show that the performance of the state-of-the-art deep neural network based ASRsystems can signiﬁcantly degrade for speech colored by either emotion or voice-quality.

Original: We showthat these natural perturbations can be modeled by using Cycle-consistent GANs (CycleGANs)[11, 12, 13], a variant of Generative Adversarial Networks (GANs) [14] which can learn distributionsof data across different domains without a parallel corpus.
Revised: We showthat these natural perturbations can be modeled by using Cycle-consistent GANs (CycleGANs)[11, 12, 13], a variant of Generative Adversarial Networks (GANs) [14] which can learn distributionsof data across different domains without a parallel corpus.

Original: Generator from our CycleGAN modelhas the ability to ﬁlter out the natural perturbations in speech and hence can be used as a front-endprocessor to improve the robustness of ASR systems.
Revised: Generator from our CycleGAN modelhas the ability to ﬁlter out the natural perturbations in speech and hence can be used as a front-endprocessor to improve the robustness of ASR systems.

Original: This front-end processing does not affect the
Revised: This front-end processing does not affect the

Original: ASR performance in absence of the perturbations.
Revised: ASR performance in absence of the perturbations.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • Analyze the performance of state-of-the-art ASR systems tested with naturally perturbedspeech, including laughter and creaky speech.• CycleGAN based front-end to convert perturbed speech into normal speech.• Analyze the CycleGAN front-end transformation and its effectiveness in ASR performance.
Revised: • Analyze the performance of state-of-the-art ASR systems tested with naturally perturbedspeech, including laughter and creaky speech.• CycleGAN based front-end to convert perturbed speech into normal speech.• Analyze the CycleGAN front-end transformation and its effectiveness in ASR performance.

Original:  Related Work
Revised:  Related Work

Original: Very few studies have analyzed the effect of emotional coloring of speech on ASR performance[15, 16, 17, 18].
Revised: Very few studies have analyzed the effect of emotional coloring of speech on ASR performance[15, 16, 17, 18].

Original: These studies have shown signiﬁcant degradation in the performance of GMMHMM-based ASR systems, when tested with emotive speech.
Revised: These studies have shown signiﬁcant degradation in the performance of GMMHMM-based ASR systems, when tested with emotive speech.

Original: Moreover, most of these approachesare based on modifying the acoustic and language models of the ASR system to handle the variationsexhibited by emotive speech.
Revised: Moreover, most of these approachesare based on modifying the acoustic and language models of the ASR system to handle the variationsexhibited by emotive speech.

Original: Recently, emotive speech utterances were converted to the neutralspeech utterances by modeling the prosody-based features [19].
Revised: Recently, emotive speech utterances were converted to the neutralspeech utterances by modeling the prosody-based features [19].

Original: But these approaches require paralleldata corpus (i.e., same utterance spoken in neutral and with emotion), which is very difﬁcult to collectfor spontaneous speech.
Revised: But these approaches require paralleldata corpus (i.e., same utterance spoken in neutral and with emotion), which is very difﬁcult to collectfor spontaneous speech.

Original: Analysis of the effect of voice qualities on ASR performance is not much explored.
Revised: Analysis of the effect of voice qualities on ASR performance is not much explored.

Original: Only a fewstudies have considered the detection of creaky voice in spontaneous speech [20, 21].
Revised: Only a fewstudies have considered the detection of creaky voice in spontaneous speech [20, 21].

Original: Further, GMMHMM-based systems were considered for synthesizing creaky speech [22, 23], but no previous workshave considered the conversion of creaky to neutral speech.
Revised: Further, GMMHMM-based systems were considered for synthesizing creaky speech [22, 23], but no previous workshave considered the conversion of creaky to neutral speech.

Original: One of the main issue in the conversionof creaky to neutral speech is the lack of parallel speech corpus for creaky and neutral speech.
Revised: One of the main issue in the conversionof creaky to neutral speech is the lack of parallel speech corpus for creaky and neutral speech.

Original: As compared to these works, we propose a CycleGAN-based [11] approach to transform speechperturbed with emotions and voice quality to normal speech.
Revised: As compared to these works, we propose a CycleGAN-based [11] approach to transform speechperturbed with emotions and voice quality to normal speech.

Original: With CycleGANs, our approach doesnot require parallel utterances of perturbed and normal speech for training and provides a front-endprocessor which can add robustness to speech recognition.
Revised: With CycleGANs, our approach doesnot require parallel utterances of perturbed and normal speech for training and provides a front-endprocessor which can add robustness to speech recognition.

Original:  Cyclic-GANS
Revised:  Cyclic-GANS

Original: GANs consist of two different networks i.e., a generator G and a discriminator D [14].
Revised: GANs consist of two different networks i.e., a generator G and a discriminator D [14].

Original: Generator isused to generate the fake samples G ( z ) , that resemble a given data distribution X , by taking randomsample z from a prior distribution p z as input, and the discriminator is used to discriminate fakesamples from real samples in the data X .
Revised: Generator isused to generate the fake samples G ( z ) , that resemble a given data distribution X , by taking randomsample z from a prior distribution p z as input, and the discriminator is used to discriminate fakesamples from real samples in the data X .

Original: Both, generator and discriminator are trained adversariallysuch that the generator will learn to generate samples which resembles the original samples in thedata by taking the feedback from the discriminator, which is a binary classiﬁer discriminating thesamples generated by the generator from the original samples.
Revised: Both, generator and discriminator are trained adversariallysuch that the generator will learn to generate samples which resembles the original samples in thedata by taking the feedback from the discriminator, which is a binary classiﬁer discriminating thesamples generated by the generator from the original samples.

Original: GANs were initially proposed for the generation of images when provided with some arbitraryrandom noise as input, and thereafter have achieved impressive results in image generation [24, 25],image-to-image translation [26], style transfer [27] and text-to-image synthesis [28, 29].
Revised: GANs were initially proposed for the generation of images when provided with some arbitraryrandom noise as input, and thereafter have achieved impressive results in image generation [24, 25],image-to-image translation [26], style transfer [27] and text-to-image synthesis [28, 29].

Original: Morerecently, unpaired image-to-image translation have been successfully performed using a variant of
Revised: Morerecently, unpaired image-to-image translation have been successfully performed using a variant of

Original: GAN, called CycleGANs [11, 12, 13] with an identity-mapping loss [30].
Revised: GAN, called CycleGANs [11, 12, 13] with an identity-mapping loss [30].

Original: We adopt the conceptof CycleGAN for performing the task of non-parallel speech-to-speech translation.
Revised: We adopt the conceptof CycleGAN for performing the task of non-parallel speech-to-speech translation.

Original: More recently,unpaired image-to-image translation was successfully learned by adopting a variant of GAN, calledcycle-consistent adversarial networks [11, 12, 13] with an identity-mapping loss [30].
Revised: More recently,unpaired image-to-image translation was successfully learned by adopting a variant of GAN, calledcycle-consistent adversarial networks [11, 12, 13] with an identity-mapping loss [30].

Original: We adoptthe concept of CycleGAN for performing the task of non-parallel speech-to-speech translation foremotion conversion.
Revised: We adoptthe concept of CycleGAN for performing the task of non-parallel speech-to-speech translation foremotion conversion.

Original: CycleGAN was earlier used for parallel-data-free speaker voice conversion [31]and speech enhancement [32].
Revised: CycleGAN was earlier used for parallel-data-free speaker voice conversion [31]and speech enhancement [32].

Original: To the best of our knowledge there is no prior work on convertingspeech perturbed with emotion or a voice quality to normal speech, by considering non-parallel data.
Revised: To the best of our knowledge there is no prior work on convertingspeech perturbed with emotion or a voice quality to normal speech, by considering non-parallel data.

Original: We are the ﬁrst to use CycleGAN for perturbed to normal speech conversion.
Revised: We are the ﬁrst to use CycleGAN for perturbed to normal speech conversion.

Original: 3.1 CycleGAN-based Speech-to-speech Conversion
Revised: 3.1 CycleGAN-based Speech-to-speech Conversion

Original: Our CycleGAN model architecture, considered in this work, is motivated from [31].
Revised: Our CycleGAN model architecture, considered in this work, is motivated from [31].

Original: The generatornetwork consists of one stride-1 convolution block, one stride-2 convolution block, 5 residual blocks[33], one 12 -stride convolution block, and one stride-1 convolution block.
Revised: The generatornetwork consists of one stride-1 convolution block, one stride-2 convolution block, 5 residual blocks[33], one 12 -stride convolution block, and one stride-1 convolution block.

Original: All convolution layersare 1 -dimensional to preserve the temporal structure [34].
Revised: All convolution layersare 1 -dimensional to preserve the temporal structure [34].

Original: Similar to [35], gated linear units, whichachieved state-of-the-art performance in language and speech modeling, are used as an activation
Revised: Similar to [35], gated linear units, whichachieved state-of-the-art performance in language and speech modeling, are used as an activation

Original: [Table]
Revised: [Table]

Original: function in the convolutional layers.
Revised: function in the convolutional layers.

Original: We also used the instance normalization, proposed for styletransfer in [27].
Revised: We also used the instance normalization, proposed for styletransfer in [27].

Original: For the discriminator network, we use a 6 × 6 patch GAN [36, 37], which classiﬁeswhether each 6 × 6 patch is real or fake.
Revised: For the discriminator network, we use a 6 × 6 patch GAN [36, 37], which classiﬁeswhether each 6 × 6 patch is real or fake.

Original: Training Details:
Revised: Training Details:

Original: In order to achieve more stable training and to generate higher quality outputs, weused the least square loss to compute the discriminator loss in place of the negative log likelihoodobjective [38, 11].
Revised: In order to achieve more stable training and to generate higher quality outputs, weused the least square loss to compute the discriminator loss in place of the negative log likelihoodobjective [38, 11].

Original: We also considered the identity-loss function [11], originally used for color preservation, which we found to be crucial for maintaining the linguistic information during conversion ofspeech.
Revised: We also considered the identity-loss function [11], originally used for color preservation, which we found to be crucial for maintaining the linguistic information during conversion ofspeech.

Original: We trained the CycleGAN models using the Adam optimizer [39] with a batch size of 1 .
Revised: We trained the CycleGAN models using the Adam optimizer [39] with a batch size of 1 .

Original: Theinitial learning rates of the generator and the discriminator are 0 . 0002 and 0 . 0001 , respectively.
Revised: Theinitial learning rates of the generator and the discriminator are 0 . 0002 and 0 . 0001 , respectively.

Original: Thelearning rates were decayed by a factor of 10 5 for each epoch.
Revised: Thelearning rates were decayed by a factor of 10 5 for each epoch.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: Analysis is performed by considering two spontaneous speech datasets, namely, AMI meetingcorpus [40] and Buckeye corpus of conversational speech [41].
Revised: Analysis is performed by considering two spontaneous speech datasets, namely, AMI meetingcorpus [40] and Buckeye corpus of conversational speech [41].

Original: Both datasets consists of dedicatedannotations along with time-stamps for speech perturbed with emotions and voice-quality.
Revised: Both datasets consists of dedicatedannotations along with time-stamps for speech perturbed with emotions and voice-quality.

Original: In thiswork, speech data collected from 40 female and 30 male speakers, in total from both datasets wasconsidered for training gender-dependent models.
Revised: In thiswork, speech data collected from 40 female and 30 male speakers, in total from both datasets wasconsidered for training gender-dependent models.

Original: For each gender and for each class (i.e., normal,laughter speech and creaky speech), 210 utterances ( 150 utterances for train and 60 utterances fortest) were considered.
Revised: For each gender and for each class (i.e., normal,laughter speech and creaky speech), 210 utterances ( 150 utterances for train and 60 utterances fortest) were considered.

Original: It is to be noted that all these utterances are non-parallel.
Revised: It is to be noted that all these utterances are non-parallel.

Revised: Each utterances is of1 - 2 sec in duration.

Original: The WORLD vocoder system [42] is used to extract features from the speech signal.
Revised: The WORLD vocoder system [42] is used to extract features from the speech signal.

Original: The speechsignals are sampled to 16 kHz, and then Mel ﬁlterbank  features, logarithmic fundamental frequency( log F 0 ) and aperiodic components (Aps) are extracted from the speech signal within a windowof length 20 msec for every 5 msec.
Revised: The speechsignals are sampled to 16 kHz, and then Mel ﬁlterbank (MFB) features, logarithmic fundamentalfrequency ( log F 0 ) and aperiodic components (APs) are extracted from the speech signal withina window of length 20 msec for every 5 msec.

Original: 24 -dimensional Mel ﬁlter bank features and 24 -dimensional Aps are modeled by the proposed CycleGAN architecture to convert the features extracted from theinput perturbed speech into features corresponding to normal speech.
Revised: 24 -dimensional MFBs and 24 -dimensional APsare modeled by the proposed CycleGAN architecture to convert the features extracted from theinput perturbed speech into features corresponding to normal speech.

Original: In previous works for speakerconversion [43, 31], only the spectral features (Mel ﬁlterbanks) were modeled.
Revised: In previous works for speakerconversion [43, 31], only the spectral features (MFBs) were modeled.

Original: But for perturbedspeech conversion, we found that modeling both, spectral features  and aperiodic components (Aps)resulted in better conversion to normal speech than considering only spectral features.
Revised: But for perturbed speechconversion, we found that modeling both, spectral features (MFBs) and aperiodic components (APs)resulted in better conversion to normal speech than considering only spectral features (MFBs).

Original: Logarithm Gaussian normalized transformation [44] was used to convert the F 0 values from the source to thetarget speech.
Revised: Logarithm Gaussian normalized transformation [44] was used to convert the F 0 values from thesource to the target speech.

Original: Table 1 shows the performance of Google and ASpIRE (based on Kaldi ASR Engine)
Revised: Table 1 shows the performance of Google and ASpIRE (based on Kaldi ASR Engine)

Original: [1, 45] ASRsystems with and without our proposed front-end system, when tested with laughter speech (speechperturbed with emotion) and creaky speech (speech perturbed with voice-quality).
Revised: [1, 45] ASRsystems with and without our proposed front-end system, when tested with laughter speech (speechperturbed with emotion) and creaky speech (speech perturbed with voice-quality).

Original: The performanceis evaluated in terms of % Word Error Rate (WER) and % Sentence Error Rate (SER).
Revised: The performanceis evaluated in terms of % Word Error Rate (WER) and % Sentence Error Rate (SER).

Original: Lower thesevalues, better is the performance.
Revised: Lower thesevalues, better is the performance.

Revised: It can be observed from Table 1 that modeling both, spectraland aperiodic components (i.e., (MFB + APs) performs better than modeling only MFBs with theproposed front-end.

Original: It can be observedfrom Table 1 that a reduction of 14 . 9 %, 21 . 0 %in WER and 16 .
Revised: It can be observed that a reduction of 14 . 9 %, 21 . 0 % in WER and 16 .

Original: 3 % , 3 . 4 % in SER, all in absolute values is achieved, when our proposed front-end isused to convert speech perturbed with laughter speech to normal speech for Google ASR and ASpIRE ASR, respectively.
Revised: 3 % , 3 . 4 %in SER, all in absolute values is achieved, when our proposed front-end is used to convert laughterspeech to normal speech for Google ASR and ASpIRE ASR, respectively.

Original: Similarly, an absolute reduction of 11 . 0 %, 7 . 9 % in WER and 22 .
Revised: Similarly, an absolutereduction of 11 . 0 %, 7 . 9 % in WER and 22 .

Original: 1 % in SER is obtained, when our proposed front-end is used 
Revised: 1 % in SER is obtained, when our proposedfront-end (MFBs+APs) is used to convert creaky speech to normal speech for Google ASR and ASpIRE ASR, respectively.  Analysis of the

Original: [Figure]
Revised: [Figure]

Original: Analysis of the Learned Front-end Transformation

Original: Figure 1 shows a 2-dimensional t-SNE projection [46] of the Mel ﬁlterbank output features for (a)normal speech, (b) laughter perturbed speech and (c) laughter perturbed speech transformed to normalspeech by the proposed front-end.
Revised: Figure 1 shows a 2-dimensional t-SNE projection [46] of the Mel ﬁlterbank output features for (a)normal speech, (b) laughter perturbed speech and (c) laughter perturbed speech transformed to normalspeech by the proposed front-end.

Original: It can be observed that the ﬁlterbank features for normal speechand normal transformed speech are quite similar to each other and that they differ signiﬁcantly fromthe ﬁlterbank features for laughter speech.
Revised: It can be observed that the ﬁlterbank features for normal speechand normal transformed speech are quite similar to each other and that they differ signiﬁcantly fromthe ﬁlterbank features for laughter speech.

Original: Additionally, the spread of the ﬁlterbank features forlaughter speech is reduced in the 2-dimensional t-SNE space.
Revised: Additionally, the spread of the ﬁlterbank features forlaughter speech is reduced in the 2-dimensional t-SNE space.

Original: We hypothesize that this may be due tothe reduction in vowel space for laughter speech [47].
Revised: We hypothesize that this may be due tothe reduction in vowel space for laughter speech [47].

Original: [Figure]
Revised: [Figure]

Original: For a more detailed analysis, Figure 2 shows a box plot for each of the 24 Mel ﬁlterbank outputfeatures, for (a) normal speech, (b) laughter perturbed speech and (c) laughter perturbed speechtransformed to normal speech.
Revised: For a more detailed analysis, Figure 2 shows a box plot for each of the 24 Mel ﬁlterbank outputfeatures, for (a) normal speech, (b) laughter perturbed speech and (c) laughter perturbed speechtransformed to normal speech.

Original: It can be observed that the feature values for normal speech and normaltransformed speech are very close and they exhibit similar variations.
Revised: It can be observed that the feature values for normal speech and normaltransformed speech are very close and they exhibit similar variations.

Original: Interestingly, for ﬁlters 1 to 8they differ signiﬁcantly from the laughter speech.
Revised: Interestingly, for ﬁlters 1 to 8they differ signiﬁcantly from the laughter speech.

Original: We hypothesize that our front-end transformationmainly operates in this region.
Revised: We hypothesize that our front-end transformationmainly operates in this region.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a novel front-end based on CycleGANs to transform naturally perturbed speech to normalspeech.
Revised: We proposed a novel front-end based on CycleGANs to transform naturally perturbed speech to normalspeech.

Original: Experiments on spontaneous laughter speech and creaky voice utterances show signiﬁcantimprovements in performance of the Google ASR and Kaldi ASR with ASpIRE model.
Revised: Experiments on spontaneous laughter speech and creaky voice utterances show signiﬁcantimprovements in performance of the Google ASR and Kaldi ASR with ASpIRE model.

Original: We foundthat adding aperiodic components to spectral features gives a better performance.
Revised: We foundthat adding aperiodic components to spectral features gives a better performance.

Original: Visualization of thelaughter speech and the converted speech features gives insights on the transformation performed bythe proposed front-end.
Revised: Visualization of thelaughter speech and the converted speech features gives insights on the transformation performed bythe proposed front-end.