D4AM: A G ENERAL D ENOISING F RAMEWORK FOR D OWNSTREAM A COUSTIC M ODELS
A BSTRACT
The performance of acoustic models degrades notably in noisy environments.
Speech enhancement (SE) can be used as a front-end strategy to serve automatic speech recognition (ASR) systems.
However, existing training objectives of SE methods are not fully effective at integrating speech-text and noise-clean paired data for training toward unseen ASRs.
In this study, we propose a general denoising framework for various downstream acoustic models, called D4AM.
Our framework fine-tunes the SE model with the backward gradient according to a specific acoustic model and the corresponding classification objective.
At the same time, our method aims to take the regression objective as an auxiliary loss to make the SE model generalize to other unseen acoustic models.
To jointly train an SE unit with regression and classification objectives, D4AM uses an adjustment scheme to directly estimate suitable weighting coefficients instead of going through a grid search process with additional training costs.
The adjustment scheme consists of two parts: gradient calibration and regression objective weighting.
Experimental results show that D4AM can consistently and effectively provide improvements to various unseen acoustic models and outperforms other combination setups.
Specifically, when evaluated on the Google ASR API with real noisy data completely unseen during SE training,
D4AM achieves a relative WER reduction of 24.65% compared to directly feeding noisy speech input.
To the best of our knowledge, this is the first work that deploys an effective combination scheme of regression (denoising) and classification (ASR) objectives to derive a general pre-processor applicable to various unseen ASR systems.
Our code is available at https://github.com/ChangLee0903/D4AM.
I NTRODUCTION
Speech enhancement (SE) aims to extract speech components from distorted speech signals to obtain enhanced signals with better properties (Loizou, 2013).
Recently, various deep learning models (Wang et al., 2020; Lu et al., 2013; Xu et al., 2015; Zheng et al., 2021; Nikzad et al., 2020) have been used to formulate mapping functions for SE, which treat SE as a regression task trained with noisy-clean paired speech data.
Typically, the objective function is formulated using a signal-level distance measure (e.g., L1 norm (Pandey & Wang, 2018;
Yue et al., 2022), L2 norm (Ephraim & Malah, 1984;
Yin et al., 2020; Xu et al., 2020), SI-SDR (Le Roux et al., 2019; Wisdom et al., 2020; Lee et al., 2020), or multiple-resolution loss (Défossez et al., 2020)).
In speech-related applications, SE units are often used as key pre-processors to improve the performance of the main task in noisy environments.
To facilitate better performance on the main task, some studies focus on deriving suitable objective functions for SE training.
For human-human oral communication tasks, SE aims to improve speech quality and intelligibility, and enhancement performance is usually assessed by subjective listening tests.
Because largescale listening tests are generally prohibitive, objective evaluation metrics have been developed to objectively assess human perception of a given speech signal (Rix et al., 2001; Taal et al., 2010;
Jensen & Taal, 2016; Reddy et al., 2021).
Perceptual evaluation of speech quality (PESQ) (Rix et al., 2001) and short-time objective intelligibility (STOI) (Taal et al., 2010; Jensen & Taal, 2016) are popular objective metrics designed to measure speech quality and intelligibility, respectively.
Recently, DNSMOS (Reddy et al., 2021) has been developed as a non-instructive assessment tool that predicts human ratings (MOS scores) of speech signals.
In order to obtain speech signals with improved speech quality and intelligibility, many SE approaches attempt to formulate objective functions for SE training directly according to speech assessment metrics (Fu et al., 2018; Fu et al., 2019; 2021).
Another group of approaches, such as deep feature loss (Germain et al., 2019) and
HiFi-GAN (Su et al., 2020), propose to perform SE by mapping learned noisy latent features to clean ones.
Experimental results show that the deep feature loss can enable the enhanced speech signal to reach higher human perception scores compared to the conventional L1 and L2 distances.
Another prominent application of SE is to improve automatic speech recognition (ASR) in noise (Seltzer et al., 2013; Weninger et al., 2015b;
Li et al., 2014; Cui et al., 2021).
ASR systems perform sequential classification, mapping speech utterances to sequences of tokens.
Therefore, the predictions of ASR systems are highly dependent on the overall structure of the input utterance.
When it comes to noisy signals, ASR performance will degrade significantly because noise interference corrupts the content information of the structure.
Without modifying the ASR model, SE models can be trained separately and “universally” used as a pre-processor for ASR to improve recognition accuracy.
Several studies have investigated the effectiveness of SE’s model architecture and objective function in improving the performance of ASR in noise (Geiger et al., 2014a; Wang et al., 2020;
Zhang et al., 2020; Chao et al., 2021;
Meng et al., 2017; Weninger et al., 2015a;
Du et al., 2019;
Kinoshita et al., 2020; Meng et al., 2018).
The results show that some specific designs, including model architecture and input format, are beneficial to improving ASR performance.
However, it has also been reported that improved recognition accuracy in noise is not always guaranteed when the ASR objective is not considered in SE training (Geiger et al., 2014b).
To tune the SE model parameters toward the main ASR task, one possible approach is to prepare the data of (noisy) speechtext pairs and backpropagate gradients on the SE model according to the classification objective provided by the ASR model.
That is, SE models can be trained on a regression objective (using noisyclean paired speech data) or/and a classification objective (using speech-text paired data).
Ochiai et al.
(2017a;b) proposed a multichannel end-to-end (E2E) ASR framework, where a mask-based
MVDR (minimum variance distortionless response) neural beamformer is estimated based on the classification objective.
Experimental results on CHiME-4 (Jon et al., 2017) confirm that under noisy conditions, the estimated neural beamformer can achieve significant ASR improvements.
Meanwhile,
Chen et al.
and
Ma et al.
(2021) proposed to train SE units by considering both regression and classification objectives, and some work (Chen et al., 2015; Ochiai et al., 2017a;b) proposed to train SE models with E2E-ASR classification objectives.
A common way to combine regression and classification objectives is to use weighting coefficients to combine them into a joint objective for SE model training.
Despite promising results, the use of combined objectives in SE training has two limitations.
First, how to effectively combine regression and classification objectives still remains an issue.
A largescale grid search is often employed to determine optimal weights for regression and classification objectives, which requires exhaustive computational costs.
Second, ASR models are often provided by third parties and may not be accessible when training SE models.
Moreover, due to various training settings in the acoustic model, such as label encoding schemes (e.g., word-piece (Schuster & Nakajima, 2012), byte-pair-encoding (BPE) (Gage, 1994; Sennrich et al., 2016), and character), model architectures (e.g., RNN (Chiu et al., 2018;
Rao et al., 2017; He et al., 2019; Sainath et al., 2020), transformer (Vaswani et al., 2017; Zhang et al., 2020), and conformer (Gulati et al., 2020)), and objectives (e.g., Connectionist Temporal Classification (CTC) (Graves et al., 2006), Attention (NLL) (Chan et al., 2016), and their hybrid version (Watanabe et al., 2017)), SE units trained according to a specific acoustic model may not generalize well to other ASR systems.
Based on the above limitations, we raise the question: Can we effectively integrate speech-text and noisy-clean paired data to build a denoising pre-processor that generalizes well to unseen ASR systems?
In this work, we derive a novel denoising framework, called D4AM, to be used as a “universal” pre-processor to improve the performance of various downstream acoustic models in noise.
To achieve this goal, the proposed framework focuses on preserving the integrity of clean speech signals and trains SE models with regression and classification objectives jointly.
By using the regression objective as an auxiliary loss, we do not require additional training costs to grid search the appropriate weighting coefficients for the regression and classification objectives; instead, D4AM applies an adjustment scheme to determine the appropriate weighting coefficients automatically and efficiently.
The adjustment scheme is inspired by the following ideas: (1) we attempt to adjust the gradient yielded by a proxy ASR model so that the SE unit can be trained to improve the general recognition ability; (2) we treat the weighted regression objective as a regularizer, thus avoiding over-fitting when training the SE unit.
For (1), we derive a coefficient α gclb (abbreviation for gradient calibration) according to whether the classification gradient conflicts with the regression gradient.
When the inner product of the two gradient sets is negative, α gclb is the projection of the classification gradient on the regression gradient; otherwise, it is set to 0 .
For (2), we derive a coefficient α srpr (abbreviation for surrogate prior) based on an auxiliary learning method called ARML (auxiliary task reweighting for minimum-data learning)
(Shi et al., 2020) and formulate the parameter distribution induced by the weighted regression objective as the surrogate prior when training the SE model.
From the experimental results on two standard speech datasets, we first notice that by properly combining regression and classification objectives, D4AM can effectively improve the recognition accuracy of various unseen ASR systems and outperform the SE models trained only with the classification objective.
Next, by taking the regression objective as an auxiliary loss, D4AM can be efficiently trained to avoid over-fitting even with limited speech-text paired data.
Finally, compared to grid search, D4AM performs mostly better.
The main contribution of this study is two-fold: (1) to the best of our knowledge, this is the first work that derives a general denoising pre-processor applicable to various unseen ASR systems; (2) we deploy a rational coefficient adjustment scheme for the combination strategy and link it to the motivation for better generalization ability.
M OTIVATION AND M AIN I DEA
Noise Robustness Strategies for ASR.
Acoustic mismatch, often caused by noise interference, is a long-standing problem that limits the applicability of ASR systems.
To improve recognition performance, multi-condition training (MCT) (Parihar et al., 2004; Rajnoha, 2009; Zhang et al., 2020) is a popular approach when building practical ASR systems.
MCT takes noise-corrupted utterances as augmented data and jointly trains an acoustic model with clean and noisy utterances.
In addition to noise-corrupted utterances, enhanced utterances from SE models can also be included in the training data for MCT training.
Recently, Prasad et al.
(2021) studied advanced MCT training strategies and conducted a detailed investigation of different ASR fine-tuning techniques, including freezing partial layers (Zhang et al., 2020), multi-task learning (Saon et al., 2017; Prasad et al., 2021), adversarial learning (Serdyuk et al., 2016; Denisov et al., 2018; Zhang et al., 2020), and training involving enhanced utterances from various SE models (Braithwaite & Kleijn, 2019; Zhang et al., 2020; Défossez et al., 2020).
The results reveal that recognition performance in noise can be remarkably improved by tuning the parameters in the ASR system.
However, an ASR system is generally formed by a complex structure and estimated with a large amount of training data.
Also, for most users, ASR systems provided by third parties are black-box models whose acoustic model parameters cannot be fine-tuned to improve performance in specific noisy environments.
Therefore, tuning the parameters in an ASR system for each specific test condition is usually not a feasible solution for real-world scenarios.
In contrast, employing an SE unit to process noisy speech signals to generate enhanced signals, which are then sent to the ASR system, is a viable alternative to externally improve the recognition accuracy in noise as a two-stage strategy of noise robustness.
The main advantage of this alternative is that SE units can be pre-prepared for direct application to ASR systems without the need to modify these
ASR systems or to know the system setups.
Furthermore, ASR is usually trained with a large amount of speech-text pairs, requiring expensive annotation costs; on the contrary, for SE, the required large number of noisy-clean training pairs can be easily synthesized.
In this study, we propose to utilize the regression objective as a regularization term, combined with the classification objective, for SE unit training to reduce the need for speech-text data and improve the generalization ability.
Regression Objective as an Auxiliary Loss.
Since our proposed framework focuses on improving
ASR performance in noise, the classification objective is regarded as the main task, and the regression objective is used as an auxiliary loss.
Generally, an auxiliary loss aims to improve the performance of the main task without considering the performance of the auxiliary task itself.
The auxiliary task learning framework searches for adequate coefficients to incorporate auxiliary information and thus improve the performance of the main task.
Several studies (Hu et al., 2019; Chen et al., 2018; Du et al., 2018; Lin et al., 2019; Dery et al., 2021; Shi et al., 2020) have investigated and confirmed the benefits of auxiliary task learning.
In D4AM, given the SE model parameter θ , we take the classification objective L cls ( θ ) as the main task and the regression objective L reg ( θ ) as the auxiliary task.
Then, we aim to jointly fine-tune the SE model with L cls ( θ ) and L reg ( θ ) , where their corresponding training data are T cls (speech-text paired data) and T reg (noisy-clean paired speech data), respectively.
Here, D4AM uses − log p ( T cls | θ ) and − log p ( T reg | θ ) , respectively, as the classification objective L cls ( θ ) and the regression objective L reg ( θ ) .
Fig.
1a shows the overall training flow of D4AM.
Notably, D4AM is partially different from traditional auxiliary task learning algorithms, where the main and auxiliary tasks commonly share the same model parameters.
In
D4AM, the main task objective L cls ( θ ) cannot be obtained by the SE model alone.
Instead, a proxy acoustic model ϕ is employed.
That is, the true classification objective L cls ( θ ) is not reachable, and alternatively, with the proxy acoustic model ϕ , a surrogate term − log p ( T cls | θ, ϕ ) , termed L ϕcls ( θ ) , is used to update the SE parameters.
Since there is no guarantee that reducing L ϕcls ( θ ) will reduce
L cls ( θ ) , a constraint is needed to guide the gradient of L ϕcls ( θ ) toward improving general recognition performance.
Meanwhile, we believe that, for any particular downstream acoustic recognizer, the best enhanced output produced by the SE unit from any noisy speech input is the corresponding clean speech.
Therefore, any critical point of L cls ( θ ) should be “covered” by the critical point space of L reg ( θ ) , and we just need guidance from speech-text paired data to determine the critical point in the L reg ( θ ) space.
That is, when an update of SE model parameters θ decreases both L ϕcls ( θ ) and L reg ( θ ) , that update should be beneficial to general ASR systems.
On the other hand, compared to L cls ( θ ) , the calculation of L reg ( θ ) does not require the proxy model when we use it to update SE model parameters.
Accordingly, we derive the update of θ at the t -th step as:
[Equation]
As pointed out in Lopez-Paz & Ranzato (2017) and Chaudhry et al.
(2019), the model function is supposed to be locally linear around small optimization steps.
For ease of computation, we replace
L reg ( θ ) ≤ L reg ( θ t ) with ⟨∇L ϕcls ( θ t ) , ∇L reg ( θ t ) ⟩ ≥ 0 .
That is, we take the alignment between ∇L ϕcls ( θ t ) and ∇L reg ( θ t ) as the constraint to guarantee the generalization ability to unseen ASR systems at each training step.
When the constraint is violated, ∇L ϕcls ( θ t ) needs to be calibrated.
Following the calibration method proposed in Chaudhry et al.
(2019) to search for the closest non-violating candidate to the original ∇L ϕcls ( θ t ) , we formulate the calibrated gradient g ∗ as:
[Equation]
In addition to guiding SE unit training toward improving ASR performance, D4AM also aims to alleviate over-fitting in the classification task.
In ARML (Shi et al., 2020), auxiliary task learning is formulated as the problem of minimizing the divergence between the surrogate prior and the true prior.
It is shown that when a suitable surrogate prior is identified, the over-fitting issue caused by the data limitation of the main task can be effectively alleviated.
Following the concept of ARML,
D4AM aims to estimate an appropriate coefficient of L reg to approximate the log prior of the main task objective L cls to alleviate over-fitting.
Finally, D4AM combines the calibrated gradient g ∗ and auxiliary task learning (ARML) to find a critical point that ensures better generalization to unseen
ASR systems and alleviates over-fitting in the classification task.
In Sec. 3, we will illustrate the details of our proposed framework.
M ETHODOLOGY
D4AM aims to estimate an SE unit that can effectively improve the recognition performance of general downstream ASR systems.
As shown in Fig.
1a, the overall objective consists of two parts, the classification objective L ϕcls ( θ ) given the proxy model ϕ and the regression objective L reg ( θ ) .
As shown in Fig.
1b, we intend to guide the update of the model from θ i to θ o with the regression objective L reg ( θ ) to improve the performance of ASR in noise.
In this section, we present an adjustment scheme to determine a suitable weighting coefficient to combine L ϕcls ( θ ) and L reg ( θ ) instead of a grid search process.
The adjustment scheme consists of two parts.
First, we calibrate the direction of ∇L ϕcls ( θ ) to promote the recognition accuracy, as described in Eq. 2.
Second, we follow the ARML algorithm to derive a suitable prior by weighting the regression objective.
Finally, the gradient descent of D4AM at the t -th step for SE training is defined as:
[Equation]
[Figure]
where ϵ t is the learning rate, and η t ∼ N (0 , 2 ϵ t ) is a Gaussian noise used to formulate the sampling of θ .
The coefficient α gclb is responsible for calibrating the direction of gradient when L ϕcls ( θ t ) does not satisfy ⟨∇L ϕcls ( θ t ) , ∇L reg ( θ t ) ⟩ ≥ 0 .
Mathematically, α gclb is the result of projecting ∇L ϕcls ( θ t ) into the space formed by using ∇L reg ( θ t ) as a basis.
The connection between the projection process and the constraint ⟨∇L ϕcls ( θ t ) , ∇L reg ( θ t ) ⟩ ≥ 0 will be introduced in Section 3.1.
The coefficient α srpr is responsible for weighting L reg ( θ t ) to approximate the log prior of parameters and is estimated to minimize the divergence between the parameter distributions of the surrogate and true priors.
The determination of α srpr , the mechanism of the sampling of θ , and the overall framework proposed in
ARML will be described in Section 3.2.
Finally, in Section 3.3, we will introduce the implementation of the D4AM framework.
3. G RADIENT C ALIBRATION FOR I MPROVING G ENERAL R ECOGNITION A BILITY
This section details the gradient calibration mechanism.
First, we assume that any critical point of L cls ( θ ) is located in the critical point space of L reg ( θ ) .
When the derived g ∗ satisfies ⟨ g ∗ , ∇L reg ( θ t ) ⟩ ≥ 0 , the direction of g ∗ can potentially bring recognition improvements to general acoustic models.
The constrained optimization problem described in Eq.
2 can be solved in a closed form while considering only a mini-batch of samples.
We then design a criterion
C = ⟨∇L ϕcls ( θ t ) , ∇L reg ( θ t ) ⟩ to check the direction of the gradient and formulate the solution as:
[Equation]
, where α gclb = −
[Equation]
where (cid:74) .
(cid:75) is an indicator function whose output is 1 if the argument condition is satisfied; otherwise, the output is 0.
The derivation of α gclb is based on the Lagrangian of the constrained optimization problem.
The details of the derivation can be found in Appendix A.7.
3. R EGRESSION O BJECTIVE W EIGHTING AS THE S URROGATE P RIOR
In this section, we discuss the relation of classification and regression objectives to the main ASR task.
A weighted regression objective is derived to approximate the log prior for SE training by modifying the ARML (Shi et al., 2020) algorithm, which takes the weighted sum of auxiliary losses as a surrogate prior and proposes a framework for reweighting auxiliary losses based on minimizing the divergence between the parameter distributions of the surrogate and true priors.
We aim to find a coefficient α srpr such that α srpr = arg min α E θ ∼ p ∗ [log p ∗ ( θ ) − α · log( p ( T reg | θ ))] , where p ∗ is the true prior distribution of θ , and p α srpr ( T reg | θ ) is the surrogate prior corresponding to the weighted regression objective − α srpr · log( p ( T reg | θ )) .
However, p ∗ ( θ ) is not practically available.
To overcome the unavailability of p ∗ ( θ ) , ARML uses an alternative solution for p ∗ ( θ ) and a sampling scheme of θ in p ∗ ( θ ) .
For the alternative solution of p ∗ ( θ ) , ARML assumes that the main task distribution p ( T cls | θ ) will cover p ∗ ( θ ) since the samples of θ from p ∗ ( θ ) likely induce high values in p ( T cls | θ ) .
Thus, ARML takes log p ( T cls | θ ) as the alternative of log p ∗ ( θ ) .
For the sampling scheme of θ in p ∗ ( θ ) , due to convenience, ARML takes the derived parameters from the joint loss at each training step and combines them with Langevin dynamics (Neal et al., 2011; Welling & Teh, 2011) to approximate the sampling of θ with an injected noise η t ∼ N (0 , 2 ϵ t ) .
Thus, the optimization problem of α srpr becomes min α E θ t ∼ p J [log( p ( T cls | θ t )) − α · log( p ( T reg | θ t ))] , where p J represents the distribution for sampling θ from the above described joint training process with respect to the original sampling.
More details about the ARML algorithm can be found in Shi et al.
Finally, at the t -th step, the overall parameter change ∆ θ ( g ) of the auxiliary learning framework given an arbitrary gradient g of the main task (ASR) can be written as:
[Equation]
Yet, the aforementioned problem still remains: we can only obtain ∇L ϕcls ( θ t ) instead of the ideal ASR task gradient ∇L cls ( θ t ) .
Therefore, the gradient should be calibrated, as described in Section 3.1.
To this end, we accordingly update θ by taking ∆ θ ( g ∗ ) , as described in Eq. 3.
Algorithm 1 D4AM (A General Denoising Framework for Downstream Acoustic Models)
Learnable-Parameter: model parameter θ (initialized by θ 0 ); task weight α srpr (initialized by 1)
Hyper-Parameter: learning rate of the t -th iteration
, learning β 1: for iteration t = do 2: α gclb ← project ( ∇L ϕ cls ( θ t ) , ∇L reg ( θ t ))
3: θ t +1 ← θ t − ϵ
[Equation]
∇L ϕcls (
[Equation]
+ (
[Equation]
gclb +
[Equation]
srpr ) · ∇L reg (
[Equation]
+
[Equation]
4: g α ←
[Equation]
5: α srpr ← α srpr − β · clamp ( g α , max =
+1 , min = − 1) 6: end for
3. A LGORITHM
In this section, we detail the implementation of the proposed D4AM framework.
The complete algorithm is shown in Algorithm 1.
The initial value of α srpr was set to 1, and the learning rate β was set to 0.05.
We first update θ t by the mechanism presented in Eq.
3 and collect θ samples at each iteration.
As described in Section 3.2, α srpr is optimized to minimize the divergence between p ∗ ( θ ) and p α srpr ( T reg | θ ) .
Referring to ARML (Shi et al., 2020), we use the Fisher divergence to measure the difference between the surrogate prior and the true prior.
That is, we optimize α by min
α E θ t ∼ p J
[ ∥∇L cls ( θ t ) − α · ∇L reg ( θ t ))
∥ 22 ] .
Because we can only obtain L ϕcls ( θ t ) , we use the calibrated version g ∗ as the alternative of ∇L cls ( θ t ) .
Then, the optimization of α becomes min α E θ t ∼ p J [ ∥∇L ϕcls ( θ t )
+ ( α gclb − α ) · ∇L reg ( θ t ))
∥ 22 ] .
In addition, we use gradient descent to approximate the above minimization problem, since it is not feasible to collect all θ samples.
Practically, we only have access to one sample of θ per step, which may cause instability when updating α srpr .
Therefore, to increase the batch size of θ , we modify the update scheme by accumulating the gradients of α srpr .
The update period of α srpr was set to 16 steps.
Unlike ARML (Shi et al., 2020), we did not restrict the value of α srpr to the range 0 to 1, because L ϕ cls and L reg have different scales.
In addition, we clamp the gradient of α srpr with the interval [ − 1 , 1] while updating α srpr to increase training stability.
Further implementation details are provided in Section 4 and Supplementary Material.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
The training datasets used in this study include: noise signals from DNS-Challenge (Reddy et al., 2020) and speech utterances from Librispeech (Panayotov et al., 2015).
DNS-Challenge, hereinafter
DNS, is a public dataset that provides 65,303 background and foreground noise signal samples.
Librispeech is a well-known speech corpus that includes three training subsets: Libri-100 (28, utterances), Libri-360 (104,014 utterances), and Libri-500 (148,688 utterances).
The mixing process (of noise signals and clean utterances) to generate noisy-clean paired speech data was conducted during training, and the signal-to-noise ratio (SNR) level was uniformly sampled from -4dB to 6dB.
The SE unit was built on DEMUCS (Défossez et al., 2019; 2020), which employs a skip-connected encoder-decoder architecture with a multi-resolution STFT objective function.
DEMUCS has been shown to yield state-of-the-art results on music separation and SE tasks, We divided the whole training process into two stages: pre-training and fine-tuning.
For pre-training, we only used L reg to train the SE unit.
At this stage, we selected Libri-360 and Libri-500 as the clean speech corpus.
The SE unit was trained for 500,000 steps using the Adam optimizer with β 1 = 0.9 and β 2 = 0.999, learning rate 0.0002, gradient clipping value 1, and batch size 8.
For fine-tuning, we used both
L reg and L ϕcls to re-train the SE model initialized by the checkpoint selected from the pre-training stage.
We used Libri-100 to prepare the speech-text paired data for L ϕcls and all the training subsets (including Libri-100, Libri-360, and Libri-500) to prepare the noisy-clean paired speech data for L reg .
The SE unit was trained for 100,000 steps with the Adam optimizer with β 1 = 0.9 and β 2 = 0.999, learning rate 0.0001, gradient clipping value 1, and batch size 16.
For the proxy acoustic model ϕ , we adopted the conformer architecture and used the training recipe provided in the SpeechBrain toolkit (Ravanelli et al., 2021).
The model was trained with all training utterances in Librispeech, and the training objective was a hybrid version (Watanabe et al., 2017) of CTC (Graves et al., 2006) and
NLL (Chan et al., 2016).
Since the main motivation of our study is to appropriately adjust the weights of the training objectives to better collaborate with the noisy-clean/speech-text paired data, we focus on comparing different objective combination schemes on the same SE unit.
To investigate the effectiveness of each component in D4AM, our comparison includes: (1) without any SE processing (termed NOIS ), (2) solely trained with L reg in the pre-training stage (termed INIT ), (3) solely trained with L ϕcls in the fine-tuning stage (termed CLSO ), (4) jointly trained with L ϕcls and L reg by solely taking α gclb in the fine-tuning stage (termed GCLB ), and (5) jointly trained with L ϕcls and L reg by solely taking α srpr in the fine-tuning stage (termed SRPR ).
Note that some of the above schemes can be viewed as partial comparisons with previous attempts.
For example, CLSO is trained only with the classification objective, the same as (Ochiai et al., 2017a;b); SRPR , GCLB , and D4AM use joint training, the same as (Chen et al., 2015; Ma et al., 2021), but (Chen et al., 2015; Ma et al., 2021) use predefined weighting coefficients.
In our ablation study, we will show that our adjustment achieves optimal results without additional training costs compared to grid search.
Since our main goal is to prepare an SE unit as a pre-processor to improve the performance of ASR in noise, the word error rate (WER) was used as the main evaluation metric.
Lower WER values indicate better performance.
To fairly evaluate the proposed D4AM framework, we used CHiME-4 (Jon et al., 2017) and Aurora4 (Parihar et al., 2004) as test datasets, which have been widely used for robust ASR evaluation.
For
CHiME-4, we evaluated the performance on the development and test sets of the 1 -st track.
For
Aurora-4, we evaluated the performance on the noisy parts of the development and test sets.
There are two types of recording process, namely wv1 and wv2.
The results of these two categories are reported separately for comparison.
For the downstream ASR systems, we normalized all predicted tokens by removing all punctuation, mapping numbers to words, and converting all characters to their upper versions.
To further analyze our results, we categorized the ASR evaluation into out-of-domain and in-domain scenarios.
In the out-of-domain scenario, we aim to verify the generalization ability of ASR when the training and testing data come from different corpora.
That is, in this scenario, the testing data is unseen to all downstream ASR systems.
In the in-domain scenario, we aim to investigate the improvements achievable in standard benchmark settings.
Therefore, we set up training and testing data for ASR from the same corpus.
Note that ASR systems typically achieve better performance in the in-domain scenarios than in the out-domain scenario (Likhomanenko et al., 2021).
4. E XPERIMENTAL R ESULTS
D4AM Evaluated with Various Unseen Recognizers in the Out-of-Domain Scenario.
We first investigated the D4AM performance with various unseen recognizers.
We prepared these recognizers by the SpeechBrain toolkit with all the training data in Librispeech.
The recognizers include: (1) a conformer architecture trained with the hybrid loss and encoded by BPE-5000 (termed CONF ), (2) a transformer architecture trained with the hybrid loss and encoded by BPE-5000 (termed TRAN ), (3) an RNN architecture trained with the hybrid loss and encoded by BPE-5000 (termed RNN ), and (4) a self-supervised wav2vec
(Baevski et al., 2020) framework trained with the CTC loss and encoded by characters (termed W2V2 ).
To avoid potential biases caused by language models, CONF ,
TRAN , and RNN only decoded results from their attention decoders, while W2V2 greedily decoded
[Table] [Table]
results following the CTC rule.
Table 1 shows the WER results under different settings.
Note that we cascaded CONF to train our SE model, so CONF is not a completely unseen recognizer in our evaluation.
Therefore, we marked the results of CONF in gray and paid more attention to the results of other acoustic models.
Obviously, NOIS yielded the worst results under all recognition and test conditions.
INIT performed worse than others because it did not consider the ASR objective in the pre-training stage.
CLSO was worse than GCLB , SRPR , and D4AM , verifying that training with
L ϕcls alone does not generalize well to other unseen recognizers.
Finally, D4AM achieved the best performance on average, confirming the effectiveness of gradient calibration and using regression objective weighting as a surrogate prior in improving general recognition ability.
D4AM Evaluated with a Standard ASR System in the In-Domain Scenario.
In this experiment, we chose the official baseline ASR system from CHiME-4, a DNN-HMM ASR system trained in an MCT manner using the CHiME-4 task-specific training data, as the evaluation recognizer.
This baseline ASR system is the most widely used to evaluate the performance of SE units.
As seen in
Table 2, the result of INIT is comparable to the state-of-the-art results of CHiME-4.
Meanwhile,
D4AM achieved notable reductions in WER compared to NOIS .
This trend is similar to that observed in Table 2 and again validates the effectiveness of D4AM.
D4AM Evaluated with a Practical ASR API.
We further evaluated D4AM using the Google
ASR API, a well-known black-box system for users.
To prepare a “real” noisy testing dataset, we merged the real parts of the CHiME-4 development and test sets.
Table 3 shows the average WER values obtained by the Google ASR API.
From the table, we observe similar trends to those observed in Table 1, but two points are particularly noteworthy.
First, NOIS still yielded the worst result, confirming that all SE units contribute to the recognition accuracy of the Google ASR API on our testing data.
Second, CLSO was worse than INIT , confirming that an SE unit trained only by the
[Table]
classification objective according to one ASR model is not guaranteed to contribute to other unseen recognizers.
In contrast, improved recognition results (confirmed by the improvements produced by
SRPR , GCLB , and D4AM ) can be obtained by well considering the combination of objectives.
4. A BLATION S TUDIES
Regression Objective Weighting for Handling Over-fitting.
As mentioned earlier, ARML addresses the over-fitting issue caused by the limited labeled data of the main task by introducing an appropriate auxiliary loss.
For D4AM, we treat the regression objective as an auxiliary loss to alleviate the over-fitting issue.
To verify this feature, we reduced the speech-text paired data to only 10% (2853 utterances) of the Libri-100 dataset.
The WER learning curves of CLSO and D4AM on a privately mixed validation set within 100,000 training steps are shown in Fig.
2a.
From the figure, we can see that the learning curve of CLSO rises after 50,000 training steps, which is a clear sign of over-fitting.
In contrast, D4AM has no such trend.
Fig.
2a confirms the effectiveness of using the regression objective to improve generalization and mitigate over-fitting.
[Figure]
D4AM versus Grid Search.
One important feature of D4AM is that it automatically and effectively determines the weighting coefficients to combine regression and classification objectives.
In this section, we intend to compare this feature with the common grid search.
Fig.
2b compares the performance of D4AM , SRPR , and grid search with different weights on the CHiME-4 dt05-real set.
For the grid search method, the figure shows the results for different weights, including 0, 0.1, 1.0, 10, 20, 30, 40, 50, and 60.
The average performance of the best seven results is represented by the dashed green line.
From Fig.
2b, we first note that D4AM clearly outperforms SRPR .
Furthermore, the result of D4AM is better than most average results of grid search.
The results confirm that D4AM can efficiently and effectively estimate coefficients without an expensive grid search process.
To further investigate its effectiveness, we tested D4AM on high SNR testing conditions, MCTtrained ASR models, and another acoustic model setup.
Additionally, we tested human perception assessments using the D4AM-enhanced speech signals.
All the experimental results confirm the effectiveness of D4AM.
Due to space limitations, these results are reported in APPENDIX.
C ONCLUSION
In this paper, we have proposed a denoising framework called D4AM, which aims to serve as a pre-processor for various unseen ASR systems to boost their performance in noisy environments.
D4AM employs gradient calibration and regression objective weighting to guide SE training to improve ASR performance while avoiding over-fitting.
Experimental results first verify that D4AM can effectively reduce WERs for various unseen recognizers, even a practical ASR application (Google API).
Meanwhile, D4AM can alleviate the over-fitting issue by effectively and efficiently combining classification and regression objectives without the exhaustive coefficient search.
The promising results confirm that separately training an SE unit to serve existing ASR applications may be a feasible solution to improve the performance robustness of ASR in noise.
REPRODUCIBILITY STATEMENT
The source code for additional details of the model architecture, training process, and pre-processing steps, as well as a demo page, are provided in the supplemental material.
Additional experiments are provided in Appendix.
ACKNOWLEDGEMENT
This work was partly supported by the National Science and Technology Council, Taiwan (NSTC 111-2634-F-002-023).
A A PPENDIX A. D4AM E VALUATED ON H IGH SNR D ATA
Most SE research focuses on “average” denoising on the target ASR task, which means that if the distortion caused by the SE model is low enough, we consider it tolerable.
Accordingly, we conducted additional experiments using cleaner speech utterances as testing data.
The WERs of the Google
ASR API are shown in Table where test-clean represents the clean LibriSpeech test set and the others are test-clean mixed with noisex92 (Varga & Steeneken, 1993) and Nonspeech (Hu & Wang, 2010) noise sources, uniformly sampled at high SNR levels between 10dB to 20dB (average 15 dB).
From Table it can be seen that the WER of the clean speech processed by D4AM is only slightly higher than that of the original clean speech (ORIG).
The result is 16.61% versus 16.27%.
However, for both types of high SNR noisy speech, using D4AM reduced the WER (19.24% versus 19.67% for noisex92 (Varga & Steeneken, 1993) and 20.19% versus 20.55% for Nonspeech (Hu & Wang, 2010)).
The results show that D4AM can yield comparable (better) performance than using the original clean (high SNR noisy) speech.
[Table]
A. D4AM E VALUATED WITH MCT- TRAINED ASR MODELS
The results of the Google ASR API in Table 3 are supposed to be representative results of the MCTtrained ASR systems.
We conducted additional ASR experiments on CHiME-4 with MCT-CONF ,
MCT-W2V2 , and MCT-TRAN corresponding to the ASR model architectures in Table 1 and trained on the LibriSpeech and DNS noise datasets with the SNR levels -4 to 6.
Note that W2V2 consists of a wav2vec2 module and a linear layer.
During training, the wav2vec2 module was fixed and used as a feature extractor, and only the linear layer was updated.
It is clear from Table 5 that D4AM consistently outperforms NOIS for all MCT-trained ASR models.
[Table]
A. D4AM T RAINED WITH A NOTHER A COUSTIC M ODEL
We chose the conformer-based ASR model as the proxy acoustic model because it has the lowest memory requirement compared with other model architectures in SpeechBrain.
In terms of implementation, especially for researchers like us with limited computing resources, model complexity is a major concern.
We believe our claim that any critical point of L cls ( θ ) should be “covered” by the critical point space of L reg ( θ ) applies to most cases.
By taking this appropriately, we can reduce the bias of different settings of the proxy acoustic model used for training.
Also, previous work (Ochiai et al., 2017a;b) has shown empirically that training an SE unit with one acoustic model can improve another ASR system.
To show that the proposed D4AM is insensitive to the choice of proxy acoustic model, we conducted additional experiments using the W2V2-CTC model as the proxy acoustic model.
The results evaluated with a LibriSpeech-trained ASR system on the CHiMEdt-05 real set are shown in Table 6, where Proxy-CONF and Proxy-W2V2-CTC denote conformerand W2V2-CTC-based proxy acoustic models, respectively.
As shown in Table 6, D4AM can effectively reduce the bias between proxy acoustic models and notably improve the performance.
Since Conformer and W2V2-CTC have different model architectures and use different label types, the results confirm that D4AM can consistently produce promising results using different types of proxy acoustic models.
A. D4AM E VALUATED WITH THE L3DAS22 MICRO -A SETS
The input of L3DAS
Guizzo et al.
(2022) is a multi-channel signal.
In our experiments, we focus on the single-channel SE task and treat different channels as separate signals for training and testing.
Also, please note that the WER definition of L3DAS22 is different from the regular version.
In
L3DAS22, the reference target for computing WER is NOT a true transcription.
The WER is the word-level distance between the ASR model’s predicted transcripts for the clean and enhanced signals.
[Table]
In addition, the WER value is calculated on a sample basis.
In contrast, the conventional WER value is calculated using ground truth transcripts as targets, and is derived by summing the errors across all testing samples.
By training our SE model with their pre-mixed training data
[Table]
we have further executed our algorithm on the set with the following results:
[Table]
A. H UMAN P ERCEPTION E VALUATION
Finally, we evaluate the enhanced speech signal using an objective perceptual metric, DNSMOS (Reddy et al., 2021).
The DNSMOS value represents the perception score of human hearing, with higher scores indicating better quality.
Table 8 shows the DNSMOS scores of enhanced speech produced by various SE methods.
The real sets of CHiME-4 described in Section 4.2 were used as the testing data.
From Table 8, we note that D4AM produces the highest DNSMOS score among all models trained with the ASR objective, and its DNSMOS score is only slightly lower than that of
INIT (only considering the regression objective for training).
The result again proves our conjecture that any critical point of L cls ( θ ) should be “covered” by the critical point space of L reg ( θ ) .
[Table]
A. T HE CHANGE IN THE α srpr VALUE DURING TRAINING
Fig.
3 shows the change in the α srpr value with and without gradient calibration applied over training steps.
It is clear that the α srpr values of both SRPR and D4AM converge rapidly, indicating that their learning process is stable.
A. T HE DERIVATION OF α gclb
For the derivation of α gclb , the Lagrangian of the constrained optimization problem defined in Eq.can be written as:
[Equation]
To find the optimal value g ∗ = arg min g L ( g, α ) , we set the derivatives of L ( g, α ) with respect to g to 0.
By solving for ∇ g L ( α, g ) = 0 , we will get g =
∇L ϕcls ( θ t ) + α · ∇L reg ( θ t ) .
Then, we can substitute g =
∇L ϕcls ( θ t ) + α · ∇L reg ( θ t ) into L ( α, g ) and get the minimum L ∗ ( α ) :
[Equation]
(7) where const is the summation of those terms regardless of α .
Here, the solution of α gclb will be the critical point of L ∗ ( α ) .
Therefore, by deriving α such that ∇ α L ∗ ( α ) = 0 , we can get the final result of α gclb :
[Equation]