Original: E XPLOITING
Revised: E XPLOITING

Original: S PATIAL
Revised: S PATIAL

Original: S EPARABILITY
Revised: S EPARABILITY

Original: FOR
Revised: FOR

Original: D EEP
Revised: D EEP

Original: L EARNING M ULTICHANNEL S PEECH E NHANCEMENT WITH AN A LIGN - AND -F ILTER N ETWORK
Revised: L EARNING M ULTICHANNEL S PEECH E NHANCEMENT WITH AN A LIGN - AND -F ILTER N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multichannel speech enhancement (SE) systems separate the target speech from background noise by performing spatial and spectral filtering.
Revised: Multichannel speech enhancement (SE) systems separate the target speech from background noise by performing spatial and spectral filtering.

Original: The development of multichannel SE has a long history in the signal processing field, where one crucial step is to exploit spatial separability of sound sources by aligning the microphone signals in response to the target speech source prior to further filtering processes. 
Revised: The development of multichannel SE has a long history in the signal processing field, where one crucial step is to exploit spatial separability of sound sources by aligning the microphone signals in response to the target speech source prior to further filtering processes. This is similar to the human listening behavior of facing toward the speaker for better perception of the speech.

Original: However, most existing deep learning based multichannel SE works have yet to effectively incorporate or emphasize this spatial alignment aspect in the network design – we postulate that it is owing to the lack of suitable datasets with sufficient spatial diversity of the speech sources.
Revised: However, most existing deep learningbased multichannel SE works have yet to effectively incorporate or emphasize this spatial alignment aspect in the network design; some of them rely on integrating conventional model-based beamformer units to extract useful spatial features implicitly while others just let the network figure everything out by itself.

Revised: However, the beamformer operation could be computationally expensive and numerically unstable when trained with the network while without it the model lacks guidance on learning meaningful spatial features.

Original: In this paper, we highlight this important but often overlooked step in deep learning based multichannel SE, i.e., signal alignment, by introducing an Align-and-Filter network (AFnet) featuring a two-stage sequential masking design.
Revised: In this paper, we highlight this important but often overlooked step in deep learning-based multichannel SE, i.e., signal alignment, by introducing an Align-and-Filter network (AFnet) featuring a two-stage sequential masking design.

Original: The AFnet estimates two sets of masks, the alignment masks and filtering masks,and multiplies the estimated masks with the respective input signals to each stage sequentially, while leveraging the relative transfer functions (RTFs) for guiding the model to align signals with various speech source locations during training. For exploration purposes, we argue that the popular CHiME-3 multichannel dataset has its own limitation in representing spatially diverse speech data as the speakers were mostly located at the front side, and thereby adopt simulated and real-world measured room impulse responses to generate multichannel recordings where the target speech sources might come from arbitrary directions. Our findings suggest that for spatially diverse speaker scenarios, careful consideration of exploiting spatial characteristics is of great importance for deep learning based multichannel SE especially when the number of microphone gets increased.
Revised: The AFnet aims at estimating two sets of masks, the alignment masks and filtering masks, to carry out temporal alignment and spectral filtering processes.

Original: We show that utilizing the RTFs for signal alignment purposes in the two-stage, sequential masking framework consistently improves the capability of the network to separate the target speech from the noise signals, supporting that spatial separability is being effectively exploited by  the proposed model. Our studies advocate for  the advantages and significance of considering the signal alignment aspect, a wisdom coming from conventional signal processing, for developing future deep based multichannel SE algorithms to improve enhancement outcomes with positional diverse target speech scenarios.
Revised: During training, we propose to supervise the learning of alignment masks by predicting the relative transfer functions (RTFs) of various speech source locations followed by learning the filtering masks for signal reconstruction. During inference, the AFnet sequentially multiplies the estimated alignment and filtering masks with the microphone signals, performing the “align-then-filter” process similar to the human listening behavior. Due to the incorporation of RTF supervision, the AFnet explicitly learns interpretable spatial features without integrating traditional beamformer operations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Signal quality could significantly deteriorate in the presence of noise and interference.

Original: Information carried by the signal, e.g., speech, could be severely lost at the receiver side (the human ear or smart assistants) in a very noisy environment, causing difficulty of communication.

Original: Speech enhancement (SE) techniques aim to resolve such issues, which had been majorly developed based on modelbased, signal processing approaches that utilize mathematical modeling of signal statistics (Loizou, 2007).

Original: In the past decade, the advent of deep learning has demonstrated the potential of advancing the development of SE systems to a higher level, making SE a topic of great interest in the machine learning field (Wang & Chen, 2018; Haeb-Umbach et al., 2019).

Original: SE systems can be categorized into single-channel (single microphone) and multichannel (multiple microphones) schemes.
Revised: Speech enhancement (SE) systems can be categorized into single-channel (single microphone) and multichannel (multiple microphones) schemes.

Original: An important aspect of multichannel SE against single-channel SE is the exploitation of spatial separability , as known as spatial filtering or beamforming , enabled by the difference between the amplitudes and times of arrival of the received microphone signals due to the different acoustic paths the sound waveform travels to the microphones.
Revised: An important aspect of multichannel SE against single-channel SE is the exploitation of spatial separability , as known as spatial filtering or beamforming , enabled by the difference between the amplitudes and times of arrival of the received microphone signals due to the different acoustic paths the sound waveform travels to the microphones.

Original: In many signal processing beamforming methods, a key step is to align the microphone signals in response to the target signal source before any further filtering processes.
Revised: In many signal processing beamforming methods (Gannot et al., 2001; Cohen, 2004; Krueger et al., 2010; Koldovsk`y et al., 2015), a key step is to align the microphone signals in response to the target signal source before any further filtering processes.

Original: This step, by steering the array toward the location of the target signal, aims to compensate for the difference of the amplitudes and time delays (or correspondingly the magnitudes and phases in the frequency domain) of the microphone signals with respect to the target source.
Revised: This step, by steering the array toward the location of the target signal, aims to compensate for the difference of the amplitudes and time delays (or correspondingly the magnitudes and phases in the frequency domain) of the microphone signals with respect to the target source.

Original: Ideally, after the alignment step, each microphone should contain the same target speech component with no difference in amplitude and time delay (or magnitude and phase) which makes  the speech extraction  easier in the later filtering process.
Revised: Ideally, after the alignment step, each microphone should contain the same target speech component with no difference in amplitude and time delay (or magnitude and phase). For a linear array in the far-filed, anechoic setting, perfectly steering the microphone array makes it as if the target signal comes from the broadside, which renders the speech extraction task easier in the later filtering stage.

Original:  Thus, an efficient SE system can first align its microphone signals in response to the target speech, followed by spectral processing for fine-tuning and enhancement.This is similar to the human ear which can do both delay estimation and frequency analysis.
Revised: Such process is similar to the human listening behavior of facing toward the speaker for better perception of the speech. Thus, an efficient SE system can first align its microphone signals in response to the target speech, followed by spectral processing for fine-tuning and enhancement.

Original: Surprisingly, though well-known in signal processing approaches, there are only few deep learning SE systems designed mainly based on this observation.
Revised: Surprisingly, though well-known in signal processing approaches, there are only few deep learning SE systems designed mainly based on this observation.

Revised: Instead, several recent works (Wang et al., 2020; 2021) have been utilizing conventional beamformer units such as the the minimum-variance-distortionless-response (MVDR) beamformer (Capon, 1969) to extract spatial characteristics implicitly in deep learning methods.

Revised: However, matrix inversion or eigendecomposition are often required which in turn increase the computation burden and numerical instability.

Original: In this paper, we revisit this important but often overlooked alignment aspect from conventional signal processing algorithms  for  deep learningbased multichannel SE network design. The main contributions of this paper are summarized as follows:
Revised: In this paper, we revisit this important but often overlooked alignment aspect from conventional signal processing algorithms and recognize its importance for efficient deep learning multichannel SE network design that requires no intermediate beamformer units such as MVDR to extract meaningful spatial features.

Original: • We propose a deep learning framework, namely Align-and-Filter network (AFnet) (Figure 1), for studying the importance of spatial separability in deep learning based multichannel SE.
Revised: Specifically, we propose the Align-and-Filter network (AFnet) shown in Figure 1 for exploiting spatial separability within multichannel data with the following main contributions: • The AFnet features a two-stage sequential masking design,

Original: The framework features a two-stage design, i.e., Align Net and Filter Net, where two sets of masks, alignment and filtering masks, aresequentially estimated and multiplied withtheir respective input signals, while leveraging the relative transfer function (RTF) information for facilitating signal alignment during the learning stage. The incorporation of RTF supervision along with the sequential masking mechanism are shown to assist the network in effectively exploiting spatial characteristics of the target speech source for improved SE outcomes.
Revised: i e., Align Net and Filter Net, where two sets of masks, alignment and filtering masks, are estimated and multiplied with the microphone signals to perform the “align-then-filter” process mimicking the human listening behavior . • During the training stage, we propose to supervise the learning of the alignment masks by estimating the relative transfer functions (RTFs) (Gannot et al., 2001; Cohen, 2004) for speech sources coming from various locations, prior to learning the filtering masks for final enhancement.

Original: • The proposed AFnetframework is utilized to analyze the spatial diversity of speech sources in different datasets.
Revised: • During inference, the AFnet is able to first align the microphone signals with respect to a speech source coming from an unknown direction due to supervised learning of alignment masks.

Original: Based on our study, we point out that some existing dataset such as CHiME- (Barker et al., 2015) lacks sufficient positional variety of target speech (mostly from the front side) and thus has its own limitation of representing broader SE scenarios, which could in turn downgrade the importance of the signal alignment aspect. On the other hand, on more spatially diverse data that we have gathered, our results indicate that remarkable improvements can be achieved by exploiting spatial separability via RTF alignment and sequential masking, while simply stacking neural network models might not work well when the number of microphones gets increased. • Through our exploration on the spatial characteristics of multichannel data with the AFnet [Figure]
Revised: Subsequently, the model performs filtering on the roughly aligned signals to achieve denoising.

Original: we highlight the wisdom of conventional signal processing methods – the importance of signal alignment – and advocate for incorporating the concept into the development of deep learning based multichannel SE systems, especially when the target speech may locate in arbitrary positions. [Figure]
Revised: • It is demonstrated that the RTF supervision incorporated with the sequential masking mechanism is the key to effectively learn useful, interpretable spatial characteristics. On situations where the target speech may come from arbitrary positions, the “align-then-filter” mechanism consistently improves the SE performance by more efficiently exploiting spatial separability of sound sources

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Multichannel SE has been a well known topic in signal processing for decades.
Revised: Multichannel SE has been a well known topic in signal processing for decades.

Original: Aside from leveraging spectral characteristics, multichannel SE can exploit positional information to perform spatial filtering , or beamforming , that allows extracting the target speech from noise based on spatial separability.
Revised: Aside from leveraging spectral characteristics, multichannel SE can exploit positional information to perform spatial filtering , or beamforming , that allows extracting the target speech from noise based on spatial separability.

Original: Conventional beamforming approaches rely on the so-called “steering vector” which carries positional information about the target speech (Doclo et al., 2015; Trees, 2004), e.g., the minimumvariance-distortionless-response (MVDR) beamformer (Capon, 1969) and its variants (Frost, 1972; Griffiths & Jim, 1982).
Revised: Conventional beamforming approaches rely on the so-called “steering vector” which carries positional information about the target speech (Doclo et al., 2015; Trees, 2004), e.g., the MVDR beamformer (Capon, 1969) and its variants (Frost, 1972; Griffiths & Jim, 1982).

Original: It is an important step in beamforming that the microphone array is steered toward the target signal location prior to further filtering processes.
Revised: It is an important step in beamforming that the microphone array is steered toward the target signal location prior to further filtering processes.

Original: To this end, certain knowledge about the acoustic paths(either impulse responses or transfer functions) between the target signal and the microphones have to be known.
Revised: To this end, certain knowledge about the acoustic paths between the target signal and the microphones have to be known.

Original: Many signal processing based multichannel SE systems utilize the ratio of the acoustic transfer functions, i.e., the relative transfer function (RTF) , that represents the coupling between sensors in response to a desired source (Cohen, 2004;
Revised: Many signal processing-based multichannel SE systems utilize the ratio of the acoustic transfer functions, i.e., the relative transfer function (RTF) , that represents the coupling between sensors in response to a desired source (Gannot et al., 2001;

Original: Gannot et al., 2001; Krueger et al., 2010; Koldovsk`y et al., 2015) for improving the denoising process.
Revised: Cohen, 2004; Krueger et al., 2010; Koldovsk ` y et al., 2015) for improving the denoising process.

Original: In the past decade,supervised learning approaches, especially deep learning, have remarkably changed the way of developing SE systems.
Revised: In the past decade, deep learning approaches have remarkably changed the way of developing SE systems.

Original: Along with the success of deep neural networks (DNNs) on single-channel SE (Lu et al., 2013; Williamson et al., 2015; Pascual et al., 2017; Luo & Mesgarani, 2019; Kim et al., 2020; Zheng et al., 2021), several multichannel SE systems have also been proposed, e.g., Erdogan et al.
Revised: Along with the success of deep neural networks (DNNs) on single-channel SE (Lu et al., 2013; Williamson et al., 2015; Pascual et al., 2017; Luo & Mesgarani, 2019; Kim et al., 2020; Zheng et al., 2021; Hu et al., 2020), several multichannel SE systems have also been proposed, e.g., Erdogan et al.

Original: ; Variani et al.
Revised: ; Variani et al.

Original: ; Sainath et al.
Revised: ; Sainath et al.

Original: (2017); Bu et al. 
Revised: ; Wang & Wang (2018); Bu et al. Koyama & Raj (2019);

Original: Luo et al.
Revised: Luo et al.

Original: ; Tolooshams et al.
Revised: ; Tolooshams et al.

Revised: (2020); Koyama & Raj (2020)

Revised: ; Wang et al.

Original:  However, the utilization of such important RTF information is often overlooked in the model design of DNN-based multichannel SE.
Revised: ; Zhang et al. ; Kim et al. ; Li et al. However, the utilization of such important RTF information is often overlooked in the model design of DNN-based multichannel SE.

Original: To the best of our knowledge, the only work that incorporates RTFs for SE is by Wang & Wang (2018), in which, however, the RTF estimation is  used as an intermediate step to assist theconventional MVDR beamformer only.
Revised: Although several SE works have incorporated RTFs (Wang & Wang, 2018; Zhang et al., 2021) the RTF estimation is mostly used as an intermediate step to assist the MVDR beamformer only.

Original: We postulate that the overlook may be due to the lack of sufficient spatial variety of the speech sources inthe popular dataset such as CHiME-3, where the benefit of utilizing RTFs could be only marginal.
Revised: We postulate that the overlook may be due to the lack of sufficient spatial variety of the speech sources in popular datasets such as CHiME-3 Barker et al. (2015), where the benefit of utilizing RTFs could be only marginal.

Original: However, many practical situations can have the target speech coming from arbitrary directions and thus a deep dive into the RTF spatial alignment aspect is still of great importance.
Revised: However, many practical situations can have the target speech coming from arbitrary directions and thus a deep dive into the RTF spatial alignment aspect is still of great importance.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: We consider an acoustic scenario with one desired speech source and several interfering noise signals in a reverberant environment.
Revised: We consider an acoustic scenario with one desired speech source and several interfering noise signals in a reverberant environment.

Original: The SE system will be developed in the time-frequency domain using the short-time Fourier transform (STFT)
Revised: The SE system will be developed in the time-frequency domain using dConv( M , C 1 ) dConv( C 1 , C 2 ) dConv( C 2 , C 3 ) dConv( C 3 , C 4 ) dConv( C 4 , C 4 ) dConv(2 C 1 , C 1 ) dConv(2 C 2 , C 1 )

Revised: dConv(2 C 3 , C 2 )

Revised: dConv(2 C 4 , C 3 )

Original: (Parchami et al., 2016) where the time-domain waveforms are transformed to complex-valued STFT representations.
Revised: ( # .. dConv( M , C 1 ) dConv( C 1 , C 2 ) dConv( C 2 , C 3 ) dConv( C 3 , C 4 ) dConv( C 4 , C 4 ) dConv(2 C 1 , C 1 ) dConv(2 C 2 , C 1 )

Revised: dConv(2 C 3 , C 2 )

Original: Let f, t stand for the frequency index and time frame index (with a total of F frequency bins and T time frames), we consider an additive noise model where the i -th microphone noisy signal X i ∈ C F × T of an N -element local microphone array can be written in the STFT domain as (Doclo et al., 2015):
Revised: dConv(2 C 4 , C 3 ) * ! +, max pool 2x up-conv 2x conv 1x skip connection element-wise (complex) multiplication summation over channels RTF Alignment

Original: X i ( f, t ) =
Revised: Loss

Original: S i ( f, t )
Revised: Signal

Original: + V i ( f, t ) , ∀ i = 1 , . . .
Revised: ReconstructionLoss [Figure] the short-time Fourier transform (STFT) (Parchami et al., 2016) where the time-domain waveforms are transformed to complex-valued STFT representations.

Original: , N , with S i ( f, t )
Revised: Let f, t stand for the frequency index and time frame index (with a total of F frequency bins and T time frames), we consider an additive noise model where the i -th microphone noisy signal STFT X i ∈ C F × T of an N -element local microphone array can be written as (Doclo et al., 2015):

Original: =
Revised: [Equation]

Original: H i ( f, t ) S 0 ( f, t ) being the speech component received by microphone i ,where H i ( f, t ) is the  acoustic transfer function between microphone i and the location of the far-field speech source signal S 

Revised: where S i ( f, t ) is the speech component received by microphone i , H i ( f, t ) is the (potentially timevarying) acoustic transfer function between microphone i and the location of the far-field speech source signal S

Original: ( f, t ) , and V i ( f, t ) is the noise component captured by microphone i .
Revised: ( f, t ) , and V i ( f, t ) is the noise component captured by microphone i .

Original: Typically, the goal of multichannel SE is to recover the speech component S( f, t ) = S r ( f, t ) of a selected reference microphone r ∈ { 1 , . . .
Revised: Typically, the goal of multichannel SE is to recover the speech component S = S r ∈ C F × T of a selected reference microphone r ∈ { 1 , . . .

Original: , N } given the noisy microphone signals X 1( f, t ) , . . .
Revised: , N } given the noisy microphone signals X 1 , . . .

Original: , X N( f, t ) .
Revised: , X N .

Original: Multichannel SE systems typically perform the “filter-and-sum” operation, or more generally known as “beamforming” – linearly combining the signals of different microphone channels to extract the target signal from background noise.
Revised: Multichannel SE systems typically perform the “filter-and-sum” operation, or more generally known as “beamforming” – linearly combining the signals of different microphones to extract the target signal from background noise.

Original: In the STFT domain, the estimation process can be expressed as:ˆ S ( f, t ) =
Revised: In the STFT domain, the estimation process can be expressed as:

Original: (cid:80) Ni =
Revised: [Equation]

Original: 1 W i ( f, t ) X i ( f, t ) , where W i ( f, t ) is the corresponding  filter weight for microphone i and ˆ S( f, t ) is the enhanced signal.
Revised: where W i ∈ C F × T is the corresponding set of filter weights for microphone i , ˆ S is the enhanced signal, and ⊙ denotes element-wise complex multiplication.

Original: 3. T WO - STAGE A LIGN - AND -F ILTER FRAMEWORK WITH SEQUENTIAL MASKING
Revised: 3. T WO - STAGE A LIGN - AND -F ILTER FRAMEWORK WITH SEQUENTIAL MASKING

Original: Inspired by the alignment concept in signal processing based algorithms, we introduce a two-stage
Revised: Inspired by the alignment concept in signal processing-based algorithms described in Section 1 and

Original:  Align-and-Filter framework for deep learning multichannel SE in Figure 1, where two modules are utilized to carry out alignment and filtering purposes respectively.
Revised: Appendix A, we introduce a two-stage Align-and-Filter framework for deep learning multichannel SE in Figure 1, where two modules are utilized to carry out alignment and filtering purposes respectively.

Original: The first module (Align Net) aims at estimating a set alignment masks for steering the input microphone signals toward the target speech location. Subsequently, the second module (Filter Net) focuses on cleaning up the interference in the aligned signals by estimating another set of filtering masks and linearly combines all channel outputs into the enhanced signal.
Revised: The first module (Align Net) aims at estimating a set alignment masks for steering the input microphone signals toward the target speech location The second module (Filter Net) focuses on cleaning up the interference in the aligned signals by estimating another set of filtering masks and linearly combines all channel outputs.

Original: Overall, the enhancement process can be expressed as:ˆ S ( f, t ) =
Revised: Overall, the enhancement process can be expressed as:

Original: (cid:80) Ni =
Revised: [Equation]

Original: 1 F i ( f, t ) A i ( f, t ) X i ( f, t ) , where A i ∈ C F × T is the alignment mask and F i ∈ C F × T is the filtering mask of the i -th microphone. 
Revised: where A i ∈ C F × T is the alignment mask and F i ∈ C F × T is the filtering mask of the i -th microphone. This corresponds to having W i = F i ⊙ A

Revised: i for the equivalent beamforming filter weights applied on X i in (2).

Original: This sequential masking scheme at the same time introduces signal level skip connections on top of the two modules for improved optimization.
Revised: Such a sequential masking scheme also introduces signal level skip connections on top of the two modules for improved optimization.

Original: 3. RTF- AWARE TRAINING
Revised: 3. RTF- AWARE TRAINING

Original: The spatial relationship between the far-field target source and the microphones along with the acoustic environment characteristics determine the inter-channel level difference (ILD) and interchannel time difference (ITD) of the microphone signals, which are important cues for sound source localization.
Revised: The spatial relationship between the far-field target source and the microphones along with the acoustic environment characteristics determine the inter-channel level difference (ILD) and interchannel time difference (ITD) of the microphone signals, which are important cues for sound source localization.

Original: In the STFT domain, the ILD and ITD correspond to the magnitude and phase of the
Revised: In the STFT domain, the ILD and ITD correspond to the magnitude and phase of the

Original: RTF respectively, which is the ratio between the acoustic transfer functions of two microphones with regards to the sound source (Li et al., 2016).
Revised: RTF respectively, which is the ratio between the acoustic transfer functions of two microphones with regards to the sound source (Li et al., 2016).

Original: Formally, we can express the RTF ˜ H
Revised: Formally, we can express the RTFs ˜ H

Original: i ∈ C F × T as:
Revised: i ∈ C F × T as:

Original: [Equation]
Revised: [Equation]

Original: ∀ i = 1 , . . . , N , where H r ( f, t ) denotes the acoustic transfer function of the selected reference microphone r ∈ { 1 , . . .
Revised: ∀ f, t , where H r ( f, t ) denotes the acoustic transfer function of the selected reference microphone r ∈ { 1 , . . .

Original: , N } .
Revised: , N } .

Original: From (1) we can see that the RTFs can be inferred from the ratios of the clean microphone signals which are often available in supervised learning SE schemes, and thus can be collected without knowing or simulating the noise.
Revised: From (4) we can see that the RTFs can be inferred from the ratios of the clean microphone signals which are often available in supervised learning SE schemes, and thus can be collected without knowing or simulating the noise.

Original: Note that the RTF information is only needed for training and is not required during inference.
Revised: Note that the RTF information is only needed for training and is not required during inference.

Original: Element-wise multiplying the RTF ˜ H 
Revised: Element-wise multiplying the RTFs ˜ H i with input X

Original: i ( f, t ) with the microphone signal X i ( f, t ) we obtain:
Revised: i we have Z i = ˜ H i ⊙ X i ∈ C F × T where:

Original: [Equation]
Revised: [Equation]

Original: ∀ i = 1 , . . .

Original: , N , where S r ( f, t ) is the speech component of the reference microphone and ˜ V
Revised: ∀ f, t , where S r ( f, t ) is the speech component of the reference microphone and ˜ V

Original: i ( f, t ) =
Revised: i ( f, t ) =

Original: [ H r ( f, t ) /H i ( f, t )] V i ( f, t ) is the noise component.
Revised: [ H r ( f, t ) /H i ( f, t )] V i ( f, t ) is the noise component.

Original: It can be seen that multiplying the RTFs with the microphone signals leads to aligning all the microphone channels with respect to the clean speech component at the reference microphone as each Z i ( f, t ) contains S r ( f, t ) .
Revised: It can be seen that multiplying the RTFs with the respective input STFT leads to aligning all the microphones with respect to the clean speech component at the reference channel as each Z i ( f, t ) contains S r ( f, t ) .

Original: In the case of far-field, anechoic setting, this means the microphone array has steered to the direction of the signal source.
Revised: In the case of far-field, anechoic setting, this means the microphone array has steered to the direction of the signal source.

Original:  Weimplicitly incorporate RTFs during the model learning phase via an RTF-aware training protocol consisting of two phases: first adjusting only the Align Net parameters so that the alignment masks A i approximate the RTFs ˜ H
Revised: For the case of linear arrays, it can be viewed as the signal is coming from the broadside after steered. We incorporate RTFs during the model learning phase via an RTF-aware training protocol consisting of two phases: First adjusting only the Align Net parameters so that the alignment mask A i approximates the RTFs ˜ H

Original: i by minimizing the following RTF alignment loss:
Revised: i by minimizing the following RTF alignment loss:

Original: L rtf = (cid:80) i,f,t | A i ( f, t ) − ˜ H
Revised: [Equation]

Original: i ( f, t ) | 2 , where we have chosen to use the mean squared error (MSE) loss.
Revised: where we have chosen to use the mean squared error (MSE) loss.

Original: Note that truncation of the magnitude of the ground-truth RTF ˜ H
Revised: Note that truncation of the magnitude of the ground-truth RTFs ˜ H

Original: i ( f, t ) to a maximum value of, e.g., 10 , can be applied for avoiding training instability.
Revised: i ( f, t ) to a maximum value of, e.g., 10 , can be applied for avoiding training instability, and the alignment mask corresponding to the reference microphone is always all ones and is possible not to be estimated.

Original: Optimizing for this loss function guides the network to learn to steer the microphone signals toward the target speech, making the model aware of the spatial information of the target source.
Revised: Optimizing for this loss function guides the network to learn to steer the microphone signals toward the target speech, making the model aware of the spatial information of the target source.

Original: The second phase performs parameter learning of the whole model (i.e., jointly adjusting both Align Net and Filter Net parameters) to reconstruct the clean speech.
Revised: The second phase performs parameter learning of the whole model (i.e., jointly adjusting both Align Net and Filter Net parameters) to reconstruct the clean speech.

Original: To this end, we minimize the signal reconstruction loss
Revised: To this end, we minimize the signal reconstruction loss:

Original: L rec =
Revised: [Equation]

Original: (cid:80) f,t 0 . 7( | ˆ S ( f, t ) | 0 . 3 − | S ( f, t ) | 0 .
Revised: where we have chosen to use the combined power-law compressed MSE loss first appeared in (Wilson et al., 2018).

Original: + 0 . 3 || ˆ S ( f, t ) | 0 .
Revised: In this work we use β = 0 . 3 and c = 0 .

Original: 3 e j ∠ ˆ S ( f,t ) − | S ( f, t ) | 0 . 3 e j ∠ S ( f,t ) | 2 , i.e., the combined power-law compressed MSE loss in Wilson et al. ; Braun & Tashev (2021).
Revised: 3 as suggested by a consolidated study in (Braun & Tashev, 2021).

Original: 3. AF NET BASED ON COMPLEX W-N ET ARCHITECTURE
Revised: 3. AF NET BASED ON COMPLEX W-N ET ARCHITECTURE

Original: We introduce the AFnet (Figure 1) for multichannel SE based on implementing a W-Net architecture suitable for realizing the two-stage framework.
Revised: We introduce the AFnet (Figure 1) for multichannel SE based on implementing a W-Net (Xia &

Original: Originally, the W-Net (Xia & Kulis, 2017) is an improved version of the U-Net structure (Ronneberger et al., 2015).
Revised: Kulis, 2017) architecture suitable for realizing the two-stage framework.

Original: For the SE application here, extension to complex operations can be more suitable when working in the STFT domain.
Revised: For the SE application here, extension to complex operations can be more suitable when working in the STFT domain.

Original: Choi et al.
Revised: Choi et al.

Original: (2018) have proposed to utilize complex-valued U-Net for the single-channel SE task.
Revised: (2018) have proposed to utilize complex-valued U-Net for the single-channel SE task.

Original: In our work, we follow this idea and extend the W-Net by incorporating complex operations forthe multichannel SE task.
Revised: We follow this idea and extend the W-Net by incorporating complex operations for multichannel SE.

Original: Note that Koyama & Raj (2019) also adopt a W-Net architecture for beamforming.

Original: However, they do not utilize complex-valued network layers.

Original: In addition, the first U-Net in their work aims to predict a single-channel enhanced output, while in our AFnet it estimates multiple masks for predicting aligned multichannel signals to exploit spatial information.

Original: In the current work, main modifications to the W-Net include: Convolutional layers of W-Net are all replaced to complex convolutional layers.
Revised: In the current work, main modifications to the W-Net include: Convolutional layers of W-Net are all replaced to complex convolutional layers.

Original: For the activation function, complex leaky ReLU, i.e., an activation function which applies leaky ReLU on both real and imaginary values, are utilized.
Revised: For the activation function, complex leaky ReLU, i.e., an activation function which applies leaky ReLU on both real and imaginary values, are utilized.

Original: The number of feature maps in each layer are also modified.
Revised: The number of feature maps in each layer are also modified.

Original: Note that in Figure 1, “dConv ( C in ,
Revised: Note that in Figure 1, “dConv ( C in ,

Original: C out )” stands for “double Convolutions” with C in input channels and C out output channels.
Revised: C out )” stands for “double Convolutions” with C in input channels and C out output channels.

Original: Each dConv unit consists of two stacks of “(complex) 3 × 3 convolution → batch normalization → leaky
Revised: Each dConv unit consists of two stacks of “(complex) 3 × 3 convolution → batch normalization → leaky

Original: ReLU”.
Revised: ReLU.”

Original: The convolution layer of the first stack takes C in channels and outputs C out channels; the convolution layer of the second  takes C out channels and outputs C out channels. 
Revised: The convolution layer of the first stack takes C in channels and outputs C out channels; the convolution layer of the second stack takes C out channels and outputs C out channels. 3. D IFFERENTIATION FROM EXISTING METHODS

Revised: Existing works such as (Wang et al., 2021; Zhang et al., 2021) leverage the conventional beamformer modules for incorporating spatial features implicitly.

Revised: However, matrix inversion or eigendecomposition are often required and additional network modules might otherwise be needed to replace such operations.

Original:   E XPERIMENTS 4. D ATASETS AND EXPERIMENTAL SETUP 4.1. P RIMARY DATASET
Revised: Other works (Koyama & Raj, 2019; 2020; Li et al., 2022) concatenate two DNN modules and jointly train both toward the enhanced speech. However, without utilizing any beamforming units within the model it is hard to interpret if the network actually learns to exploit spatial features. Differently, our method introduces a simple skip connection of alignment masks trained via supervised learning of the RTFs to explicitly incorporate meaningful spatial characteristics.  E XPERIMENTS 4. D ATASETS AND EXPERIMENTAL SETUP 4.1. P RIMARY DATASET

Original: Speech corpus: We leverage the AVSpeech dataset which is a large collection of video clips of single speakers talking with no audio background interference introduced in Ephrat et al.
Revised: Speech corpus: We leverage the AVSpeech dataset which is a large collection of video clips of single speakers talking with no audio background interference introduced in Ephrat et al.

Original: The dataset is based on public instructional YouTube videos, from which short, 3-10 second clips were automatically extracted, where the only audible sound in the soundtrack belongs to a single speaking person.
Revised: The dataset is based on public instructional YouTube videos, from which short, 3-10 second clips were automatically extracted, where the only audible sound in the soundtrack belongs to a single speaking person.

Original: We downloaded 8308 clips from the training set and another 1199 clips from the test set for our experiments, using only the audio portion (converted to 16 kHz .wav files).
Revised: We downloaded 8308 clips from the training set and another 1199 clips from the test set for our experiments, using only the audio portion (converted to 16 kHz .wav files).

Original: Room impulse response  data: We mix the speech files with noise files of several types also downloaded from YouTube and consider different reverberation conditions for generating the noisy data using simulated impulse responses via Pyroomacoustics (Scheibler et al., 2018), which is a Python-based acoustic simulator for generating different acoustic scenes.
Revised: Room impulse response (RIR) data: We mix the speech files with noise files of several types also downloaded from YouTube and consider different reverberation conditions for generating the noisy data using simulated impulse responses via Pyroomacoustics (Scheibler et al., 2018), which is a Python-based acoustic simulator for generating different acoustic scenes.

Original: We create a room of size 8 × 8 × 3 (length × width × height in meters).
Revised: We create a room of size 8 × 8 × 3 (length × width × height in meters).

Original: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.
Revised: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.

Original: The simulating approach for estimating room impulse responses is based on the image source method.
Revised: The simulating approach for estimating RIRs is based on the image source method.

Original: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array with arbitrary directions of arrival.
Revised: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array with arbitrary directions of arrival.

Original: The microphones form a 2-by-4 planar array placed vertically at the center of the room, and the spacing between two adjacent microphones is 5 cm.
Revised: The microphones form a 2-by-4 planar array placed vertically at the center of the room, and the spacing between two adjacent microphones is 5 cm.

Original: 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) are used for testing.
Revised: 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) are used for testing.

Original: Each type of noise is multiplied with a scale randomly chosen from { 0, 0.5, 1, 1.5, 2 } before added up, and the combined noise is then added to the clean speech according to a specificSNR level randomly selected from { -10, -6, -3, 0, 3, 6, 10 } dB. In this way, we are adopting the supervised learning SE scheme of mix-and-separate, where the clean and noise signals are separately collected and subsequently mixed together to become the noisy signals for training purposes. This mix-and-separate scheme by assuming additive noise model is still being used in the majority of SE works (not only signal processing but also deep learning based approaches).
Revised: Each type of noise is multiplied with a scale randomly chosen from { 0, 0.5, 1, 1.5, 2 } before added up, and the combined noise is then added to the clean speech according to a specific

Original: To apply it to real data, one may collect the clean and noise recordings using the target device in real-world environments and mix them up to generate noisy mixtures for training. The main reason why we have chosen to use the simulated multichannel data as the primary dataset for exploration purposes is to have sufficient spatial diversity of the target speech and also be able to label the speech source location for visualization of the intermediate variables learned by the AFnet (i.e., the alignment masks) and other in-depth study. The acoustic room simulator provides the flexibility of modifying several factors for generating representative multichannel data, which has also been a common practice in many multichannel audio processing works.
Revised: SNR level randomly selected from { -10, -6, -3, 0, 3, 6, 10 } dB.

Original: 4.1. A UXILIARYDATASET :
Revised: 4.1. A UXILIARY

Revised: DATASET :

Original: The CHiME-3 dataset (Barker et al., 2015), made available as part of a speech separation and recognition challenge, is used for training and evaluating SE performance.
Revised: The CHiME-3 dataset (Barker et al., 2015), made available as part of a speech separation and recognition challenge, is used for training and evaluating SE performance.

Original: The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled at 16 kHz.
Revised: The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled at 16 kHz.

Original: It consists of 7,138 and 1,320 simulated utterances with an average length of 3 seconds for training and testing, respectively.
Revised: It consists of 7,138 and 1,320 simulated utterances with an average length of 3 seconds for training and testing, respectively.

Original: We take this dataset only as auxiliary due to limited positional variety of the target speech – most of the speakers have been speaking from the front side with respect to the microphones.
Revised: We take this dataset only as auxiliary due to limited positional variety of the target speech – most of the speakers have been speaking from the front side with respect to the microphones.

Original: As such, the dataset is less ideal for exploration of spatial characteristics and signal alignment purposes.
Revised: As such, the dataset is less ideal for exploration of spatial characteristics and signal alignment purposes

Original: 4.1. M ODEL SETTINGS AND EVALUATION METRICS
Revised: 4.1. M ODEL SETTINGS AND EVALUATION METRICS

Original: In this paper, we set the number of channels of the AFnet (Figure 1) as: C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 , 64 , resulting in a model size of approximately 2.6M parameters. 
Revised: In this paper, we set the number of channels of the AFnet (Figure 1) as: C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 , 64 , resulting in a model size of approximately 2.6M parameters. The slope of leaky

Original:  Regarding the STFT processing of the input signal, for the primary dataset we use the Hann window with 512-point fast Fourier transform (FFT) and for CHiME-3 with 1024-point FFT, both using a hop size of 256.
Revised: ReLU is set to 0 . Regarding the STFT processing of the input signal, for the primary dataset we use the Hann window with 512-point fast Fourier transform (FFT) and for CHiME-3 with 1024-point FFT, both using a hop size of 256.

Original: During training, 4-second long segments are randomly cropped from the training data samples while during testing the whole utterances are used.
Revised: During training, 4-second long segments are randomly cropped from the training data samples while during testing the whole utterances are used.

Original: The RTFs are obtained by computing the ratios between the clean speech signals at all microphones according to (1).
Revised: The RTFs are obtained by computing the ratios between the clean speech signals at all microphones according to (4).

Original: We use the following three commonly seen speech enhancement metrics for evaluation purposes.
Revised: We use the following three commonly seen SE metrics for evaluation purposes.

Original: PESQ:
Revised: PESQ:

Original: Perceptual Evaluation of Speech Quality (ITU-T Recommendation P.862.2, 2005).
Revised: Perceptual Evaluation of Speech Quality (ITU-T Recommendation P.862.2, 2005).

Original: STOI: Short-Time Objective Intelligibility (Taal et al., 2011).
Revised: STOI: Short-Time Objective Intelligibility (Taal et al., 2011).

Original: SSNR: Segmental Signal-to-Noise Ratio (Hansen & Pellom, 1998).
Revised: SSNR: Segmental Signal-to-Noise Ratio (Hansen & Pellom, 1998) (segment length = 30 msec, 75% overlap, SNR min = − 10 dB, SNR max = 35 dB).

Original: For all the metrics, the higher the score, the better the performance.
Revised: For all the metrics, the higher the score, the better the performance.

Revised: More details of experiential settings can be found int Appendix B.

Original: 4. R ESULTS 4.2. A LIGNED VS . UNALIGNED SIGNAL ENHANCEMENT
Revised: 4. R ESULTS 4.2. A LIGNED VS . UNALIGNED SIGNAL ENHANCEMENT

Original: [Table]

Original: To get an understanding of the importance of alignment, we first conduct an experiment to compare the difference between enhancement outcomes of aligned and unaligned input signals with the primary dataset.
Revised: To get an understanding of the importance of alignment, we first conduct an experiment to compare the difference between enhancement outcomes of aligned and unaligned input signals with the primary dataset.

Original: To this end, two schemes are considered, where in the first scheme we manually align the noisy microphone signals by multiplying them with the corresponding ground truth speech RTFs, while in the other scheme we directly use the original unaligned signals.
Revised: To this end, two schemes are considered, where in the first scheme we manually align the noisy microphone signals by multiplying them with the corresponding ground truth speech RTFs, while in the other scheme we directly use the original unaligned signals.

Original: In each scheme, a single U-Net model, or equivalently, the Filter Net (the second half of AFnet), is trained for the signal reconstruction loss L rec and tested on the corresponding aligned or unaligned data.
Revised: In each scheme, a single U-Net model, or equivalently, the Filter Net (the second half of AFnet), is trained for the signal reconstruction loss L rec and tested on the corresponding aligned or unaligned data.

Original: The performance numbers of the two cases are presented in Table 1.
Revised: The performance numbers of the two cases are presented in Table 1.

Original: It can be seen that when the signals are aligned, the Filter Net model can do a much better job of separating the target signal from background noise and achieves significantly better scores than unaligned results in all three metrics.
Revised: It can be seen that when the signals are aligned, the Filter Net model can do a much better job of separating the target signal from background noise and achieves significantly better scores than unaligned results in all three metrics.

Original: This observation is consistent across all three number of microphone settings.
Revised: This observation is consistent across all three number of microphone settings.

Original: The results suggest that for a spatially diverse dataset where the target speech may come from various locations, having a signal alignment module in front of the signal filtering module can be beneficial.
Revised: The results suggest that for a spatially diverse dataset where the target speech may come from various locations, having a signal alignment module in front of the signal filtering module can be beneficial.

Original: We also present the results for the CHiME-3 dataset in Table 1.
Revised: We also present the results for the CHiME-3 dataset in Table 1.

Original: We can see that the margin between the results of using the aligned and unalinged signals is much smaller here.
Revised: We can see that the margin between the results of using the aligned and unalinged signals is much smaller here.

Original: It could be attributed to that most of the CHiME-3 recording data samples have the speaker almost speaking from the front side with respect to the microphones which results in less variety in directional angles (the common situation when the speaker was asked for holding a tablet to speak the sentence listed in the tablet, and the speech data were recorded through 6 embedded microphones in the same tablet.
Revised: It could be attributed to that most of the CHiME-3 recording data samples have the speaker almost speaking from the front side with respect to the microphones which results in less variety in directional angles (the common situation when the speaker was asked for holding a tablet to speak the sentence listed in the tablet, and the speech data were recorded through 6 embedded microphones in the same tablet.

Original: See the recording setup and recording demonstration picture in CHiME-3 official website) and thus the less spatial diversity.
Revised: See the recording setup and recording demonstration picture in CHiME-3 official website) and thus the less spatial diversity.

Original: Moreover, as the speech sources mostly come from the front side, they are in a sense already roughly aligned.
Revised: Moreover, as the speech sources mostly come from the front side, they are in a sense already roughly aligned.

Original: Thus, there is the less benefit brought by performing the alignment, meaning that the requirement of aligning signals is bypassed to some extend. 
Revised: Thus, there is the less benefit brought by performing the alignment, meaning that the requirement of aligning signals is bypassed to some extend. [Table] [Table]

Original: 4.2. E XPLOITING SPATIAL SEPARABILITY WITH AF NET
Revised: E XPLOITING SPATIAL SEPARABILITY WITH AF NET

Original: Having the above insight, we move on to studying the proposed AFnet for exploitation of spatial separability on the primary dataset with different model/training settings and comparing with typical one-stage deep learning framework approaches.
Revised: Having the above insight, we move on to studying the proposed AFnet for exploitation of spatial separability on the primary dataset with different model/training settings and comparing with typical direct filter weight estimation approaches.

Original: For detailed analysis, we consider four cases below:
Revised: For detailed analysis, we consider four cases below:

Original: i) U-Net for filter weights: using only a single U-Net (same model as the Filter Net of AFnet) to  estimate the  filter weights W i in typical(one-stage) multichannel SE ii) W-Net for filter weights: using a W-Net (same model as the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to  estimate the  filter weights W i in typical(one-stage) multichannel SE iii) AFnet w/o RTF loss: the two-stage sequential masking AFnet model trained by normal training scheme without incorporating RTFs iv) AFnet: the two-stage sequential masking AFnet model trained with RTF-aware scheme Cases i) and ii) represent the typicalone-stage approach which uses a DNN to  estimate the filter weights for beamforming purposes, serving as baseline methods here.
Revised: i) U-Net for filter weights: using only a single U-Net (same model as the Filter Net of AFnet) to directly estimate the sets of filter weights W i in typical multichannel SE ii) W-Net for filter weights: using a W-Net (same model as the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to directly estimate the sets of filter weights W i in typical multichannel SE iii) AFnet w/o RTF loss: the two-stage sequential masking AFnet model trained by normal training scheme without incorporating RTFs iv) AFnet: the two-stage sequential masking AFnet model trained with RTF-aware scheme Cases i) and ii) represent the typical approach which uses a DNN to directly estimate the beamformer weights.

Original: Cases iii) and iv) are considered to observe the role of RTF alignment in the proposed AFnet.
Revised: Cases iii) and iv) are considered to observe the role of RTF alignment in the AFnet

Original: Figure 2 presents the enhancement results.
Revised: Figure 2 presents the enhancement results.

Original: One can see that the AFnet with RTF supervision achieves the best performance overall.
Revised: One can see that the typical approaches of U-Net and

Original: More importantly, the performance improves as the number of microphone increases, indicating that AFnet indeed exploits the spatial characteristics of sound sources within the multichannel data for enhancement purposes. In contrast, the typical one-stage approaches of U-Net and W-Net  do not seem to actually benefit from the multichannel input data, as the performance is not necessarily improving with increased number of microphones.In fact, the
Revised: W-Net for direct filter weights estimation do not seem to actually benefit from the multichannel input data, as the performance is not necessarily improving with increased number of microphones.

Original:  W-Net even performs worse than the single U-Net model for the cases with more microphones (4mic, 8-mic) though doubling the model size, with its best result surprisingly reached for the 2-mic case.
Revised: In fact, the W-Net even performs worse than the single U-Net model for the cases with more microphones (4-mic, 8-mic) though doubling the model size, with its best result surprisingly reached for the 2-mic case.

Original: This observation implies that to further take advantage of audio data with more microphone channels, simply stacking U-Net models may not straightforwardly work well.
Revised: This observation implies that to further take advantage of audio data with more microphone channels, simply stacking U-Net models may not straightforwardly work well.

Original: This might be due to the potential optimization difficulty of having a deeper model and more complicated multichannel input data to extract useful information.
Revised: This might be due to the potential optimization difficulty of having a deeper model and more complicated multichannel input data to extract useful information.

Original: On the other hand, the AFnet seems to mitigate these issues by introducing the sequential masking mechanism which also provides certain input signal level skip connections, as we can see that even without RTF supervision it is able to perform much better than the W-Net. 
Revised: On the other hand, the AFnet seems to mitigate these issues by introducing the sequential masking mechanism which also provides certain input signal level skip connections, as we can see that even without RTF supervision it is able to perform much better than the W-Net. Nevertheless, the performance of AFnet without RTF supervision does not necessarily improve with increased number of microphones.

Original: With the RTF loss incorporated for  leveraging spatial separability more easily, the AFnet is able to achieve even better performance. From these observations, we can conclude that the positional variety of  the target speaker can be an important factor to consider in multichannel
Revised: This may be a result of lacking guidance on learning meaningful spatial features. Finally we see that with the RTF loss incorporated for explicitly leveraging spatial separability, the AFnet is able to obtain better performance as the number of microphones increases, and achieves the best performance overall.

Original: SE, and careful consideration of the spatial alignment for model design, e.g., the RTF loss within the sequential masking scheme, is critical for deep learning based multichannel SE systems for diverse speaker position scenarios.

Original: [Figure]
Revised: [Figure]

Original: 4.2. V ISUALIZING THE ALIGNMENT OUTCOMESAND DATASET SPATIAL DIVERSITY
Revised: 4.2. V ISUALIZING THE ALIGNMENT OUTCOMES

Original: To further observe the effect of RTF supervision, we visualize the distributions of the alignment masks in Figure 3 from AFnet trained with RTF loss and that without RTF loss.
Revised: To further observe the effect of RTF supervision, we visualize the distributions of the alignment masks in Figure 3 from AFnet trained with RTF loss and that without RTF loss.

Original: To obtain these plots, we take noisy signal samples from the test set in which the speech source in each sample is located at one of three pre-defined positions in the room.
Revised: To obtain these plots, we take noisy signal samples from the test set in which the speech source in each sample is located at one of three pre-defined positions in the room.

Original: We feed the taken noisy samples to the trained AFnet models (both with and without RTF supervision) and obtain the alignment masks for each sample.
Revised: We feed the taken noisy samples to the trained AFnet models (both with and without RTF supervision) and obtain the alignment masks for each sample.

Original: The distribution of the principal components (PCs) of the obtained masks are plotted.
Revised: The distribution of the principal components (PCs) of the obtained masks are plotted.

Original: It can be seen that with RTF-aware training, the distributions show clearer clusters of speech sources corresponding to the three different positions than with normal training.
Revised: It can be seen that with RTF-aware training, the distributions show clearer clusters of speech sources corresponding to the three different positions than with normal training.

Original: In addition, with increased number of microphones the clusters also become more separate.
Revised: In addition, with increased number of microphones the clusters also become more separate.

Original: The results together with improved enhanced speech quality provide evidence of the fact that the system is actually exploiting spatial separability of the audio sources for achieving better enhancement performance.
Revised: The results together with improved enhanced speech quality provide evidence of the fact that the system is actually exploiting spatial separability of the audio sources for achieving better enhancement performance.

Original: Another angle to inspect the spatial diversity of the dataset can also be provided by observing the alignment masks of the data points.

Original: In Figure 4 we compare the distributions of the intermediate alignment masks learned by the AFnet models trained on the primary dataset (high spatial diversity) and the CHiME-3 dataset (low spatial variety).

Original: Here, we plot the distributions of the alignment masks estimated by the trained models for all the test data samples of the two datasets.

Original: It can be seen that for the primary dataset, the RTF-aware training leads to a more widespread distribution than that of the normal training case, corresponding to the greater spatial diversity of the target speech source positions.

Original: In contrast, in the CHiME-3 dataset the RTF-aware training actually results in a more aggregated cluster than the normal training scheme, indicating that the target speech sources are coming from similar directions.

Original: This visualization of the alignment masks provides another angle to inspect the data properties from the spatial diversity aspect for exploring SE performance.

Original: 4.2. C OMPARISON TO SOTA METHODS
Revised: 4.2. C OMPARISON TO SOTA METHODS

Original: To demonstrate the superiority of our method, we compare the AFnet with several SOTA architectures, including: Conv-TasNet (Luo & Mesgarani, 2019): a single-channel SE system performing masking based separation method in the learned transform domain via learnable encoder and decoder.
Revised: To demonstrate the superiority of our method, we compare the AFnet with several SOTA architectures, including: Conv-TasNet (Luo & Mesgarani, 2019): a single-channel SE system performing masking-based separation method in the learned transform domain via learnable encoder and decoder.

Original: DCUnet (Choi et al., 2018): a phase-aware single-channel SE approach utilizing complexvalued operations with the U-Net architecture.
Revised: DCUnet (Choi et al., 2018): a phase-aware single-channel SE approach utilizing complexvalued operations with the U-Net architecture.

Original: FaSNet (Luo et al., 2020): a time-domain, multichannel SE system exploiting cross-channel correlation based on filter-and-sum method.
Revised: FaSNet (Luo et al., 2020): a time-domain, multichannel SE system exploiting cross-channel correlation based on filter-and-sum method.

Original:  We train the SOTA models by ourselves.
Revised: [Figure] We train the SOTA models by ourselves.

Original: For fairness, all models are trained by minimizing the reconstruction loss L rec of the  power-law compressed MSE, using 512-point FFT with a hop size of[Figure] [Figure] 256 via the Hann window.
Revised: For fairness, all models are trained by minimizing the reconstruction loss L rec of the combined power-law compressed MSE of (7), using 512-point FFT with a hop size of 256 via the Hann window.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: It can be seen that the single-channel methods do not perform well even with a larger model size.
Revised: It can be seen that the single-channel methods do not perform well even with a larger model size.

Original: The FaSNet, with a lighter model, performs slightly better than the two single-channel approaches when the number of microphones is larger.
Revised: The FaSNet, with a lighter model, performs slightly better than the two single-channel approaches when the number of microphones is larger.

Original: Finally, our proposed AFnet outperforms all the SOTA approaches by significant amount in all the metrics used, while being of the smallest model size.
Revised: Finally, our proposed AFnet outperforms all the SOTA approaches by significant amount in all the metrics used, while being of the smallest model size.

Original: The results demonstrate the superiority ofthe AFnet for enhancing speech in noisy environments.
Revised: Note that the performance of AFnet lies in between the aligned and unaligned signal cases in Table 1, which is reasonable as the alignment is never perfect but only to certain degree.

Original: Although a less ideal dataset for our study on spatial characteristics, we still benchmark the AFnet with several SOTAs on the CHiME-3 dataset: Nerual BF (Erdogan et al., 2016), MVDR GC (Bu et al., 2019), rSDFCN (Liu et al., 2020), CA Dense U-Net (Tolooshams et al., 2020), and IC ConvTasNet (Lee et al., 2021).
Revised: Although a less ideal dataset for our study on spatial characteristics, we still benchmark the AFnet with several SOTAs on the CHiME-3 dataset, including: Neural BF (Erdogan et al., 2016), MVDR GC (Bu et al., 2019), rSDFCN (Liu et al., 2020), CA Dense U-Net (Tolooshams et al., 2020), and IC Conv-TasNet (Lee et al., 2021).

Original: In Table 3 we report the PESQ and STOI taken from the corresponding papers and the missing entries in the table indicate that the metric is not reported in the reference paper (we do not report SSNR as most papers do not report it).
Revised: In Table 3 we report the PESQ and STOI taken from the corresponding papers and the missing entries in the table indicate that the metric is not reported in the reference paper (we do not report SSNR as most papers do not report it).

Original: We present both results of the AFnet trained with and without RTF supervision (with the 5-th channel selected as the reference microphone).
Revised: We present both results of the AFnet trained with and without RTF supervision (with the 5-th channel selected as the reference microphone).

Original: One can see that the two-stage, sequential masking design of the AFnet is efficient in that it achieves comparable or better performance to other SOTA approaches while maintaining a small model size.
Revised: One can see that the two-stage, sequential masking design of the AFnet is efficient in that it achieves comparable or better performance to other SOTA approaches while maintaining a small model size.

Original: Notably, the RTF supervision helps improve the AFnet performance, while the gains are minor in this case due to the less spatial diversity of the speech data.
Revised: Notably, the RTF supervision helps improve the AFnet performance, while the gains are minor in this case due to the less spatial diversity of the speech data.

Original: Nevertheless, the results still show the benefit of incorporating signal alignment into the network design, even though on a dataset with limited positional variety of the speech target.
Revised: Nevertheless, the results still show the benefit of incorporating signal alignment into the network design, even though on a dataset with limited positional variety of the speech target.

Original: [Table]
Revised: The Align Net trained with the RTF loss performs both temporal alignment as well as level adjustment. It would be interesting to look at how the magnitude and phase components of the RTFs, which correspond to ILD and ITD in time domain, affect the enhancement performance of the AFnet.

Original: [Table]
Revised: In

Original: The results presented in previous sections on the primary dataset are based on the multichannel audio data synthesized by using the image source method via the Pyroomacoustics library, which is useful for methodological study but also has its limitation to represent real data.
Revised: Figure 4 we compare four different RTF alignment schemes – no alignment, aligning RTF magnitude only, aligning RTF phase only, and aligning everything for the RTF loss L rtf . From the results it can be seen that phase alignment is the key to obtaining improved performance over the alignmentagnostic system. This is reasonable as aligning phase in the frequency domain corresponds to compensating the delay in the time domain, which is crucial for any beamforming algorithms that rely on spatial separability.

Original: To further validate the proposed approach’s generalization capabilities to more realistic scenarios, we also conduct experiments on multichannel audio data generated using real-world measured acoustic room impulse responses taken from the work of Hadad et al.
Revised: The results suggest that the AFnet indeed exploits spatial information for improved enhancement performance.

Original: (2014), where the impulse responses of several 8channel linear microphone arrays with respect to various target source directions of arrival (DoAs) (from -90 to 90 degrees) are provided.
Revised: [Table] [Table] [Figure]

Original: We utilize the 3-3-3-8-3-3-3 array impulse responses of the dataset including reverberation time of 0.16, 0.36, and 0.61 second cases, together with the AVSpeech utterances and the YouTube noise profiles to generate the multichannel noisy data for experiments. To observe the relation of the AFnet performance with the speech source spatial diversity, we generate a spatially diverse dataset where the speech DoA ranges from -90 to 90 degrees, and another spatially constant dataset where the speech comes from only one direction.
Revised: 4.2. A RE INTERMEDIATE ALIGNMENT MASKS NECESSARY ?

Original: Table 4 and Table 5 present the results for the two datasets respectively.
Revised: Given the results in Figure 4 where the phase importance is recognized [Table]

Original: One can see that for for the spatially diverse dataset ofTable 4, the AFnet clearly benefits from the sequentially masking mechanism and the RTF loss supervision, while for the spatially constant dataset of Table 5 the three models perform comparably with the RTF loss marginally improves AFnet.
Revised: one would ask if the intermediate alignment mask design of AFnet is necessary [Table] and if we can directly apply phase alignment at the output filtering masks?

Original: These results show that a model (such as the W-Net) that performs well on a dataset lacking spatial variety of the target speech may not generalize well to a spatially diverse dataset, unless further attention is paid to exploiting spatial characteristics of the speech sources. The RTF information leveraged by the twostage sequential masking design of the AFnet is shown to take advantage of the spatial separability for enhanced SE performance on spatially diverse datasets.
Revised: To answer this question

Original: [Table]
Revised: [Table]

Original: DoA is fixed to one
Revised: we train the W-Net for predicting the filter weights (i [Table] e., the filtering masks) to minimize L rec + λ L rtf, phase , where L rec is the same signal reconstruction loss as (7) and L rtf, phase is the RTF alignment loss similar to (6) but imposed on the filtering masks given by the W-Net for the phase portion only and regularized by λ > 0 . Several values of λ are considered and the results are compared to AFnet in Table 4.

Revised: We can see that over a wide range of λ the W-Net is still not able to achieve as good performance as AFnet which utilizes the alignment masks with RTF supervision.

Revised: This implies that the intermediate masking mechanism for performing alignment is critical for achieving improved SE with two-stage network design.

Original: [Table]
Revised: [Table]

Revised: Additional results on a variety of testing scenarios, e.g., SNR conditions, room configurations, training protocols, time-varying RIR and realistic RIR data are provided in Appendix C. Comparison of

Revised: Align Net with signal processing-based algorithms for RTF estimation is presented in Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented the AFnet, a deep learning based multichannel SE approachfeaturing a two-stage, sequential masking design, that leverages spatial information inherent in the RTFs for signal alignment, an essential factor in many conventional signal processing SE algorithms.
Revised: In this paper, we presented the AFnet, a deep learning-based multichannel SE approach that performs signal alignment followed by filtering for extracting the target speech from noisy observations. We showed that leveraging spatial information inherent in the RTFs for signal alignment purposes of the two-stage, sequential masking network during training is the key to remarkable improvements.

Original: The effect of target speech spatial diversity of the dataset and the importance of spatial alignment on deep learning based methods were analyzed and discussed.
Revised: Our findings suggest that RTF alignment, especially the phase estimation, plays a crucial role in deep learning multichannel SE for target speech that may come from arbitrary locations. A PPENDIX A C ONVENTIONAL SIGNAL PROCESSING BEAMFORMING ALGORITHMS BASED ON ALIGN - THEN - FILTER PROCESS

Original: Our findings suggest that signal alignment plays a crucial role in deep learning based multichannel SE approaches when the target speech may come from arbitrary locations, especially when there are more microphones, and that it would be beneficial to consider such alignment aspect when developing future multichannel SE systems.
Revised: To support the motivation behind the proposed AFnet which is designed based on the align-thenfilter concept well-known in signal processing, we review several spatial filtering algorithms based on using this design principle. The intuition for performing alignment followed by filtering is relatively simple. Consider the RTF aligned signal expression in (5). We can see that in the aligned signals each Z i ( f, t ) contains

Original: A I MPLEMENTATION DETAILS A. P RIMARY DATASET
Revised: S r ( f, t ) . Since all the S r ( f, t ) among the N channels are in-phase (and actually are identical if magnitudes are also aligned in this case), summing up all the Z i ( f, t ) over the N channels will not be destructive to the speech component. However, the noise components ˜ V i ( f, t ) are usually not in-phase (unless coming from the same direction as the speech source), destructive combination can happen by summing them up.

Revised: Consequently, combining all the Z i ( f, t ) across the N channels will boost the SNR as the speech component is preserved while the noise components are suppressed.

Revised: The above process is actually the classic “delay-and-sum (DS)” beamformer Trees (2004); Doclo et al.

Revised: In a DS beamformer, a steering vector containing the time delay information between the microphone sensors with respect to the target speech is first applied to the input multichannel signals to compensate the time delay (or equivalently the phase difference) of the speech components in all channels.

Revised: This step is called “alignment.”

Revised: Then a summation operation over the aligned channels is performed to suppress noise for boosting SNR.

Revised: This step is called “filtering.”

Revised: Mathematically, the enhancement process of the DS beamformer can be expressed as:

Revised: [Equation]

Revised: 8) ∀ f, t , where ˜ h ( f, t ) =

Revised: [ ˜ H 1 ( f, t ) , . .

Revised: ˜ H N ( f, t )]

Revised: T is the steering vector consisting of RTFs (potentially only the phase components) and x

Revised: ( f

Revised: t

Revised: ) =

Revised: X

Revised: ( f, t )

Revised: . . , X N ( f, t )

Revised: ] T is

Revised: the

Revised: vector

Revised: of

Revised: microphone signals.

Revised: Another famous beamformer utilizing the align-then-filter concept is the “generalized sidelobe canceller (GSC)”

Revised: (Griffiths & Jim, 1982; Frost, 1972).

Revised: In a typical GSC, the input noisy signals are first aligned by the steering vector.

Revised: Then, two branches of processing follow, where one branch sums up all the aligned microphone signals to perform a DS beamforming, while the other branch introduces a “blocking matrix” to extract noise references while blocking out the target speech component.

Revised: Then, the noise-only output of the blocking matrix will go through an adaptive filtering algorithm to estimate the residual noise component in the DS beamforming output signal in an adaptive manner.

Revised: Note that the alignment step plays an important role here.

Revised: First, it leads to the DS beamformer for boosting SNR.

Revised: In addition, it is beneficial for the blocking matrix stage to extract noise-only references.

Revised: For example, a simple blocking mechanism can be just subtracting all other Z i ( f, t ) , i ̸

Revised: = r from the reference channel Z r ( f, t ) so that the S r ( f, t ) will be cancelled out and only the noise remains.

Revised: The GSC is obviously another “align-then-filter” example with a more sophisticated filtering stage.

Revised: Finally, the popular “minimum-variance-distortionless-response (MVDR)” beamformer (Capon, 1969; Doclo et al., 2015) can also belong to the align-then-filter framework.

Revised: The derivation of the MVDR filter weights w =

Revised: [ W 1 ( f, t ) , . .

Revised: . , W N ( f, t )] T ∈ C N which applies to the noisy signals to obtain the enhanced speech as ˆ S ( f, t ) =

Revised: w H ( f, t ) x ( f, t ) starts from considering the following optimization problem:

Revised: [Equation]

Revised: where Φ v ( f, t ) =

Revised: E [ v ( f, t ) v ( f, t ) H ] ∈ C N × N is the power spectral density of the vector of noise signals v ( f, t ) =

Revised: [ V 1 ( f, t ) , . .

Revised: , V N ( f, t )] T and ¯ h ( f, t ) =

Revised: [ ˜ H − 1 1 ( f, t ) , . . . ,

Revised: ˜ H − 1 N ( f, t )]

Revised: T is the (reciprocal) vector of RTFs.

Revised: Solving the above we obtain the MVDR filter as:

Revised: [Equation]

Revised: ∀ f, t .

Revised: We see that the RTFs which are related to signal alignment are involved in the computation of the MVDR filter weights.

Revised: From the above, we can see that the RTF information plays an essential role in many conventional beamforming algorithms.

Revised: This has motivated us to devise the AFnet following the alignment principle.

Revised: Note that the signal processing-based beamforming algorithms typically rely on modeling assumptions of signal or noise statistics to perform denoising, and therefore the performance is limited due to mismatch of real-world observations and modeling assumptions.

Revised: Our approach is based on data-driven, deep learning techniques to overcome such limitations while taking advantage of the

Revised: RTFs.

Revised: A PPENDIX B I MPLEMENTATION DETAILS B. RTF- AWARE TRAINING ALGORITHMS

Revised: The main RTF-incorporated training scheme is summarized in Algorithm 1, which performs twostep training consisting of i) training the Align Net for the RTF loss followed by ii) training both

Revised: Align and Filter Nets jointly for the signal reconstruction loss.

Revised: inputs

Revised: Model( X i ∈ C F × T , ∀ i = 1 , . . .

Revised: , N ; Align Net, Filter Net)

Revised: Initialize Align Net, Filter Net for number of training iterations do

Revised: Sample a minibatch of new input spectrograms

Revised: X i , ∀ i = 1 . . .

Revised: , N

Revised: Update Align Net by minimizing L rtf end for for number of training iterations do

Revised: Sample a minibatch of new input spectrograms

Revised: X i , ∀ i = 1 . . .

Revised: , N

Revised: Update Align Net and Filter Net by minimizing L rec end for

Revised: Another way of RTF-aware training for the AFnet is by combining the RTF loss and signal reconstruction loss together and jointly train the entire AFnet from scratch.

Revised: In this sense, the RTF loss serves as a regularization term weighted by a scaler λ > 0 .

Revised: The training scheme is depicted in

Revised: Algorithm 2.

Revised: Later in Section C.1 we will show that this is less effective than Algorithm 1.

Revised: inputs

Revised: Model( X i ∈ C F × T , ∀ i = 1 , . . .

Revised: , N ; Align Net, Filter Net)

Revised: Initialize Align Net, Filter Net for number of training iterations do

Revised: Sample a minibatch of new input spectrograms

Revised: X i , ∀ i = 1 . . .

Revised: , N

Revised: Update Align Net and Filter Net by minimizing L rec + λ for

Original: Corpus: We use Audio-Visual Speech Dataset (AVSpeech)
Revised: Corpus: We use Audio-Visual Speech Dataset (AVSpeech)

Original: (Ephrat et al., 2018) available online.
Revised: (Ephrat et al., 2018) available online.

Original: AVSpeech is a large collection of video clips of single speakers talking with no audio background interference.
Revised: AVSpeech is a large collection of video clips of single speakers talking with no audio background interference.

Original: The audio files of the AVSpeech dataset serve as proper target speech files due to their variety (diversity of language, gender, age, etc.)
Revised: The audio files of the AVSpeech dataset serve as proper target speech files due to their variety (diversity of language, gender, age, etc.)

Original: The dataset can be found at: https://looking-tolisten.github.io/avspeech/.
Revised: The dataset can be found at: https:// looking-to-listen.github.io/avspeech/ .

Original: Noise data: We collected 8 types of noise profiles from YouTube which are commonly seen sound sources in daily life.
Revised: Noise data: We collected 8 types of noise profiles from YouTube which are commonly seen sound sources in daily life.

Original: The noise types include blender, vacuum, washer, baby cry (for training) and dog barking, kids playing sound, hair dryer, food sizzling (for testing).
Revised: The noise types include blender, vacuum, washer, baby cry (for training) and dog barking, kids playing sound, hair dryer, food sizzling (for testing).

Original: Each type of noise profile contains recordings from different sound sources of the same type.
Revised: Each type of noise profile contains recordings from different sound sources of the same type.

Original: Data pre-processing: All the raw audio data are first converted to 16 kHz mono .wav
Revised: Data pre-processing: All the raw audio data are first converted to 16 kHz mono .wav

Original: files before feeding to the room acoustic simulator to generate the multichannel data samples for experimentation (the AVSpeech audio originally may contains 1 or 2 channels of audio).
Revised: files before feeding to the room acoustic simulator to generate the multichannel data samples for experimentation (the AVSpeech audio originally may contains 1 or 2 channels of audio).

Original: For spectral processing, we use the Hann window with 512-point FFT and a hop size of 256 for the STFT.
Revised: For spectral processing, we use the Hann window with 512-point FFT and a hop size of 256 for the STFT.

Original: By conjugate symmetry of the FFT, only half of the frequency bins (i.e., 257) are actually needed for the network to process the spectrograms.
Revised: By conjugate symmetry of the FFT, only half of the frequency bins (i.e., 257) are actually needed for the network to process the spectrograms.

Original: Room acoustic simulator: Pyroomacoustics is utilized to generate multichannel audio which can be found at https://github.com/LCAV/pyroomacoustics/tree/pypi-release.
Revised: Room acoustic simulator: Pyroomacoustics is utilized to generate multichannel audio which can be found at https://github.com/LCAV/pyroomacoustics/tree/pypi-release .

Original: We create a room of size 8 × 8 × 3 (length × width × height in meters).
Revised: We create a room of size 8 × 8 × 3 (length × width × height in meters).

Original: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.
Revised: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.

Original: The room impulse responses are estimated based on the image source method.
Revised: The RIRs are estimated based on the image source method.

Original: 8 microphones are placed to form a planar array (4 columns, 2 rows) vertically, where the distance between adjacent microphones is set to 5 cm.
Revised: 8 microphones are placed to form a planar array (4 columns, 2 rows) vertically, where the distance between adjacent microphones is set to 5 cm.

Original: The microphone array center is placed at the center of the room.
Revised: The microphone array center is placed at the center of the room.

Original: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array center.
Revised: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array center.

Original: In the experiments, the upper leftmost and rightmost microphones are used for the 2-mic scheme, the microphones at the four corners are used for the 4-mic scheme, and all microphones are used for the 8-mic scheme.
Revised: In the experiments, the upper leftmost and rightmost microphones are used for the 2-mic scheme, the microphones at the four corners are used for the 4-mic scheme, and all microphones are used for the 8-mic scheme.

Original: Measured room impulse responses: To further validate the proposed approach’s generalization capabilities to more realistic scenarios, we also conduct experiments on multichannel audio data generated using real-world measured acoustic room impulse responses.
Revised: RIRs: To further validate the proposed approach’s generalization capabilities to more realistic scenarios, we also conduct experiments on multichannel audio data generated using real-world measured acoustic RIRs.

Original: A popular multichannel impulse response dataset is from the work by Hadad et al.
Revised: impulse response dataset is from the work by Hadad et al.

Original: (2014), where the impulse responses of several 8channel microphone arrays with respect to various target source directions (from -90 to 90 degrees) are provided.
Revised: (2014), where the impulse responses of several 8-channel microphone arrays with respect to various target source directions (from -90 to 90 degrees) are provided.

Original: We utilize the 3-3-3-8-3-3-3 array impulse responses of the dataset including reverberation time of 0.16, 0.36, and 0.61 second cases (https://www.iks.rwth-aachen.de/en/research/toolsdownloads/databases/multi-channel-impulse-response-database/), together with the AVSpeech utterances and the YouTube noise profiles to generate the multichannel noisy data for experiments. 
Revised: We utilize the 3-3-3-8-3-3-3 array impulse responses of the dataset including reverberation time of 0.16, 0.36, and 0.61 second cases ( https://www.iks.rwth-aachen.de/en/research/tools-downloads/ databases/multi-channel-impulse-response-database/ ), together with the AVSpeech utterances and the YouTube noise profiles to generate the multichannel noisy data for experiments. The experimental results and related discussion are presented in Appendix C.7.

Original: Network architectures:
Revised: Mixing speech with noise: We mix the multichannel speech files with noise profiles to generate noisy-clean data pairs. Each type of noise is multiplied with a scale randomly chosen from { 0, 0.5, 1, 1.5, 2 } before added up, and the combined noise is then added to the clean speech according to a specific SNR level randomly selected from { -10, -6, -3, 0, 3, 6, 10 } dB. In this way, we are adopting the supervised learning SE scheme of mix-and-separate, where the clean and noise signals are separately collected and subsequently mixed together to become the noisy signals for training purposes. This mix-and-separate scheme by assuming additive noise model is still being used in the majority of SE works (not only signal processing- but also deep learning-based approaches). To apply it to real data, one may collect the clean and noise recordings using the target device in real-world environments and mix them up to generate noisy mixtures for training.

Original: • AFnet: the implementation of AFnet (Figure 1) is based on the deep complex U-Net model at: https://github.com/sweetcocoa/DeepComplexUNetPyTorch, for both the Align Net and Filter Net parts.
Revised: • AFnet: the implementation of AFnet (Figure 1) is based on the deep complex U-Net model at: https://github.com/sweetcocoa/DeepComplexUNetPyTorch , for both the Align Net and Filter Net parts.

Original: The number of layers and the number of features in each layer are modified to the one shown in Figure 1, where C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 , 64 .
Revised: The number of layers and the number of features in each layer are modified to the one shown in Figure 1, where C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 , 64 .

Original: • Conv-TasNet : the experimental results of Conv-TasNet is based on the implementation from https://github.com/kaituoxu/Conv-TasNet using their own model settings instead of the original Conv-TasNet paper.
Revised: • Conv-TasNet : the experimental results of Conv-TasNet is based on the implementation from https://github.com/kaituoxu/Conv-TasNet using their own model settings instead of the original Conv-TasNet paper.

Original: Note that although the model hyperparameters were designed based on 8 kHz sampling rate, we directly use them for 16 kHz sampling rate of our data without further modifications.
Revised: Note that although the model hyperparameters were designed based on 8 kHz sampling rate, we directly use them for 16 kHz sampling rate of our data without further modifications.

Revised: Hyperparameters: N = 256 , L = 20 , B = 256 , H = 512 , P = 3 , X = 8 , R = 4 , Norm=gLN, Noncausal.

Original: • DCUnet : the experimental results of DCUnet is based on the implementation of the code at https://github.com/mhlevgen/DCUNetTorchSound.
Revised: • DCUnet : the experimental results of DCUnet is based on the implementation of the code at https://github.com/mhlevgen/DCUNetTorchSound .

Original: We choose the Large-DCUnetmodel which realizes the 20-layer model in the original DCUnet paper.
Revised: We choose the LargeDCUnet-20 model which realizes the 20-layer model in the original DCUnet paper.

Original: Note that although in the original system a 1024-point FFT was used, here we use 512-point FFT instead.
Revised: Note that although in the original system a 1024-point FFT was used, here we use 512-point FFT instead.

Original: • FaSNet : the experimental results of FaSNet is based on the implementation of the code at: https://github.com/yluo42/TAC.
Revised: • FaSNet : the experimental results of FaSNet is based on the implementation of the code at: https://github.com/yluo42/TAC .

Original: We choose to use the FaSNet TAC model where TAC stands for transform-average-concatenate.
Revised: We choose to use the FaSNet TAC model where TAC stands for transform-average-concatenate.

Revised: Hyperparameters: enc dim= 64 , feature dim= 64 , hidden dim= 128 , layer= 4 , segment size= 50 , nspk= 1 , win len= 4 , context len= 16 , sr= 16000 .

Original: Model optimization: All experiments were run on one NVIDIA Tesla V100 GPU of 32 GB CUDA memory
Revised: Model optimization: All experiments were run on one NVIDIA Tesla V100 GPU of 32 GB CUDA memory.

Original: • AFnet training: the AFnet is trained via the RTF-aware training scheme.
Revised: • AFnet training: the AFnet is trained via the RTF-aware training scheme.

Original: The Adam optimizer is used for minimizing L rtf with a total of 80 epochs, where for the firstepochs a learning rate of 0.001 is used and for the rest 30 epochs it is decreased to 0.0001.
Revised: The Adam optimizer is used for minimizing L rtf with a total of 80 epochs, where for the firstepochs a learning rate of 0.001 is used and for the rest 30 epochs it is decreased to 0.0001.

Original: For minimizing L rec we train the network for another 55 epochs with Adam, where for the first 50 epochs a learning rate of 0.001 is used and for the rest 5 epochs it is decreased to 0.0001.
Revised: For minimizing L rec we train the network for another 55 epochs with Adam, where for the first 50 epochs a learning rate of 0.001 is used and for the rest 5 epochs it is decreased to 0.0001.

Original: The total number of epochs is 80+55=135.
Revised: The total number of epochs is 80+55=135.

Original:  • AFnet training (w/o RTF supervision): the AFnet trained without RTF loss is just minimizing L rec .
Revised: A batch size of 4 is used. • AFnet training (w/o RTF supervision): the AFnet trained without RTF loss is just minimizing L rec .

Original: In this case, the Adam optimizer with a learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with a total of 80 epochs. 
Revised: In this case, the Adam optimizer with a learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with a total of 80 epochs. A batch size ofis used.

Original: • Conv-TasNet, DCUnet, and FaSNet training: for fair comparison purposes, the training of the SOTA methods is same by minimizing the reconstruction loss L rec using 512-point FFT with a hop size of 256 via the Hann window (same STFT setting as AFnet).
Revised: • Conv-TasNet, DCUnet, and FaSNet training: for fair comparison purposes, the training of the SOTA methods is same by minimizing the reconstruction loss L rec using 512-point FFT with a hop size of 256 via the Hann window (same STFT setting as AFnet).

Original: For the time-domain approaches (Conv-TasNet and FaSnet), the waveform of the model output is converted to STFTs for computing the loss.
Revised: For the time-domain approaches (Conv-TasNet and FaSnet), the waveform of the model output is converted to STFTs for computing the loss.

Original: In all cases, the Adam optimizer with a learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with a total of 80 epochs. 
Revised: In all cases, the Adam optimizer with a learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with a total of 80 epochs. A batch size of 4 is used.

Original: A. CH I ME-3 DATASET
Revised: B. CH I ME-3 DATASET

Original: Dataset: The publicly available CHiME-3 dataset (Barker et al., 2015), made available as part of a speech separation and recognition challenge, is used for training and evaluating SE performance.
Revised: Dataset: The publicly available CHiME-3 dataset (Barker et al., 2015), made available as part of a speech separation and recognition challenge, is used for training and evaluating SE performance.

Original: The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled at 16 kHz.
Revised: The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled at 16 kHz.

Original: It consists of 7,138 and 1,320 simulated utterances with an average length of 3 seconds for training and testing, respectively.
Revised: It consists of 7,138 and 1,320 simulated utterances with an average length of 3 seconds for training and testing, respectively.

Original: SOTAs to compare:
Revised: SOTAs to compare:

Original: • Nerual BF (Erdogan et al., 2016):
Revised: • Neural BF (Erdogan et al., 2016):

Original: An MVDR beamforming with mask estimation through bidirectional-LSTM.
Revised: An MVDR beamforming with mask estimation through bidirectional-LSTM.

Original: • MVDR GC (Bu et al., 2019):
Revised: • MVDR GC (Bu et al., 2019):

Original: An MVDR beamforming using a neural network based method to identify and correct phase errors in the steering vector.
Revised: An MVDR beamforming using a neural network-based method to identify and correct phase errors in the steering vector.

Original: • rSDFCN (Liu et al., 2020):
Revised: • rSDFCN (Liu et al., 2020):

Original: A time-domain, fully convolutional network (FCN) with sinc and dilated convolutional layers for multichannel SE.
Revised: A time-domain, fully convolutional network (FCN) with sinc and dilated convolutional layers for multichannel SE.

Original: • CA Dense U-Net
Revised: • CA Dense U-Net

Original: (Tolooshams et al., 2020):
Revised: (Tolooshams et al., 2020):

Original: A time-frequnecy domain multichannel SE model that combines the merits of DenseNet, U-Net, and channel attention (CA) mechanism.
Revised: A time-frequnecy domain multichannel SE model that combines the merits of DenseNet, U-Net, and channel attention (CA) mechanism.

Original: • IC Conv-TasNet
Revised: • IC Conv-TasNet

Original: (Lee et al., 2021):
Revised: (Lee et al., 2021):

Original: The Inter-Channel (IC) Conv-TasNet is the extension of the time-domain Conv-TasNet for single-channel SE to the multichannel SE case.
Revised: The Inter-Channel (IC) Conv-TasNet is the extension of the time-domain Conv-TasNet for single-channel SE to the multichannel SE case.

Original: Spectral processing We use the Hann window with 1024-point FFT and a hop size of 256 for the STFT
Revised: Spectral processing: We use the Hann window with 1024-point FFT and a hop size of 256 for the STFT

Original: By conjugate symmetry of the FFT, only half of the frequency bins (i.e., 513) are
Revised: By conjugate symmetry of the FFT, only half of the frequency bins ( i.e.

Original:  actually
Revised: are actually

Original: needed for the network to process the spectrograms
Revised: needed for the network to process the spectrograms.

Original: Model optimization: The AFnet training was run on one NVIDIA Tesla V100 GPU of 32 GB
Revised: Model optimization: The AFnet training was run on one NVIDIA Tesla V100 GPU of 32 GB

Original: CUDA memory using the same model optimization scheme discussed in Section A.1.
Revised: CUDA memory using the same model optimization scheme discussed in Section B.2.

Original: A. E VALUATION METRICS
Revised: B. E VALUATION METRICS

Original: PESQ: a speech quality measure using the wide-band version recommended in ITU-T P.862.2.
Revised: PESQ: a speech quality measure using the wide-band version recommended in ITU-T P.862.2.

Original: It basically models the mean opinion scores (MOS) that cover a scale from 1 (bad) to 5 (excellent).
Revised: It basically models the mean opinion scores (MOS) that cover a scale from 1 (bad) to 5 (excellent).

Original: We use the Python-based PESQ implementation from: https://github.com/ludlows/python-pesq.
Revised: We use the Python-based PESQ implementation from: https://github.com/ludlows/ python-pesq .

Original: STOI: a function that well represents the average intelligibility of the degraded speech.
Revised: STOI: a function that well represents the average intelligibility of the degraded speech.

Original: It provides a value from 0 to 1, which can be interpreted as the percentage of correctly recognized words by normal-hearing people.
Revised: It provides a value from 0 to 1, which can be interpreted as the percentage of correctly recognized words by normal-hearing people.

Original: We use the Python-based STOI implementation from: https://github.com/mpariente/pystoi.
Revised: We use the Python-based STOI implementation from: https://github.com/mpariente/pystoi .

Original: SSNR: an SNR measure, instead of working on the whole signal, that calculates the average of the SNR values of short segments.
Revised: SSNR: an SNR measure, instead of working on the whole signal, that calculates the average of the SNR values of short segments (segment length = 30 msec, 75% overlap, SNR min = − 10 dB, SNR max = 35 dB).

Original: We use the Python-based SSNR implementation from: https://github.com/schmiph2/pysepm.
Revised: We use the Python-based SSNR implementation from: https://github. com/schmiph2/pysepm .

Original: B A DDITIONAL EXPERIMENTAL RESULTS B. I MPORTANCE OF MAGNITUDE AND PHASE ESTIMATION IN RTF ALIGNMENT
Revised: A PPENDIX C A DDITIONAL EXPERIMENTAL RESULTS C. C OMPARISON OF RTF- AWARE TRAINING SCHEMES ( TWO - STEP VS . REGULARIZED )

Original: It would be interesting to look at how the magnitude and phase components of  the RTFs, which correspond to ILD and ITD in time domain, affect the enhancement performance of the AFnet.
Revised: In this section we try to answer whether the two-step RTF-aware training scheme (Algorithm 1) or the regularized training scheme (Algorithm 2) is better. To this end, we train the AFnet model with several values of λ using Algorithm 2 and the results are compared to AFnet trained with Algorithm 1 in Table 5. It can be seen that both training schemes help improve over the RTF-agnostic results (i.e., λ = 0 ) as they both incorporate RTF information for explicitly learning spatial characteristics. However, the two-step approach performs consistently better than the regularized approach.

Revised: The results suggest that learning to align prior to learning to denoise during the training stage is more beneficial.

Revised: [Table]

Revised: [Table]

Revised: Algorithm 2, λ =

Revised: [Table]

Revised: [Table]

Revised: Algorithm 2, λ =

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Algorithm 2, λ =

Revised: [Table]

Revised: Algorithm 2,

Original: In Figure 5 we compare four different RTF alignment schemes  – no alignment, aligning RTF magnitude only, aligning RTF phase only, and aligning everything for the RTF loss L rtf  .
Revised: [Table] [Table] C. F IXING VS . UNFIXING A LIGN N ET WHILE MINIMIZING RECONSTRUCTION LOSS We compare two additional training schemes of the AFnet – fixing or unfixing the Align Net parameters after it has been trained via the RTF loss L rtf , when we train the full AFnet for the reconstruction loss L rec .

Original: From the results it can be seen that phase alignment is  the key to obtaining improved performance over the alignment-agnostic system.
Revised: Table 6 presents the results. It can be seen that adjusting the Align Net parameters together with the Filter Net parameters for optimizing the reconstruction loss is crucial for achieving better performance compared to fixing the Align Net after the RTF loss minimization phase. This indicates that both stages may share the task of performing spatial and spectral denoising jointly, where the first stage shares more loading on the spatial denoising part while the second stage is more on the spectral filtering portion. Dividing the SE task into the two subtasks makes the network learn to more efficiently denoise the speech.

Revised: [Table]

Original: This is reasonable as aligning phase in the frequency domain corresponds to compensating the delay in the time domain, which is crucial for any beamforming algorithms that rely on  spatial separability.
Revised: Given the ability of AFnet to exploit spatial separability, it is also interesting to utilize the AFnet for inspecting the spatial diversity of the dataset. Such information can be provided by observing the alignment masks of the data points. In Figure 5 we compare the distributions of the intermediate alignment masks learned by the AFnet models trained on the primary dataset (high spatial diversity) and the CHiME-3 dataset (low spatial variety).

Original: The results suggest that the AFnet indeed exploits spatial information for improved enhancement performance.
Revised: Here, we plot the distributions of the alignment masks estimated by the trained models for all the test data samples of the two datasets. It can be seen that for the primary dataset, the RTF-aware training leads to a more widespread distribution than that of the normal training case, corresponding to the greater spatial diversity of the target speech source positions. In contrast, in the CHiME-3 dataset the RTF-aware training actually results in a more aggregated cluster than the normal training scheme, indicating that the target speech sources are coming from similar directions. This visualization of the alignment masks provides another angle to inspect the data properties from the spatial diversity aspect for exploring SE performance.

Original: [Figure]
Revised: [Figure]

Original: B. E NHANCEMENT PERFORMANCE VS . INPUT SNR CONDITION COMPARISON
Revised: C. E NHANCEMENT PERFORMANCE VS . INPUT SNR CONDITION COMPARISON

Original: To look more into the testing results, we present the PESQ, STOI, and SSNR performance numbers vs. input SNR conditions in Figure 6 for the three schemes:
Revised: To look more into the testing results, we present the PESQ, STOI, and SSNR performance numbers vs. input SNR conditions in Figure 6 for the three schemes:

Original: i)
Revised: i)

Original: AFnet: the two-stage AFnet model trained with the RTF-aware training scheme ii) AFnet w/o RTF loss: the two-stage AFnet model trained with normal training scheme without incorporating RTFs iii) W-Net for filter weights: using a W-Net (same model as the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to  estimate the  filter weights W i in typical(one-stage) multichannel SE
Revised: AFnet: the two-stage AFnet model trained with the RTF-aware training scheme ii) AFnet w/o RTF loss: the two-stage AFnet model trained with normal training scheme without incorporating RTFs iii) W-Net for filter weights: using a W-Net (same model as the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to directly estimate the sets of filter weights W i in typical multichannel SE

Original: We take 160 utterances for each specified input SNR value from the test set and obtain the corresponding performance number of each input SNR value.
Revised: We take 160 utterances for each specified input SNR value from the test set and obtain the corresponding performance number of each input SNR value.

Original: The results for the 8-mic systems are presented.
Revised: The results for the 8-mic systems are presented.

Original: From the results we can see that the proposed AFnet consistently improves the baseline method over a wide range of input SNR conditions. 
Revised: From the results we can see that the proposed AFnet consistently improves the baseline method over a wide range of input SNR conditions. [Figure]

Revised: C. G ENERALIZATION TO OTHER ROOM CONFIGURATIONS

Revised: To further test the generalization of the proposed AFnet to unseen room configurations

Original: [Figure]
Revised: [Figure]

Revised: we utilize the Pyroomacoustic simulator to generate test data randomly sampled from three different room sizes: ( 8 × 5 × 3 ), ( 7 × 6 × 4 ), and ( 8 × 6 × 3 ) in terms of (length × width × height) in meters

Revised: [Figure]

Revised: which are different from the training room

Revised: (i

Revised: [Figure]

Original: C A LIGN N ET VS . CONVENTIONAL SIGNAL PROCESSING APPROACHES FOR RTF ESTIMATION
Revised: e., ( 8 × 8 × 3 )). Figure 7 presents the SE results under the unseen room configurations. We can see the the proposed AFnet generalizes well to unseen room schemes as it still improves over the typical direct filter weight estimation approaches in most cases. Moreover

Revised: [Figure]

Revised: the performance of AFnet consistently improves as the number of microphones increases

Revised: [Figure]

Revised: C. SE UNDER TIME - VARYING RIR SCENARIOS

Revised: The above results have been tested on a relatively static environments where the target speech position of an utterance, once given, is not changing.

Revised: However, in most real-world scenarios the speaker may naturally move around and therefore the RIRs will not always be time-invariant.

Revised: To test the proposed AFnet under such circumstances, we leverage the Pyroomacoustic simulator to generate another set of test data (same amount of data as the static RIR case in the previous sections) in which during an utterance the RIRs are changed from one set to another to emulate speaker movements.

Revised: We evaluate the algorithms on such time-varying RIR generated multichannel dataset and the results are presented in Figure 8.

Revised: Note that the testing is performed on the same models that have been trained only with the data generated with static RIRs before.

Revised: From the results we can see that the proposed

Revised: AFnet has a better tracking ability to tackle time-varying target speech locations, even though the model has not observed any changing RIR data during the training stage.

Revised: This indicates that the RTF supervision of the alignment masks is also crucial for SE under time-varying RIR environments – by incorporating phase alignment into the model learning, the positional information of the target speech can be better captured for improved SE even under the moving target scenario.

Revised: C. R ESULTS ON REAL - WORLD MEASURED RIR GENERATED DATA

Revised: The results presented in previous sections on the primary dataset are based on the multichannel audio data synthesized by using the image source method via the Pyroomacoustics library, which is useful for methodological study but also has its limitation to represent real data.

Revised: To further validate the proposed approach’s generalization capabilities to more realistic scenarios, we also conduct

Revised: 2-Mic 4-Mic 8-Mic 1.

Revised: P E S Q

Revised: U-

Revised: [Figure]

Revised: W-

Revised: [Figure]

Revised: AFnet w/

Revised: [Figure]

Revised: RTF

Revised: [Figure]

Revised: AFnet 2-Mic 4-Mic 8-Mic 0.

Revised: S T O I

Revised: U

Revised: [Figure]

Revised: AFnet w/o RTF

Revised: [Figure]

Revised: AFnet 2-Mic 4-Mic 8-Mic 4.

Revised: [Figure]

Revised: [Figure]

Revised: experiments on multichannel audio data generated using real-world measured acoustic RIRs taken from the work of Hadad et al.

Revised: (2014), where the impulse responses of several 8-channel linear microphone arrays with respect to various target source directions of arrival (DoAs) (from -90 todegrees) are provided.

Revised: We utilize the 3-3-3-8-3-3-3 array impulse responses of the dataset including reverberation time of 0.16, 0.36, and 0.61 second cases, together with the AVSpeech utterances and the YouTube noise profiles to generate the multichannel noisy data for experiments.

Revised: To observe the relation of the AFnet performance with the speech source spatial diversity, we generate a spatially diverse dataset where the speech DoA ranges from -90 to 90 degrees, and another spatially constant dataset where the speech comes from only one direction.

Revised: Table 7 and Table 8 present the results for the two datasets respectively.

Revised: One can see that for for the spatially diverse dataset of Table 7, the AFnet clearly benefits from the sequentially masking mechanism and the RTF loss supervision, while for the spatially constant dataset of Table 8 the three models perform comparably with the RTF loss marginally improves AFnet.

Revised: These results show that a model (such as the W-Net) that performs well on a dataset lacking spatial variety of the target speech may not generalize well to a spatially diverse dataset, unless further attention is paid to exploiting spatial characteristics of the speech sources.

Original: In this section, we study how effective the deep leaning based Align Net for estimating the RTFs compared to conventional model-based signal processing methods in  the presence of noise.
Revised: The RTF information leveraged by the twostage sequential masking design of the AFnet is shown to take advantage of the spatial separability for enhanced SE performance on spatially diverse datasets. [Table] [Table] Similar to what have been done in Figure 3 for the simulated RIR case, we also visualize the estimated alignment masks for the spatially diverse dataset of the realistic RIR case.

Revised: The results in

Revised: Figure 9 once again show that the RTF-aware training results in clear clusters of the speech sources coming from difference directions as compared to the normal training case, confirming that the network is more aware of the spatial separability of the sound sources.

Revised: A PPENDIX D A LIGN N ET VS . CONVENTIONAL SIGNAL PROCESSING APPROACHES FOR RTF ESTIMATION

Revised: In this section, we study the effectiveness of the deep leaning-based Align Net for estimating the

Original:  We show that the Align Net performs reasonably well in the low SNR regime where signal processing approaches fail to, justifying its usage of alignment purposes for later enhancement processing in
Revised: RTFs compared to conventional model-based signal processing methods in the presence of noise. We show that the Align Net performs reasonably well in the low SNR regime where signal processing approaches fail to, justifying its usage of alignment − −   4 PC- − −

Original: AFnet.
Revised: P C -

Original: To our best knowledge, it is the first work that utilizes the neural network for direct RTF estimation in the SE task 1 .
Revised: [Figure] With RTF Supervision speech at loc 1 speech at locspeech at loc− −     4 PC- − − P CNormal Training speech at loc 1 speech at locspeech at loc

Original: C. RTF ESTIMATION PROBLEM
Revised: D. RTF ESTIMATION PROBLEM

Original: RTFs or their time-domain counterparts, relative impulse responses (ReIRs) are important tools in several multichannel audio processing tasks (Gannot & Cohen, 2004; Laufer et al., 2013).
Revised: RTFs or their time-domain counterparts, relative impulse responses (ReIRs) are important tools in several multichannel audio processing tasks (Gannot & Cohen, 2004; Laufer et al., 2013).

Original: The estimation of RTFs has been studied for a while in the audio signal processing field (Gannot et al., 2001;
Revised: The estimation of RTFs has been studied for a while in the audio signal processing field (Gannot et al., 2001;

Original: Koldovsk ` y et al., 2015; Giri et al., 2016; 2018; Srikrishnan et al., 2018), where the task is to estimate the correlation between the target source components received by the two microphones, either in terms of RTFs or ReIRs, given the noisy recordings.
Revised: Koldovsk ` y et al., 2015; Giri et al., 2016; 2018; Srikrishnan et al., 2018), where the task is to estimate the correlation between the target source components received by the two microphones, either in terms of RTFs or ReIRs, given the noisy recordings.

Original: Following the previous works, we will discuss the scenario with two microphones (i.e., left and right microphones) while the idea may be generalized to more microphones.
Revised: Following the previous works, we will discuss the scenario with two microphones (i.e., left and right microphones) while the idea may be generalized to more microphones.

Original: Consider a two-channel noisy recording of a target speech in a noisy environment, whose position is fixed for a certain time interval.
Revised: Consider a two-channel noisy recording of a target speech in a noisy environment, whose position is fixed for a certain time interval.

Original: This situation can be represented as:
Revised: This situation can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where n is the time sample index taking values 1 , . .
Revised: where n is the time sample index taking values 1 , . .

Original: , N ; ∗ denotes the convolution; x L and x R are, respectively, the signals from the left ( L ) and right ( R ) microphones; h L and h R are the impulse responses between the target and the two microphones; s is the far-field target speech; and v L and v R are the noise components.
Revised: , N ; ∗ denotes the convolution; x L and x R are, respectively, the signals from the left ( L ) and right ( R ) microphones; h L and h R are the impulse responses between the target and the two microphones; s is the far-field target speech; and v L and v R are the noise components.

Original: Let h rel represent the ReIR between the speech signal components arriving at the two microphones, we have following relation:
Revised: Let h rel represent the ReIR between the speech signal components arriving at the two microphones, we have following relation:

Original: [Equation]
Revised: [Equation]

Original: The main goal is to estimate h rel given x L and x R .
Revised: The main goal is to estimate h rel given x L and x R .

Original: The issue is in the presence of noise v the estimation becomes challenging.
Revised: The issue is in the presence of noise v the estimation becomes challenging.

Original: The oracle ReIR is given as h rel =
Revised: The oracle ReIR is given as h rel =

Original: h L ∗
Revised: h L ∗

Original: h − 1 R , where h − 1 R denotes the filter inverse to h R .
Revised: h − 1 R , where h − 1 R denotes the filter inverse to h R .

Original: To ensure that the solution is causal, a fixed delay of a few milliseconds can be introduced (Lin et al., 2007; Koldovsk`y et al., 2013), i.e., h rel =
Revised: To ensure that the solution is causal, a fixed delay of a few milliseconds can be introduced (Lin et al., 2007; Koldovsk`y et al., 2013), i.e., h rel =

Original: h L ∗
Revised: h L ∗

Original: h − 1 R ∗ δ ( n − D ) , where δ ( · ) is the unit impulse function and D is the delay in samples.
Revised: h − 1 R ∗ δ ( n − D ) , where δ ( · ) is the unit impulse function and D is the delay in samples.

Original: In the STFT domain, (4) can be equivalent to: 
Revised: In the STFT domain, (12) can be equivalent to: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the oracle RTF is given by:
Revised: where the oracle RTF is given by:

Original: [Equation]
Revised: [Equation]

Original: The goal again is to estimate H rel given the microphone signals X L and X R .
Revised: The goal again is to estimate H rel given the microphone signals X L and X R .

Original: Note that under the assumption that the source position is fixed, we have the transfer functions independent of the time (thus omitting the index t ).
Revised: Note that under the assumption that the source position is fixed, we have the transfer functions independent of the time (thus omitting the index t ).

Original: C. M ETHODS Signal processing based methods
Revised: D. M ETHODS Signal processing-based methods

Original: • Frequency-domain approaches

Original:  The RTF estimation problem can be addressed in the frequency domain by utilizing signal and noise statistics,i.
Revised: • Frequency-domain approaches: The RTF estimation problem can be addressed in the frequency domain by utilizing signal and noise statistics,

Original: e.
Revised: i

Original: , using the power spectral density (PSD) entities.
Revised: e., using the power spectral density (PSD) entities.

Original: To deal with noise, Gannot et al.
Revised: To deal with noise, Gannot et al.

Original: (2001) propose the non-stationarity based frequency domain (NSFD) method relying on the assumption that noise signals are stationary, or less dynamic, when compared to the target speech signal.
Revised: (2001) propose the non-stationarity based frequency domain (NSFD) method relying on the assumption that noise signals are stationary, or less dynamic, when compared to the target speech signal.

Original: To be more exact, let Φ AB ( f, t ) denote the (cross)-PSD between A and B during the t -th frame, we have:
Revised: To be more exact , let Φ AB ( f, t ) denote the (cross)-PSD between A and B during the t -th frame, we have:

Original: [Equation]
Revised: [Equation]

Original: Assuming over t = 1 , . .
Revised: Assuming over t = 1 , . .

Original: . , P frames the the noise is stationary, we can write Φ V X R ( f, t ) =
Revised: . , P frames the the noise is stationary, we can write Φ V X R ( f, t ) =

Original: Φ V X R ( f ) and solve the overdetermined set of equations:
Revised: Φ V X R ( f ) and solve the overdetermined set of equations:

Original: [Equation]
Revised: [Equation]

Original: In practice the PSDs in the above set of equations are replaced by their sample estimates.
Revised: In practice the PSDs in the above set of equations are replaced by their sample estimates.

Original: • Time-domain approaches: The ReIR estimation problem can be formulated in the time domain as:
Revised: • Time-domain approaches: The ReIR estimation problem can be formulated in the time domain as:

Original: h rel =
Revised: h rel =

Original: arg min h ∈ R M L ( x L , X R h ) , where x L = [ x L (1 − D ) , . . .
Revised: arg min h ∈ R M L ( x L , X R h ) , where x L = [ x L (1 − D ) , . . .

Original: , x L ( N − D )]
Revised: , x L ( N − D )]

Original: T and D is an integer delay for causality, and X R is the convolution matrix of dimensions N × M constructed from x R .
Revised: T and D is an integer delay for causality, and X R is the convolution matrix of dimensions N × M constructed from x R .

Original: When the cost function L ( · , · ) is the squared Euclidean distance, the problem can be solved by least squares (noise-free) or regularized least squares (noisy) methods.
Revised: When the cost function L ( · , · ) is the squared Euclidean distance, the problem can be solved by least squares (noise-free) or regularized least squares (noisy) methods.

Original: To better handle the noise, prior knowledge about the ReIR can be leveraged to improve the estimation, e.g., utilizing the structural sparsity of ReIRs.
Revised: To better handle the noise, prior knowledge about the ReIR can be leveraged to improve the estimation, e.g., utilizing the structural sparsity of ReIRs.

Original: One popular approach is the weighted ℓ 1 approach (Benichoux et al., 2014; Koldovsk`y et al., 2015; Giri et al., 2018):
Revised: One popular approach is the weighted ℓ 1 approach (Benichoux et al., 2014; Koldovsk ` y et al., 2015; Giri et al., 2018):

Original: [Equation]
Revised: [Equation]

Original:  where w = [ w 1 , . . 
Revised: (17) where w = [ w 1 , . . .

Original: . , w M ] T is a vector of non-negative weights and ⊙ denotes the Hadamard product.
Revised: , w M ] T is a vector of non-negative weights and ⊙ denotes the Hadamard product.

Original: To mimic the expected structure of a ReIR, the weights are chosen as follows:
Revised: To mimic the expected structure of a ReIR, the weights are chosen as follows:

Original: [Equation]
Revised: [Equation]

Original:  where k 1 , k 2 , and k 3 are positive constants and D is the integer delay.
Revised: 18) where k 1 , k 2 , and k 3 are positive constants and D is the integer delay.

Original: Deep learning based approach (proposed Align Net):
Revised: Deep learning-based approach (proposed Align Net):

Original: In the proposed AFnet, the first half Align
Revised: In the proposed AFnet, the first half Align

Original: Net is utilized to perform signal alignment by estimating RTFs, which can be viewed as a deep learning based RTF estimation method.
Revised: Net is utilized to perform signal alignment by estimating RTFs, which can be viewed as a deep learning-based RTF estimation method.

Original: By exploiting the power of complex-valued deep networks for predicting the RTFs, the data-drivenapproach Align Net  can be shown to outperform conventional model-based signal processing approaches under noisy environments.
Revised: By exploiting the power of complex-valued deep networks for predicting the RTFs, the data-driven Align Net approach can be shown to outperform conventional model-based signal processing methods under noisy environments.

Original: C. E XPERIMENTS
Revised: D. E XPERIMENTS

Original: Settings: We generate 2-mic noisy data using Pyroomacoustics library for conducting the RTF estimation experiments.
Revised: Settings: We generate 2-mic noisy data using Pyroomacoustics library for conducting the RTF estimation experiments.

Original: We create a room of size 8 × 8 × 3 (length × width × height in meters).
Revised: We create a room of size 8 × 8 × 3 (length × width × height in meters).

Original: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.
Revised: Reverberation time is randomly chosen from { 0.16, 0.32, 0.48, 0.64 } second.

Original: The simulating approach for estimating room impulse responses is based on the image source method.
Revised: The simulating approach for estimating RIRs is based on the image source method.

Original: The two microphones are placed with a distance of 15 cm in between, where the center is placed at a distance of 1 meter to one of the walls, with a height of 1.525 meters, and the array is positioned horizontally to the wall with equal distance to the two sides.
Revised: The two microphones are placed with a distance of 15 cm in between, where the center is placed at a distance of 1 meter to one of the walls, with a height of 1.525 meters, and the array is positioned horizontally to the wall with equal distance to the two sides.

Original: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array.
Revised: Speech and noise signals are randomly positioned at a distance of { 1, 2, 3, 4 } meters from the microphone array.

Original: 649 files from the training set of AVSpeech dataset are used for training the Align Net and 160 files from the testing set are used for evaluating the competing algorithms (i.e., Align Net, time-domain weighted ℓ 1 approach, and NSFD approach).
Revised: 649 files from the training set of AVSpeech dataset are used for training the Align Net and 160 files from the testing set are used for evaluating the competing algorithms (i.e., Align Net, time-domain weighted ℓ 1 approach, and NSFD approach).

Original: 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) are used for testing.
Revised: 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) are used for testing.

Original: Each type of noise is multiplied with a scale randomly chosen from { 0, 0.5, 1, 1.5, 2 } before added up, and the combined noise is then added to the clean speech according to a specific SNR level randomly selected from { -10, -6, -3, 0, 3, 6, 10 } dB for training and { 0, 5, 10, 15 } dB for testing.
Revised: Each type of noise is multiplied with a scale randomly chosen from { 0, 0.5, 1, 1.5, 2 } before added up, and the combined noise is then added to the clean speech according to a specific SNR level randomly selected from { -10, -6, -3, 0, 3, 6, 10 } dB for training and { 0, 5, 10, 15 } dB for testing.

Original: For the freqency-domain approaches (Align Net and NSFD) we use the Hann window with 512point FFT and a hop size of 256 for the STFT.
Revised: For the freqency-domain approaches (Align Net and NSFD) we use the Hann window with 512point FFT and a hop size of 256 for the STFT.

Original: For Align Net we use C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 ,and train for the RTF loss L rtf .
Revised: For Align Net we use C 1 , C 2 , C 3 , C 4 = 32 , 64 , 64 ,and train for the RTF loss L rtf .

Original: For NSDF we use 250 frames as a processing block and smoothing over 5 frames for computing the PSD entities.
Revised: For NSDF we use 250 frames as a processing block and smoothing over 5 frames for computing the PSD entities.

Original: For the time-domain weighted ℓ 1 we use k 1 = 0 .
Revised: For the time-domain weighted ℓ 1 we use k 1 = 0 .

Original: 1 , k 2 = 0 .
Revised: 1 , k 2 = 0 .

Original: 11 and k 3 = 0 .
Revised: 11 and k 3 = 0 .

Original: 3 with λ = 0 . 0001 ; and a ReIR length of M = 1024 for each processing block of 2048 samples.
Revised: 3 with λ = 0 . 0001 ; and a ReIR length of M = 1024 for each processing block of 2048 samples.

Original: Evaluation metric:
Revised: Evaluation metric:

Original: To quantitatively evaluate the competing algorithms, we use a well-known and widely used performance metric called the Attenuation Rate (ATR) (Koldovsk ` y et al., 2015), which can be evaluated as the ratio between SNR out and SNR in in dB scale, where:
Revised: To quantitatively evaluate the competing algorithms, we use a well-known and widely used performance metric called the Attenuation Rate (ATR) (Koldovsk`y et al., 2015), which can be evaluated as the ratio between SNR out and SNR in in dB scale, where:

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: 19) and

Original: [Equation] 
Revised: [Equation] [Equation]

Original: The numerator of SNR out measures the leakage of the target signal whereas the denominator measures the attenuation of the noise signal.
Revised: The numerator of SNR out measures the leakage of the target signal whereas the denominator measures the attenuation of the noise signal.

Original: The more negative the value (in dB) of ATR is, the better the evaluated algorithm performs.
Revised: The more negative the value (in dB) of ATR is, the better the evaluated algorithm performs.

Original: Results: The ATR results are presented in Table 6.
Revised: Results: The ATR results are presented in Table 9.

Original: It can be seen that for all the methods the performance degrades as SNR decreases, since it becomes more challenging to estimate the RTF when noise is stronger.
Revised: It can be seen that for all the methods the performance degrades as SNR decreases, since it becomes more challenging to estimate the RTF when noise is stronger.

Original: However, the proposed Align Net significantly outperforms the conventional model-based approaches over all SNR settings, indicating the effectiveness of the data-driven approach for the RTF estimation task against existing methods.
Revised: However, the proposed Align Net significantly outperforms the conventional model-based approaches over all SNR settings, indicating the effectiveness of the data-driven approach for the RTF estimation task against existing methods.

Original: In future study, improving the design of Align Net for better RTF estimation as well as its applications to other audio processing tasks (e.g., sound localization, speech separation, etc.) can be interesting topics to explore.
Revised: In future study, improving the design of Align Net for better RTF estimation as well as its applications to other audio processing tasks (e.g., sound localization, speech separation, etc.) can be interesting topics to explore.

Original: [Table]
Revised: [Table]