Improving Generative Adversarial Networks for Speech Enhancement through Regularization of Latent Representations Abstract   Speech enhancement aims to improve the quality and intelligibility of speech signals, which is a challenging task in adverse environments. Speech enhancement generative adversarial network (SEGAN) that adopted a generative adversarial network (GAN) for speech enhancement achieved promising results. In this paper, a new network architecture and loss function based on SEGAN are proposed for speech enhancement. Different from most network structures applied in this field, the new network, called highlevel GAN (HLGAN), uses parallel noisy and clean speech signals as input in the training phase instead of only noisy speech signals, which enables us to make full use of the information carried by the clean speech signals. Additionally, we introduce a new supervised speech representation loss, also known as highlevel loss, in the middle hidden layer of the generative network. The highlevel loss function is advantageous to HLGAN in speech enhancement under low signaltonoise (SNR) environments and lowresource environments. We evaluate the performance of HLGAN over a wide range of experiments, in which our model produces significant improvements. Extensive experiments further demonstrate the generality of our model in a variety of speech enhancement cases. The issue of SEGAN losing speech components while removing noise in low SNR environments is improved. In addition, HLGAN can effectively enhance the speech signals of two lowresource languages simultaneously. The reasons for the superior performance of HLGAN are discussed.

Multiscale Generative Adversarial Networks for Speech Enhancement The generative adversarial networks can be used to recognize and eliminate noise from noisy speech after extensive training. The most representative model is Speech Enhancement Generative Adversarial Network (SEGAN). However, eliminating the noise without distortion is still a challenging task especially in a low SNR environment. To solve such problems, this paper proposes Speech Enhancement Multiscale Generative Adversarial Networks (SEMGAN), whose generator and discriminator networks are structured on the basis of fully convolutional neural networks (FCNNs). Compared with SEGAN, the generator generates speeches in three different dimensions and makes multiple judgments in the discriminator. In addition, multiple types of noise and signalnoise ratios (SNRs) are used to train our model for improving the generalization capability. In the stage of testing, we further propose pre SEMGAN, which solve the problem that the last frame of speech data was not processed well. As the experimental results indicated, the architecture (SEMGAN and pre SEMGAN) proposed gain a superior performance in comparison with the optimally modified logspectral amplitude estimator (OMLSA) and SEGAN in different noisy conditions. It is worth mentioning that SEMGANu0027s PESQ and STOI score increase about 7% and 3.6% over SEGAN respectively in the case of 2.5 dB SNR.