  Existing speech enhancement methods mainly separate speech from noises at the
signal level or in the time-frequency domain. They seldom pay attention to the
semantic information of a corrupted signal. In this paper, we aim to bridge
this gap by extracting phoneme identities to help speech enhancement.
Specifically, we propose a phoneme-based distribution regularization (PbDr) for
speech enhancement, which incorporates frame-wise phoneme information into
speech enhancement network in a conditional manner. As different phonemes
always lead to different feature distributions in frequency, we propose to
learn a parameter pair, i.e. scale and bias, through a phoneme classification
vector to modulate the speech enhancement network. The modulation parameter
pair includes not only frame-wise but also frequency-wise conditions, which
effectively map features to phoneme-related distributions. In this way, we
explicitly regularize speech enhancement features by recognition vectors.
Experiments on public datasets demonstrate that the proposed PbDr module can
not only boost the perceptual quality for speech enhancement but also the
recognition accuracy of an ASR system on the enhanced speech. This PbDr module
could be readily incorporated into other speech enhancement networks as well.
