We introduce Wavesplit, an end-to-end speech separation system. From a single recording of mixed speech , the model infers and clusters representations of each speaker and then estimates each source signal conditioned on the inferred representations. The model is trained on the raw waveform to jointly perform the two tasks . Our model infers a set of speaker representations through clustering, which addresses the fundamental permutation problem of speech separation. Moreover, the sequence-wide speaker representations provide a more robust separation of long, challenging sequences, compared to previous approaches. We show that Wavesplit outperforms the previous state-of-the-art on clean mixtures of 2 or 3 speakers ( WSJ0-2mix, WSJ0-3mix ), as well as in noisy (WHAM!) and reverberated (WHAMR!)conditions. As an additional contribution, we further improve our model by introducing online data augmentation for separation .
We introduce Wavesplit, an end-to-end source separation system. From a single mixture , the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers ( WSJ0-2/3mix ), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram .
R
speech
source
R
recording of mixed speech
mixture
R
and clusters representations of each speaker
a representation for each source
R
conditioned on
given
D
on the raw waveform
R
the two tasks . Our model
both tasks from the raw waveform. Wavesplit
R
speaker representations through
source representations via
R
speech separation. Moreover, the
separation. For speech separation, our
R
sequences, compared to previous approaches. We show that Wavesplit outperforms the previous
recordings compared to prior work. Wavesplit redefines the
R
WSJ0-2mix, WSJ0-3mix
WSJ0-2/3mix
R
(WHAM!) and reverberated (WHAMR!)conditions. As an additional contribution, we further improve our model by introducing online data augmentation for separation
and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram