Multi-accent Speech Separation with One Shot Learning (2021)

Abstract: Speech separation is a problem in the field of speech processing that has been studied in full swing recently. However, there has not been much work studying a multi-accent speech separation scenario. Unseen speakers with new accents and noise aroused the domain mismatch problem which cannot be easily solved by conventional joint training methods. Thus, we applied MAML and FOMAML to tackle this problem and obtained higher average Si-SNRi values than joint training on almost all the unseen accents. This proved that these two methods do have the ability to generate well-trained parameters for adapting to speech mixtures of new speakers and accents. Furthermore, we found out that FOMAML obtains similar performance compared to MAML while saving a lot of time.

## introduction speech separation has been a well-known task to solve in the speech processing field. many model architectures mentioned in section 2 have been proposed and achieved high performance. this suggests that deep learning based methods are suitable for the speech separation task. despite having promising results, the generalizability of these models is still questionable. the performance of switching to different datasets or environments is not guaranteed. a straightforward solution is to exhaustively collect data under all kinds of environment settings and train a model with these data jointly. although this may sound reasonable, it is difficult to always consider every situation during training. to make sure that models can be quickly adapted to mixtures spoken by new speakers with not many samples, metalearning comes to the rescue. meta-learning has been widely applied on different speech tasks, especially on speech recognition mentioned in section 2. nonetheless, there is not much work that applied meta-learning on the speech separation task. in our previous work, @xcite , we first proposed to solve the speech separation problem with meta-learning. their setting is viewing utterance mixtures of two different speakers as a meta task. these speakers have the same accents. however, we hope that a speech separation model can have the ability to adapt to mixtures with accents never seen before. thus, besides the setting of two different speakers forming a meta task, we also added a setting that meta tasks with speakers of same accents form an accent task set. section 4 and 5.1 describe more about the dataset and task construction procedure. our contributions are listed below: the remaining sections of this paper are organized as follows. in section 2, we give a brief overview of existing works related to speech separation and meta-learning. in section 3, we elaborate the problem formulation of speech separation in detail. in section 4, we list out the two phases of maml, including the meta training phase and meta testing phase. additionally, we show how fo-maml is modified from maml. the experimental setup, dataset, and model we used are presented in section 5. finally, results and conclusions are given in section 6 and 7. 2 related work speech separation end-to-end separation models have shown great success in separating speech mixtures of the wsj0-2mix dataset designed by @xcite which is generated from the wsj0 corpus @xcite . @xcite came up with a time-domain audio separation network (tasnet) that takes waveforms as input to alleviate the separation model from dealing with time-frequency representations. they further proposed convolutional tasnet @xcite which substitutes the lstm layers in tasnet with convolutional layers. this overcame the problem of long temporal dependencies of lstm and reduced the model size. before long, they came up will the dual-path rnn model, which used intra-and inter-blocks to capture local and global information dependencies within the speech mixtures. @xcite utilized the idea of dual-path rnn and added a speaker identity loss to improve performance on separating mixtures with an unknown number of speakers. @xcite proposed to use a separator constructed with u-convblocks which can not only reduce the number of layers while still having high performance but also require less computational resources and time. this helped the model to more likely be used in real-time speech separation. @xcite integrated speaker identity information into the separating process, and obtained state-of-the-art performance. meta-learning meta-learning has recently become a trend when it comes to solving multitask problems. this training method has been widely applied in the computer vision field, for instance, @xcite @xcite . meta-learning is also used in the natural language processing field. @xcite used maml @xcite for low-resource neural machine translation (nmt). moreover, in the speech processing domain, some speech-related problems are solved with metalearning, too. @xcite applied metatransfer learning on code-switched speech recognition. @xcite applied meta-learning to solve the multilingual lowresource speech recognition problem. ( winata et al., 2019) also used maml to adapt models to unseen accents on speech recognition. @xcite adopted meta-learning algorithms to perform speech translation on speech-transcript paired low-resource data. @xcite came up with some improvements of meta-learning to help the speaker verification task. ## speech separation in this work, we perform single channel speech separation. given a mixture the conv-tasnet model is a mask-based model which consists of an encoder, separator, and decoder. the encoder encodes the mixture x to a latent space as shown in eq.( 5 ). the masks are then multiplied with the encoder output, forming separated features d i shown in eq.( 7 ), at this point, before measuring the estimated sources with si-snr, there is a label permutation problem. an align between {ŝ 1 , • • • , ŝc } and {s 1 , • • • , s c } needs to be decided. we used the utterance-level permutation invariant training(upit) method described in @xcite to solve this problem. ## maml the procedure of maml @xcite is stated as follows. given a set of multi-accent tasks ## conclusion our results show that maml and fomaml training methods are effective on multi-accent speech separation. more specifically, it is confirmed that these two methods are better than joint training when adapting to new speakers with new accents and even noisy environments. besides, fomaml is shown to be sufficient for dealing with the multiaccent speech separation task and can reduce a large amount of training time. despite the fact that fomaml outperforms joint training on the testing set, we can still see that the performance of each accent task set varies a lot from fig. (4 ). this is probably due to the task-difficulty imbalance issue described in @xcite , perhaps some speakers with special accents may be hard to separate. thus, in the future, we will try to solve this problem with meta sampling methods mentioned in @xcite .