Speaker Verification Experiments for Adults and Children Using Shared Embedding Spaces (2021)

Abstract: For children, the system trained on a large corpus of adult speakers performed worse than a system trained on a much smaller corpus of children’s speech. This is due to the acoustic mismatch between training and testing data. To capture more acoustic variability we trained a shared system with mixed data from adults and children. The shared system yields the best EER for children with no degradation for adults. Thus, the single system trained with mixed data is applicable for speaker verification for both adults and children.

## introduction the use of speaker verification (sv) technology for children has many beneficial application areas, such as child security and protection, entertainment, games and education. for example, in an interactive class the teacher could identify each child, by continuing a previous lecture and adapt its content with the child's speech, and log the child's responses without a conventional login process @xcite . the acoustic and linguistic characteristic of children's speech differ from adults' speech @xcite . the main differences are in pitch, speaking rate and formant frequencies @xcite . these acoustic differences together with the lack of training data make sv more challenging. little work has been reported in this area. in @xcite in-domain and out-of-domain data augmentation are used to improve a child sv system in a limited data scenario. in @xcite vocal tract information is used for children's sv. explanation for degraded recognizer scores through acoustic changes resulting from voice disguise is presented in @xcite . in this work, we explore how recent advances in (adult) sv could aid in child sv, as well. in particular, we combine adult and child sv into a single task, by using a shared embedding space for adult and child speakers. this allows us to leverage the large resources available for adult speakers for the low-resource child speaker verification task. in applications where both adult and child speakers can be expected, it is also natural to use a single shared system for both groups; we find that a shared system can be used which benefits child sv without degrading adult sv performance. contributions. we construct a neural sv system, which leverages recent advancements in the field. in particular, we find improvements from using the additive-margin softmax loss and the netvlad time aggregation methods. in contrast to most recent literature, which uses convolutional neural layers, we apply recurrent layers, motivated by success in speaker diarization @xcite . we compare our results to recent high-performing systems of similar complexity. though we do not outperform the top results, the comparison validates our approach. we then apply the proposed sv system to adult and child sv and find that using a shared embedding for both adult and children improves child sv drastically without affecting adult sv performance. ## related speaker verification work in recent years, deep learning motivated approaches have shown significant progress in sv. we consider three main reasons for their success. firstly, larger and more realistic speakersin-the-wild speaker recognition datasets have become available to the public @xcite @xcite . secondly, the loss functions used in the training of neural networks have advanced. in general, the main objective of the neural networks designed for sv is to transform a given recording into a speaker embedding which embodies the speaker characteristics of the recording @xcite @xcite . in the most current methods, the embeddings are learned in a speaker identification process, were original softmax loss is modified by adding a margin to the class decision boundaries @xcite @xcite . this allows efficient training and reduces the intra-class variance of the created embeddings @xcite @xcite . finally, the neural network architectures have developed. one of the most prominent discoveries has been x-vectors, speaker embeddings which are extracted from an architecture based on time-delay neural networks (tdnns) @xcite @xcite . x-vectors have been shown to outperform i-vectors, which have enjoyed a stateof-the-art status in sv for a long time @xcite . in some cases, i-vectors have also been inferior to the sv systems which utilize convolutional neural networks (cnns) @xcite . furthermore, novel aggregation methods for neural networks have been proposed. whereas average pooling has been used extensively before, the most recent approaches include statistics pooling, attentive statistics pooling and netvlad (vector of locally aggregated descriptors) @xcite @xcite . in addition, recurrent neural networks (rnns) with long-short term memory (lstm) cells @xcite have been experimented with @xcite @xcite . most importantly, they have shown success in a related task, online speaker diarization @xcite @xcite . in this task, lstms have been been able to create compact speaker embeddings from very short segments. our approach has some similarities with wan et al. @xcite . as in their work, we use lstms in sliding windows. however, unlike them, we do not apply the generalized endto-end loss for neural network training. instead, we use the am-softmax loss @xcite . furthermore, unlike them, we combine lstms with netvlad. although netvlad layer has been previously used for sv @xcite , in that study, the layer was connected to a cnn. netvlad has been originally designed for aggregation of cnns @xcite and to the best of our knowledge, we are the first to use it with lstms in any application. ## proposed methods in this section, we detail our sv system which consists of three stages: splitting, embedding and averaging, as illustrated in fig. 1 . split. first, the audio input is split into overlapping windows with short, roughly 2 seconds or less, duration. time-varying features are then extracted from each frame, resulting in a set of feature sequences x. the sequences consist of 30 mel-frequency cepstral coefficients (mfcc) which are extracted every 10ms with 25ms frame length. every x is normalized with zero mean and unit variance. embed. in the next step, each x is transformed into a speaker embedding. this can be further divided into two distinct steps: frame-level processing and aggregation. in aggregation, the higher level features h are compressed into a speaker embedding f . we compare two aggregation approaches: average pooling and netvlad. the aggregation components are illustrated in fig. 3 . note that the aggregation component with average pooling has a slightly different configuration than its netvlad motivated counterpart. this choice was based on balancing the number of parameters in both neural networks. we force the embeddings to be l 2 normalized in both components. as a result, cosine distance is the most natural distance metric between different embeddings. a rectified linear unit activation is used in all of the fully connected (fc) layers. we also apply batch normalization @xcite after each layer except l 2 normalization layers. this means that the last two layers of both components perform normalization. although this might seem strange, we discovered it to be beneficial in the preliminary experiments. the operation of the netvlad layer can be summarized as follows. let us denote the output of the preceding fc layer as @xmath0. first, v is transformed into @xmath1 average. in the final stage, we compute a single embedding f @xmath2. when considering f c1 and f c2 extracted from two different recordings, our system performs sv by computing cosine distance between the embeddings and by thresholding the obtained value. another popular method for comparing the embeddings is probabilistic discriminant analysis (plda) @xcite . plda could result in performance improvements @xcite , but also increase the complexity of our system, and we do not apply it in this work. ## experiments data. we use two training sets for adult speakers which are both generated from voxceleb2 @xcite . in the first, abbreviated as v c2, all recordings in voxceleb2 are windowed into 2 second samples with 1 second overlap. the reason for this choice is the training objective of our neural networks that is to identify a speaker from a given training set based on a 2 second segment of speech. the duration was not selected arbitrarily: we experimented also with setting it to 1 and 2.5 seconds. the former was too short for neural networks to learn speaker characteristics properly and the latter did not generally improve the performance of the networks. v c2 consists of roughly 6.83 million training samples from 5994 speakers. the second set, v c2 c , is otherwise the same as v c2 but excludes a portion of the samples based on a heuristic cleaning algorithm. the motivation for this algorithm came from our listening tests which confirmed that voxceleb2 included wrongly labeled speaker identities in some cases. the exclusions removed approximately 46k samples from v c2 but retained the number of speakers, 5994. given samples s i belonging to i-th speaker in v c2, the cleaning algorithm operates in four steps: we evaluate our models also using the cleaned versions of voxceleb1 verification test sets, voxceleb1-test (v c t ), voxceleb1-h (v c h ) and voxceleb1-e (v c e ) @xcite . the recordings in these sets are framed to 2 second duration segments with 1.5 seconds overlap. the overlap duration was determined in the preliminary experiments. we construct also our own verification set from the development set of voxceleb1. this set is used for model evaluation during training. the set consists of speech segments with a fixed 2 seconds duration, and which each are extracted from a unique session and speaker. the number of extracted segments is close to 20k and they belong to 1211 speakers. we form close to 150k segment pairs where half of the pairs correspond to the same speaker and the other half to different speakers. we name this verification set as v c 2sec . the set is disjoint in speakers with v c t but not with v c h and v c e . however, we consider this evaluation set to be valid since the pair compositions and segment durations of v c 2sec differ significantly from v c h and v c e . for child speech experiments we used cslu kids @xcite database for training. it has 1110 speakers of english language with age range from 6 to 16 years and sampling rate 16 khz. for testing the system we used pf-star @xcite and the finnish speechdat @xcite datasets. pfstar has 134 speakers of english with age range from 4 to 14 years, originally sampled at 22,050 hz. the downsampling at 16 khz was performed for consistency with the model. speechdat has 354 speakers of finnish with age range from 6 to 14 years, originally data sampled at 8 khz. the up-sampling at 16 khz was performed for consistency. for children's experiments we used the same speaker embedding method as adults. training. in training, the output of the aggregation component is connected to a fully connected layer which is used for a speaker identification task. training has two stages: warm-up with the softmax loss and fine-tuning with the am-softmax loss @xcite . in the warm-up, the neural network is trained for 5 epochs, using adam optimizer with 0.01 learning rate. batch size is chosen as 512. we generally observed that the performance of the neural networks on the v c 2sec would not improve after the fifth epoch when using the softmax loss. the main point of using the am-softmax loss is to decrease intra-class variance, which is generally difficult with the softmax loss @xcite @xcite . in other words, the higher the margin m is set, the more closer, in terms of cosine distance, the speaker embeddings belonging to the same class are forced. the cosine distance metric arises from the l 2 normalizations of both f and the rows of w. the scale of s is generally set to a some high value to ensure convergence @xcite . in recent years, the am-softmax loss and other similar methods @xcite have emerged as state-of-the-art approaches in speaker verification @xcite @xcite . the fine tuning is continued for 10 epochs with otherwise the same setting as in warm-up. we monitor the progress of the training by first computing cosine distances between the embeddings of each pair in v c 2sec and then calculating equal error rate (eer) on these distances after each epoch. eer is a standard error metric in speaker verification @xcite @xcite . although the v c 2sec contains over 150k pairs, the evaluation on this set is efficient during the training since it consists of short, equal length segments which can be embed-ded rapidly. we save the weights of the neural network after each epoch, and choose the configuration with the best eer value as our final model. ## results first in section 5.1 we validate our sv approach on adult speech. then in section 5.2 we apply the system with children. ## conclusion we have presented a speaker verification system based on a shared neural embedding space for adults and children. the neural network consists of a cascade of lstm layers and a netvlad aggregation layer, and uses the am-softmax loss in training. we have demonstrated that the system achieves promising results with adults and children. because the child data is limited, we trained a shared system with mixed adult and child data to capture more acoustic variability. the shared sys-tem gives a 54% and 43% relative improvement for children compared to the separate children's and adult systems. for adults, the shared system gives the same performance as compared to the adult system. finally, we can conclude that this shared system can be now used for both children and adults.