Cross-Lingual Speaker Identification for I ndian Languages (2023)

Abstract: The paper introduces a cross-lingual speaker identification system for Indian languages, utilising a Long Short-Term Memory dense neural network (LSTM-DNN). The system was trained on audio recordings in English and evaluated on data from Hindi, Kannada, Malayalam, Tamil, and Telugu, with a view to how factors such as phonetic similarity and native accent affect performance. The model was fed with MFCC (mel-frequency cepstral coefficient) features extracted from the audio file. For comparison, the corresponding mel-spectrogram images were also used as input to a ResNet-50 model, while the raw audio was used to train a Siamese network. The LSTM-DNN model outperformed the other two models as well as two more traditional baseline speaker identification models, showing that deep learning models are superior to probabilistic models for capturing low-level speech features and learning speaker characteristics.

## introduction ascertaining the identities of the writers and speakers are important tasks in language and speech processing. the vocabulary a person uses as well as the ways a person writes and talks can give us information about their identity or their background. furthermore, people's voices are unique identifiers, just like their retinas and fingerprints, making speaker recognition (the task of recognising the voice of a speaker based on audio input) applicable to building human-to-machine interaction and biometric solutions such as voice assistants, voice-controlled services, and speech-based authentication products @xcite . there are two basic speaker recognition tasks: speaker recognition can be monolingual as well as cross-lingual @xcite . for monolingual tasks, the same language is used to both train and test models. in cross-lingual speaker recognition, a model is trained on one language, e.g., english, and tested on a different language, e.g., arabic. in a multilingual country like india, with more than 120 languages having tens of thousands of speakers and some 50 languages having official status at national or regional level, most citizens speak several languages fluently. due to this plethora of multilingual speakers, it is not feasible to train a speaker recognition model in one language and re-train the model in a new language. therefore, the development of cross-language speaker recognition models has become a salient task. intuitively, language mismatch in training and test language should not be a problem, since a person's vocal traits have nothing to do with what they are saying, but in general, the performance of a speaker recognition system still degrades when a model is trained on one language and verification is done on another @xcite . probabilistic models like gaussian mixture model (gmm; @xcite and gaussian mixture model-universal background model (gmm-ubm; @xcite have traditionally been used for speaker recognition; however, in recent years deep learning-based approaches have outperformed probabilistic-based ones for both speaker identification and speaker verification. this paper reports on research conducted on five indian languages: hindi, kannada, malayalam, telugu, and tamil. english was used as the training language for the models. previous research has shown that extracting features from the audio signal and using them as input to the model will produce much better performance than directly considering raw audio signal as input. here raw audio, mel-frequency cepstral coefficients (mfccs; dave, 2013), and spectrogram images were utilised as input. the impact of language mismatch, the number of speakers, and the duration of utterances were studied while comparing the performances of the three input methods. the rest of the paper is structured as follows: section 2 describes related work in the domain, while section 3 presents the methodology and proposed neural network architecture. experimental results are discussed in section 4 and further analysed in section 5, while section 6 concludes the observations. ## related work cross-lingual speaker recognition has been in focus for researchers for some time because of the abundance of bilingual speakers in the world. @xcite studied the enrollment-test mismatch and found that it caused significant performance degradation for speaker recognition. @xcite investigated the mismatch between training and operation, within a gmm-ubm architecture, finding considerable performance degradation if the speech data used to train the universal background model and the data used to validate/test speakers were in different languages. @xcite drew similar conclusions when utilizing a model based on i-vectors @xcite , an intermediate vector representation between gaussian mixture models and mfcc. several deep neural network (dnn) models have been proposed for the speaker recognition task, with @xcite arguing that the reason for performance degradation in the cross-lingual environment is the use of probabilistic-based modelsas in all the above-mentioned methods-and showing considerable improvement when using a dnn model. @xcite proposed a textdependent speaker verification architecture utilising an lstm to extract d-vectors, i.e., embeddings over the averaged activation from the network's last hidden layer, with deep speaker by @xcite showing better results than i-vector based methods. @xcite introduced the concept of xvector embeddings, a model based on a time-delay deep neural network architecture that computes speaker embeddings from variable-length acoustic segments. the network consists of layers that operate on speech frames, a statistics pooling layer that aggregates over the frame-level representations, additional layers that operate at the segment level, and finally a softmax output layer. the embeddings are extracted after the statistics pooling layers. koluguri et al. (2020) described speakernet, an architecture using an x-vector-based statistics pooling layer to map variable-length utterances to a fixed-length embedding. @xcite presented a transformer-based speaker recognition system using wav2vec 2.0 @xcite . this paper broadly discusses two main approaches to feature extraction: (i) mfcc-based and (ii) spectrogram-based. due to its computational simplicity and robustness to multicollinearity, mfcc is the most popular feature extraction technique among researchers. mfcc yields uncorrelated features which are favorable for linear models like support vector machines (svm) and gaussian mixture models. in the mfcc-based approach, filter banks are designed in a manner to operate in a similar way to the human auditory frequency perception. many fusions of mfcc-based features have been studied. combining two different sets of features from mfccs and perceptual linear predictive coefficients (plpc) using ensemble classifiers in conjunction with principal component transformation can significantly improve the performance of mfcc-gmm speaker recognition systems @xcite . combining mfcc features with residual phase cepstrum coefficients (rpcc) also offers significant overall improvement to the robustness and accuracy of speaker identification tasks @xcite . @xcite used mfcc incorporated into a histogram transform feature for text-independent speaker identification. spectrogram images as a feature for convolutional neural network (cnn) models have also been explored @xcite , by extracting spectrogram images from audio files and feeding them to a cnn. the network's performance improved significantly when there were short utterances and a moderate amount of audio files present per speaker. ## methodology the national institute of technology karnataka's speaker profiling dataset (nisp; kalluri et al., 2021) was used for the experiments. it contains recordings of some 4-5 minutes each of speakers talking in both english and their mother tongues. the corpus includes hindi, which is an indo-aryan language, together with four dravidian languages: kannada, malayalam, tamil, and telugu. the text prompts used for the recordings were presented in two different sessions to the speakers, in their native language and in english, respectively. the data was sampled at 44.1 khz with a bitrate of 16 bits per sample. each speaker's data consists of 30 to 40 audio files in .wav format. a subset of the original nisp dataset was used to train the models, due to limitations of available computing resources. the dataset statistics are summarised in ## results and discussion the results of the experiments are summarised in the cross-lingual experiments with only five speakers per language ( in general, we can notice that the speaker identification accuracy drops for all models when the number of speakers is increased. this means that as the number of speakers in the dataset increases, it becomes more difficult for the models to accurately identify individual speakers. the variations in accuracy over the five languages show the effect of the native accent of speakers and the phonetic similarity @xcite between training and test languages. the native accent of speakers refers to the way in which they pronounce words and phrases based on their regional or cultural background. the phonetic similarity between languages refers to the degree to which the sounds and pronunciation of words in one language are similar to those in another language. the learning curves in figure 3 show the performance of the model during training and testing across all five languages. ## ablation study to evaluate the lstm-dnn model, several parameter variations were tested, analysing changes in one parameter at the time, while keeping the other parameters constant. four groups of ablations were examined. first, different feature extraction techniques. second, to explore the effects of regularization in lstm layers, recurring dropout rates were set to none, 0.2, and 0.5, respectively. third, the impact of reducing the number of lstm layers. finally, the learning rates, with two constant learning rates of 0.001 and 0.0001, and an exponential schedule with an initial rate of 0.01 and a decay rate of 0.9. as the accuracy results in ## conclusion an lstm dense neural network model for crosslingual speaker identification is proposed in this work. the model was trained using speaker recordings in english and cross-lingual speaker identification was performed on five indian languages: hindi, kannada, malayalam, tamil, and telugu. there was a clear variation in speaker identification accuracy across the different languages. since english was used for training for all speakers, the variation in accuracy is arguably due to variations in phonetic features of the native test languages, as well as any phonetic similarity between those languages and english. the average classification accuracy on the test data for the lstm-dnn method was 83.51%, with 68.29% for gmm-ubm, and 80.12% for svm, with those three learners trained using mfcc (mel-frequency cepstral coefficient) features. a siamese network using raw audio input reached 74.31% accuracy and a resnet-50 trained on melspectrograms 80.59% accuracy. the lstm-dnn model thus yielded better average accuracy than the other models, showing the efficiency of an lstm-dnn trained using mfcc features input under the constraint of limited data. the siamese network few-shot learning approach using simple raw audio input is good when there are few speakers but fails to generalise over a significant number of speakers. a complex cnnbased model with spectrogram inputs like resnet-50 gives better results than mfcc feature extraction when there are sufficient images available to train the model; however, the scarcity of image data is a bottleneck for that approach. finally, the traditional probabilistic gmm-ubm performed worst of all models in the cross-lingual environment. while this research focused on speaker identification, the work can also be used as a springboard to develop more advanced frameworks like x-vectors for indian languages and apply the methods to the speaker verification problem. the models developed can furthermore be utilised in isolation or together with text-based feature extractors for similar digital forensic tasks such as author profiling or native language identification, i.e., to recognize a person's l1 (native language) based on text and speech produced in a foreign language (l2).