Advancing Multilingual Speaker Identification and Verification for I ndo- A ryan and D ravidian Languages (2025)

Abstract: Multilingual speaker identification and verification is a challenging task, especially for languages with diverse acoustic and linguistic features such as Indo-Aryan and Dravidian languages. Previous models have struggled to generalize across multilingual environments, leading to significant performance degradation when applied to multiple languages. In this paper, we propose an advanced approach to multilingual speaker identification and verification, specifically designed for Indo-Aryan and Dravidian languages. Empirical results on the Kathbath dataset show that our approach significantly improves speaker identification accuracy, reducing the performance gap between monolingual and multilingual systems from 15% to just 1%. Additionally, our model reduces the equal error rate for speaker verification from 15% to 5% in noisy conditions. Our method demonstrates strong generalization capabilities across diverse languages, offering a scalable solution for multilingual voice-based biometric systems.

## introduction in today's world, biometric recognition is revolutionizing how we identify and verify individuals. traditional methods, such as passwords, personal identification numbers, or signatures, are often inconvenient because they can be forgotten, stolen, or forged @xcite . in contrast, biometric traits are unique to each individual, making them difficult to replicate or steal. these systems rely on either physiological characteristics, such as fingerprints, iris patterns, or facial features, or behavioral traits, such as handwriting, voice, or keystroke patterns, to identify a person @xcite . among these biometric traits, voice-based recognition offers clear advantages. two factors make it a strong choice: first, speech is a natural and easy signal for users to provide. second, the wide availability of phones and low-cost microphones make voice capture accessible and convenient for many applications @xcite . in voice-based biometric recognition, there are two distinct modes of operation: speaker identification, which typically involves recognizing an individual from a larger pool, and speaker verification, which focuses on validating a specific identity claim @xcite . voice-based recognition systems can be classified by their language handling capabilities into monolingual and multilingual systems @xcite . monolingual systems are trained and tested within a single language, offering high accuracy but limited flexibility outside that specific linguistic context. multilingual systems, on the other hand, are designed to recognize speakers across multiple languages within a single model, eliminating the need for separate models for each language. this versatility makes multilingual systems well-suited for environments where multiple languages are spoken. recent advancements in self-supervised learning (ssl) have significantly enhanced the performance and robustness of voice-based recognition systems. ssl models, particularly in the context of the upstream model, play a crucial role in feature extraction. here, rich speech features are captured and transferred to a downstream model, which is responsible for tasks such as speaker identification and verification @xcite . by separating the feature extraction and task-specific components, ssl models offer greater flexibility, improving the performance of voice recognition systems, particularly in multilingual applications. despite these advances, multilingual systems still lag behind their monolingual counterparts in terms of accuracy @xcite . this performance gap is particularly significant in multilingual countries such as india, where linguistic diversity presents a unique challenge. india's population speaks languages from four main language famformat (sampled at 16 khz) using the pre-trained indicwav2vec model @xcite to generate 24 framelevel representations. these are then mean-pooled along the time axis to create utterance-level representations. a weighted average pooling is applied across the 24 utterance-level representations to produce the final representation, which is passed through two fully connected layers to predict speaker identity. layer dimensions and additional hyper-parameters are detailed in section 3. in this paper, we propose a novel architecture for voice-based biometric recognition using the pretrained indicwac2vec model @xcite to enhance both speaker identification and verification. our model was tested under two conditions: clean and noisy environments. while there was a slight improvement in monolingual speaker identification accuracy, the major gain was in multilingual speaker identification accuracy, where the performance gap between monolingual and multilingual systems decreased from around 15% to 1%. additionally, instead of creating a separate speaker verification model, we used the speaker embeddings from our speaker identification model for verification. compared to the standard approach, our method reduced the equal error rate from 15% to 5% on unknown data in both clean and noisy conditions, demonstrating improved multilingual voice-based recognition. ## methodology our speaker identification model builds upon the architecture proposed by javed et al.. to enhance the model's performance on speaker identification and verification tasks, we have introduced two key modifications, as illustrated in figure 1 . ## experimental setup our speaker identification model consists of two main components: an upstream model and a downstream model. the upstream model, in-dicwav2vec, is a wav2vec-based model pretrained on half a million hours of raw speech data across 128 indian languages @xcite . following standard practice in speech processing benchmarks, such as superb @xcite , we freeze the upstream model and train only the downstream model. this approach allows us to use the rich, pre-trained representations while reducing computational complexity, as only the downstream model is updated to predict speaker in the sid-multi task, languages are grouped into three categories: dravidian (a model trained on all dravidian languages and tested on each language individually), indo-aryan (a model trained on all indo-aryan languages and tested on each language individually), and all (a model trained on all languages combined and tested on each language individually) identity. 1 2 . as the speaker identification task is framed as a classification problem, we use cross-entropy as the loss function and accuracy as the evaluation metric. in the speaker verification task, we first train the model for multilingual speaker identification, then extract speaker embeddings. these embeddings are compared using cosine similarity, and performance is evaluated using the equal error rate (eer), which represents the point at which the false acceptance rate equals the false rejection rate. hyper-parameter tuning, performed using grid search, was applied to both tasks to optimize the model's performance 3 . for evaluating our model's performance, we select the kathbath dataset @xcite , which is particularly well-suited for speaker identification tasks involving indo-aryan and dravidian languages. this dataset is the largest available for indian languages, making it an ideal choice for multilingual speaker identification. it includes 8 indo-aryan languages-gujarati, marathi, bengali, odia, hindi, punjabi, sanskrit, and urdu-and 4 dravidian languages-kannada, malayalam, tamil, and telugu. all 12 languages 1 our model was implemented using pytorch. 2 all experiments were conducted on an nvidia quadro rtx 6000 gpu with 30gb of ram. 3 the fully connected layer has a dimension of 1500, with a batch size of 32 and a learning rate of 2.5 × 10 -3 . are widely spoken, ensuring the model's generalization across a diverse set of linguistic and acoustic features. the dataset is divided into four categories: clean known, noise known, clean unknown, and noise unknown, which allows for robust evaluation under varying conditions of noise and speaker familiarity. the "clean" and "noise" labels distinguish between clean and noisy audio, while "known" and "unknown" indicate whether the speaker is seen or unseen during training. we follow the recommended train-test splits for each dataset. ## results and discussion a key architectural modification in our model is the use of weighted average pooling for features extracted from the pre-trained indicwav2vec model, replacing traditional mean pooling. figure 2 demonstrates that layer contributions are not uniform; notably, layers 9 through 12 consistently receive higher weights across all models. this suggests that these deeper layers play a substantial role in encoding speaker identity, as they may capture more abstract, speaker-specific features that are essential for accurate identification. furthermore, there is a strong correlation between the weight patterns in monolingual and multilingual models. layers with relatively small weights in monolingual models appear even smaller in the multilingual model, while those with higher in contrast, the multilingual setting shows a slight decrease in accuracy compared to the monolingual case, which is expected due to the added complexity of handling multiple languages. nevertheless, the model trained on the "all" languages category maintains relatively high performance across languages, demonstrating strong generalization. the dravidian and indo-aryan subsets perform similarly, with the indo-aryan model slightly outperforming others in some cases. notably, the multilingual models exhibit better resilience to noise compared to the monolingual models, suggesting that training with multiple languages helps the model learn more robust speaker features. however, noise remains a challenge, and further improvements in noise robustness are needed for better performance in real-world conditions. next, however, when it comes to asv, our model lags behind the baselines in terms of equal error rate (eer). while xls-r and indicwav2vec achieve eer values ranging from 2.08 to 2.83 for clean conditions and 11.58 to 15.39 for noisy conditions, our model exhibits better eer values, particularly in unknown conditions, with the best value being 5.15 for unknown speakers in clean conditions and 5.55 for unknown speakers in noisy conditions. these results suggest that while our model excels in speaker identification tasks, further improvements in asv, especially under known conditions, are necessary. despite the performance gap in asv, the results highlight the robustness of our model in sid tasks across both monolingual and multilingual settings, making it a promising candidate for practical voice recognition applications. ## conclusion in this work, we presented a novel approach for multilingual speaker identification and verification using a modified indicwav2vec-based model. our model integrates self-supervised learning techniques to extract rich, robust speech features, which substantially improve speaker identification performance, especially in multilingual settings. key innovations include a weighted average pooling mechanism for better aggregation of transformer layer representations and an additional embedding layer to refine speaker-specific features. these modifications led to significant improvements, reducing the performance gap between monolingual and multilingual systems from 15% to 1%, and lowering the equal error rate for speaker verification from 15% to 5% under noisy conditions. our experiments, conducted with the kathbath dataset, demonstrated the model's ability to generalize effectively across multiple languages. the simplicity of the model structure, combined with its robust performance, positions it as an efficient and scalable solution for voice-based biometric recognition.