SV eritas: Benchmark for Robust Speaker Verification under Diverse Conditions (2025)

Abstract: Speaker verification (SV) models are increasingly integrated into security, personalization, and access control systems, yet their robustness to many real-world challenges remains inadequately benchmarked. Real-world systems can face diverse conditions, some naturally occurring, and others that may be purposely, or even maliciously created, which introduce mismatches between enrollment and test data, affecting their performance. Ideally, the effect of all of these on model performance must be benchmarked; however existing benchmarks fall short, generally evaluating only a subset of potential conditions, and missing others entirely. We introduce SVeritas, the Speaker Verification tasks benchmark suite, which evaluates the performance of speaker verification systems under an extensive variety of stressors, including “natural” variations such as duration, spontaneity and content of the recordings, background conditions such as noise, microphone distance, reverberation, and channel mismatches, recording condition influences such as audio bandwidth and the effect of various codecs, physical influences, such as the age and health conditions of the speaker, as well as the suspectibility of the models to spoofing and adversarial attacks. While several benchmarks do exist that each cover some of these issues, SVeritas is the first comprehensive evaluation that not only includes all of these, but also several other entirely new, but nonetheless important real-life conditions that have not previously been benchmarked. We use SVeritas to evaluate several state-of-the-art SV models and observe that while some architectures maintain stability under common distortions, they suffer substantial performance degradation in scenarios involving cross-language trials, age mismatches, and codec-induced compression. Extending our analysis across demographic subgroups, we further identify disparities in robustness across age groups, gender, and linguistic backgrounds. By standardizing evaluation under realistic and synthetic stress conditions, SVeritas enables precise diagnosis of model weaknesses and establishes a foundation for advancing equitable and reliable speaker verification systems.

## introduction speaker verification technology has achieved remarkable accuracy under controlled conditions, driven by advances in deep neural embeddings, margin-based losses, and self-supervised pretraining. however, real-world deployments -from secure access control and telephony authentication, to personalized assistants, and law enforcementconfront a broad spectrum of challenges that degrade performance, including degradations to the signal itself, and mismatches between the test utterances and the enrollment recordings they are compared to. these mismatches arise from natural variability (e.g., spontaneous versus read speech, cross-language trials, or temporal drift), environmental distortions (e.g., reverberation, background noise, far-versus near-field capture), device and codec artifacts, demographic factors (age, health or physical condition), and even malicious manipulations such as spoofing or adversarial attacks. without comprehensive, standardized evaluation across these diverse stressors, it remains unclear which aspects of sv systems are robust in practice and where critical vulnerabilities lie. existing benchmarks each target a narrow subset of these challenges. for example, common-bench @xcite offers large-scale multilingual text-independent trials, yet it relies on an ecapa-based outlier filter that may prune precisely the hardest cases (e.g., heavy accents or noisy recordings), and omits deliberate distortions such as codec compression or spoofed audio. in-dicsuperb @xcite highlights performance on twelve indian languages but focuses exclusively on scripted, read speech in clean or synthetic noise conditions, neglecting cross-language scenarios, far-field capture, or adversarial manipulations. other benchmarks examine specific dimensions in isolation -far-field effects in multisv @xcite , age variation in time-varying sv @xcite , or spoofing attacks in asvspoof @xcite . to the best of our knowledge, no prior suite spans the full gamut of natural, environmental, demographic, codec, and adversarial factors under a unified framework. fur-thermore, many datasets lack sufficient metadata to analyze fairness across age, gender, or linguistic subgroups, rely on relatively large enrollment durations, or assume static, text-independent protocols that do not reflect modern low-resource, multi-file, or cross-domain requirements. to address these gaps, we introduce sveritas, the first comprehensive speaker verification benchmark suite that systematically evaluates state-ofthe-art models across an extensive set of real-world and synthetic stressors. sveritas assembles trials spanning (i) content and style variations (read vs. spontaneous, same vs. different sentences, multilanguage), (ii) acoustic and channel mismatches (noise types and levels, far-vs. near-field, codec and bandwidth variations), (iii) demographic and physical factors (age-group mismatches, health or emotional states), (iv) enrollment/test duration and multi-file enrollment, (v) security threats (spoofing via tts/vc pipelines, universal and adversarial perturbations), and (vi) speaking-style adaptation (lombard speech under noise-induced conditions). by unifying these dimensions, sveritas not only measures aggregate metrics such as equal error rate and detection cost function, but also facilitates fine-grained analyses of performance disparities across demographic subgroups and operating conditions. through extensive evaluation of leading architectures, we uncover systemic weaknessesparticularly in cross-language, age-mismatch, and codec-compressed trials -and expose fairness gaps that vary nontrivially by gender and language background. in summary, sveritas establishes a rigorous, reproducible foundation for diagnosing robustness and equity in speaker verification. by revealing hitherto uncharacterized vulnerabilities and enabling targeted stress-testing, our benchmark paves the way for developing more reliable, inclusive, and secure sv systems suitable for deployment in the complex acoustic and demographic landscapes of real-world applications. our code is publicly available with documentation, fostering straightforward reproducibility and extensibility. https://github.com/massabaali7/sveritas . ## sveritas benchmark sveritas aims to provide a thorough benchmarking of sv systems, evaluating its performance under various degradations, mismatches, sources of bias, and attacks, providing both detailed and summarized evaluations, along with statistical significance reports where appropriate. the tests are not only intended to evaluate the performance of the system under various conditions and threats that may be expected in real-life deployments, but to also provide a diagnostic tool to identify weaknesses, and detect any systematic biases or vulnerabilities. sveritas evaluates the robustness of sv models through a structured three-stage pipeline: (1) scenario simulation, (2) embedding extraction, and (3) performance evaluation. as illustrated in figure 1 , the first stage introduces a wide range of real-world and synthetic perturbations to both enrollment and test audio. these include natural variations (e.g., speaking style, duration, and linguistic content), environmental conditions (e.g., noise, reverberation, and microphone distance), and recording artifacts (e.g., codec compression and bandwidth limitations). sveritas also incorporates physical and demographic variability, such as speaker age, health, and accent, as well as adversarial factors including spoofing attempts and both black-box and white-box attacks. the second stage applies multiple state-of-the-art embedding models to extract speaker representations. finally, performance is evaluated using metrics such as equal error rate (eer) across matched/mismatched scenarios and demographic subgroups, enabling a comprehensive assessment of model robustness and fairness. ## evaluation we evaluate several state-of-the-art sv models using sveritas and analyze their robustness across a broad range of challenging scenarios. we further extend this analysis to examine model behavior across various demographic subgroups, including speaker age, language background, ethnicity, and gender. prior work (hutiri and ding, 2022) has noted the presence of biases in sv systems, and our findings corroborate and expand upon these observations by revealing that disparities in robustness can emerge across subgroups. these results highlight the importance of standardized evaluation under real-world conditions and underscore the utility of sveritas in advancing fair and reliable speaker verification. to further quantify fairness and robustness, we conduct a series of pairwise statistical tests across demographic groups using eer as the primary metric. while each group yields a single eer value per model, we leverage the diversity of five models to enable paired comparisons between groups. this design allows us to assess whether performance disparities are consistent across architectures. full statistical test tables, including t-statistics, p-values, and significance levels, are provided in appendix a. ## conclusion we present sveritas, a comprehensive and extensible benchmark for evaluating speaker verification models under diverse real-world and synthetic stressors. unlike prior work, it covers environmental noise, channel mismatches, codecs, cross-lingual variation, demographic shifts, adversarial attacks, and importantly, tts-based spoofing-an often overlooked but growing threat. sveritas enables fine-grained robustness and fairness analysis across gender, age, ethnicity, and language, while offering a modular framework that supports easy integration of new models and evaluation settings. it provides a unified, reproducible foundation for building sv systems that are not only accurate, but also resilient, equitable, and ready for real-world deployment.