Factors Affecting the Performance of Automated Speaker Verification in A lzheimer’s Disease Clinical Trials (2023)

Abstract: Detecting duplicate patient participation in clinical trials is a major challenge because repeated patients can undermine the credibility and accuracy of the trial’s findings and result in significant health and financial risks. Developing accurate automated speaker verification (ASV) models is crucial to verify the identity of enrolled individuals and remove duplicates, but the size and quality of data influence ASV performance. However, there has been limited investigation into the factors that can affect ASV capabilities in clinical environments. In this paper, we bridge the gap by conducting analysis of how participant demographic characteristics, audio quality criteria, and severity level of Alzheimer’s disease (AD) impact the performance of ASV utilizing a dataset of speech recordings from 659 participants with varying levels of AD, obtained through multiple speech tasks. Our results indicate that ASV performance: 1) is slightly better on male speakers than on female speakers; 2) degrades for individuals who are above 70 years old; 3) is comparatively better for non-native English speakers than for native English speakers; 4) is negatively affected by clinician interference, noisy background, and unclear participant speech; 5) tends to decrease with an increase in the severity level of AD. Our study finds that voice biometrics raise fairness concerns as certain subgroups exhibit different ASV performances owing to their inherent voice characteristics. Moreover, the performance of ASV is influenced by the quality of speech recordings, which underscores the importance of improving the data collection settings in clinical trials.

## introduction healthcare systems are increasingly relying on automatic speaker verification (asv) models to ensure secure and accurate identification of patients and healthcare providers, with the aim of preventing fraud, safeguarding patient privacy, and ensur-ing the accuracy of medical records @xcite . conducting large-scale clinical trials, involving numerous patients, doctors, clinics, and even different countries can pose significant challenges in identifying instances of duplicate participation, which occurs when a single individual joins the same study more than once, either at different sites or time points, leading to skewed results and undermining the validity of study findings @xcite . @xcite discovered that as much as 7.78% of patients involved in a clinical study were duplicated across different sites. in some cases, individuals participate in multiple clinical trials concurrently in order to earn more money. when a trial enrolls an adequate number of substandard participants, it risks not meeting the primary endpoints and ultimately causing a multimillion-dollar study to fail. @xcite examined the financial effect of duplicate participants on the pharmaceutical companies conducting a set of short-term study programs across psychiatric disorders including schizophrenia, major depressive disorder, and bipolar depression. based on their results, enrolling ineligible subjects in the selected studies results in a loss of around $29,680,000 for the sponsor pharmaceutical company. in addition, duplicate participation results in higher placebo rates and compromised data integrity. these findings highlight the importance of addressing the duplicate participant problem and underscore the need for reliable and accurate asv methods in healthcare systems to verify whether an unknown voice belongs to a known enrolled individual @xcite . cognitive impairment has been linked to a decline in vocabulary richness, syntactic complexity, and speech fluency, according to previous research @xcite @xcite . therefore, it is important to investigate whether the abnormal speech patterns exhibited by individuals with cognitive impairment can affect asv performance. despite this concern, there is a paucity of research examining the relationship between cognitive impairment and asv in the existing literature. this research gap motivated us to address this issue by examining the effect of alzheimer's disease (ad) severity level on asv performance. furthermore, external factors such as participants' demographic information @xcite , recording environment, or data collection procedure @xcite may also have an impact on asv performance, but their impact is not well-studied in the healthcare industry. an extensive analysis of these external factors could provide valuable insights into the accuracy and reliability of asv models and identify potential sources of bias due to differences in inherent voice characteristics among subgroups @xcite . the purpose of this study is to investigate the effectiveness of asv models in identifying duplicate patient participation in large-scale clinical trials, and to explore the factors that influence asv performance in such settings. to this end, we utilize a longitudinal clinical dataset of english speech recordings obtained through multiple speech tasks from 659 participants with varying levels of ad. we employ the titanet model, an end-to-end deep learning text-independent asv model pre-trained on a large volume of speech recordings of english speakers. asv models can be classified into two groups: text-dependent (td) and text-independent (ti). ti asv models allow for more flexibility in the enrollment and verification phases without constraints on the speech content. when pre-trained on extensive audio datasets, ti models demonstrate a comparable level of accuracy to td models. we evaluate the performance of titanet on our dataset in a zero-shot setting, achieving a 3.1% equal error rate (eer). in addition, we analyze the impact of various external factors on asv performance, including participant demographic characteristics (i.e., age, and gender), audio quality criteria (i.e., clinician interference, background noise, participant accent, and participant clarity), as well as ad severity level. this study aims to provide valuable insights into the factors that can affect the performance of asv models in clinical trial environments, with the goal of improving the accuracy, fairness, and reliability. our findings indicate that voice biometrics may present fairness issues, as certain subgroups demonstrate differing speaker verification performances due to their inherent voice characteristics. in addition, the quality of speech recordings can impact asv performance, highlighting the importance of monitoring and enhancing data collection and recording settings during clinical trials. ## related work speaker verification technology has been increasingly utilized in various domains, including healthcare systems. several studies have analyzed the feasibility and effectiveness of speaker verification models in healthcare settings @xcite . however, the external factors that can affect the performance of asv models has not been extensively studied through research in the healthcare field. race and gender effect: @xcite utilized three state-of-the-art asv models including the xvector-tdnn @xcite , ecapa-tdnn @xcite , and dtw @xcite models to explore demographic effects on speaker verification. for this purpose, they used a subset of the mpower study @xcite , a parkinson's disease mobile dataset, comparing a diverse group of 300 speakers by race and gender. their results demonstrated that the latinx subgroup indicates the worst asv performance among the four major races in the dataset (i.e., white, black, latinx, and asian). based on their findings, gender represents minor differences in asv performance between male-only and femaleonly subgroups. we did similar gender-level and accent-level analyses on patients with alzheimer's disease to detect the potential sources of bias in asv due to the inherent voice characteristics of distinct genders or english accents. age effect: @xcite analyzed the effect of long-term ageing on asv performance. they utilized a conventional gmm-ubm system @xcite on a longitudinal voice dataset of a cohort of 13 adult speakers, whose recordings were collected over a time span of 30-40 years. according to their results, short-term aging (less than 5 years) does not have a significant impact on verification performance, compared to normal inter-session variations. however, for longer periods, aging has a negative effect on veri-fication accuracy. moreover, the researchers found that the rate of verification score decline is more rapid for speakers aged 60 years and above. however, they only evaluated their models on small cohorts and inter-speaker differences across different age groups were not further analyzed, while in the present work, we evaluate the asv performance across different age groups over 55 years old and incorporate a larger clinical dataset with 659 speakers in total, while we controlled for ad effect. taylor et al. (2020) also demonstrated that some speech and vocal characteristics (e.g., the spectral center of gravity, spectral skewness, or spectral kurtosis) undergo alterations with aging, and these changes can vary between men and women. these findings suggest that age is an effective factor in speaker's voice characteristics and this underscores the importance of assessing age effect on our asv model to ensure the fairness of the model across different age groups. noise effect: wan (2017) applied librispeech corpus @xcite of english novel reading speech with varying lengths and tested asv performance across different types and levels of background noise (e.g., babble, car, office and airplane noise) with a great mismatch between training and testing speech. based on their findings, performance varies across different types of noise and the number of errors grow with a decrease in the sound-to-noise-ratio value. however, other metrics of audio quality were not considered in their study and their models were only trained and tested on healthy speech recordings. in the present study, we assess the effect of other audio quality aspects, such as participant clarity, clinician interference, and background noise, on asv performance using a dataset of speakers with varying severity levels of ad, which was collected in clinical environment. ## results and discussion in order to have a baseline level of asv performance, we evaluated the titanet model on all the speech recordings of the adct dataset and obtained a 3.1% eer. to further analyze the impact of participant demographic characteristics, audio quality and ad severity level, we then recalculated eer and compared the asv performance across different subgroups. ## conclusion large-scale clinical trials require accurate verification of participants, as duplicate participation may lead to substandard data quality and significant financial and health risks. therefore, developing accurate asv models for verifying participant identity is essential in these settings. external fac-tors such as participant profile or audio quality can cause errors and biases in asv performance during the trials, but limited research has been conducted in this area. in the present work, we utilize a longitudinal speech dataset of participants with varying levels of ad severity and investigate the impact of external factors, such as different participant demographic characteristics, audio quality criteria, and ad severity levels, on the performance of an end-to-end ti asv model. our findings show that variations in asv performance can be attributed to the inherent voice characteristics of different subgroups (e.g., different ages, genders, accents, or ad severity levels) that are likely to be confused with the identity of the speaker. hence, it is critical to reassess this technology to mitigate the risk of potential biases toward certain subgroups. based on our results, poor audio quality with unclear speech, noisy background, and clinician interference also negatively impacts asv performance. this highlights the importance of quality assurance for the speech recordings during the trials. in future work, we aim to automate the audio quality assessment process by leveraging existing automated methods such as perceptual evaluation of speech quality (pesq) @xcite or short-time objective intelligibility (stoi) @xcite , which would reduce the human effort required for this task.