On the Utility of Audiovisual Dialog Technologies and Signal Analytics for Real-time Remote Monitoring of Depression Biomarkers (2020)

Abstract: We investigate the utility of audiovisual dialog systems combined with speech and video analytics for real-time remote monitoring of depression at scale in uncontrolled environment settings. We collected audiovisual conversational data from participants who interacted with a cloud-based multimodal dialog system, and automatically extracted a large set of speech and vision metrics based on the rich existing literature of laboratory studies. We report on the efficacy of various audio and video metrics in differentiating people with mild, moderate and severe depression, and discuss the implications of these results for the deployment of such technologies in real-world neurological diagnosis and monitoring applications.

## introduction diagnosis, detection and monitoring of neurological and mental health in patients remain a critical need today. this necessitates the development of technologies that improve individuals' health and well-being by continuously monitoring their status, rapidly diagnosing medical conditions, recognizing pathological behaviors, and delivering just-intime interventions, all in the user's natural information technology environment @xcite . however, early detection or progress monitoring of neurological or mental health conditions, such as clinical depression, amyotrophic lateral sclerosis (als), alzheimer's disease, dementia, etc., is often challenging for patients due to multiple reasons, including, but not limited to: (i) lack of access to neurologists or psychiatrists; (ii) lack of awareness of a given condition and the need to see a specialist; (iii) lack of an effective standardized diagnostic or endpoint for many of these health conditions; (iv) substantial transportation and cost involved in conventional or traditional solutions; and in some cases, (v) shortage of medical specialists in these fields to begin with @xcite . we developed nemsi @xcite , or the neurological and mental health screening instrument, to bridge this gap. nemsi is a cloud-based multimodal dialog system that conducts automated screening interviews over the phone or web browser to elicit evidence required for detection or progress monitoring of the aforementioned conditions, among others. while intelligent virtual agents have been proposed in earlier work for such diagnosis and monitoring purposes, nemsi makes novel contributions along three significant directions: first, nemsi makes use of devices available to everyone everywhere (web browser, mobile app, or regular phone), as opposed to dedicated, locally administered hardware, like cameras, servers, audio devices, etc. second, nemsi's backend is deployed in an automatically scalable cloud environment allowing it to serve an arbitrary number of end users at a small cost per interaction. thirdly, the nemsi system is natively equipped with real-time speech and video analytics modules that extract a variety of features of direct relevance to clinicians in the neurological and mental spaces. a number of recent papers have investigated automated speech and machine vision features for predicting severity of depression (see for example @xcite @xcite @xcite @xcite . these include speaking rate, duration, amplitude, and voice source/spectral features (fundamental frequency (f0), amplitude modulation, formants, and energy/power spectrum, among others) computed from the speech signal, and facial dynamics (for instance, landmark/facial action unit motions, global head motion, and eye blinks) and statistically derived features from emotions, action units, gaze, and pose derived from the video signal. we use these studies to inform our choices of speech and video metrics computed in real time, allowing clinicians to obtain useful analytics for their patients moments after they have interacted with the nemsi dialog system. we need to factor in additional considerations while deploying analytics modules as part of scalable real-time cloud-based systems in practice. many of the studies above analyzed data recorded either offline or in laboratory conditions, implicitly assuming signal conditions which may hold differently or not at all during real world use. these considerations include, but are not limited to: (i) wide range of acoustic environments and lighting conditions resulting in variable background noise and choppy/blocky video at the user's end foot_0 , (ii) limitations on a given user's network connection bandwidth and speed; (iii) the quantum of server traffic (or the number of patients/users trying to access the system simultaneously); and (iv) device calibration issues, given the wide range of user devices. this paper investigates the utility of a subset of audio and video biomarkers for depression collected using the nemsi dialog system in such real-world conditions. the rest of this paper is organized as follows: sections 2 and 3 first present the nemsi dialog system and the data collected and analyzed. section 4 then details the speech and video feature extraction process. section 5 presents statistical analyses of different groups of depression cohorts as determined by the reported phq-8 score, before section 6 rounds out the paper, discussing the implications of our observations for real-world mental health monitoring systems. nemsi end users are provided with a website link to the secure screening portal as well as login credentials by their caregiver or study liaison (physician or clinic). once appropriate microphone and camera checks that the captured audio and video are of sufficient quality are complete, users hear the dialog agent's voice and are prompted to start a conversation with the agent, whose virtual image also appears in a web window. users are also able to see their own video, if so needed, in a small window in the upper right corner of the screen. the virtual agent then engages with users in a conversation using a mixture of structured speaking exercises and open-ended questions to elicit speech and facial behaviors relevant for the type of condition being screened for. analytics modules extract multiple speech (for instance, speaking rate, duration measures, f0, etc.) and video features (such as range and speed of movement of various facial landmarks) and store them in a database, along with information about the interaction itself such as the captured user responses, call duration, completion status, etc. all this information can be accessed by the clinicians after the interaction is completed through an easy-to-use dashboard which provides a high-level overview of the various aspects of the interaction (including the video thereof and analytic measures computed), as well as a detailed breakdown of the individual sessions and the underlying interaction turns. ## data depending on the health condition to be monitored and on the clinician's needs, different protocols can easily be employed in the nemsi system. for the present study, we designed a protocol targeting the assessment of depression severity, based on @xcite . the protocol elicits five different types of speech samples from participants that are consistently highlighted in the literature: (a) free speech (open-ended questions about subjects' emotional and physical state), (b) automated speech (counting up from 1), (c) read speech, (d) sustained vowels, and (e) measure of diadochokinetic rate (rapidly repeating the syllables /pa ta ka/). after dialog completion, participants are asked to answer the patient health questionnaire eightitem depression scale (phq-8), a standard scoring system for depression assessment @xcite . the self-reported phq-8 score serves as a reference point for our analysis. further, we ask for information about age, sex, primary language and residence. in total, we collected data from 307 interactions. after automatic data cleaning 2 , 208 sessions re-mained for analysis. from those 208 participants, 98 were females, 97 were males and 13 did not specify. mean participant age is 36.5 (@xmath0.1). 184 participants specified english as their primary language, 9 other languages and 15 did not specify. 176 participants were located in the us, 8 in the uk, 5 in canada, 4 in other countries and 15 did not specify. figure 1 shows the distribution of phq-8 scores among women and men. 4 signal processing and metrics extraction 4.1 speech metrics for the speech analysis, we focus on timing measures, such as speaking rate and percentage of pause duration, frequency domain measures, such as fundamental frequency (f0) and jitter, and energy-related measures, such as volume and shimmer. we have selected commonly established speech metrics for clinical voice analysis @xcite @xcite . as described in section 3, there are different types of speech samples, e.g. free speech and sustained vowels. not all acoustic measures are meaningful for each type of stimuli. all metrics are extracted with praat @xcite . for the following measures, heuristics have been used to ignore obvious outliers in the analysis: articulation rate (@xmath1, speaking rate (@xmath2, percent pause time (@xmath3. cutpoint free speech held vowels (group sizes) 5 (127/81) percent pause volume (a,f), hnr (m), time (a,f) mean f0 (m) 10 (168/40) -jitter (f) 15 (193/15) volume (a,f,m) mean f0 (a), volume (f) ## analyses and observations the central research question of this study is the following: for a given metric, is there a statistically significant difference between participant cohorts with and without depression of a given severity (i.e., above and below a certain cut-point phq-8 score)? the phq-8 has established cutpoints above which the probability of a major depression increases substantially @xcite . ten is commonly recommended as cutpoint for defining current depression (see @xcite for a comprehensive overview). for our analysis, we use the cutpoints 5, 10, and 15 which align with the phq-8 score intervals of mild, moderate and moderately severe depression. concretely, for each metric and cutpoint, we divide the data into two sample populations: (a) phq-8 score below and (b) phq-8 equal or above the cutpoint. we conducted a non-parametric kruskal-wallis test for every combination to find out whether certain obtained metrics show a statistically significant difference between cohorts. foot_3 ## conclusion and outlook we investigated whether various audio and video metrics extracted from audiovisual conversational data obtained through a cloud-based multimodal dialog system exhibit statistically significant differences between depressed and non-depressed populations. for several of the investigated metrics such differences were observed indicating that the employed audiovisual dialog system has a potential to be used for remote monitoring of depression. however, more detailed investigations on the nature of value distributions of metrics, their dependency on subject age or native language, the quality of input signals or used devices, among other studies, are necessary to to see to which degree the results are generalizable. additionally, the used phq-scores were self-reported and might therefore be less accurate than scores obtained under the supervision of a clinician. in future work, we will also collect additional interactions from larger and more diverse populations. furthermore, we will perform additional analysis on the obtained data, such as regression analysis. finally, we will extend the set of investigated metrics and investigate their efficacy for other neurological or mental health conditions.