Towards the Development of Speech-Based Measures of Stress Response in Individuals (2021)

Abstract: Psychological and physiological stress in the environment can induce a different stress response in different individuals. Given the causal relationship between stress, mental health, and psychopathologies, as well as its impact on individuals’ executive functioning and performance, identifying the extent of stress response in individuals can be useful for providing targeted support to those who are in need. In this paper, we identify and validate features in speech that can be used as indicators of stress response in individuals to develop speech-based measures of stress response. We evaluate effectiveness of two types of tasks used for collecting speech samples in developing stress response measures, namely Read Speech Task and Open-Ended Question Task. Participants completed these tasks, along with the verbal fluency task (an established measure of executive functioning) before and after clinically validated stress induction to see if the changes in the speech-based features are associated with the stress-induced decline in executive functioning. Further, we supplement our analyses with an extensive, external assessment of the individuals’ stress tolerance in the real life to validate the usefulness of the speech-based measures in predicting meaningful outcomes outside of the experimental setting.

## introduction various psychological and physiological stress conditions, e.g., an approaching deadline, an interview not going well, a combat situation, or extreme temperatures, can have an impact on an individual due to various (maladaptive) physiological and mental processes @xcite . long-term exposure to stress can play a significant role in the formation and exacerbation of mental disorders, such as anxiety disorders, depression, and schizophrenia @xcite @xcite . stress in one's environment may result in a relatively immediate (whether short-term or longlasting) effect on performance. for example, a condescending interviewer may lead an interviewee to not be able to respond at all or a sudden combat situation may lead an individual to make more errors. however, different individuals respond differently to the same stress conditions depending on their mental and physiological constitution, experiences, training and preparedness, among other factors. identifying the degree of stress response in individuals would be helpful in reducing stress' impact on their health and performance both at the individual and at the community level. for example, an automatic measure of stress response can be used by an individual for self monitoring and deciding to use a management strategy of daily stress reduction exercises when needed. similarly, community members can be supported through targeted allocation of mental health resources. an automatic measure of stress response can provide additional information about individuals' response to stress to the clinicians treating them so that appropriate and timely therapeutic support can be provided. previously, self-report inventories of stressors and their symptoms have been used to measure individuals' stress response (e.g., @xcite @xcite . however, these inventories are limited in their scope (e.g., sports, school) and utilities. further, self-report inventories have inherent problems. for example, individuals may not be fully aware of the effect stressors have on them, or they may not answer questions truthfully. therefore, development of more objective yet accessible measures of stress response are needed. an extensive body of research has shown the impact of stress on speech, e.g., @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite have associated certain changes in speech with exposure to stress. for example, @xcite reported an increase in fundamental frequency, amplitude and speech rate in extreme levels of stress. they also reported changes in the energy distribution, frequency jitter and amplitude shimmer in stressed speech. features, such as mel-frequency cepstrum coefficients, have been found to be affected by emotional states including anxiety/stress to be useful for identifying or classifying these emotional states, e.g., see @xcite al. (2020); dhole and kale (2020); tomba et al. (2018) ; @xcite . the primary goal of the current project is to extend this work by identifying and validating a set of speech-based features that can be used as individual difference measures of stress response. to achieve this goal, we use two continuous speech sample collection methods, namely a read speech task and an open-ended question task. read speech provides much cleaner data than spontaneous speech and hence can be very useful for modeling speech related phenomena. it has been extensively used in speech processing studies, for example, see @xcite ; @xcite ; @xcite ; @xcite . open-ended question task, on the other hand, provides more naturalistic data, which can complement the information available in read speech. the notion that they may provide different types of information is confirmed by works such as @xcite which discussed the need for developing methods for different speaking styles instead of just focusing on read speech. for the purpose of developing measures for stress response, we consider stress response and stress tolerance to be two facets of the same phenomenon, where stress response refers to how an individual responds to or is affected by stress, whereas stress tolerance refers to how tolerant an individual is to stress (i.e., how well they can still perform tasks under stress). we focus on investigating speech and identifying relevant acoustic features to develop a speech-based measure of stress response. we expect such a measure to also be informative about an individual's stress tolerance. to this end, we establish the relationship of speech features with a complex, ecologically valid stress tolerance measure based on trained judges' stress tolerance assessment of individuals as described in section 2.3. the rest of the paper is organized as follows: in section 2, we describe our data and data collection procedures. in section 3, we discuss our methodology to extract acoustic features from the speech produced by individuals in stress conditions and to prepare the performance assessment measures for evaluating the usefulness of the extracted features. in section 4, we present our findings based on the analysis of extracted features from speech in terms of their relationship with stress conditions as well as with cognitive performance and real-life stress tolerance measures. in section 5, we discuss our results and implications for development of speechbased measures for stress response in individuals. in addition to the features identified by our investigation to be informative of an individual's stress response, we also provide recommendations with respect to the types of tasks used to collect speech samples to extract these features. in section 6, we conclude and briefly discuss future work. this is followed by section 7 where we discuss a few use cases and ethical considerations for this work. ## data the data used for our investigation were collected from 13 male participants. they were recruited from among the candidates going through a weeklong selection assessment process at a us military unit who had provided consent prior to the selection week and remained on-site until the end of the selection week. they were provided a description of the study before obtaining their consent. the data collection was conducted the day after the selection week was over. all protocols for data collection were approved by the institutional review board at the appropriate branch of the us military (where data had to be collected) as well as the institutional review board at the florida institute for human and machine cognition (where the research activity had to take place) prior to data collection. two types of data were collected: speech samples and selection assessment data. the speech samples were recorded in a lab setting in two stress conditions, namely neutral and stress. section 2.1 provides details about stress induction and section 2.2 provides more details about speech data collection. in addition to the speech samples, participants' scores that were assigned during the selection week by trained and experienced us mil-figure 1 : a demonstration of salivary gluco-corticol stress response induced by mast @xcite itary assessment personnel were obtained to augment our analyses. more details about the selection assessment data are provided in section 2.3. ## analysis and findings one can expect that the features in speech indicative of an individual's stress response can also be taken to indicate that the corresponding speech was produced in a stress condition. hence, one strategy to identify potentially relevant features associated with stress response from among the 45 extracted features (44 in case of open-ended question task) is to find the ones that can distinguish between the neutral and the stress conditions. hence, we performed a paired sample t-test on our data where the extracted features in the two stress conditions were taken as the two sets of observations pre-and post-stress induction. 3 we found that a number of speech features, extracted from read speech task and open-ended question task, showed statistically significant difference between the means in the observations corresponding to the two stress conditions, as shown in we found that duration and a time domain feature zero crossing rate extracted from read speech task showed significant difference between the two stress conditions with @xmath0.005. 4 the rest of the features that showed significant difference with varying p-values (@xmath1.01 or @xmath2.05) were all associated with the frequency domain features mfccs or their derivatives. some features showed significant difference when extracted from either of the two speech tasks. for example, the time domain feature zero crossing rate showed significant difference when extracted from the open-ended question task speech samples as well, although with comparatively less significance than when extracted from the read speech task speech samples. in contrast, the frequency domain feature mfcc2 showed more significance when extracted from the open-ended question task samples than when extracted from the read speech task samples. the frequency domain features delta mfcc4 (first derivative of mfcc4) and delta delta mfcc4 (second derivative of mfcc4) showed significant difference between the means for observations in the two stress conditions when they were extracted from read speech task, but not when they were extracted from open-ended question task. similarly, second derivatives of mfcc12, mfcc11, mfcc2 and mfcc10 showed significant difference in the two stress conditions when they were extracted from open-ended question task but not when they were extracted from read speech task. while more features extracted from the speech samples collected for open-ended question task showed significant difference in the means between the two stress conditions, the features extracted from the read speech task speech samples showed a higher significant difference between the two conditions, in general. we then correlated the extracted features with verbal fluency task-based scores representing the change in cognitive performance due to stress to provide a measurement of stress response in participants (performance score), as described in section 3.2. similarly, we correlated these acoustic features with the selection assessment-based scores that provided a measurement of stress tolerance in participants (stress tolerance score), as described in section 2.3. in regards to the hypothesis about the relationship between acoustic features and performance scores/stress tolerance scores, we found that there were a number of acoustic features in both speech tasks that showed high to moderate correlations with performance scores as well as with stress tolerance scores as shown in in regards to the second hypothesis about the highly differential features for stress conditions to also be correlated with the performance and the stress tolerance scores, we found that there is some overlap between the two sets of features. however, not all features that significantly distinguished the stress conditions were also strongly/moderately correlated with the performance and the stress tolerance scores for the two speech tasks. duration extracted from read speech task was found to be significantly differential for stress conditions and it showed moderate correlation with the performance score in read speech task which was also significant. however, it was not found to be correlated with stress tolerance (with r(13) = .184 for read speech task and .313 for open-ended question task). zero crossing rate, on the other hand, was found to be significantly differential for the stress conditions for both read speech task and open-ended question task, but it did not show significant correlations with the performance and the stress tolerance scores for either of the tasks. similarly, mfcc2, while significantly differential for stress conditions for both the tasks, showed low correlations with the performance and the stress tolerance scores. delta mfcc4 and delta delta mfcc4 were significantly differential for read speech task but showed low correlations with both the scores for both the tasks. delta delta mfcc12, delta delta mfcc11, delta delta mfcc2 and delta delta mfcc10 were significantly differential for open-ended question task, but showed low correlations with both the scores for both the tasks. while exploring the above two hypotheses, we found a subset of acoustic features (e.g., duration, magnitude spectrum, some of the mfccs or their derivatives) that showed strong to moderate correlations with the performance score (stress response) or the stress tolerance score, answering our first question setforth in section 1. next, we explored the effectiveness of speech tasks in indicating an individual's stress response to answer the second question setforth in section 1. based on the correlations in ## towards building a speech-based measure of stress response/tolerance in section 4, we identified a number of acoustic features that strongly/moderately correlated with verbal fluency task-based performance scores and external assessment-based stress tolerance scores. we found that duration (speech rate) and frequency domain features, such as magnitude spectrum and some of the mfccs or their derivatives, showed strong to moderate correlations with stress re-sponse/stress tolerance that were significant. time domain features, on the other hand, did not show significant correlations with either stress response or stress tolerance. although this finding needs to be confirmed with larger data, this may be taken to indicate the effectiveness of duration and frequency domain features over time domain features in a speech-based measure for stress response/tolerance. it should be noted that some of these duration and frequency domain features, e.g., speech rate, mfccs and their derivatives have also been found to be indicative of stress in prior works on stress detection from the speech signal, e.g., @xcite ; dhole and kale (2020); @xcite ; @xcite , to name a few. also, opportunities to validate stress response or any predictive features against rich, ecologically valid datasets like the selection assessment data used in our experiments are rare, and it is encouraging to see significant correlations between some of these features and the stress tolerance score based on the selection assessment data ( we compared the two speech collection tasks, read speech task and open-ended question task, for their effectiveness in providing useful informa-tion in speech features for stress response/tolerance. the findings showed that these two tasks provided complementary information in that a different set of features correlated with stress response/tolerance when samples were collected through the two tasks. this makes sense given the fact that read speech task provides cleaner speech samples whereas open-ended question task provides more naturalistic speech. this suggests that a speech-based measure for stress response/tolerance would benefit from using both these tasks for data collection. however, if only one task needs be used in order to minimize time, effort, and other resources expended in data collection, open-ended question task should be used since open-ended question task samples led to a larger number of correlated acoustic features which showed strong correlations with high significance values. another finding from section 4 was that the features helpful in distinguishing between the stress conditions may not necessarily be the features that indicate stress response/tolerance. this may reflect that certain aspects of an individual's speech, while being affected by environmental stressors, may not involve the same mechanism as performance decline. however, there may be other aspects of speech that are involved in this way and hence correlated with stress response/tolerance. this may suggest that all speech features are not equal when identifying their relationship with stress response (performance decline) or stress tolerance. environmental stress' effect on an individual's speech does not imply an effect on their cognitive performance to the same extent or in the same way. this finding supports a recommendation that while speech features employed for stress detection can be potential candidates for stress response/tolerance prediction, they do not provide an exhaustive set of features useful for such predictions and hence further features should be explored while developing measures for stress response/tolerance. the flipside of the above finding is that a number of features that did not show significant difference between the neutral and stress conditions nevertheless showed a good correlation with the stress-induced change in executive functioning (i.e., verbal fluency task) and with the external assessment of stress tolerance. to provide further support for the recommendation made above, this finding may suggest that subtle differences that do not reach significance in distinguishing between the neutral and stress conditions may still be useful as an indicator for stress response. alternatively, this mismatch between the features that appear more frequently in the stress conditions and the features that predict stress response/tolerance elsewhere could be due to the difference in the type of stress between our stress induction method and the stress experienced by the participants during the selection week (for the selection assessment data) or the small sample size. hence, for development of a reliable speech-based measure for stress response/tolerance, further exploration would be useful to test the mismatch hypothesis above, and with larger datasets. ## conclusion and future work the main contribution of this paper is to present a proof of concept identifying potential language markers for stress response which we plan to extend and refine in the future with more focused and larger trials. specifically, we have explored the usefulness of specific acoustic features extracted from speech produced in two stress conditions for their relationship with stress response and stress tolerance. we identified duration and a number of frequency domain features that are significantly correlated with stress response or tolerance. in the future, we plan to extract further acoustic features to test a more extensive list of features as potential candidates for involvement in the development of the speech-based measure. we also tested the effectiveness of two continuous speech sample collection tasks, read speech task and open-ended question task, in developing speech-based measures for stress response and tolerance. we found that both tasks provided complementary information in the speech features and hence it can be beneficial to use both these tasks for data collection. however, if one of the tasks needs to be selected, open-ended question task would provide more information that has consequences for stress response/tolerance. since this study involved only 13 participants, in order to confirm the current results and develop a more reliable and robust measure of stress response/tolerance, we plan to extend it further by increasing the sample size. the participants in the current study belong to a very particular population, young males aspiring to serve in a military unit. one can reasonably assume that this special population tends to have high stress tolerance. testing general population with the current experimental design might reveal greater range of stress response scores. hence, training on the general population may lead to greater predictive validity. additionally, in this study, we focused on correlations as a measure of the relationship between extracted acoustic features and stress response/tolerance. in our future work, we plan to use the identified acoustic features to develop and train machine learning models to predict the stress response/tolerance scores. additionally, openended question task's capability of extracting semantics-based features, such as sentiment and topic, will also be used to develop robust models for stress response and stress tolerance predictions.