MUSER : MU ltimodal Stress detection using Emotion Recognition as an Auxiliary Task (2021)

Abstract: The capability to automatically detect human stress can benefit artificial intelligent agents involved in affective computing and human-computer interaction. Stress and emotion are both human affective states, and stress has proven to have important implications on the regulation and expression of emotion. Although a series of methods have been established for multimodal stress detection, limited steps have been taken to explore the underlying inter-dependence between stress and emotion. In this work, we investigate the value of emotion recognition as an auxiliary task to improve stress detection. We propose MUSER – a transformer-based model architecture and a novel multi-task learning algorithm with speed-based dynamic sampling strategy. Evaluation on the Multimodal Stressed Emotion (MuSE) dataset shows that our model is effective for stress detection with both internal and external auxiliary tasks, and achieves state-of-the-art results.

## introduction stress is a feeling of emotional or physical tension, as a response to the environment when people have difficulty dealing with the conditions @xcite . stress detection is a classification task that predicts whether a certain target is under stress. the task has drawn research attention for two reasons: first, stress detection plays an important role in applications related to psychological well-being @xcite , cognitive behavior therapies @xcite , and safe driving @xcite ; second, stress is a known regulator of human emotion mechanisms @xcite , and thus research on stress detection can potentially benefit the development of emotionally intelligent agents. the impact of stress on human behavior can be observed through various modalities. previ-ous work has considered both unimodal and multimodal stress detection using acoustic, video and physiological sensor signals @xcite @xcite @xcite . however, text-based stress detection remains vastly underexplored, with some studies @xcite showing the potential for further research. in recent years, the surge of advanced natural language understanding models and structures provides a great opportunity for stress detection systems, especially using the textual modality. in this work, we focus on the textual and acoustic modalities. for the model architecture, we use transformers @xcite as a textual encoder and multi-layer perceptrons (mlp) as an acoustic encoder. the majority of existing stress detection methods are based on single-task learning with the binary stress/non-stress labels. however, stress is not an isolated affective state, but closely related to the expression and regulation of human emotions. physiological studies @xcite have demonstrated that emotion and stress share some neural structures, including prefrontal cortex @xcite , anterior cingulate cortex @xcite , and amygdala @xcite . acoustic studies @xcite have shown that the pitch and amplitude of human emotional prosody is different under stress and non-stressed status. inspired by these studies, our work aims to exploit the inter-dependence between emotion and stress. specifically, we investigate the value of emotion recognition as an auxiliary task for stress detection. multi-task learning @xcite @xcite has proven to be effective for transferring knowledge between different tasks. dynamic sampling strategies, which aim at adaptively adjusting the ratio of samples from different tasks, are widely used to balance the training schedule. however, strategies based on gradients @xcite , uncertainty @xcite or loss @xcite cannot leverage the validation performances, while some performance-based strategies @xcite are impractical if the metrics for different tasks are not directly comparable (i.e., with different scale ranges). to this end, we propose a novel speed-based strategy that is both effective and efficient in the multi-task learning for stress and emotion. our method is evaluated on the multimodal stressed emotion (muse) dataset @xcite , which includes both stress and emotion labels, making it the ideal benchmark for an in-depth analysis of their inter-dependence. to test the generalization ability of our method, we also use an external emotion dataset for the auxiliary task. multimodal emotion recognition is a wellstudied field with many existing datasets @xcite @xcite @xcite . we choose the omg-emotion dataset @xcite as the external auxiliary task because it is representative and challenging, with numerical emotion scores instead of categorical labels. our paper makes four main contributions. first, we show the inter-dependence between stress and emotion via quantitative analyses on linguistic and acoustic features, and propose to use emotion recognition as an auxiliary task for stress detection. second, we establish a stress detection model with a transformer structure, as well as a novel speed-based dynamic sampling strategy for multi-task learning. we name our framework the multimodal stress detector with emotion recognition (muser). third, we achieve stateof-the-art results on the muse dataset via multitask training with stress and emotion labels. we also achieve competitive results when we use the omg-emotion @xcite dataset as an external auxiliary task. finally, experimental results show that our speed-based dynamic sampling significantly outperforms other widely-used methods. ## conclusions in this work, we uncovered the connections and differences between stress detection and emotion recognition using textual and acoustic features, and proposed to use emotion recognition as an auxiliary task for stress detection. we proposed muser: a transformer-based model structure, together with a novel speed-based dynamic sampling strategy for multi-task learning. experimental results support the inter-dependence of stress and emotion (activation/valence), and proves the effectiveness and robustness of our methods. muser achieved state-of-the-art results on the muse stress detection task both when internal (muse) and when external (omg-emotions) emotion data and annotations were used. our code is publicly available at https://lit.eecs.umich.edu/ downloads.html#muser