Multimodal Topic-Enriched Auxiliary Learning for Depression Detection (2020)

Abstract: From the perspective of health psychology, human beings with long-term and sustained negativity are highly possible to be diagnosed with depression. Inspired by this, we argue that the global topic information derived from user-generated contents (e.g., texts and images) is crucial to boost the performance of the depression detection task, though this information has been neglected by almost all previous studies on depression detection. To this end, we propose a new Multimodal Topic-enriched Auxiliary Learning (MTAL) approach, aiming at capturing the topic information inside different modalities (i.e., texts and images) for depression detection. Especially, in our approach, a modality-agnostic topic model is proposed to be capable of mining the topical clues from either the discrete textual signals or the continuous visual signals. On this basis, the topic modeling w.r.t. the two modalities are cast as two auxiliary tasks for improving the performance of the primary task (i.e., depression detection). Finally, the detailed evaluation demonstrates the great advantage of our MTAL approach to depression detection over the state-of-the-art baselines. This justifies the importance of the multimodal topic information to depression detection and the effectiveness of our approach in capturing such information.

## introduction depression detection is a task of determining a human being is depressed or non-depressed by automatically analyzing user-generated contents (ugc). due to its crucial role in assessing mental health, it has recently received considerable attention from several research communities, such as nlp @xcite and cv @xcite . these studies mainly utilize ugc (e.g., texts and images) on social media to perform depression detection and achieve promising results since ugc instantly reflects not only the daily lives but also the mental states of users. despite the progress of prior studies, they always focus on leveraging rnn variants (e.g., gru) to model texts or images along the timeline (see figure 1 ) posted by users, suffering from the problem of ignoring the global topic information inside these texts and images, though obviously this global topic information can mitigate the notorious difficulty of rnn variants w.r.t. modeling long-range dependencies @xcite . more importantly, from the point of health psychology, humans plagued by the negative emotion for a prolonged period of time (generally longer than two weeks), leading to their inability of carrying out daily activities, are highly probable to have the depression symptom @xcite . for instance, figure 1 illustrates a month-long timeline of a depressed user. from this figure, we can see that this person is highly possible to be depressed since he/she discloses negative emotions lasting for almost a month. this conforms to his/her depression symptom and indicates the importance of considering the global semantics for depression detection. inspired by the above observations, this paper hypothesizes that it is desirable to consider the global topic information inside multiple modalities (i.e., texts and images) for depression detection. still take figure 1 as an example, leveraging a proper topic model to mine global clues inside the texts, e.g., words ## related work depression detection is an interdisciplinary research task and has been drawing ever-more attention in nlp with a focus on extracting various types of features from text modality @xcite . compared with the studies on the text modality, the studies on multimodality (e.g., both the text and image modalities) like @xcite are much less and limited to neglect the topic information inside multiple modalities. in the following, we will first review the depression detection task and then introduce the related studies on neural topic models. ## multimodal topic-enriched auxiliary learning figure 2 shows the framework of our proposed multimodal topic-enriched auxiliary learning (mtal) approach for depression detection, which consists of one primary task and two auxiliary tasks. the primary task is exactly the depression detection task (introduced in section 3.1). two auxiliary tasks are the textual and visual topic modeling respectively (together with the proposed modality-agnostic topic model are introduced in section 3.2). finally, a topic-enriched auxiliary learning strategy is proposed to combine the primary task with auxiliary tasks (introduced in section 3.3). ## experimentation to validate the effectiveness of our approach, we evaluate the performance of the proposed and baseline approaches and show the details in to facilitate this corresponding research, the dataset with the new data setting is released as the new benchmark dataset for multimodal depression detection via github foot_1 . implementation details. in our experiments, all hyper-parameters are tuned according to the development set. specifically, bert is optimized by the adam optimizer @xcite , where β 1 = 0.9 and the initial learning rate is 10 -4 . other parameters of bert are following @xcite . for our mtal approach, we set the dimensions of lstm hidden states to be 256 and adopt another adam optimizer @xcite with an initial learning rate of 10 -2 and β 1 = 0.9 for cross-entropy training. the regularization weight is 10 -5 . the dropout rate is 0.5. for cnn, we set the widths of filters to 3, 4, 5 with 100 features each. the dimension m of topic embeddings in our topic model is 128 and the number of topics (k) is 20. besides, if a tweet includes no image, the image vector will be initialized as a zero vector. evaluation metrics. the performance is evaluated using standard accuracy (acc.) and macro-f1 (f1) by following @xcite . moreover, t-test is used to evaluate the significance of the performance difference between two approaches by following @xcite . baselines. for comparison, we re-implement several approaches as baselines for depression detection. 1) h-lstm. this is a hierarchical lstm approach to aspect sentiment classification. in our implementation, we use it to model word sequence and text sequence for depression classification. 2) bert+lstm. this is a bert model for encoding each text, followed by an lstm to encode the text sequence for depression classification. 3) bert+lstm+textual topic modeling. this is an extension of bert+lstm with textual topic information. 4) vgg+lstm. this is a vgg model for encoding each image, followed by an lstm to encode the image sequence. 5) vgg+lstm+visual topic modeling. this is an extension of vgg + lstm with visual topic information. 6) ef-lstm @xcite . this is a state-of-the-art multimodal approach to human communication comprehension task. 7) comemory @xcite . this is a state-of-the-art multimodal approach to the multimodal sentiment analysis task. 8) coatt @xcite . this is a state-of-the-art multimodal approach to the named entity recognition task. 9) hybrid attention @xcite . this is a state-of-the-art multimodal approach using modality attention to learn modality-specific and modality-fusion features for the spoken language classification. note that the above four multimodal approaches can only encode single text+image pair. in our implementation, we also use an lstm to encode the final vector sequence of all text+image pairs for depression classification. 10) comma @xcite . this is exactly a state-of-the-art multimodal approach to depression detection. in this study, we re-implement it based on our new data setting. 11) primary task. our approach w/o integrating two auxiliary tasks. ## conclusion in this paper, we propose a novel multimodal topic-enriched auxiliary learning approach to depression detection. the main idea of the proposed approach is to incorporate not only the textual topic information but also the visual topic information for depression detection. experimental results demonstrate that the proposed approach significantly outperforms a number of competitive baselines, including the representative textual depression detection approaches and the state-of-the-art multimodal-based approaches. in our future work, we would like to explore more information, such as the user personal attribute information (such as profession, location and age) and the social attribute information (e.g., timeline, social behavior and social relationships), to assist depression detection. in addition, we would like to apply our approach to other psychological analysis tasks, such as multimodality-based emotion analysis, anxiety detection and personality inference.