U SING G RAPH R EPRESENTATION L EARNING WITH S CHEMA E NCODERS TO M EASURE THE S EVERITY OF D EPRESSIVE S YMPTOMS
A BSTRACT
Graph neural networks (GNNs) are widely used in regression and classiﬁcation problems applied to text, in areas such as sentiment analysis and medical decisionmaking processes.
We propose a novel form for node attributes within a GNN based model that captures node-speciﬁc embeddings for every word in the vocabulary.
This provides a global representation at each node, coupled with node-level updates according to associations among words in a transcript.
We demonstrate the efﬁcacy of the approach by augmenting the accuracy of measuring major depressive disorder (MDD).
Prior research has sought to make a diagnostic prediction of depression levels from patient data using several modalities, including audio, video, and text.
On the DAIC-WOZ benchmark, our method outperforms state-of-art methods by a substantial margin, including those using multiple modalities.
Moreover, we also evaluate the performance of our novel model on a
Twitter sentiment dataset.
We show that our model outperforms a general GNN model by leveraging our novel 2-D node attributes.
The performance of our work demonstrates the generality of the proposed method.
I NTRODUCTION
Nowadays, deep learning techniques have been frequently used in sentiment analysis (Tan et al., 2013; Mukhtar & Khan, 2018;
Liao et al., 2021), particularly in movie recommendations and product reviews and ratings.
In the healthcare domain, sentiment analysis with deep learning provides many beneﬁts such as utilizing medical information to achieve the best result to increase healthcare quality (Abirami & Askarunisa, 2017; Bi et al., 2020).
We focus our work on applications related to mental health such as using deep learning to make a diagnostic prediction.
There is a pressing need to ﬁnd a convenient and automated method to assess depression severity.
Worldwide, more than 300 million people are suffering depression (Organization et al., 2017).
Apart from the high prevalence of Major Depressive Disorder, overall 85% of depressed individuals are underdiagnosed (Falagas et al., 2007).
Research has shown that about 30% of patients suffering from an episode of major depression do not seek treatment, with only 10% of them being adequately treated (Falagas et al., 2007).
We validate our approach based on its application to the diagnosis of
MDD.
We demonstrate that our approach can result in more accurate predictions of the severity of depression, which can enhance automated health care decision making.
Medical research (Tsakalidis et al., 2018) shows that text sentiment analysis methods can be effective in making inferences about people’s mental states, such as MDD.
For example, depressed people use more ﬁrst person pronouns than non-depressed people because they are more focused on themselves and less connected with others.
These may not be reﬂected in a person’s visual appearance, i.e., visual behaviors.
Visual data alone will therefore be insufﬁcient for capturing depressive symptoms that are only manifested in a person’s verbal utterances.
Mental disorders are difﬁcult to detect.
In practice, clinicians ﬁrst measure the severity of depressive symptoms in patients and then identify depression in them.
During in-person clinical interviews, clinicians widely use a structured Patient Health Questionnaire (PHQ) — a clinically validated tool determining the severity of depressive symptoms across several personal dimensions (Kroenke &
Spitzer, 2002; Arseniev-Koehler et al., 2018).
Inspired by the observation of the provided data set of such interviews, we formulated the hypothesis that the context of words in a transcript can be used to generate PHQ scores (Kroenke et al., 2009).
This PHQ metric for depressive disorder forms a context constructed with eight items, such as “sleep problem”, “anxiety problem”, “fatigue problem”, “depression problem”, and “no motivation or interest in things” (Arseniev-Koehler et al., 2018).
We observed that each text (see ﬁgure 1) covers information relating to at least one of these eight topics.
This motivates use of a deep learning model that represents these eight PHQ topics, thereby introducing an inductive bias into the learning process.
Thus the contextual information about these eight PHQ topics should be encoded to determine depression states.
We motivate generating graph-level representations for transcripts (as inputs) to encode contextual information from these transcripts (we assume a transcript encodes facts representing depressive symptoms).
To achieve this, we propose a novel form for node attributes within a GNN based model that captures node-speciﬁc embeddings for every word in the vocabulary.
The representations of each word is shared globally and can be updated according to associations among words in a transcript.
We summarize the representations of all the words in the transcript to predict depression states.
Intuitively, the generated node-level embeddings maintain records linking certain existing facts that indicate the known symptoms of a subject.
When subsequent facts are discovered which indicate additional depressive symptoms, records will be updated by aggregating both old and new facts.
In other words, those records will be updated via a message passing mechanism over the transcript context until ideally all major depressive features (characterizing depressive symptoms) have been discovered.
In this way, we can represent the most relevant contextual information which is universal across all transcripts – possibly involving the context of eight PHQ topics we discussed above.
Using graph structures to capture context-level features is an innovative pathway which we hypothesise can be used to measure different levels of depression.
In our experiments, our approach realized a good generalization on a limited, incomplete, and unevenly distributed dataset (see ﬁgure 3a).
1. R ELATED W ORK
Research in measuring the severity of depressive symptoms aims to train a regression model to predict depression scores (Valstar et al., 2016; Alhanai et al., 2018;
Tsakalidis et al., 2018).
Some prior work (Ringeval et al., 2017; Alhanai et al., 2018; Haque et al., 2018) applies a sequencelevel deep learning model to capture implicit depressive signals.
Such models, in general, use a multi-modal sentence embedding, rather than a mapping of a whole interview, to predict a PHQ score.
Other prior work (Valstar et al., 2016; Cummins et al., 2017) implements statistical functions (e.g., max, min) on short-term features over an entire interview, but this may fail to preserve useful temporal information across an entire interview, such as some short-term signs of regret, anxiety, etc.
Current machine learning methods (Valstar et al., 2016; Alhanai et al., 2018; Song et al., 2018; Du et al., 2019) concentrate on learning features from multi-modalities –– as determinants for estimating the severity of depressive disorders.
The method (Lin et al., 2020; Solieman & Pustozerov, 2021) applies a multi-modal fusion network that summarizes all depressive features from both text and audio.
The work (Lam et al., 2019) uses multi-head attention modules to extract contextual information from clinical text to depression detection.
The methods (Ray et al., 2019; Mallol-Ragolta et al., 2019; Zhang et al., 2020; Solieman & Pustozerov, 2021) utilize deep learning models with pretrained global word embeddings to extract contextual features from text.
They only apply their work to the depression detection task.
In principle, we should be able to improve performance further using all available modalities, however feature-fusion method may lead to learn a wrong representation encoding inter-subject variability unrelated to depression (Williamson et al., 2016).
The feature-fusion approach to learning from small data may cause poor generalization: with a small sample size, the number of features should also be small to avoid the problems of dimensionality and overﬁtting.
However, given that the dimensions of audio and video features are very large, sparse parametrization needs to be taken into account when training machine learning models for learning a joint multi-modal feature vector (Williamson et al., 2016).
M ETHOD
We generate a node-level embedding matrix for each word of a transcript.
These node attributes represent an underlying relationship between the current word and every other word in the transcript.
In this way, a graph neural network (GNN) schematizes underlying associations among words via their internal representations.
We propose to aggregate their internal representations to encode information from the context of a transcript generalizing all major depressive symptoms 1 .
Building a Text Graph: For a given transcript, we build a text graph.
We regard all unique words appearing in a transcript as the nodes of the graph.
Each edge starts from a word in the transcript and ends with a word within a ﬁxed window on either side of the word.
Let G =
( V, E ) denote a text level graph.
V is a set of nodes representing all the unique words in a given text, and E is a set of undirected edges between pairs of these nodes, each represented by a set of the two nodes at either end of the edge.
We build edges by using a ﬁxed size sliding window in a transcript to collect co-occurrence statistics.
Each node has an attribute which is a matrix.
To emphasise this representational structure of the attribute matrix, we refer to it as a ‘schema’ (Hammen & Zupan, 1984;
Rudolph et al., 1997; Soyg ¨ ut & Savas¸ir, 2001; Dozois & Beck, 2008)
U i ∈ R n × d .
The j th row of U
i is a vector of length d containing the representation that node v
i has of v j ; and n denotes the total number of unique words (the vocabulary size) in a corpus.
2. S CHEMAS
We generate a schema for each word node which performs a role of recording a global context.
This global context retains information from interactions between the current word and every other word.
In this way, each word node maintains “a dynamic record” (in the form of a schema) of the context from the given transcript.
The schemas are progressively updated by a GNN model.
This resulting model produces ﬁnal embeddings of the words in a transcript in relation to all words in the vocabulary, including the other words that make up the current training transcript.
Schemas preserve structures that represent relationships between the identiﬁer word and every other word.
Thus we can exploit these schemas to capture context-level features in an explicit way by learning a GNN model.
Our innovation is to represent word proximity through the graph structure and co-occurrence within the same transcript within the schema at each node.
2. S CHEMA - BASED G RAPH N EURAL N ETWORK (SGNN)
We use multiple passes (layers) of the message passing mechanism (MPM)
(Gilmer et al., 2017; Xu et al., 2018) to update the schema
Initialization: We ﬁrst initialise an n × d 1 matrix (in our experiments, we set d 1 = 32 ) as the schema U (1) i at each node v i ∈ V .
The schema is all zeros apart from the row corresponding to the word associated with this node, which is a random d-dimensional vector using a linear transformation.
Message passing layer: In this work, the operation of message passing is split into two steps to update the schema at each node.
We ﬁrst modify the schema at each node v i :
[Equation]
where 1 ∈ R
n is a vector of ones.
The ﬁrst term updates each row
Second we compute the message function, which is deﬁned as:
[Equation]
where || denotes concatenation along the second axis.
(cid:0) W ( k ) m (cid:1) 1 ≤ m ≤ 5 are learnable parameters.
This is essentially a two-layer neural network.
In the next step, each node’s schema is updated as:
[Equation]
We apply equation (3) as the sum aggregator over the k -th layer of the SGNN.
After all K message-passing layers have been applied, the schemas are pooled to get representations of all nodes in the graph.
We apply an elementwise max-pooling operater to extract node features.
We use the READOUT (Ying et al., 2018; Xu et al., 2018) function which aggregates node features by summing them together:
[Equation]
We use h G to predict a PHQ score for each transcript.
In our experiments, we apply a 2-layer multi-layer perceptron (MLP).
2. D ATASETS
We performed experiments on two datasets, DAIC-WOZ and a twitter sentiment dataset.
Whilst our principal interest is in depression prediction, we included the second dataset to show the generality of the method we present here.
• The DAIC-WOZ dataset contains video-based facial actions, audio and the conversation transcribed to text for each participant.
Both the 6 th and 7 th International Audio/Video Emotion Challenge (AVEC) (Valstar et al., 2016) used this dataset.
We utilized only the text transcripts of all 142 individuals within the dataset.
43 out of 142 subjects (30 % ) were labeled as depressed.
The provided dataset has been split into a training set having 107 patients and a development set containing 35 patients.
In line with prior work (Valstar et al., 2016; Alhanai et al., 2018; Cummins et al., 2017; Haque et al., 2018) and to ensure comparable results, we test on the “development set” from the original competitions (Valstar et al., 2016), since the actual test set is not in the public domain.
Privacy: This data 2 does not contain protected health information(PHI).
Personal names, speciﬁc dates and locations were removed from the audio recording and transcription by the dataset curators.
• Borderlands Sentiment Twitter 3 dataset includes 74682 tweets labeled as positive (28 % ), negative (30 % ) and neutral (42 % ) respectively.
Tweets range in length from 2 to 126 words with an average length of 25.3 words per tweet.
We extracted 10 % of full data for training.
E XPERIMENTS
In this section, we show the results generated by our method on two distinct tasks of depression prediction and twitter sentiment classiﬁcation respectively.
[Table] [Table]
i) Fast(schema-GNN)
[Table]
ii) Without equivariant linear layers (Maron et al.
[Table]
[Table] iii)
Mean
[Table]
3. THE S EVERITY OF D EPRESSIVE S YMPTOMS P REDICTION
In this experiment, we predict the PHQ score for each participant.
The loss objective is mean absolute error (MAE) for regression.
We evaluate our SGNN model on DAIC-WOZ benchmark and we then compare our method to the state-of-art works centering on feature fusion learning algorithms.
We compare our method to prior work on measuring depressive symptom severity.
The performance of our method and eleven other methods, including the state of the art method, is set out in Table 1.
It is noted that results of some models are directly taken from their original papers.
Our method outperforms all other methods, despite using only the textual modality.
We found that our model performs better than the standard GNN (Gilmer et al., 2017).
The
GNN without our schemas utilizes node representations which are initialized by pre-trained 300dimensional GloVe word embeddings (Mikolov et al., 2013).
However, our method constructs schemas which are initialized with random vectors.
Moreover the standard GNN propagates node features in terms of vectors while our model uses 2-D node attributes.
This change increases the expressive power of the MPM, resulting in better results.
The machine learning algorithms of (Valstar et al., 2016; Williamson et al., 2016) perform modeling statistics using handcrafted features derived from audio, text and (or) visual inputs.
However, our method uses just raw text.
We also note that our model performs better than other prior work (Alhanai et al., 2018; Song et al., 2018; Haque et al., 2018; Du et al., 2019; Lin et al., 2020; Ray et al., 2019).
That is likely due to the difference of representation learning.
Their work uses a multi-modal sentence-level embedding to predict a PHQ score while our model is trained to learn a graph-level embedding of each transcript.
The method (Lin et al., 2020) based on the BiLSTM model reaches to a similar lower MAE.
They apply pre-trained w2v embeddings.
However, we train word embeddings with random vectors.
3.1. A BLATION S TUDY
To further analyze our model, we perform three ablation studies (see Table 2).
To give more reliable measures of performance, we perform 10-fold cross validation on the combined training and development dataset.
We concatenate the training and development set as one set and then divide it into 10 folds in a stratiﬁed manner.
Each time one fold is used for testing and the other 9 folds are used for training.
In (i), we give equations 1-4 (in section 2.2) for ‘point-wise’ multiplication.
According to the results in Table 2, we can see that using convolution layers can better model the relations between words compared with point-wise multiplication layers.
In (ii), we remove the second and third (equivariant) terms in (1).
There was a signiﬁcant reduction in performance (Table 2).
As a result, the representation power of our model can increase when parametrized with these layers, thereby improving the model’s performance on the data set.
In (iii), we replace the max-pooling operator by a mean-pooling operator.
From Table 2, we can see that the result was not good when applying mean operator of the node.
Max operator highlighting features can enhance discriminating depressive features, which helps to achieve a better result.
Qualitative Analysis: Our model has some incorrect predictions which may be caused by the unequal distribution of dataset (see ﬁgure 3a).
In ﬁgure 3b, our SGNN model made more biased predictions on those subjects with higher PHQ depression scores (plotted in blue):
For instance, the model predicted a PHQ score of around 16 to the patient with an actual PHQ score of 13, and the patient who has the PHQ score of 23 has been overpredicted with a score of around 26.
To improve the learning capacity of the model, more labeled training samples with PHQ scores higher thanwould be needed.
3.1. I NTERPRETABILITY ANALYSIS OF WORD EMBEDDINGS
We give an illustrative visualization of the word embeddings learned by SGNN model.
Figurecan tell us what the content the model mainly focuses on, therefore helping us to understand its learning process to some extent.
We show two examples with predictive depression scores higher than 15.
The depression symptoms (of a patient) with corresponding PHQ-scores higher than 14 are classiﬁed as moderate-severe or severe by a domain expert (Kroenke et al., 2009).
Compared with the content of the transcripts illustrated by word clouds (ﬁgure 4a, ﬁgure 4c) of the left-side column, we found that word clouds (ﬁgure 4b, ﬁgure 4d) on the right-side column represent very different content of the same transcripts.
We note that the model can learn to discriminate semantic features relating to depressive symptoms or non-depressive symptoms from the text. [Table] [Figure] [Table]
[Figure] [Table] [Figure] Word Visualization: We also qualitatively visualize word embeddings, in the form of bi-grams, learned by SGNN model.
Table 3 shows the most important word pairs that have been learned to capture context-level semantic features, such as “feel, tough”, “therapy, asleep”, “sleeping, depression” or “depression, psychiatrist” related to PHQ topics.
This demonstrates that the high depression scores are predicted on the basis of appropriate semantic.
3. T WITTER S ENTIMENT C LASSIFICATION
We evaluate the performance of our SGNN model on classifying three different sentiments on Twitter data (Naseem et al., 2019; 2020; Yenduri et al., 2021; Janjua et al., 2021).
We train our model with a total of 7800 tweets.
10-fold stratiﬁed cross-validation is also adopted for our experiments.
We compare our SGNN model with a general GNN model.
Table 4 reports accuracies on both training and testing set.
The row one method is initialized with pretrained 300-dimensional GloVe embeddings.
The row two method initialises all the nodes with random vectors (we set 32-dimensional random vectors for experiments).
We observed that our low dimensional 2-D node embeddings can [Table] propagate contextual information to the whole text well, while high dimensional pretrained embed- do not improve performances.
Table 4 also shows the performance gains of the proposed model on the Borderlands Sentiment
Twitter data when 2-D node embeddings applied.
This gain demonstrates that a GNN model with 2-D node attributes can capture more context-level features of sentiments from text, and thus can greatly improve the accuracy of this domain task.
3. A NALYSIS OF 2- DIMENSION NODE ATTRIBUTES
We compared results of performances between a standard GNN model and our SGNN model on both the depression prediction task and the twitter sentiment classiﬁcation task using 2-dimensional (2-D) node attributes.
On the ﬁrst experiment, our graph representation learning framework using schemas achieves better generalisation on long text.
Figure 3b shows that our SGNN model (plotted in blue) generalized the development set much better, particularly for the small group of samples in the class of having scores higher than 13.
On the second experiment, we ﬁnd our SGNN model also performs better on the short text Twitter dataset using schemas, showing that it can also model consecutive and short-distance semantics well.
We suggest that using 2-D node attributes (i.e. schemas) can improve the expressive power of the
MPM.
By setting a global context (matrices) of each node (as the input), we can obtain some parameters, in an explicit way, that capture the context of a word using each node’s schema.
On the contrary, using simple 1D node embedding such as GloVe may result in losing contextual information when they are processed by several layers of a learning model.
Settings: We set the learning rate as 10 − 3 , L 2 weight decay as 10 − 4 , the dropout rate as 0.5, and the window size as 4 to gather word-word occurrence statistics for both experiments.
The loss objective for the ﬁrst experiment is mean absolute error (MAE); the loss objective for the second experiment is cross-entropy loss.
We trained the SGNN for a maximum of 500 epochs using the
SGD optimizer (Kingma & Ba, 2014) and stopped training if the validation loss does not decrease for 10 consecutive epochs.
C ONCLUSIONS
Our work shows a novelty of improving the performance of predicting depression states by training a deep graph learning model to learn contextual features from the text.
Our results have demonstrated that it is possible to apply deep learning methods to tackle more speciﬁc problems within the ﬁeld, and not just a more general problem, even with limited data.
Future work might address ﬁnding a way of explaining a GNN model, such as, visualizing the relationship between the underlying depression features and depression scores, which helps us better understand clinical context behind the data.