Improving Classification of Infrequent Cognitive Distortions: Domain-Specific Model vs. Data Augmentation (2022)

Abstract: Cognitive distortions are counterproductive patterns of thinking that are one of the targets of cognitive behavioral therapy (CBT). These can be challenging for clinicians to detect, especially those without extensive CBT training or supervision. Text classification methods can approximate expert clinician judgment in the detection of frequently occurring cognitive distortions in text-based therapy messages. However, performance with infrequent distortions is relatively poor. In this study, we address this sparsity problem with two approaches: Data Augmentation and Domain-Specific Model. The first approach includes Easy Data Augmentation, back translation, and mixup techniques. The second approach utilizes a domain-specific pretrained language model, MentalBERT. To examine the viability of different data augmentation methods, we utilized a real-world dataset of texts between therapists and clients diagnosed with serious mental illness that was annotated for distorted thinking. We found that with optimized parameter settings, mixup was helpful for rare classes. Performance improvements with an augmented model, MentalBERT, exceed those obtained with data augmentation.

## introduction data augmentation first became a popular topic in computer vision, where deep neural networks have performed remarkably well. complex architectures, such as alexnet @xcite , vgg-16 @xcite , resnet @xcite , densenet @xcite , generally require sufficient training data for model convergence, even with the help of dropout regularization and batch normalization. this situation also occurs in natural language processing (nlp) with deep learning methods and can become more problematic when limited to small datasets by data collection or data annotation constraints. in imaging, data augmentation, involving transformations such as cropping and shearing, is a common strategy to expand the amount of data available for training. analogously, several methods have been proposed to perform data augmentation in nlp, including easy data augmentation @xcite , back translation @xcite , gpt-2 augmentation @xcite , and mixup @xcite . @xcite applied some of these methods to pretrained transformer models and showed an average improvement in accuracy of 1-6%. however, the lowresource scenario was simulated by simply constraining the training data from large corpuses. it remains unclear how these methods might perform when used in realistic applications, where certain classes may be of very low frequency. one exemplary case concerns nlp analysis of online therapy sessions, where large amounts of patient-generated texts must be classified, but only well-trained specialists with relevant mental health domain knowledge can perform annotation manually to ensure clinical accuracy. in this study, we used a dataset from text message conversations between clients and therapists, previously used for detecting distorted thoughts @xcite . besides the limitation in size, we found that some types of distorted thinking are very rare, resulting in worse classification performance. to address these issues, we investigate the extent to which data augmentation methods can improve performance of the best-performing bert model from these experiments. we compare the utility of this augmentation approach to the use of a domain-specific pretrained language model, mentalbert. in doing so, we evaluate the utility of data augmentation techniques and a domain-specific model to improve the identification of rare classes in the context of real-world data. our main contributions are as follows: ## low-resource corpus from our previous work @xcite , we utilized data from a randomized controlled trial of a community-based text-message intervention for individuals with serious mental illness @xcite . data were collected from 39 participants enrolled in the active intervention arm of this trial between december 2017 and october 2019. as part of the study, clients participating in standard care engaged with trained clinicians in text-message conversations up to three times a day for 12-weeks. in total, 14,312 messages were sent between clinicians and clients with 7,354 coming from clients. to build a predictive model for distorted thoughts, five common distortions were selected (burns, 1999): mental filter (mf), jumping to conclusions (jc), catastrophizing (c), should statements (sm), overgeneralization (o). in addition, we added the label any distortion (ad), generated in accordance with the other assigned distortions. two mental health specialists annotated all messages from clients by assigning these six categories, which are not mutually exclusive @xcite . this provided ground truth for labels. it is worth noting that any message could be identified as having multiple distortions, or no distortions at all, making this a multi-label multiclass problem. ## methods based on results by @xcite , we used bert as a starting point for our study, since it outperformed support vector machines and logistic regression (with l2 regularization), which had been used in prior work @xcite . all models in this study were trained with the previously identified best hyperparameter settings for the dataset @xcite (section 3.1). given the observed frequencies ( we used two approaches to data augmentation, differing in the point at which augmentation occurs. the first involves directly augmenting the original text and outputting augmented examples as plain text, to be added to the original data (section 3.2). the second approach involves augmentation in the hidden spaces of a deep neural network, and its outputs are vectors in the hidden space, rather than plain text (section 3.3). for domain-specific model, we utilized a domain-specific pretrained language model with additional linguistic knowledge pertinent to the task at hand (section 3.4). ## results performance for all models is shown in bert: for the baseline bert model, bert (no aug), we obtain an auprc of 0.5179 for the most frequent classes (ad,c). when frequency decreases (classes mf,jc), the auprc also drops to 0.3718, and it drops further to 0.2139 for the rarest class of o,sm. this trend applies to all models. when data augmentation is applied to the base bert model, we see improved results with different models. for the most frequent class of ad,c, back translation using spanish achieves the highest auprc of 0.5208, followed by mixup: α = 0.02. however, none of these results are significant improvements over baseline bert. for the less frequent classes (mf,jc), back translation outperforms baseline bert by 1.5%. mixup does not offer a performance boost here. when it comes to the rarest classes (o,sm), improvement is clearer: eda, back translation (spanish), and most settings of mixup can offer a boost in auprc. among them, mixup (α = 4) shows the biggest improvement in auprc by around 1.6%, which is statistically significant (t(8) = 3.24, p-@xmath0.012 from t test). it is also notable that both gpt-2 based data augmentation methods decrease the performance of the base bert model substantially (0.47 vs 0.52 for ad,c and 0.14 vs 0.21 for o,sm). overall model performances is consistent with some of the preceding observations: (1) data augmentation improves overall performance, but only by a small margin; (2) in-domain pretraining of the language model (mentalbert) provides the most improvement in performance; (3) for mixup, a small α is favored (0.02 for bert and 0.2 for mentalbert). ## discussion we examined several data augmentation methods and explored their applications in bert and mentalbert for detecting distorted thinking in a modestly-sized set of text-based therapy messages. grouping distortion classes by frequency, we found that most of data augmentation methods do not improve performance for frequent classes (frequency: 8-25%). for rare classes (3%), mixup significantly improved auprc results by 1.6%. in comparison, the domain-specific pretrained language model, mentalbert, offered the highest benefit for dominant classes. however, mentalbert also performs relatively poorly with rare classes. this may be due to the limited number of training examples. another reason might be the fact that our text messages sometimes represent general conversations related to case management (e.g. appointment reminders) rather than the specific mental health related concerns that predominate in mental-health-related subreddits. we also explored different settings for the hyperparameter α for the mixup method. for dominant classes, mixup favors a small α, which corresponds with previous work @xcite . this indicates the model performs better with limited mixing of two random samples, generating cases where only one example predominates. in comparison, a larger α is favored for rare classes. according to supplementary figure 1 , this means the model tends toward mixes in which the influence of individual texts is diluted, a possible way to create more variation in this low-resource scenario for the model to learn from. however, progressing to more extreme values (α = 8) harms performance, and this cutoff point may change in other settings. taken together, our results suggest that mixup is helpful for rare classes, but may compromise performance on frequent classes. future work with mixup should include increasing the number of training epochs, since @xcite sug-label generated text jc yes you understand that it's incredibly frustrating and a lot of hard work but it's not at all stressful c okay, i will do that, eventually ## conclusions we compared a range of data augmentation strategies and a domain-specific pretrained language model for their utility in improving identification of infrequently observed cognitive distortions. using a domain-specific pretrained language model (men-talbert) provided the greatest improvements, especially for dominant classes, whereas data augmentation did not improve performance with this model. in contrast, some data augmentation methods significantly improved performance with the base bert model, but we did not find a method to improve performances for all classes universally, nor did we find a consistent hyperparameter setting to improve performance across these class frequencies. mixup appears helpful for rare classes, but a relatively large hyperparameter setting for α should be used. however, this may compromise the performance on frequent classes to some degree. taken together our results suggest that the domain-specific model may be a better strategy for frequent classes, and that the best data augmentation strategy for infrequently observed classes varies across frequency ranges. as future work, two areas of interest include: (1) modified loss functions, such as the label-distribution-aware margin (ldam) loss @xcite and class-balanced (cb) loss @xcite , which have been proposed in the field of computer vision to address class imbalance; (2) unsupervised learning frameworks to address the inherent uncertainty of labels for augmented data, such as confident learn-ing @xcite and unsupervised data augmentation (uda) @xcite .