Team 9: A Comparison of Simple vs. Complex Models for Suicide Risk Assessment (2021)

Abstract: This work presents the systems explored as part of the CLPsych 2021 Shared Task. More specifically, this work explores the relative performance of models trained on social me- dia data for suicide risk assessment. For this task, we aim to investigate whether or not simple traditional models can outperform more complex fine-tuned deep learning mod- els. Specifically, we build and compare a range of models including simple baseline models, feature-engineered machine learning models, and lastly, fine-tuned deep learning models. We find that simple more traditional machine learning models are more suited for this task and highlight the challenges faced when trying to leverage more sophisticated deep learning models.

## introduction globally 800,000 people die from suicide each year, which makes it one of the leading causes of death @xcite . despite decades of substantial efforts to analyze risk factors for suicidal thoughts and behaviors @xcite , models have produced predictions only slightly better than random chance (@xmath0.56-0.58) @xcite . recent progress in natural language processing (nlp) and machine learning systems to predict suicide risk have been shown to have higher auc 0.9 @xcite , however it is still a complicated task particularly due to the sensitivity and difficulty in obtaining high quality labeled datasets. this work is part of the 2021 clpsych shared task @xcite , which provides secure and ethical access to sensitive data in order to work on the problem of predicting suicide risk from social media data. the shared task has two main objectives: prediction of a suicide attempt 30 days prior, and prediction of a suicide attempt 6 months prior. in this paper, we present our team's results from the shared task using a variety of methods to improve performance. we focus on exploring various machine learning ensemble models, feature engineering approaches and compare to deep learning architectures and transfer learning methods in nlp. we find that baseline models such as term frequency, used in combination with simple machine learning models outperform fine-tuned deep learning transformer-based models. ## methods our goal for this task was to compare the results of models across different levels of complexity, and see how they perform in the context of a small dataset in the mental health space. all tweets were aggregated at the user level, and each of the classification methods were implemented and compared at that level. ## results in subtask 1, our models are as follows: run 1 refers to the character tf-idf gradient boosting model, run 2 refers to the syntax gradient boosting model and run 3 refers to the bertweet model using fine-tuned method 1. in the validation experiments, we found bertweet fine-tuned method 1 to outperform method 2. in subtask 2, run 1 refers to the syntax gradient boosting model, run 2 the character tf-idf model, and run 3 the voting classifier. we see that in the case where the bertweet model could be applied, it outperformed more simple machine learning models. however, although the bertweet model had a high f1, f2, and tpr, it has a high fpr and a low auc score -this implies that the model is overfitting, and has a tendency to predict 1s. in the case where bertweet could not be applied (subtask 2), having an ensemble model fared better than the single gradient boosting models. the voting classifier outperformed the baseline in most metrics (f1, f2, tpr) but also had a nominally higher fpr and lower auc score than the baseline. the increased fpr corresponds to misclassifying one negative sample as a positive sample. for assessing suicide risk though, we feel that it is better to overpredict suicide risk than underpredict, since the consequences of underpredicting are much more severe. f2 score gives less weight to precision and more weight to recall therefore prioritizing the proportion of actual positives that were correctly identified. both bertweet (subtask 1) and the voting classifier (subtask 2) have higher f2 score than the baseline, however f2-score alone is an unsuitable metric as a classifier that predicts all 1s would have a recall of 1. the auc is widely used to as a measure for predictive modeling accuracy, however, auc is not recommended for small sample sizes @xcite . overall, looking at all the metrics in ## discussion for subtask 1, in the transfer learning methods, we tried two fine-tuning techniques. in the first approach, i.e. method 1, we instantiate a bertweet model with an added single linear layer on top for classification. in this approach, the entire pretrained bertweet model and the additional untrained classification layer is trained on our specific task. the average accuracy with the validation set was 0.51 and 0.45 for the test set, suggesting overfitting of the model. for the second approach, i.e. method 2, we freeze all the layers of bertweet and only update the weights of the attached layers. while the training loss decreased for the first 4 epochs, it did not decrease further, suggesting that the model was trained for too long and is also overfitting on the training data. while both approaches suggested that such a small dataset caused overfitting, a simple fine-tuning approach through adding one fully-connected layer to bertweet and training the whole model end-to-end for a few epochs (method 1) showed better results than appending a custom architecture to the frozen bertweet model (method 2). as all tweets were aggregated into one large tweet at the user level and the sequence length was limited to 128, effectively this approach reduced the dataset from tweets of the last 30 days to the last 1-3 days depending on the tweet length. this causes loss of potentially valuable data and features that may be missed as these particular models cannot learn from the older tweets. as the machine learning models do not have these limiting properties, they are more suitable for this task. a recommendation for future work is to transform the dataset in an alternate manner, for example, creating a classification task at the tweet level instead of the aggregated user-tweet level. ## conclusion the main question we sought to explore in this paper was the following, would a classical machine learning model approach outperform a more sophisticated deep learning model for the suicide risk assessment task? given past research in this space that struggled with this task as well as the small nature of the datasets, it was our hypothesis that keeping it simple would lead to better performance. our findings support this hypothesis. we found that bertweet struggled with overfitting and demonstrated limitations, such as sequence length, that made it difficult to leverage for this task. in our evaluations, we found that a simple baseline model, or an ensemble of machine learning models can outperform the more sophisticated models. in addition, the short time period inherent in building a model for a shared task made it difficult to investigate alternate data transformations that are more appropriate for a complex model like fine-tuned bert/bertweet. however, we do find some promise in the test performance of bertweet for subtask 1 and believe with more time and exploration a variation of transfer learning models can be built and leveraged in a task of this nature.