Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on T witter (2024)

Abstract: Social media data has been used for detecting users with mental disorders, such as depression. Despite the global significance of cross-cultural representation and its potential impact on model performance, publicly available datasets often lack crucial metadata relatedto this aspect. In this work, we evaluate the generalization of benchmark datasets to build AI models on cross-cultural Twitter data. We gather a custom geo-located Twitter dataset of depressed users from seven countries as a test dataset. Our results show that depressiondetection models do not generalize globally. The models perform worse on Global South users compared to Global North. Pre-trainedlanguage models achieve the best generalization compared to Logistic Regression, though still show significant gaps in performance on depressed and non-Western users. We quantify our findings and provide several actionable suggestions to mitigate this issue

## introduction according to the data from world health organization, depression is a global issue affecting 280 million people worldwide foot_1 . in response to these trends, in the last decade, there has been a surge in studying the mental health status of users from social media based on their content and interaction @xcite . research has focused on various disorders, including depression, anxiety, and eating disorders, and has used many methods @xcite . specifically -depression is among the most widely studied disorders (and the most commonly diagnosed), and twitter is a common source of data in these studies @xcite . this work tries to predict if someone may have depression based on data from social media @xcite . given this area's popularity and potential reach to clinical settings, nlp has also called for careful evaluations of bias, performance gaps, and generalizability of claims from small datasets @xcite @xcite . one source of underexplored bias in these datasets and models is the impact of a person's geographic location (and consequently, their culture) on their communication style. the importance of cultural consideration in social media studies about mental health is critical @xcite . in prior work, de choudhury et al. showed cross-cultural differences in mental health communication styles in cultures such as the us, uk, india, and south africa. cross-cultural users also have different identity dimensions, language use, and support behavior @xcite ; sentiment detection can vary across cultures @xcite . however, interaction in international forums does not affect their clinical mental health language use @xcite . recent literature reviews @xcite and persuasive calls @xcite point to the need to study the generalizability of models to distinctive user populations for mental health research. similar audits have been instrumental in identifying gaps in performance across clinical/non-clinical populations @xcite and in gender and racial groups @xcite @xcite . building on this prior research, in this paper, we analyze the generalization of depression detection models on cross-cultural data trained on existing benchmark datasets. inspired by @xcite , we ask: do models built on popular social media benchmark datasets to predict depression generalize to people who live in different countries, yet speak english? if the prior work is correct about geographic and cultural biases impacting predictions, what countries may be most affected? how stark are the performance differences between countries? we audit depression detection models generalization on cross-cultural data gathered from twitter (now x). we collected data from seven countries using a strict location verification technique and the prevalence of english content in users' feeds, using keyword matching and manual annotation to identify genuine depression disclosures. we trained two models, logistic regression and men-tallongformer, on benchmark depression datasets (clpsych and multi-task learning). we assessed their generalization by both country and socioeconomic development classification (global north vs. global south). we show that models trained on broad twitter benchmarks do not generalize well to the crosscultural data. models generalize much better to evaluation data from users in the global north (us, uk, australia) than to users in the global south that use english as a national language (india, nigeria, philippines, south africa). distinct gaps emerge between countries, with models generalizing very poorly to posters from nigeria and india. our findings demonstrate that existing benchmark datasets are not representative of training generalized models that could detect depressed users from various cultures. we provide suggestions for building better datasets and models. ## datasets we carefully selected two popular benchmark datasets for constructing depression models on twitter data and then created a geolocated dataset of depression posts. datasets. clpsych: this dataset comes from the clpsych 2015 shared task @xcite . the shared task contains two mental disorder identifications, identified with keywords and manual annotation: depression and ptsd, of which we use the depression treatment data and control. the data comprises the users' most recent posts around the date of depression disclosure, up to a maximum of 3000 posts per user. multi task learning (mtl): this dataset is from @xcite , which contains twitter user profile information and their posts within one month. this dataset also identified people who may be depressed in twitter. both clpsych and mtl are gathered based on a strict set of keywords/keyphrases such as "(i'm/i was/ i am/ i've been) diagnosed depression," etc., to identify the candidate depressed users. we leverage the text data only (as this dataset does contain images). our global dataset: at the time of writing, there are no public benchmark datasets of global expressions of depression in twitter data. therefore, we collect a corpus from public posts from twitter using the twitter research api (now defunct) 3 . we used the search terms/phrases from de choudhury et al.'s cross-cultural depression study on twitter to identify people discussing depression or suicidality (a common co-morbid symptom of depression). these include phrases such as "i am/i'm depressed" and "i want to hurt myself", and were verified by psychologists by the collaborators of de choudhury et al. we searched the sample of the twitter data made available between january 2015 and december 2022 4 . given our focus on cross-cultural content, we leveraged geotagged tweets. we specifically gathered users from seven countries: australia, south africa, nigeria, the philippines, india, the united kingdom, and the united states. we selected these countries for geographic diversity, their large volume of geotagged disclosures, and the fact that english is a first language or is a business/governmentlisted language in those countries. to verify that a user was in the country, we looked at their 3200 geotagged posts before disclosure and took the country where the user posted the most. we manually verified each user's veracity of depression disclosure with human raters, similar to the process in @xcite . we developed and applied a codebook to identify users who had genuine disclosures of depression (see appendix a.1 for details). we then gathered control users with similar demographics whose posts did not include the search terms but with the same geotagged tweet rules as discussed. therefore, we made a matched, "control" sample of users from the same country who disclosed having depression and those who did not. at the outset, our dataset comprised 16,112 potentially depressed users from the seven countries. of these, 1,556 were manually reviewed, leading to the identification of 267 authentic disclosures. the annotation encompassed all original sample users from the global south countries but did not cover all users from the global north due to the substantial volume of data. @xcite between the raters resulted in 0.65, showing a substantial agreement. the disagreements were resolved through two rounds of discussion. appendix a.3 shows genuine and non-genuine posts obtained through human annotation. these examples are paraphrased and lightly edited to protect the identity of the posters @xcite . our geo-located dataset encompasses a total of 531 users, with 267 users identified as having depression through manual verification. we define two groups of countries based on the united nations categorization of countries foot_5 -the global north and the global south. 140 users were in the global north (64 united states, 45 united kingdom, 31 australia), while 127 users hail from the global south (58 philippines, 35 south africa, 19 india, 15 nigeria). ## baseline models for mental health prediction tasks, logistic regression is a popular and performant statistical baseline due to its quick training time, success in prediction, and highly interpretable feature relevance @xcite @xcite . in this experiment, we extracted the features using the term frequency-inverse document frequency (tf-idf). using scikit-learn, we applied grid search on 5-fold cross-validation. the best hyperparameters for the logistic regression are 'penalty': 'l2', 'solver': 'lbfgs', 'max_iter': 10000, and 7000 tf-idf features. pretrained language models such as bert (bidirectional encoder representations from transformers) @xcite have significantly improved text classification on many general (murarka et al., 2020) and domain-specific tasks @xcite . we finetune the mentallongformer language model for our baseline, which outperforms the other pre-trained models on this specific task and has an extended sequence modeling capacity @xcite . we use it to investigate its generalization capabilities to our task. for this experiment, the pre-trained head of the mentallongformer is replaced with a randomly initialized classification head. we set the learning rate to 5e-5. adam is used as an optimizer (kingma and ba, 2014). we trained for '@xmath0. the remaining parameters were set to the default hyperparameters of mentallongformer on huggingface. ## results in ## recommendation and conclusion in this work, we quantified the generalization capability of depression detection models in crosscultural data. we specifically quantified that models have higher discrepancies in identifying users from different cultures. we provide the following suggestions for improving the identified gaps. construct datasets with more geographical examples. similar to @xcite , we hypothesize that mental health detection from social media suffers from small datasets. existing benchmark datasets lack the location meta-data of users @xcite and lack different demographic representations @xcite , meaning that fairness audits are challenging to execute post-hoc. we propose a few solutions to this problem. first, researchers could adapt techniques to infer geo-location if larger datasets were available @xcite to conduct audits. larger datasets could be composited from comparable sources, pointing to evidence from @xcite ) that more data helps alleviate racial disparities in predictions. balancing the datasets effectively makes the algorithms fair in different groups @xcite . ultimately, the field needs to find paths forward to identify and supplement datasets for this task. as an initial stride in this direction, we provide the details for our dataset and how other users may replicate our findings 6 investigating the cross-cultural detection capabilities of proposed models. current work has a considerable gap in ethical consideration and transparent reporting @xcite . fine-grained subgroup analysis reporting leads towards building more inclusive and transparent models @xcite . we call for critical consideration when reporting these metrics when introducing algorithms.