Generalizable Cross-Lingual Cognitive Distortion Detection with Standardized Annotations and Multi-Task Learning (2025)

Abstract: Cognitive distortion is a critical issue in psychology, with most existing studies based on Burns’ cognitive distortion theory. However, differences in annotation standards lead to variations in building analysis tools, resulting in inconsistent analyses and limiting the generalizability of findings, especially in large-scale and cross-linguistic contexts. To address this issue, we collected all publicly available datasets (four in total) and conducted a series of experiments to evaluate the generalizability of various cross-linguistic models. The results indicate that models exhibit significant performance differences across datasets, highlighting the generalization problem. To mitigate this issue, we propose two solutions. First, we propose a multi-task learning model based on teacher student architecture solution, which demonstrates improved generalization performance in our experiments. Second, we introduce a new dataset (~5,000 samples) derived from reannotating existing open datasets to ensure standardized alignment. The annotation process we provided is interpretable and grounded in psychological principles. Based on this, we constructed large language models with cognitive reasoning chains, enhancing both generalizability and interpretability. This study identifies the generalization challenge in cognitive distortion research, and our experiments show that the proposed solutions significantly improve model performance. The dataset and code are publicly available at: https://github.com/HongzhiQ/CrossLinCD.

## introduction with the rapid pace of modern life and the increase in stress levels, psychological health issues have become increasingly prevalent. according to the world health organization (who), one in eight people globally suffers from mental disorders @xcite . among them, depression is a major contributor to the global burden of diseases. individuals with major depressive disorder (mdd) may experience profound emotional distress and, in severe cases, self-harm or suicide @xcite . thus, early identification of depressive traits is essential for targeted interventions and suicide prevention @xcite . cognitive factors play a central role in the onset and progression of depression @xcite . beck's theory posits that maladaptive cognitive structures lead to negative biases, causing individuals to focus on adverse experiences, interpret events pessimistically, and engage in rumination, suppressing positive memories and exacerbated @xcite . detecting cognitive distortions early is, therefore, critical for effective prevention strategies. using artificial intelligence (ai) for largescale data analysis offers a promising approach to identify such distortions early and provide timely psychological support @xcite . despite these advances, developing tools for cognitive distortion recognition still faces several major challenges. first, the scarcity of publicly available datasets limits model training and evaluation @xcite @xcite . second, while existing datasets are based on the same theoretical foundations (burns' cognitive distortion theory (burns, 1981)), annotation discrepancies exist due to subjective differences in human labeling, leading to poor model generalization across datasets. for example, in chinese datasets, @xcite use a multi-label classification with twelve categories, while @xcite adopt a multiclass approach with seven. in contrast, @xcite define fourteen categories for an english dataset. these issues hinder the scalability and reliability of cognitive distortion detection models, especially in multilingual contexts. to address these challenges, we collected all publicly available cognitive distortion datasets and conducted a series of cross-lingual experiments. our findings reveal substantial performance inconsistencies across datasets, highlighting the need for improved generalization. to mitigate this issue, we proposed two solutions. first, we developed a multi-task learning framework that enables models to capture language-invariant cognitive distortion features, improving cross-dataset performance. second, we propose a new dataset of 5,702 samples by reannotating existing open datasets for standardized alignment. we provide a psychological principled annotation process to support the training of large language models with reasoning chains. experimental results demonstrate that our proposed dataset leads to a significant improvement in model performance. this study underscores the challenge of generalization in cognitive distortion recognition and presents two feasible solutions. by releasing our datasets, we aim to facilitate future research in this domain and contribute to the development of more robust and interpretable cognitive distortion detection models. ## related work cognitive distortion research is vital for understanding flawed thinking patterns that contribute to mental health issues @xcite . numerous studies have focused on utilizing ai methods to identify cognitive distortions within text, such as the research by @xcite @xcite . however, these studies have not made their cognitive distortion datasets publicly available, thereby limiting further research. among the existing studies, only four datasets have been made public @xcite @xcite , and these are insufficient for comprehensive research. moreover, some of the available datasets are relatively small, such as the study by @xcite with only 698 samples. differences in annotation principles lead to variations in task definitions and standards. for example, in chinese, the dataset proposed by @xcite differs from that of @xcite : the former is a twelve-category multi-label classification problem, while the latter is a seven-category multi-class classification problem. in english, the problem still exist. for example, @xcite define it with fourteen categories, while @xcite use ten categories. regarding performance, @xcite proposed a model and evaluated it on two datasets, observing a significant performance gap (macro f1-score: 0.68 on the crowddist dataset and 0.23 on mh-c). the gap becomes even larger when analyzing across languages, making it difficult to develop a single, generalizable tool that performs well in both settings. although these datasets come from different sources, all these studies @xcite @xcite are based on the same psychological theory. therefore, we believe they share a common underlying concept for identifying cognitive distortions. in this study, we propose two solutions to address these issues. the first involves a multi-task teacher-student framework, while the second focuses on reannotating data for standardized alignment. experimental results show that our solutions significantly improve generalization in this task. we acknowledge the diversity of annotation principles and leave this issue as an open topic while providing our proposed solutions. ## dataset we collected all publicly available datasets related to cognitive distortions to evaluate generalization across datasets. then, we invited annotators to label them, creating a standardized and unified dataset to enhance generalization. the dataset distribution and characteristic can be seen in ## experimental methods this study does not focus on algorithmic innovation but rather on addressing the generalization problem across datasets and languages in the cognitive distortion classification task. therefore, for deep learning models, our experimental approach uses xlm @xcite as the baseline due to its strong cross-lingual ability. for llms, we selected several popular open-source models. ## results cross-dataset generalization performance our study investigates the cross-dataset transferability of models trained on different cognitive distortion datasets. the results can be seen in the result reveal that while models generally perform well on their respective training datasets, their performance drops significantly when applied to other datasets. this suggests that the models lack robustness in cross-dataset generalization, likely due to inconsistencies in annotation standards across different datasets. for instance, the model trained on d r achieves 56.17% accuracy when evaluated on the same dataset but performs significantly worse on d t1 (31.33%), d t2 (13.99%), and d s (26.33%). similar trends are observed across other models, indicating that the learned representations are highly dataset-specific and do not generalize well to other data distributions. this highlights the challenge posed by nonuniform labeling criteria, which introduce variations in data interpretation and hinder knowledge transfer. we also incorporated a teacher-student training framework to enhance generalizability. while some improvements are observed, such as a slight increase in average performance across datasets (from 31.96% to 33.51% for m dr and from 38.69% to 42.18% for m ds ), the overall gains remain limited. notably, the ts approach improves performance on certain datasets (e.g., m ts ds achieves 75.28% on d s compared to 74.52% in the baseline), but fails to provide substantial improvements in cross-dataset settings, with performance remaining relatively low. these findings suggest that the core issue lies in the lack of standardized labeling across datasets, which restricts model transferability. while teacher-student training offers marginal benefits, it does not fully mitigate the inconsistencies caused by dataset-specific annotation schemes. this underscores the necessity of developing a unified annotation framework or multi-task learning strategies to enhance model generalization across cognitive distortion datasets. ## discussion in the task of cognitive distortion recognition, the challenge of generalization arises due to significant annotation differences across datasets, despite being grounded in the same theoretical framework. these inconsistencies limit model transfer and require strategies to enhance generalization. to address this, we explored multiple approaches, including multi-task learning, standardized annotation alignment. multi-task learning was investigated as a means to enhance generalization by enabling the model to learn from multiple datasets simultaneously. given that different datasets may emphasize distinct aspects of cognitive distortion, training a shared model across multiple tasks allows for a more comprehensive representation. this approach mitigates dataset-specific biases and encourages the model to learn transferable features that improve robustness across varied data distributions. to further reduce inconsistencies, we introduced a standardized dataset by aligning annotation criteria across different sources. standard alignment plays a crucial role in reducing variability introduced by subjective annotation differences. by defining a unified labeling scheme, we ensured that the model is trained on a consistent dataset, minimizing annotation-induced discrepancies. however, while standardization improves uniformity, excessive alignment may inadvertently obscure dataset-specific nuances, necessitating a careful balance between standardization and preserving valuable diversity in the data. to preserve such nuances, one promising direction is to retain original annotation metadata during training-such as label variants or annotator notes-as auxiliary signals. additionally, future work could explore multi-view contrastive learning frameworks, where models simultaneously learn from both the standardized and the original label structures. such designs may offer a compromise between generalizability and granularity, enabling better cultural and contextual sensitivity in cross-lingual cognitive distortion analysis. beyond data alignment, we explored interpretable llms to enhance generalization by increasing transparency in decision-making. interpretability allows models to provide explicit reasoning for their predictions, which not only aids in model validation but also increases trust and usability. moreover, an interpretable llm can help uncover potential biases in training data and suggest corrective measures to refine learning strategies. however, despite the benefits of interpretability, our experiments have shown that the current performance is suboptimal. it highlights the need for future research to explore ways to enhance model performance without sacrificing transparency. our findings indicate that generalization in cognitive distortion recognition requires a multifaceted approach, combining structured learning strategies, data standardization, and model interpretability. while each of these strategies contributes to mitigating dataset-specific limitations, their combined effect is likely to yield the most robust improvements in generalization. future research can further investigate the interactions between these methods and explore additional domain adaptation techniques to refine generalization across diverse datasets. in addition, it is crucial to consider ethical and practical risks when handling sensitive mental health data. although our study relies on publicly available and anonymized sources, we acknowledge the possibility of reidentification and unintended harm. thus, we advocate for stricter anonymization protocols, humanin-the-loop deployment, and future research into privacy-preserving techniques and model auditing tools to ensure responsible use of such technologies. ## conclusion this study explored strategies to improve generalization in cognitive distortion recognition, addressing annotation inconsistencies and model limitations. we propose a multi-task learning approach and a standardized dataset to enhance generalization. our results demonstrate that these two solutions significantly improve generalization, and the proposed open-source dataset can further support research in this domain. future work should focus on domain adaptation, and further improving model interpretability and robustness.