P sycho L ex: Unveiling the Psychological Mind of Large Language Models (2024)

Abstract: This paper explores the intersection of psychology and artificial intelligence through the development and evaluation of specialized Large Language Models (LLMs). We introduce PsychoLex , a suite of resources designed to enhance LLMs’ proficiency in psychological tasks in both Persian and English. Key contributions include the PsychoLexQA dataset for instructional content and the PsychoLexEval dataset for rigorous evaluation of LLMs in complex psychological scenarios. Additionally, we present the PsychoLexLLaMA model, optimized specifically for psychological applications, demonstrating superior performance compared to general-purpose models. The findings underscore the potential of tailored LLMs for advancing psychological research and applications, while also highlighting areas for further refinement. This research offers a foundational step towards integrating LLMs into specialized psychological domains, with implications for future advancements in AI-driven psychological practice.

## introduction the rise of large language models (llms) has significantly advanced artificial intelligence (ai), providing remarkable capabilities in natural language processing and understanding @xcite @xcite . these models have shown proficiency in generating human-like text, translating languages, and engaging in sophisticated dialogues @xcite . however, as users increasingly rely on llms for psychological and therapeutic questions @xcite , the limitations of these models in specialized domains have become apparent. notably, there is a critical absence of datasets designed to evaluate and enhance llms' performance in the field of psychology. despite considerable progress in general ai research, the integration of psychological expertise into llms remains underdeveloped. existing methodologies often lack the depth required to understand and respond accurately to complex psychological inquiries. moreover, the field is hindered by the lack of comprehensive datasets that include not only questions and answers but also instructional content tailored to psychological contexts. this gap is significant because it restricts the practical applications of llms in psychological research, therapy, and education, where nuanced and precise information is essential. our research seeks to address this gap by introducing psycholex, a suite of resources and models specifically designed for psychological applications in both persian and english. the primary objectives of this study are to develop and evaluate specialized datasets, namely psycholexqa and psycholexeval, and to introduce psycho-lexllama, an llm developed for psychologyical tasks. these contributions include: (i) psy-cholexqa, which provides comprehensive instructional content and detailed questions and answers to enhance llm training; (ii) psycho-lexeval, a multiple-choice question and answer (mcqa) dataset designed for rigorous evaluation of llms in psychological contexts, ensuring they can handle complex psychological queries accurately and contextually;(iii) psycholexllama, which improves the performance of llms in psychological tasks through continual pre-training and fine-tuning of llama 3.1. @xcite together, these contributions aim to provide robust solutions to existing challenges, enhancing the accuracy and relevance of ai-driven psychological tools and paving the way for future advancements in integrating ai with psychological practice. the structure of this paper is organized as follows: section 2 reviews related work in llms and their applications in psychology. section 3 details the datasets developed for this study, including their creation and validation processes. section 4 discusses the development and fine-tuning of the psycholexllama model. section 5 presents the evaluation methodology and results, comparing psycholexllama with other state-of-the-art models. section 6 provides a comprehensive discussion of the findings, and section 7 concludes the paper with insights into future research directions and potential applications. by exploring the intersection of ai and psychology, this paper aims to unveil the psychological capabilities of llms and demonstrate their potential to advance both fields significantly. ## related works in this section, we review existing research that benchmarks the capabilities of large language models (llms) in persian, followed by studies that explore the integration of llms into psychological research and applications. this dual-focus review establishes the context for our work, emphasizing both the linguistic challenges specific to persian and the broader implications of applying llms in the field of psychology. ## dataset this section outlines the datasets developed to investigate the application of large language models (llms) in psychology. we detail the creation and utilization of three pivotal datasets: the foundational pretraining data, the psycholexqa dataset for instructional content, and the psycholexeval dataset for evaluating model comprehension and performance. ## psycholexllama in this section, we detail the development of psy-cholexllama, a specialized large language model (llm) designed explicitly for psychology. our goal was to surpass the performance of general-purpose models by optimizing our model to require minimal data and hardware resources. the model benefits significantly from the integration of the psycholexqa dataset, which includes tasks such as case study analysis, experiment design, and diagnostic evaluation. this dataset allows psycho-lexllama to handle complex psychological scenarios across a variety of domains like clinical, cognitive, and developmental psychology, making it a highly valuable tool for both practical and research-oriented applications. the integration of psycholexqa significantly enhances psycholexllama's role in applied psychology and psychological science. it assists clinical practitioners with diagnostics and interventions, supports research with experiment design and data interpretation, and enriches education by generating tailored instructional content. overall, it provides specialized, accurate responses, making it a valuable tool across clinical, research, and educational domains in psychology. we utilized the transformers 2 library for model development. the process of constructing our model is illustrated in figure 1 . ## evaluation in this study, we conducted a comprehensive evaluation of various language models that operate in both persian and english, focusing on their ability to understand and accurately respond to psychological questions. the models assessed include include qwen2 @xcite , aya-23 @xcite , phi-3 @xcite , llama-3, llama-3.1 @xcite , gemma 1.1 @xcite , persianmind @xcite , and psycholexllama. our focus on open-source models was intended to enhance the accessibility and reproducibility of our findings. the generation configuration for all the llms evaluated is consistent across the experiments and is detailed in ## results this section outlines the outcomes of our evaluation of selected large language models (llms) using the psycholexeval dataset in both persian and english. the primary focus was on assessing the models' proficiency in understanding and responding to psychological questions. tables 3 and 4 illustrate the accuracy results of the models on the psycholexeval dataset for persian and english, respectively. these tables quantify how effectively each model comprehends and addresses psychology-related questions across languages. ## conclusion this study has significantly advanced our understanding of how large language models (llms) can be effectively tailored for applications within psychology. through the integration of specialized psychological content, the development of the psycholexqa and psycholexeval datasets, and the creation of the psycholexllama model, we have demonstrated the substantial benefits of targeted model training and fine-tuning. our findings indicate that specific pretraining and fine-tuning strategies substantially enhance the performance of llms in psychological settings, underscoring the critical role of thoughtful model architecture and training approaches. notably, while larger models typically show strong performance, our results reveal that even smaller models can achieve exceptional outcomes when subjected to precise, domain-specific adjustments. this suggests a scalable potential for llms in psychological applications that can be adapted to different contexts and constraints. in conclusion, this research not only sheds light on the current capabilities and challenges of using llms in psychology but also sets a foundation for future work. it encourages ongoing refinement of these models to improve their relevance and accuracy, thereby enhancing their utility in real-world psychological applications. moving forward, we anticipate that continued advancements in model training methodologies and evaluation strategies will drive significant progress in the field, making llms an indispensable tool in the arsenal of psychological research and practice.