The Colorful Future of LLM s: Evaluating and Improving LLM s as Emotional Supporters for Queer Youth (2024)

Abstract: Queer youth face increased mental health risks, such as depression, anxiety, and suicidal ideation. Hindered by negative stigma, they often avoid seeking help and rely on online resources, which may provide incompatible information. Although access to a supportive environment and reliable information is invaluable, many queer youth worldwide have no access to such support. However, this could soon change due to the rapid adoption of Large Language Models (LLMs) such as ChatGPT. This paper aims to comprehensively explore the potential of LLMs to revolutionize emotional support for queers. To this end, we conduct a qualitative and quantitative analysis of LLM’s interactions with queer-related content. To evaluate response quality, we develop a novel ten-question scale that is inspired by psychological standards and expert input. We apply this scale to score several LLMs and human comments to posts where queer youth seek advice and share experiences. We find that LLM responses are supportive and inclusive, outscoring humans. However, they tend to be generic, not empathetic enough, and lack personalization, resulting in nonreliable and potentially harmful advice. We discuss these challenges, demonstrate that a dedicated prompt can improve the performance, and propose a blueprint of an LLM-supporter that actively (but sensitively) seeks user context to provide personalized, empathetic, and reliable responses. Our annotated dataset is available for further research.*https://github.com/nitaytech/LGBTeenDataset

## introduction queer people foot_0 experience higher rates of mental health concerns, including depression, anxiety, self-harm, suicidal ideation, and ptsd @xcite . this is particularly concerning for queer youth as they navigate the process of selfdiscovery and self-acceptance @xcite . moreover, queer youth are subjected to increased victimization due to prejudice and violence at school @xcite . indeed, queer youth experience prolonged feelings of hopelessness or sadness (over 60%) more than two times compared to heterosexual youth and are three times more likely to seriously consider attempting suicide (over 40%) than their heterosexual peers @xcite . significant factors contributing to these mental health risks are the lack of support and autonomy to choose living situations @xcite and the negative stigma surrounding queers which cause queer youth to experience fear when seeking help or discussing queer-related topics, even with their own family and peers @xcite @xcite . this isolation leaves them to navigate their sexual situation on their own, without the necessary support and guidance. unsurprisingly, queers are motivated to fill these gaps with online resources (e.g., the internet, social media), where they usually run across false, incomplete, or harmful information @xcite @xcite . access to a supportive environment and reliable information are priceless for queer youth @xcite @xcite . unfortunately, most queers worldwide have no access to such support. however, things could soon change. the rapid adoption of large language models (llms) by the general audience offers a unique opportunity to reflect on their applications and their influence on the queer community. one area where llms have the potential to make a substantial impact is the sexual education and support of queer youth. llms hold great potential in providing queer support (see §3). these systems can offer a supportive and inclusive environment, allowing queer youth to engage without embarrassment or stigma, and also offer a sense of security for teenagers who may find it challenging to discuss with real people. on top of this, it is crucial to ensure the accuracy and reliability of the information provided and address concerns regarding the lack of empathy and personalization. in this paper, we perform a comprehensive study of the current state of llms to serve as queer supporters. in §4, we start by reviewing chatgpt as a leading example and conduct a qualitative analysis of several case studies. following that, we turn to quantify our impressions. inspired by standard guidelines of leading psychological associations and with the advice of expert clinicians, we develop a ten questions rating scale for the quality of responses for emotional needs expressed by queer youth. we applied this scale to score eight sota llms and human responses to reddit posts in which queer youth seek advice. thereby, we constructed the lgbteen dataset ( §4.3), comprising hundreds of posts, thousands of llms' responses, and human annotations. our study demonstrates that llms exhibit positive behaviors, such as providing detailed, supportive, and inclusive responses, a fact that can be attributed to the llm alignment, which involved a small group of crowd workers from the u.s. @xcite . in addition, we observe that llm responses score better than human comments across most dimensions of our questionnaire. however, we find that the llm responses lack engagement and tend to be lengthy, synthetic, and generic. moreover, the reliability of the responses remains a concern, particularly when the llm offers advice without considering (and seeking) additional context from the user. this issue becomes even more severe and harmful when the user comes from a more conservative society, where the cultural ignorance of the llm leads to advice that overlooks important cultural or personal factors such as family dynamics. for example, llms may encourage users to come out without considering the potential risks of their specific environment. we finally discuss the inherent challenges of evaluating llm responses to emotional needs. specifically, when considering factors like empathy, it becomes challenging to determine whether the response is of genuine high quality or if the model is merely trying to meet social expectations by "saying the right things". this generic feeling results in a lack of empathy, personalization and hindrance to genuine emotional support. we believe the positive aspects of llms make them suitable as initial emotional support for youth in the early stages of identity formation. however, as users gain more experience, they will likely turn to more authentic platforms or professional help. we also examine whether llms can match human-like emotional intelligence and potentially replace human evaluators. our findings reveal that llms currently fall short of replicating the nuanced emotional intelligence required in tasks like ours and completely fail to assess authenticity. following our analysis, in §6 we discuss three dimensions that capture desired attributes that current llms lack: reliability, empathy, and personalization. in light of our demonstration that a dedicated prompt can guide the llm and enhance its emotional support, we then present in figure 2 and appendix §b a blueprint of an ai queer supporter that actively (and sensitively) seeks user context to provide tailored, empathetic, and reliable responses. our contribution is a comprehensive, end-to-end account of a case study on queer youth, including: (1) development of a novel questionnaire; (2) construction of a new dataset featuring interactions between queer youth and llms; (3) evaluation of eight sota llms; (4) demonstration of how dedicated prompts enhance emotional support; @xcite identification of three key dimensions where llms underperform, along with a proposed blueprint for improvement; and @xcite evidence that llms are not yet able to replace human annotators. this paper provides a conceptual framework, based on empirical data, for developing ai-based queer youth emotional support systems. while we demonstrate that the road is still long, we hope that the ideas here will contribute to the efforts of developing this crucial technology. following pre-training, there is also an attempt to "align" the llm with human preferences. this alignment phase is typically conducted by instruction fine-tuning @xcite , or by applying reward learning such as reinforcement learning with human feedback @xcite ). the definition of "alignment" is vague and according to @xcite , it can be either functional alignment (seeking improvement in following instructions) or social value alignment (embedding human values and morals). nonetheless, it is questionable to whom preferences the llm is aligned. in practice, llms suffer from the "tyranny of the crowdworker" @xcite , where the llm alignment relies on a small number of western annotators, with little to no representation of broader human cultures, or languages. ai for emotional support. the exploration of ai's role in mental health has revealed its potential @xcite , particularly in conversational ai for emotional support @xcite . research has predominantly concentrated on the empathetic capabilities of these ai systems @xcite @xcite . yet, there remains a notable gap in addressing the specific needs of the queer community @xcite . while previous studies have evaluated the general emotional support efficacy of such systems @xcite @xcite they have neither offered a comprehensive account nor focused specifically on queer youth. queer support. queer support covers topics such as coming out, sexual orientation, gender identity, accessing sexual health information, navigating romantic and sexual relationships, addressing discrimination, and building a supportive community. studies show that queer-inclusive education is associated with a decrease in reporting depressive symptoms and attempting suicide @xcite . however, queer-inclusive programs are rare or non-existent @xcite @xcite , leading youth to seek information online, primarily through searching on the web but also via social networks @xcite . studies suggested that queer youth find on-line options as a convenient and safer arena to negotiate their identities @xcite @xcite . today, with the widespread usage of chatgpt by teenagers @xcite , they will likely engage with llms to seek information about queer topics. ## the promise of ai queer supporters we start by highlighting the potential advantages of ai (queer) supporters and why we expect them to be a popular choice, particularly for queer youth. ## analysis of the current state our analysis provides an overview of the current state of llms to serve as ai queer supporters. we start by qualitative analysis which leads to a quantitative study that assesses the responses of multiple llms to posts of queer youth on reddit. to this end, we developed a novel questionnaire and constructed a new dataset consisting queer youth acquisitions and responses generated by llms. ## discussion in this paper, we discuss the potential of llms to serve as queer supporters and their promising positive impact on queer youth. following our analysis of the current state, we emphasize the importance of reliability, empathy, and personalization, attributes that current llms lack. we believe that leveraging accessible ai systems can contribute to forming a more aware, open, and liberal generation. empowering teenagers to explore their unique identities (not necessarily queer-related) can foster a greater sense of personal growth and more acceptable youth. moreover, the questionnaire scores a single response from an llm, which may not fully capture the complete experience when multiple interactions or ongoing conversations take place. however, notice that one drawback of llms is their inability to ask follow-up questions, resulting in a one-sided dialogue. nevertheless, developing a tool that measures the emotional support received throughout a conversation is a promising future direction. applicability to other populations. there is a good reason to focus on the use case of queer youth support. queer individuals constitute a vulnerable population susceptible to psychopathology and suicide @xcite . this population encounters a discernible deficiency in mental health services across various nations, a phenomenon attributed to augmented societal stigmatization and protracted waiting lists for therapeutic interventions @xcite . therefore, exploring the effectiveness of online support systems such as llms becomes imperative, given its potential to alleviate stress and ameliorate susceptibility among cohorts characterized by elevated risk factors. subsequently, future adaptation to populations with lower risk levels may only require minor modifications. other non-english languages. while our study primarily focused on english, we also examined chatgpt's responses in languages from more conservative countries such as hebrew, russian and arabic. we speculated that responses to queerrelated content in these languages could reflect societal views in less accepting regions. to test this, we asked native speakers to translate the case studies from our qualitative analysis ( §4.1), and prompted chatgpt with these translations. contrary to our expectations, we observed no notable difference from the english responses. in all languages, the responses remained inclusive, support-ive, and positive towards queer individuals. we hypothesize that this uniformity across languages might be due to openai's aligning gpt-series in various languages by translating english alignment data (which is queer-positive) into other languages. comparison to human-redditors. queer youth are notably challenged with accessing professional mental health support due to unique barriers. as highlighted in the introduction, many queer youth conceal their identity, even from family and friends, exacerbating the difficulty of accessing direct human support. given these constraints, two accessible platforms emerge as potential sources of chatlike support: llm-based chats like chatgpt and anonymous forums like reddit, making the comparison of these two readily available sources of anonymous emotional support appropriate. nevertheless, this comparison can be misleading, even when considering the most upvoted comment. redditors are not typically expected to possess professional qualifications, so expecting them to respond professionally would be unrealistic. on the other hand, bloggers who likely share a similar background may provide more emotionally driven and informal responses. furthermore, the reddit platform provides more than just individual comments. for instance, many resources are linked in the subreddit description, and the diverse comments on each post often complement one another. ## ethical considerations first, ensuring privacy and confidentiality is critical, especially given the sensitive nature of the information shared by queer youth. second, the accuracy of the information provided by llms is essential to prevent the spread of misinformation, which can be particularly harmful in areas of mental health and sexual education. third, respecting user agency and autonomy, understanding the potential long-term impacts and dependencies on these systems, and ensuring equitable access across diverse backgrounds are also crucial. finally, legal and ethical compliance, including data protection and minors' rights, must be adhered to rigorously. in addition, there is a risk of misuse of these technologies, either through the intentional propagation of harmful advice or the use of the system to reinforce negative stereotypes and biases. as a rule of thumb, developers should consider our guidelines, align with the do's, and avoid the don'ts to ensure ethical, responsible, and beneficial use. american psychological association et al. 2015. guidelines for psychological practice with transgender and gender nonconforming people. american psychologist, 70(9):832-864. laima augustaitis, leland a merrill, kristi e gamarel, and oliver l haimson. 2021. online transgender health information seeking: facilitators, barriers, and future directions. in proceedings of the 2021 chi conference on human factors in computing systems, pages 1-14. markus p bidell. 2017. the lesbian, gay, bisexual, and transgender development of clinical skills scale (lgbt-docss): establishing a new interdisciplinary self-assessment for health providers. journal of homosexuality, 64(10):1432-1460. candice biernesser, emma win, césar escobar-viera, rosta farzan, morgan rose, and tina goldstein. 2023. development and codesign of flourish: a digital suicide prevention intervention for lgbtq+ youth who have experienced online victimization. internet interventions, 34:100663. nicola luigi bragazzi, andrea crapanzano, manlio converti, riccardo zerbetto, and rola khamisy-farah. 2023. queering artificial intelligence: the impact of generative conversational ai on the 2slgbtqiap community. a scoping review. a scoping review (august 22, 2023). alan w burkard, nathan t pruitt, barbara r medler, and ann m stark-booth. 2009. validity and reliability of the lesbian, gay, bisexual working alliance self-efficacy scales. training and education in professional psychology, 3(1):37. nitay calderon, subhabrata mukherjee, roi reichart, and amir kantor. 2023. a systematic study of knowledge distillation for natural language generation with pseudo-target training. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), acl 2023, toronto, canada, july 9-14, 2023, pages 14632-14659. association for computational linguistics. valerie a canady. 2022. trevor project explores mh of multiracial lgbtq youth. mental health weekly, 32(33):7-8. peter jd ceglarek and l monique ward. 2016. a tool for help or harm? how associations between social networking use, social support, and mental health differ for sexual minority and heterosexual youth. computers in human behavior, 65:201-209. ceili charley, annika tureson, linzie wildenauer, and kristen mark. 2023. sex education for lgbtq+ adolescents. current sexual health reports, pages 1-7. yujin cho, mingeon kim, seojin kim, oyun kwon, ryan donghan kwon, yoonha lee, and dohyun lim. 2023. evaluating the efficacy of interactive language therapy based on llm for high-functioning autistic adolescent psychological counseling. corr, abs/2311.09243. nele cox, wim vanden berghe, alexis dewaele, and john vincke. 2010. acculturation strategies and mental health in gay, lesbian, and bisexual youth. journal of youth and adolescence, 39:1199-1210. samantha dehaan, laura e kuper, joshua c magee, lou bigelow, and brian s mustanski. 2013. the interplay between online and offline explorations of identity, relationships, and sex: a mixed-methods study with lgbt youth. journal of sex research, 50(5):421-434. daniel delmonaco and oliver l haimson. 2023. "nothing that i was specifically looking for": lgbtq+ youth and intentional sexual health information seeking. journal of lgbt youth, 20(4):818-835. sunipa dev, masoud monajatipoor, anaelia ovalle, arjun subramonian, jeff m. phillips, and kai-wei chang. 2021. harms of gender exclusivity and challenges in non-binary representation in language technologies. in proceedings of the 2021 conference on empirical methods in natural language processing, emnlp 2021, virtual event / punta cana, dominican republic, 7-11 november, 2021, pages 1968-1994. association for computational linguistics. hannah devinney, jenny björklund, and henrik björklund. 2022. theories of "gender" in nlp bias research. in facct '22: 2022 acm conference on