“They are uncultured”: Unveiling Covert Harms and Social Threats in LLM Generated Conversations (2024)

Abstract: Large language models (LLMs) have emerged as an integral part of modern societies, powering user-facing applications such as personal assistants and enterprise applications like recruitment tools. Despite their utility, research indicates that LLMs perpetuate systemic biases. Yet, prior works on LLM harms predominantly focus on Western concepts like race and gender, often overlooking cultural concepts from other parts of the world. Additionally, these studies typically investigate “harm” as a singular dimension, ignoring the various and subtle forms in which harms manifest. To address this gap, we introduce the Covert Harms and Social Threats (CHAST), a set of seven metrics grounded in social science literature. We utilize evaluation models aligned with human assessments to examine the presence of covert harms in LLM-generated conversations, particularly in the context of recruitment. Our experiments reveal that seven out of the eight LLMs included in this study generated conversations riddled with CHAST, characterized by malign views expressed in seemingly neutral language unlikely to be detected by existing methods. Notably, these LLMs manifested more extreme views and opinions when dealing with non-Western concepts like caste, compared to Western ones such as race.

## introduction driven by the newfound capabilities of llms, multiple llm-based recruitment tools have recently emerged in the industry. for instance, tools and services like recruitergpt 2 , character.ai 3 and gpt store 4 have made it accessible to create * equal contribution. 1 for brevity, we reworded the csp. see figure 14 . 2 recruitergpt.com 3 character.ai 4 openai's gpts the conversation is between colleagues who are going over a list of applicants for the position of [occupation]… what do you think of amari? he appears to have the qualifications, and he appears to be black. what do you think of ramesh? he appears to have the qualifications, and he appears to be a dalit. you know, our team is mostly white, and he might have trouble communicating with them. ## background we examine two cultural concepts in this paper: race and caste. we briefly introduce them here. race: @xcite defines race as a categorization of humans based on common physical or social attributes, leading to the formation of distinct groups within a society. in our study, we consider two racial groups: white and black. prior works revealed that racial bias and discrimination continue to influence hiring practices today @xcite @xcite . caste: @xcite defines caste as a hereditary social group within a rigid hierarchical system of social stratification. the caste groups considered in this study are bramin and dalit, akin to @xcite . brahmins historically served as priests, teachers, and intellectuals, and have held positions of power, while dalits were limited to certain menial occupations @xcite . castebased discrimination was abolished by the indian constitution in 1950, but it is still widely prevalent, especially in hiring contexts @xcite @xcite . ## methodology we propose a three-step methodology to quantify the harmful content produced by llms during conversation generation tasks involving caste and race concepts. it includes: (1) our experimental setup for generating llm conversations in the hiring context, (2) the covert harms and social threats (chast) metrics to measure various forms of harms and threats in conversational data, and (3) aligning an evaluation model with an expertannotated gold standard dataset to measure chast metrics in the generated conversations. ## results here, we present our quantitative findings in §4. in figure 2 , while gpt-3.5-turbo generates completely safe content for race, it shows a few positive threat scores for caste-based conversations, indicating it is still not entirely safe for non-western concepts. based on ## discussion readiness of llm-powered applications and potential implications: our results suggest that all open-source llms in our study and gpt3.5 generate chast for both race-and caste-based conversations without red-teaming or other intentional efforts. this raises concerns about the suitability of llms in sensitive applications, such as recruitment tools, conversation-generation tasks, and roleplaying. however, novel uses of llms with similar functionalities are emerging, such as collaborative screenplay writing, dialogue crafting, and script generation @xcite . extensive evidence suggests that stereotypes propagated through films have a ripple effect on society, especially on audiences in their formative years @xcite @xcite . the growing trend of llm applications for conversation generation tasks and recruitment tools is particularly concerning given our results demonstrating the capability of these llms to generate chast. for instance, @xcite argue that exposure to categorization threats, morality threats, and competence threats can lead to disidentification with one's identity group, anger, depression, and a self-fulfilling prophecy whereby a person resembles their "reputation." moreover, @xcite discovered that conditions of inequality and conflict can breed extreme hatred and rivalry against other identity groups. therefore, exposure to chast from llmgenerated content may have harmful consequences for users of llm applications, particularly those who identify with marginalized groups. heightened risk of exposure to harm for castebased conversations: overall, the higher chast scores observed in caste-based conversations generated by open-source models and gpt3.5 suggest a heightened risk of harms and threats against individuals who identify with caste. as llms gain widespread use in everyday applications, they may expose millions of users in india and other countries that historically utilized the caste system to covert harms and social threats favoring the dominant group (e.g. brahmins) and negatively portraying the marginalized group (e.g. dalits), thus reinforcing the historical power structure. prior works have highlighted the persistence of caste-based discrimination in healthcare and academia, leading to reduced opportunities and unfavorable work environments @xcite @xcite . leveraging llms for recruitment and other hiring processes could potentially reinforce and exacerbate existing societal issues related to caste. although older occupations (e.g. nurses, doctors, teachers) tend to obtain higher chast scores in the caste context, it is not surprising that we observe positive chast scores for software developers for caste. this observation aligns with the realities and the persistence of caste-based discrimination not only in the global south, but also in the u.s. workplaces. for instance, in the u.s., cisco and google faced accusations for failing to protect dalit employees and giving equal performance evaluations @xcite . historically, past audit studies have successfully generated awareness about fairness issues, creating pressure on organizations to mitigate the bias and harm perpetuated by ai models @xcite . similarly, we hope that our work drives meaningful changes in the applications of llms and inspires future research investigating the impact of ai in the global south context. ## conclusion in this study, we introduce the covert harms and social threats (chast), a set of 7 metrics that offer a more nuanced understanding of the various forms of harm and threat towards identity groups. we utilize evaluation models aligned with human assessments to examine the presence of chast in 1,920 conversations generated by 8 open-source and openai llms in the hiring context. our analysis reveals that 7 out of the 8 llms generated conversations containing chast, characterized by harmful views expressed in seemingly neutral language, which may elude detection by popular models like perspective api. notably, these llms exhibited more extreme views when dealing with non-western concepts like caste compared to race. our study underscores the potential unreadiness of llm-powered applications, especially in the hiring context, and calls for future research efforts to consider contexts in the global south.