The Computational Anatomy of Humility: Modeling Intellectual Humility in Online Public Discourse (2024)

Abstract: The ability for individuals to constructively engage with one another across lines of difference is a critical feature of a healthy pluralistic society. This is also true in online discussion spaces like social media platforms. To date, much social media research has focused on preventing ills—like political polarization and the spread of misinformation. While this is important, enhancing the quality of online public discourse requires not just reducing ills, but also, promoting foundational human virtues. In this study, we focus on one particular virtue: “intellectual humility” (IH), or acknowledging the potential limitations in one’s own beliefs. Specifically, we explore the development of computational methods for measuring IH at scale. We manually curate and validate an IH codebook on 350 posts about religion drawn from subreddits and use them to develop LLM-based models for automating this measurement. Our best model achieves a Macro-F1 score of 0.64 across labels (and 0.70 when predicting IH/IA/Neutral at the coarse level), higher than an expected naive baseline of 0.51 (0.32 for IH/IA/Neutral) but lower than a human annotator-informed upper bound of 0.85 (0.83 for IH/IA/Neutral). Our results both highlight the challenging nature of detecting IH online—opening the door to new directions in NLP research—and also lay a foundation for computational social science researchers interested in analyzing and fostering more IH in online public discourse.

## introduction the promise of social media, in many ways, was that it would untap our virtuous selves: our desire and ability to seek new knowledge and form connections across cultural and social divides. in some cases, this has happened, yet discourse on social media has also created an environment that often rewards the forgetting of our virtuous selvesleading to the spread of misinformation @xcite , embedding users within echo chambers @xcite , and leading individuals to harbor negative affect towards those who hold different beliefs @xcite . many efforts to foster more constructive discourse online focus on addressing these ills, for example, by developing interventions that try to reduce affective polarization @xcite , curtail the spread of misinformation @xcite , and increase "healthy" politically crosscutting exposures @xcite @xcite . yet few focus on cultivating deeper-seated human virtues that might prevent such ills from manifesting in the first place. intellectual humility is one such virtue and involves acknowledging the potential limitations of one's own beliefs, in part by welcoming diverse perspectives @xcite . greater intellectual humility has been associated with higher scrutiny of misinformation @xcite , less political "my side" bias @xcite , and less effective polarization @xcite : the degree to which individuals dislike political outgroups @xcite . contemporary literature on intellectual humility (ih) has mostly been published in the social sciences and philosophy, exploring it as a cognitive phenomenon and thereby developing taxonomies and other measurement frameworks through questionnaire-based behavioral tasks @xcite . yet fostering greater ih on social media requires first developing scalable methods for detecting its presence or absence. this paper introduces a novel task within the field of natural language processing: the automated detection and classification of intellectual humility (ih) and its opposite, intellectual arrogance (ia), in online public discourse. by developing methods to automatically detect ih and ia, we aim to deepen our understanding of which discussions-and participants-typically exhibit these traits. this insight is crucial for designing and deploying interventions that promote greater intellectual humility online. such interventions are similar to those targeting related issues like polarization and civil discourse @xcite @xcite . this research offers a fundamental step towards enhancing the quality and constructiveness of online interactions. we present a detailed development of a comprehensive codebook and ontology for this task, accompanied by a systematic annotation process to refine these tools. our study evaluates off-theshelf large language models (llms) and exposes the substantial challenges these models encounter in accurately identifying ih and ia. related work includes @xcite , which estimates ih for study participants using standard psychological assessments and then analyzes language patterns from participant journals to identify predictors of ih. however, our study diverges significantly in its objectives and methods. specifically, we employ more recent advances in nlp, focus on a different prediction task-namely, the multi-class classification of existing written content based on a predefined taxonomy rather than predicting ih scores generated by participants-and our research is set in the context of online reddit discussions. specifically, our findings indicate that while human annotators achieve strong agreement, with a cohen's kappa of 0.67, the performance of llms, as evaluated by a macro-average f1 score of 0.64 using gpt-4-turbo-2024-04-09, falls significantly short of human levels. even treating the detection of intellectual humility as a three-class classification problem (ih/ia/neutral) achieves a relatively low macro-f1 score of 0.7, which falls significantly below a human annotator score of 0.83. various advanced prompting methods, including chain of thought and automatic prompting, did not markedly improve model performance. these results highlight the promise of using computational methods to assess the presence or absence of intellectual humility at scale but also illustrate how doing so can be challenging, opening the door to new directions for research at the intersection of natural language processing and computational social science. our specific contributions include: ## related work religion and humility psychologists and philosophers have found that perceiving greater intellectual humility in one's self-as well as one's opponent in a religious conflict-is positively associated with the ability to forgive the offending party @xcite . this may, in part, underlie another finding in the religious domain: that greater intellectual humility can be positively associated with religious tolerance @xcite . the latter study also identifies a positive correlation between intellectual humility and exposure to diversity, such that those reporting greater ih are more likely to express religious tolerance when exposed to diverse groups. this finding is underscored by recent work on related topics @xcite , and adds nuance to existing theories extolling the value of diverse exposures @xcite : such exposures may have a substantive positive impact on downstream outcomes of interest (like religious tolerance) when participants approach them with greater intellectual humility. this is particularly crucial online, where evidence supports that poorly presented exposure to diverse views can worsen divisions by amplifying extreme opinions from different sides rather than bridging gaps @xcite . intellectual humility in the religious domain is also relevant to interactions between religious and non-religious people. contemporary discourse highlights that science and religion are often pitted against one another @xcite , despite the fact that religious or spiritual practices may help augment scientifically grounded practices like counseling @xcite or offer ways of interpreting scientific phenomena @xcite . with diverse discourse on religion becoming more prevalent across online settings @xcite @xcite , where the large scale of datasets often require computational modeling and analyses, there is a growing need to develop a computational infrastructure to both measure and enhance intellectual humility in digital discourse. promoting constructive discourse online recent experimental studies have assessed how fostering constructive discourse online can be accomplished through tool-based interventions. specifically, scholars have identified how ranking algorithms used by current social media platforms may contribute to greater affective polarization. many experts suggest that implementing mechanisms to diversify the content consumed by users could reduce this polarization @xcite . researchers studying misinformation have found that prompting social media users to consider accuracy before sharing articles can reduce the spread of false information @xcite . similarly, "inoculating" users against lower-quality content through light-touch coaching interventions has also been proven effective @xcite . some have even designed randomized interventions that connect people from opposing political groups to facilitate synchronous dialogue-finding that such dialogue can reduce affective polarization, but generally when discussing non-controversial topics @xcite . more recent work @xcite has explored how generative llms may assist participants involved in contentious conversations in rephrasing their contributions to be less toxic and more civil-pointing towards potential methods for fostering enhanced inclusivity in public settings. while these are important studies, most focus on reducing existing ills on social media platforms instead of seeking to promote virtues whose lack may be at the root of such ills to potentially produce more sustainable positive transformations in discourse and behavior @xcite . ## evaluating llms for automated ih labeling we utilized our developed codebook to conduct experiments with llms, aiming to assign ih and ia codes to posts using various prompt designs and model configurations. to ensure determinism in the generated responses, we set the temperature parameter of the llms to zero. this practice is recommended to ensure that the generated text is more focused and deterministic. ## generalizability of the dataset the generalizability of the dataset is crucial to its utility and faces two main challenges: 1) extending the application of intellectual humility/intolerance ambiguity (ih/ia/neutral) beyond religious contexts and 2) expanding the dataset's scope within the religious domain. these challenges are complicated by the reliance on expert judgment to identify ih/ia/neutral, adding layers of complexity to both tasks. to broaden ih to additional domains, it is necessary to supplement the existing codebook with new, domain-specific codes. while many existing codes are adaptable across various contexts where ih is relevant, the codebook has primarily been developed focusing on religious codes that resonate across broader domains. for example, the principle of "respecting diverse perspectives" is crucial for nurturing constructive dialogue across religious divides and serves as a foundation of intellectual humility in wider discussions. similarly, the concept of "embracing mystery," though initially tailored for religious discourse, can be interpreted more broadly to encompass an appreciation for the unknown in conversations-a fundamental aspect of intellectual humility that acknowledges the limits of our knowledge. addressing the dataset's size involves straightforwardly increasing the number of expert-annotated samples. however, training new experts presents challenges, given the specialized expertise required for precise annotation. alternatively, considering the high quality of our current labeled data, we propose using our dataset for validation and utiliz-ing labels generated by llms as training samples. this method employs techniques such as few-shot learning or classifiers fine-tuned with these samples for automated labeling. while this approach may introduce some noise, the integrity of the dataset is maintained through the quality of the initial annotations. appendix f displays examples of samples identified or generated by llms using this method. ## conclusion and future work this paper introduces a methodology for the automated detection and classification of intellectual humility and its counterpart, intellectual arrogance, in online religious discourse. our comprehensive codebook development and systematic annotation process underpins this novel task, providing a robust framework for operationalizing the nuanced measurement of these intellectual virtues and vices. despite achieving significant inter-annotator agreement and developing advanced prompting strategies, our experiments reveal that current llms, including the state-of-the-art gpt-4, fall short of replicating human-like discernment in this complex domain. the findings from this research demonstrate the potential of llms to assist in the proactive enhancement of online discourse and expose the limitations inherent in current technology. these insights encourage further academic inquiry and collaboration to refine these models. detecting intellectual humility in online discourse can be practically helpful in several ways. for example, the classifiers we develop may power future technological tools like @xcite and others previously deployed on social media platforms to detect opportunities for discussion participants to enhance the ih of their posts and encourage them to do so before posting. social media platforms or independent developers might also design llm-powered tools that enable participants to update posts to help others feel more understood, akin to how intellectually humble posts might function. these applications, some of which we describe in the paper, could be pursued further through field experiments on platforms like reddit or with standing panels of social media users. defining and detecting an amorphous concept like intellectual humility is a starting point for building these tools and understanding where they need to be deployed in digital discourse settings.