OATH -Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants (2024)

Abstract: Warning: Contents of this paper may be upsetting.Public attitudes towards key societal issues, expressed on online media, are of immense value in policy and reform efforts, yet challenging to understand at scale. We study one such social issue: homelessness in the U.S., by leveraging the remarkable capabilities of large language models to assist social work experts in analyzing millions of posts from Twitter. We introduce a framing typology: Online Attitudes Towards Homelessness (OATH) Frames: nine hierarchical frames capturing critiques, responses and perceptions. We release annotations with varying degrees of assistance from language models, with immense benefits in scaling: 6.5× speedup in annotation time while only incurring a 3 point F1 reduction in performance with respect to the domain experts. Our experiments demonstrate the value of modeling OATH-Frames over existing sentiment and toxicity classifiers. Our large-scale analysis with predicted OATH-Frames on 2.4M posts on homelessness reveal key trends in attitudes across states, time periods and vulnerable populations, enabling new insights on the issue. Our work provides a general framework to understand nuanced public attitudes at scale, on issues beyond homelessness.

## introduction public opinion towards social issues is used as a signal by advocacy organizations to garner support for their relief and reform efforts @xcite . online social media platforms, such as twitter foot_0 , serve as a great repository of public opinion from a diverse (e.g., socio-economic and political) user base @xcite . however, understanding broader public opinion necessitates processing online discourse at a large scale, a daunting challenge in social science research. large language models (llms) have recently been used in social science @xcite @xcite , and may offer a promising solution to the scale challenge. however, can llms be reliably used to process discourse with nuanced latent attitudes towards complex societal issues? one such issue is homelessness in the u.s. foot_1 , which inspires complex and nuanced public attitudes and beliefs @xcite . for instance, it is common for people to simultaneously feel both sympathy towards the plight of people experiencing homelessness (peh), as well as anger and resentment towards them @xcite @xcite . a majority of work from social science has studied discourse about homelessness via ethnographic studies, qualitative surveys, and smaller scale content analyses @xcite @xcite . however, understanding the societal attitudes about homelessness at a large scale via online discourse has rarely been attempted in prior efforts. our work investigates the role of llms in reliably assisting social work experts in characterizing such large-scale online discourse on homelessness. in this work, we introduce a novel framework to understand, synthesize and characterize large-scale public attitudes towards societal issues through a collaboration between social work experts and llms, illustrated in figure 1 . our framework consists of three components designed to study written discourse on social issues: (i) the discovery of a framing typology ( §2), (ii) annotation of data at scale with this typology ( §3), and (iii) the analysis of these collected annotations ( §4). while we focus our study on homelessness, our framework is generalizable to other domains of social discourse. at the heart of our framework is our novel framing typology to characterize online attitudes towards homelessness (oath-frames) in the u.s., as expressed in social media posts from twitter ( §2). our frames are developed using framing theory @xcite and grounded theory @xcite from social science, and in collaboration with social work researchers studying homelessness. oath-frames can be broadly categorized into three themes: critiques , perceptions and responses to peh, and further into nine finer-grained issue-specific frames; we observe that the most salient theme in expert-only annotations is perceptions of peh while the rest of our annotations highlight critiques of the government, normative structures and (lack of) aid distribution (figure 2 ). our 2.4m oath-frames-annotated posts enable a large-scale analysis across states and time periods, revealing changing trends in attitudes with key sociopolitical events ( §4). oath-frames surface harmful language towards peh, which we show are often mislabeled by popular sentiment and toxicity classifiers, highlighting the value of our typology. we empirically establish trends where differences in attitudes are observed, when vulnerable populations are compared with peh; for instance, comparisons with immigrants elicit harmful rhetoric whereas those with ukrainians elicit critiques towards the government for funding allocation. oath-frames can assist in understanding nuanced issues concerning marginalized peh populations, often ignored in language technologies @xcite . overall, our framework provides a blueprint for fast and scalable analysis of public opinion on key social issues of our time; we believe this will aid advocacy groups and political constituents in work towards meaningful social definition example and labels from different annotators govcrit. government critique: criticism about government body, policies and laws including discussion of homelessness through the lens of political parties and values. moneyaid. money aid resource allocation: discussion of money, aid or resource disbursement on addressing the homelessness issue. examples include the allocation of necessities such as essential items for emergency relief supplies, or government budgeting with respect to competing priorities. you propose atlanta/fulton county fund a homeless city. i'm asking what return are we getting for our tax dollars? police officers and firefighters provide a service which can justify us building a training for them. what justification do you have for your city for the homeless? all: moneyaid. , govcrit. / expert, gpt-4, expert+gpt-4: solnint. / expert: (un)deserv. soccrit. societal critique: criticism of social norms, systems and society at large in how homelessness is being addressed and perceived. also includes pointing out hypocrisy and performative activism. ## oath-frames discovery we discovered a novel typology of pragmatic frames for capturing and characterizing societal attitudes towards homelessness and people experiencing homelessness (peh). to this end, we applied concepts from framing theory @xcite , which has been used to identify how certain aspects in an issue's portrayal are emphasized to influence broader public opinion for topics like immigration @xcite , politics in media @xcite , eating disorders @xcite , and so on. in social media discourse, prior work has focused on the framing of stigma towards peh @xcite and surveys that study attitudes and perceptions towards peh @xcite . in our work, we drew in-3 https://dill-lab.github.io/oath-frames/ spiration from @xcite and their application of grounded theory @xcite to characterize societal attitudes on homelessness at scale, going beyond stigma. following boydstun and gross (2013), our frame discovery process uncovered salient themes (frame types) in a text corpus on a topic; in the context of our study, this resulted in oath-frames (frames for online attitudes towards homelessness). using a keyword search for "homeless" on the twitter api, foot_2 we sample 5m posts. we preprocessed this sample (i.e. deduplicated, replaced emojis with unicode, anonymized user tags, and removed non-english posts) to retain 3.1m posts from 2.1m unique users spanning from 2021 to 2023. as a first step in frame discovery, we only considered posts discussing public attitudes and opinions on homelessness, following boydstun and gross's (2013) public opinion frame. we discarded posts which we deemed ambiguous, underspecified or unrelated (e.g. political homelessness). we also discarded posts containing first person lived experiences of peh, since our focus was on studying public attitudes on homelessness. next, we followed an extensive iterative 'coding' process based on grounded theory from social science @xcite to surface our oath-frames where 'codes' are 1-2 word descriptors of the data. the coding process involved our four domain experts independently analyzing a sample of 100 posts to conceptualize a unique categorization of attitudes. this was followed by a discussion where all posts were reviewed and similar frames were grouped together till we reached consensus on definitions, edge cases and representative examples. we repeated this process thrice on different sets of posts with refined iterations of our frames until we reached thematic saturation which surfaced nine fine-grained issue-specific frames under homelessness; see our oath-definitions in our issue-specific frames can be categorized into three overarching themes, as illustrated in figure 1 (frame discovery). first, we find critiques of systems and institutions enabling homelessness and also the use of homelessness as a vehicle to critique others (e.g. actors, programs or spending). second, we find posts reflecting different perceptions of peh which attribute undesirable characteristics to peh to make harmful generalizations. third, we find many responses to the issue, posed as solutions, interventions, calls to action and charitable acts. our three themes are consistent with those in prior work on homelessness @xcite , but our work is the first to surface nine issue-specific (homelessness) frames providing more fine-grained insights. ## analyzing posts with oath-frames synthesizing public opinion using online discourse is useful for advocacy groups and policymakers to inform homelessness services and equitable housing policies to local communities @xcite . we demonstrate the utility of oath-frames to such end users by analyzing attitudes with respect to state mentions, marginalized populations, and time period on the set of 2.4m posts. we also highlight the utility of our oath-frames over sentiment and toxicity labels. ## conclusion and future work we introduce oath-frames which characterize online attitudes towards homelessness under three overarching themes and nine issue-specific frames. we annotate posts using expert, gpt-4 and expert+gpt-4 and use them to train a model to predict oath-frames on a large collection of 2.4m posts. we show that modeling oath-frames offers distinct advantages over existing models for toxicity and sentiment classification, and reveals nuanced attitudes in relation to states, marginalized populations, and over time. we hope to expand oath-frames to include more context such as target groups, or correlated factors (e.g., substance use and mental illness). our work serves as a blueprint for building frameworks that incorporate insights from social science experts for different large-scale socially sensitive annotation tasks. we expect oath-frames to serve as a tool for social science practitioners in analyzing attitudes towards complex social issues at scale.