Generating Ethnographic Models from Communities’ Online Data (2020)

Abstract: In this paper we describe computational ethnography study to demonstrate how machine learning techniques can be utilized to exploit bias resident in language data produced by communities with online presence. Specifically, we leverage the use of figurative language (i.e., the choice of metaphors) in online text (e.g., news media, blogs) produced by distinct communities to obtain models of community worldviews that can be shown to be distinctly biased and thus different from other communities’ models. We automatically construct metaphor-based community models for two distinct scenarios: gun rights and marriage equality. We then conduct a series of experiments to validate the hypothesis that the metaphors found in each community’s online language convey the bias in the community’s worldview.

## introduction recent advances in machine learning, particularly deep learning, have led to successful exploitation of vast amounts of human-generated internet data and have produced remarkably accurate computational models of complex semantic and social phenomena in language, speech, vision, and other media, thus bringing us closer to the practical reality of artificial intelligence. these models are often considered objective and universal because the volume of data on which they are based is so vast that it is believed to be free of sampling limitations plaguing earlier research. and yet, the models that can be derived are only as good as the data from which they are built; the data, however vast, may still be biased. for one, people who post on the internet are not necessarily representative of the general population. furthermore, society is composed of various communities and groups whose opinions and worldviews differ dramatically on a range of important issues. when data are oversampled from some sources over others (which can easily happen due to different rates of production), the resulting model is bound to be biased accordingly. this bias can lead to unwanted consequences, for example in government planning or resource allocation. the flip side of the data bias, however, is that each community produces a unique information footprint that may be used to understand how its members perceive the world. the objective of our project is to investigate whether online information generated by various communities can serve as raw data for developing reliable and accurate ethnographic models of these communities, thus augmenting costly and limitedscale field studies. clearly, the methods of computational ethnography will be different from its traditional counterpart and will rely extensively on substantial volumes of largely undirected data, from which the critical information (e.g., relationships, opinions, conceptualizations) can be learned. one rich source of data is language, which serves as a communication vehicle, but also, as it evolves, encodes social, cultural, and often physical experiences of its user communities. these experiences are often vividly captured in the use of figurative language constructs, such as metaphors, that directly link abstract notions to collective physical experiences @xcite . this observation has been confirmed in earlier work that constructed metaphor repositories across language and cultural dimensions (western/american, latin american/mexican, eastern european/russian, middle eastern/persian) related to such notions as government, economic inequality, and democracy (e.g., @xcite ; ## initial case study and approach our initial approach was to develop and evaluate ethnographic models of two u.s. communities: (1) a community whose members prefer individual oversight of guns and prioritize gun rights, which we refer to as indo; and (2) another community whose members prefer government oversight of guns and prioritize gun control, which we refer to as govto. our research demonstrates that these two communities' cultural models differ fundamentally from one another in their representation and valuation of concepts related to the gun debate. these concepts include broad notions such as gun rights and gun control, as well as narrower issues such as the second amendment, school shootings, and assault weapons. each community is defined by the set of valuations they assign to these concepts. in order to extract these valuations, we identify culturally biased correlations, expressed in the use of metaphorical language, between key gun-related concepts and more basic, concrete, and imageable source domains, such as war, barrier, disease, animals, natural force, water, and foodstuffs. additionally, we capture the prevalent sentiment that members of each community apply when referring to these concepts, in both literal and metaphorical contexts. we leverage language data available through public online sources produced by the target communities. these sources include mainstream media as well as public community blogs, newsletters, and websites. data are collected and processed automatically using simple internet crawlers and natural language processing software capable of analyzing sentences for grammatical components, sentiment, and presence of metaphors. the processed data are then deposited in searchable structured repositories that are unique to each community. we estimate that the amount of data required to support both confirmatory and exploratory studies is approximately 5 million words per community. in the initial feasibility demonstration stage of this project, we collected 6 million words for both the indo and govto community. in the remainder of this section, we provide detailed descriptions of data collection and processing procedures, as well as the repository construction process. these steps prepare us for the experiments described in sections 5 and 6. ## metaphor extraction approach we distinguish two levels of metaphor identification: (1) text (or linguistic) metaphors that consist of a metaphorical target, typically an abstract concept, and a relation adopted from a concrete source domain; and (2) conceptual metaphors that generalize across multiple occurrences of text metaphors involving the same target. while text metaphors have the semantic form of shared-property (target, source), the conceptual metaphor usually conveys a more definite mapping @xmath0. for example, the textual metaphor "erosion of gun rights" alludes a shared property between gun rights and a geological landmark, thus invoking "gun rights is a geological landmark" conceptual metaphor. we note that conceptual metaphors are often implied rather than directly stated. accordingly, the metaphor extraction process follows these two steps: we extract text metaphors first and then fuse them into conceptual ones. for ethnographic modeling purposes we use conceptual metaphors espoused in the language generated by a particular group of people. we have developed a data-driven computational approach to extracting text metaphors that combines topical structure and imageability analysis in order to locate the candidate metaphorical expressions within text @xcite . to analyze topical structure, we identify nouns and verbs in a text passage and link their repeated occurrences, including co-references, synonyms, and hyponyms, and combine them into topic chains. content words (e.g., verbs, nouns, adjectives) found outside these topical chains are candidate source relations if they also carry high imageability scores. imageability ratings of most lexical items are looked up in an expanded mrc psycholinguistic database, which were built for several languages @xcite . the candidate relations are then used to compute and rank possible source domains in an emerging conceptual metaphor. full details of the metaphor extraction process can be found in the cited papers. our approach to metaphor extraction is contrasted with more traditional computational approaches based on selectional restriction violations @xcite @xcite @xcite @xcite for an overview) which do not scale well due to their heavy reliance on domain knowledge. more recent variants of this general approach (e.g., @xcite utilize more robust deep learning methods but their utility remains limited to only some forms of text metaphors. ## metaphor-based ethnographic models in this section, we outline the ethnographic models derived for each of the two scenarios. we provide only top-level characterization of each domain in terms of selection and distribution of metaphors that define each community's viewpoint: two communities for the gun debate scenario and three communities for the marriage equality scenario. ## confirmatory studies on the gun debate we conducted a study to experimentally confirm the bias in community data in relation to the gun debate scenario. a subset of passages containing metaphors was selected from both the indo and govto communities' metaphor repository and were displayed to human participants, whose task was to categorize each passage as advocating for either individual or government oversight of guns. the objective of this study was to confirm that the bias was captured in the metaphors used by each community and that this bias can be detected by human raters. we thus predicted that participants would be able to categorize the passages as representing the intended community viewpoints at rates above chance. this result would confirm that our metaphor repositories accurately reflect the language use of the two communities relevant to this target scenario (i.e., indo and govto). a sample of 338 respondents completed the study via amazon mechanical turk. raters viewed 20 passages from indo sources and 20 passages from govto sources. overall accuracy scores were calculated by dividing the total number of correct categorizations by the total number of passages (i.e., 40). as predicted, participants categorized passages with abovechance accuracy (mean @xmath1, @xmath2, t(337)=21.94, p<.001, @xmath3.19. indo and govto categorization accuracy scores were calculated by dividing the number of correct categorizations for each passage type by 20. participants categorized passages from govto sources (mean @xmath4, @xmath5, @xmath6, t(337)=7.62, p<.001, @xmath7.41. thus, human raters were able to determine whether passages came from indo or govto media sources at reliably greater-thanchance (above 50% accuracy) rates. we replicated this study with another sample of 906 participants who rated 40 randomly selected (vs. researcher-selected, as in the previous study) passages from the same metaphor repositories on a continuous scale (0=definitely in favor of government oversight to 100=definitely in favor of individual oversight). as predicted, participants rated indo passages as being reliably more in favor of individual oversight than the scale midpoint (@xmath8.48, @xmath9.26), t(905)=17.47, p<.001, @xmath10.58. participants also rated govto passages as being reliably more in favor of government oversight than the scale midpoint (@xmath11.31, @xmath12.71), t(905)=-27.62, p<.001, @xmath13.92. these results again suggest that participants were able to detect the bias present in the passages and rated them accordingly. ## confirmatory study on the marriage equality debate we conducted a study to confirm the bias in community data in relation to the marriage equality scenario. following the same procedure as in the first study, 285 participants categorized a total of 45 passages automatically selected from progressive, moderate, and traditional sources as representing a progressive, moderate, or traditional stance on marriage. as predicted, participants categorized passages with above chance accuracy, with 33% accuracy representing chance (mean @xmath14, @xmath15, t(284)=11.49, p<.001, @xmath16.68. moreover, participants categorized passages from moderate sources (@xmath17, @xmath18, @xmath19, t(568)=-3.70, pbonferroni<.001, @xmath20.20. accuracy scores for passages from traditional sources (mean @xmath21, @xmath22. thus, human raters were able to determine whether passages regarding marriage equality originated from progressive, moderate, or traditional media sources at reliably greater-than-chance rates. ## conclusion in this paper, we presented results from a project in which we built ethnographic models of communities based on the choice of metaphors in online language use. we investigated two distinct scenarios, a binary gun rights vs. gun control debate and a non-binary marriage equality debate. we demonstrated in both cases that metaphor choice provides strong clues to a community's identity and bias, and that automatically derived models can adequately delineate target communities. future work will focus on improving the accuracy of metaphor classification and exploiting other forms of figurative language that capture deeply held collective meanings and stances. this research explored a new avenue of computational ethnography, conducted entirely online. the long-standing benchmark is traditional field ethnography that involves typically smallscale field work, which takes months or years to complete. such studies, while producing "thick" models, are not always feasible, especially in the areas of conflict or disturbance nor when a rapid response is required. furthermore, field ethnography by its nature is heavily reliant on regional, local language speaking subject matter experts who may be hard to find due to lack of expertise or risks involved. current approaches to online ethnography include application of traditional methods of observation of online behavior aimed at modeling online communities, as opposed to the real-world communities (e.g., @xcite . efforts aimed at deriving offline models from online data are either small-scale @xcite or limited to superficial analysis of social media (e.g., sentiment extraction) that cannot easily separate transient views from entrenched opinions (e.g., @xcite . these approaches produce low-quality results also due to over-sensitivity to data noise and are thus unreliable. we believe that our research shows the value of advanced sociolinguistic analysis and natural language processing in studying the online human terrain.