Situated Data, Situated Systems: A Methodology to Engage with Power Relations in Natural Language Processing Research (2020)

Abstract: We propose a bias-aware methodology to engage with power relations in natural language processing (NLP) research. NLP research rarely engages with bias in social contexts, limiting its ability to mitigate bias. While researchers have recommended actions, technical methods, and documentation practices, no methodology exists to integrate critical reflections on bias with technical NLP methods. In this paper, after an extensive and interdisciplinary literature review, we contribute a bias-aware methodology for NLP research. We also contribute a definition of biased text, a discussion of the implications of biased NLP systems, and a case study demonstrating how we are executing the bias-aware methodology in research on archival metadata descriptions.

## introduction analysis of computer systems has raised awareness of their biases, prompting researchers to make recommendations to mitigate harms that biased computer systems cause. analysis has shown computer systems exhibiting biases through racism foot_0 (noble, 2018), sexism 2 (perez, 2019), and classism 3 (d' @xcite . this list of harms is not exhaustive; biased computer systems may also harm people based on ability, citizenship, and any other identity characteristic. to mitigate harms from biased computer systems, researchers have recommended actions, methods, and practices. however, none of the recommendations comprehensively address the complexity of the problems bias causes. considering the numerous types of bias that may enter a natural language processing (nlp) system, places that bias may enter, and harms that bias may cause, we propose a bias-aware methodology to comprehensively address the consequences of bias for nlp research. our methodology integrates critical reflection on social influences on and implications of nlp research with technical nlp methods. to scope our research direction and inform our methodology, we draw on an interdisciplinary selection of literature that includes work from the humanities, arts, and social sciences. we intend the methodology to (a) support the reproducibility of nlp research, enabling researchers to better understand which perspectives were considered in the research; and (b) diversify perspectives in nlp systems, guiding researchers in explicitly communicating the social context their research so others can situate future research in contexts that have yet to be investigated. we begin with our bias statement ( §2) and motivations for proposing a bias-aware nlp research methodology ( §3). next, we summarize the interdisciplinary literature informing our methodology ( §4), explain the methodology ( §5), and demonstrate it with a case study of our ongoing research with bias in archival metadata descriptions ( §6). we end with a summary and vision for future nlp research ( §7). ## bias statement we situate this paper in the united kingdom (uk) in the 21 st century, writing as authors who primarily work as academic researchers. we identify as three females and one male; and as american, german, and scots. together we have experience in natural language processing, human-computer interaction, data visualization, digital humanities, and digital cultural heritage. in this paper, we propose a biasaware methodology for nlp researchers. we define biased language as written or spoken language that creates or reinforces inequitable power relations among people, harming certain people through simplified, dehumanizing, or judgmental words or phrases that restrict their identity; and privileging other people through words or phrases that favor their identity. biased language causes representational harms @xcite , or the restriction of a person's identity through the use of hyperbolic or simplistic language @xcite . nlp systems built on biased language become biased computer systems, which "systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others" @xcite . representational harms may cause inequitable system performance for different groups of people, leading to allocative harms @xcite , or the denial of a resource or opportunity @xcite . the people who experience harms from biased nlp systems varies with the context in which people use the system and with the language source on which the system relies. moreover, people may not be aware they are being harmed given the black-box nature of many systems @xcite . that being said, whether or not people realize they are being prejudiced against, the people harmed will be those excluded from the most powerful social group. statistics report a homogeneity of perspectives among students in computer-related disciplines that do not reflect the diversity of people affected by computer systems, risking a homogeneity of perspectives in the technology workforce and the computer systems that workforce develops. for academic year 2018/19, statistics on students in the uk foot_2 report that the dominant group of people studying computer-related subjects overwhelmingly are white males without a disability. foot_3 moreover, differences in total numbers of surveyed students across identity characteristics (e.g. sex, ethnicity, disability) skew the statistics in favor of those reported as white, male, and without a disability. lack of diverse perspectives among students in computer-related disciplines may limit the diversity of perspectives in the workforce, where the development of nlp and other computer systems occurs. as of 2019, the wise campaign reported that women comprise 24% of the core-stem workforce in the uk. 6 lack of diverse perspectives in the development of nlp and other computer systems risks technological decisions that exclude groups of people ("technical bias"), as well as applications of computer systems that oppress groups of people ("emergent bias") @xcite . that being said, even if student demographics in nlp and computer-related disciplines become more balanced, the data underlying nlp systems will still cause bias. theories of discourse state that language (written or spoken) reflects and reinforces "society, culture and power" @xcite . in turn, nlp systems built on human language reflect and reinforce power relations in society, inheriting biases in language @xcite such as stereotypical expectations of genders @xcite and ethnicities @xcite . drawing on feminist theory, we argue that all language is biased, because language records human interpretations that are situated in a specific time, place, and worldview @xcite . consequently, all nlp systems are subject to biases originating in the social contexts in which the systems are built ("preexisting bias") @xcite . psychology research suggests that biased language causes representational harms: @xcite studied how masculine-generic language (e.g. "he") versus gender-neutral language (e.g. "he or she") affected participants' responses to questionnaires. the authors report that women gave themselves lower scores on intrinsic goal orientation and task value in questionnaires using masculine-generic language in contrast to questionnaires using gender-neutral language. 7 the study provides an example of how biased language may harm select groups of people, because the participants reported as women experienced a restriction of their identity, influencing their behavior to conform to stereotypes. acknowledging the harms of biased language and biased nlp systems, researchers have proposed approaches mitigating bias, though no approach has fully removed bias from an nlp dataset or algorithm. to mitigate bias in datasets, @xcite produced a dataset of gendered ambiguous pronouns (gap) to provide an unbiased text source on which to train nlp algorithms. however, the gap dataset reverses gender roles, assuming that gender is a binary rather than a spectrum. 8 any nlp system that uses the gap dataset thus adopts its preexisting gender bias. efforts to mitigate bias in algorithms are similarly limited, focusing on technical performance rather than performance in social contexts. @xcite describe an approach to debias word embeddings, writing, "finally we show that given sufficiently strong alternative cues, systems can ignore their bias" (p. 16). however, the paper does not explain the intended social context in which to apply the authors' approach, risking emergent bias. 9 additionally, @xcite demonstrate how this debiasing approach hides, rather than removes, bias. in our bias-aware methodology, we describe documentation and user research practices that facilitate transparent communication of biases that may be present in nlp systems, facilitating reflection on how to include more diverse perspectives and empower underrepresented people. ## interdisciplinary literature review to inform our proposed bias-aware nlp research methodology, we draw on an interdisciplinary corpus of literature from computer science, data science, the humanities, the arts, and the social sciences. nlp and ml scholars have recommended actions to diversify perspectives in technological research, recognizing the value of diversity to bias mitigation. @xcite and crawford (2017) recommend interdisciplinary collaboration so researchers can learn from humanistic, artistic, and sociological disciplines regarding human behavior, helping researchers to more effectively anticipate harms that computer systems may cause, in addition to benefits they may bring, addressing risks of emergent bias. they also recommend engaging with the people affected by nlp and other computer systems, testing on more diverse populations to address the risk of technical bias, and rethinking power relations between those who create and those who are affected by computer systems to address the risk of preexisting bias. though these recommendations address the three types of bias that may enter an nlp system, they do not articulate how to identify relevant people to include in the development and testing of nlp systems. our bias-aware methodology builds on recommendations from @xcite and @xcite by outlining how to identify and include stakeholders in nlp research ( §5.1). d @xcite propose data feminism as an approach to addressing bias in data science. they define data feminism as, "a way of thinking about data, both their uses and their limits, that is informed by direct experience, by a commitment to action, and by intersectional feminist thought" (p. 8). 10 data feminism has seven principles: examine power, challenge power, elevate emotion and embodiment, rethink binaries and hierarchies, embrace pluralism, consider context, and make labor visible. these principles facilitate critical reflection on the impacts of data's collection and use in social contexts. our bias-aware methodology tailors these principles to nlp research, outlining activities that encourage researchers to consider influences on and implications of their work beyond the nlp community ( §5.1). 7 the authors report that men showed no difference in their intrinsic goal orientation and task value scores with masculinegeneric versus gender-neutral language in the questionnaires; impacts on people who do not identify as either a man or a woman are unknown as the study groups participants into these two gender categories @xcite . within the nlp research community, @xcite recommend improved documentation practices to mitigate emergent, technical, and preexisting biases. they recommend all nlp research includes a "data statement," which they describe as, "a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software" (p. 587). aimed at developers and users of nlp systems, data statements reduce the risk of emergent bias. the authors also note: "as systems are being built, data statements enable developers and researchers to make informed choices about training sets and to flag potential underrepresented populations who may be overlooked or treated unfairly" (p. 599), helping authors of data statements reduce the risk of technical and preexisting biases. a data statement serves as guiding documentation for the case study approach we propose in our bias-aware methodology ( §5.2), documenting the specific context in which nlp researchers work. our bias-aware methodology guides research activities before, during, and after the writing of a data statement: for researchers reading data statements to find a dataset for an nlp system, our methodology guides their evaluation of a dataset's suitability for research; for researchers writing data statements, our methodology guides their documentation of the data collection process. in addition to technological disciplines, our methodology draws on critical discourse analysis @xcite , participatory action research @xcite , intersectionality @xcite , feminism @xcite @xcite , and design @xcite . participatory action research provides a way for nlp researchers to diversify perspectives in their research, engaging with the social context that influences and is affected by nlp systems. intersectionality reminds researchers of the multitude of experiences of privilege and oppression that bias causes, because no single identity characteristic determines whether a person is "dominant" (favored) or "minoritized" (harmed) (d' @xcite . the case study approach common to design methods enables a researcher to make progress on addressing bias through explicitly situating research in a specific time and place, and conducting user research with people to understand their power relations in that time and place. feminist theory values perspectives at the margins, encouraging researchers to engage with people who are excluded from the dominant group in a social context. feminist theorist @xcite writes, "in order to gain a causal critical view of the interests and values that constitute the dominant conceptual projects...one must start from the lives excluded as origins of their design -from 'marginal' lives" (p. 341). our bias-aware research methodology includes collaboration with people at the margins of nlp research in an effort to empower minoritized people. ## a bias-aware methodology our bias-aware methodology has three main activities: examining power relations ( §5.1), explaining the bias of focus ( §5.2), and applying nlp methods ( §5.3). though we discuss the activities individually, we recommend researchers execute them in parallel because each activity informs the others. we aim for the methodology to include activities that researchers may adapt to their own research context, be their focus on algorithm development, adaptation, or application; or on dataset creation. we hope for this paper to begin a dialogue on tailoring a bias-aware methodology to different types of nlp research. ## case study in this section we describe how we are implementing the bias-aware nlp research methodology in a case study on bias in metadata descriptions from the online archival catalog of the centre for research collections at the university of edinburgh ("the archive"). 15 for consistency with the outline of a bias-aware methodology ( §5), we group our case study into the same three activities, explaining our examination of power relations ( §6.1), our bias of focus ( §6.2), and then our application of nlp methods ( §6.3). each subsection includes accomplished, ongoing, and planned future work. to demonstrate how we execute the three activities in parallel, as proposed in §5, we first provide a chronological overview. initially, our research began with information gathering linked to a participatory action research (par) methodology. we reviewed literature on bias in nlp and archives, and on digital humanities research (collaborations between technologists and humanists that often analyze data sources with historical language). we also met with employees at the archive to better understand the archive's policies, which guide the writing of metadata descriptions and documentation practices, such as the metadata standards used. the employees described how they are proactively challenging the inherited metadata and inherited practices of the archive, which date back to the 16 th century. after the literature review and meeting we began writing data statements for the archive's metadata descriptions and for our research. due to the limited research on nlp methods applied to archival metadata, and limited large-scale analysis of metadata descriptions, we undertook a pilot data project, 16 walking through the process of extracting metadata descriptions from a single archival collection, adding historical context to our documentation of the extracted descriptions, and calculating corpus analytics (using elementtree 17 and nltk 18 in a jupyter notebook 19 ). after establishing a workflow to extract metadata descriptions from the archive's online catalog, we again met employees at the archive to discuss the challenges that biased language poses to their work and to their visitors. this meeting helped us add to our data statements, identify stakeholders in our research, and begin describing the stakeholders' power relations. moreover, the meeting confirmed the value of an nlp system that detects and classifies bias, as the archive does not currently have a systematic approach to measuring bias in its catalog's metadata descriptions. ## conclusion in this paper we propose a bias-aware methodology for nlp research to mitigate harms from biased nlp systems. the methodology integrates practices and methods from nlp, ml, data science, gender and feminist studies, linguistics, and design. due to the numerous types of bias, the intersectional nature of oppression, and the possibility of direct and indirect harms from bias, detecting and measuring bias is a complex process. our methodology encourages nlp researchers to situate their work in case studies, explicitly describing the context of and stakeholders in their research. we advise nlp researchers to build the time and resources needed to undertake such work into project plans, and to put eliminating bias at the center of their research. documenting instances of bias and their associated power relations will enable the nlp community to look for patterns across different contexts that use nlp systems. amassing case studies in order to look for such patterns will guide nlp research towards generalizable approaches to bias mitigation, approaches that do not unintentionally minoritize people whose perspectives were unknowingly excluded.