Mind the data gap(s): Investigating power in speech and language datasets (2022)

Abstract: Algorithmic oppression is an urgent and persistent problem in speech and language technologies. Considering power relations embedded in datasets before compiling or using them to train or test speech and language technologies is essential to designing less harmful, more just technologies. This paper presents a reflective exercise to recognise and challenge gaps and the power relations they reveal in speech and language datasets by applying principles of Data Feminism and Design Justice, and building on work on dataset documentation and sociolinguistics.

## introduction algorithmic systems disproportionately harm marginalised communities by reproducing existing structures of oppression within a society in a process called algorithmic oppression @xcite . these harms occur in all contexts where ai is applied to people, including speech and language technologies (slts) @xcite . understanding power relations in the datasets used to train and test slts is essential to designing fundamentally more just and less harmful technologies. in this paper, i suggest reflecting on the gaps in the content and documentation of language datasets as a way to guide data compilation @xcite and the re-use of existing datasets in appropriate contexts @xcite . the aim of this paper is to contribute to a (long overdue) conversation about power, representation and bias in slts (see e.g., @xcite @xcite . it is grounded in the understanding that (language) technologies are political tools which cannot be "neutral". unless they are explicitly designed to benefit marginalised communities, they will (re)produce existing structures of oppression and cause harm @xcite @xcite . one way of approaching algorithmic oppression has been to carefully document the datasets used to train and test machine learning systems. @xcite provide a highly influential documentation framework which can be applied to all ai datasets and @xcite introduce an approach to documentation specific to datasets for natural language processing, which i draw on here. this transparency can help to anticipate "predictive bias", a systematic difference in error rates for different groups @xcite , which is one (but not the only) outcome of algorithmic oppression. detailed documentation is absolutely crucial to not just equitable, but fundamentally useful slts because it allows practitioners to choose appropriate datasets for a particular task. by definition, documentation is interested in what is included in a dataset. to highlight power inequities, it's also useful to think about what is missing from a dataset. in slts, the exclusion of particular ways of using language (accents, dialects, etc.) can lead to the exclusion of communities. this paper is an invitation to reflect on why these "data gaps" exist, who is harmed by them and how this harm could be prevented. the questions i propose here are not exhaustive or definitive, and addressing them may be difficult in many cases. the point is not to create the "perfect" dataset but to highlight that all (language) datasets involve power relations. in the context of limiting harm and challenging power, thinking carefully about the appropriateness of any (language) technology in a particular context is fundamental foot_0 . in some cases, the most effective way to challenge power is to refuse to build the technology or compile the dataset @xcite . just as technologies are not "neutral", they are also not inevitable. a technological "fix" to a structural social problem will often fall short @xcite . moreover, entirely "unbiased" (in the narrow sense of predictive bias) and "inclusive" language technologies can be at least equally harmful to marginalised communities, as "inclusion" can expose communities to further marginalisation and violence @xcite . for example, automatic speech recognition systems are used in us prisons to monitor phone calls between incarcerated people and their friends, families and legal support @xcite . in this context, "better" or "more accurate" speech recognition based on "more diverse" or "inclusive" speech datasets may make it easier for authorities to harm incarcerated people and their communities. inclusion in datasets owned by technology corporations or public or governmental institutions can further mean that the "data", i.e. voices of these communities, is no longer owned by or even accessible to them. as a first step in any slt data compilation process it is therefore crucial to consider and ideally directly involve the affected language communities to understand their own needs and desires with respect to language technology, and to avoid perpetuating a long history of colonial approaches to data and language in which communities, especially in the global south, are exploited by academic institutions, (neo)colonial states and multinational corporations @xcite @xcite . in contexts where we do choose to use or compile a dataset, we need to be aware of how power operates within it. the goal is not just to identify or mitigate biases once a system is ready for deployment, to for example, "retrofit against racism" @xcite . instead, similarly to @xcite , i argue that these questions should guide the (dataset) design process. although it may be too late to change the way the data was compiled when reusing a dataset @xcite , it is still useful to critically reflect on the contents and context of the dataset, to ensure it is appropriate. since it's impossible to evaluate potential or actual harms of data gaps in isolation, this should be done with a particular deployment context in mind. i consider two examples, not to prove that datasets contain imbalances, but to illustrate the framework: mozilla's common voice english (release 7.0) @xcite and the linguistic data consortium's switchboard-2 @xcite used to train and test automatic speech recognition (asr) systems. i chose these datasets because they were compiled in quite different ways, by different types of institutions, for different purposes and contain different data gaps as a result: commonvoice is a crowd-sourced speech dataset compiled by mozilla with the explicit aim to create "diverse" speech datasets for asr development, while switchboard-2 is a collection of telephone conversations collected by the linguistic data consortium, an academic institution, to develop speaker recognition systems. i use the term "power" to refer to the structural position a particular social group occupies in relations to others. because these social hierarchies as well as relevant categories or groups within them are socially constructed, they vary depending on the cultural and historical context (see e.g., saini, 2019, on race). over the past century, constructs of race, gender and sexuality, (dis)ability, class, age and nationality have been used in a global and many local contexts to secure and uphold the dominant position of white people, in particular those who are cisgender, heterosexual, able-bodied, wealthy, men, and/or from the global north. hill collins @xcite , 227) introduces the concept of the matrix of domination to describe "the overall social organization within which intersecting oppressions originate, develop, and are contained". it encompasses social, cultural and legal institutions which uphold the dominant position of some groups, while marginalising others, for example through laws and policies (or their enforcement and application), as well as cultural discourses and ideologies and everyday social interaction @xcite , pp 282). by "intersecting oppressions", hill @xcite ) refers to fact that these categories are not separate or separable, but rather produced by interlocking systems of oppression such as white supremacy and patriarchy (see also "intersectionality" as coined by @xcite ). 2 this complex understanding of power also accounts for the fact that groups who are marginalised by one of those systems, can be privileged by another system and hold power, for example white women (see @xcite ). this paper draws on a feminist perspective on data and power, in particular as articulated by d' @xcite . feminism is not an unproblematic framing. many feminists and feminisms (past and present) exclude, ignore and/or harm marginalised people of all genders, in particular people of colour, black people and trans* and non-binary people @xcite @xcite . in academia and other (neoliberal) institutions the concept of intersectionality is further frequently co-opted and misrepresented in a, ahistorical, "depoliticised" and often explicitly deracialised fashion @xcite . the invocation of and commitment to "ornamental intersectionality", and notions of "equality", "diversity" and "inclusion" can further serve to symbolically address structural inequalities without in any way redressing them @xcite . mindful of both this misuse of radical frameworks to which praxis is central, and the genuine harm that has been perpetrated under the guise of "feminism", i understand "feminist work [as] justice work" @xcite which seeks to challenge all systems of oppression. it is a way of making sense of the world(s) we live in and of organising (for) world(s) we can and want to flourish in. as such, it is for everyone and (potentially) by everyone who wants to understand and challenge existing power structures. i build directly on d'ignazio and klein's seven principles of "data feminism": "examine power", "challenge power", "elevate emotion and embodiment", "rethink binaries and hierarchies", "embrace pluralism", "consider context" and "make labor visibile" (d' @xcite . i am also drawing on "design justice" as a way of understanding how (technology) design reproduces structural oppression and an approach to reimagining those design processes (see costanza-chock, 2020, 23) foot_2 . the principles of design justice focus on using design to empower communities, centering the voices of those who are impacted by (tech-nology) design and working towards sustainable and community-controlled designs. ## power in language datasets what does it mean to "challenge power" when compiling or using datasets then? d' @xcite showcase projects which compile "counterdata" filling (deliberate) gaps. for example a 1971 map compiled by the detroit geographic expedition and institute to highlight the disproportionate rate at which black children were killed by white drivers (d' @xcite . another way of challenging power using data is to analyse the way oppression is manifested in data, but importantly (data) feminism also encourages us to go beyond critiques of the world as it currently is to imagining the world as it ought to be. as noted above, sometimes the way to challenge power is refusal: refusal to compile data, refusal to share data or refusal to (re)use data @xcite . however, when we choose to engage with data(sets), we can challenge power by investigating and highlighting power relations. while this is unlikely to prevent all harm, it allows use to act more carefully and hopefully reduce harm. i outline three steps in reflecting on power relations reproduced in slt datasets to guide the compilation or selection of a dataset. the first is to identify gaps in data and documentation and their consequences to analyse power relations. the second involves asking why those gaps exist (and persist) given the broader context. the final step is about imagining alternative ways of compiling and using the dataset to create more just, less harmful technologies.