Right the docs: Characterising voice dataset documentation practices used in machine learning (2023)

Abstract: Voice-enabled technologies such as virtual assistants are quickly becoming ubiquitous. Their functionality relies on machine learning (ML) models that perform tasks such as automatic speech recognition (ASR). These models, in general, currently perform less accurately for some cohorts of speakers, across axes such as age, gender and accent; they are biased. ML models are trained from large datasets. ML Practitioners (MLPs) are interested in addressing bias across the ML lifecycle, and they often use dataset documentation here to understand dataset characteristics. However, there is a lack of research centred on voice dataset documentation. Our work makes an empirical contribution to this gap, identifying shortcomings in voice dataset documents (VDD), and arguing for actions to improve them. First, we undertake 13 interviews with MLPs who work with voice data, exploring how they use VDDs. We focus here on MLP roles and trade-offs made when working with VDDs. Drawing from the literature and from interview data, we create a rubric through which to analyse VDDs for nine voice datasets. Triangulating the two methods in our findings, we show that VDDs are inadequate for the needs of MLPs on several fronts. VDDs currently codify voice data characteristics in fragmented ways that make it difficult to compare and combine datasets, presenting a barrier to MLPs’ bias reduction efforts. We then seek to address these shortcomings and “right the docs” by proposing improvement actions aligned to our findings.

## introduction, motivation and previous work voice-enabled technologies, such as virtual assistants and smart speakers, are "going to scale" through axes such as volume @xcite @xcite , geographies @xcite @xcite , miniaturisation @xcite , expanding use cases @xcite @xcite and use in multiple modalities @xcite . speech technology has become part of the fabric of modern information infrastructures -the technical capabilities, social norms, organisational practices and economic mechanisms @xcite that collectively allow us to speak with machines and have them do our bidding. as voice technology becomes ubiquitous, so too does the potential societal impact of its bias. a person's poor voice interaction experience is no longer confined to a virtual assistant in the home, or to a mobile phone, but extends to the workplace, the car, healthcare, and customer service settings. these systems use machine learning (ml)enabled components like automatic speech recognition (asr). however, they don't yet work well for everyone @xcite @xcite . they exhibit bias -defined here as systematic and unfair discrimination against individuals or cohorts of individuals in favour of others @xcite foot_0 -across axes such as age @xcite , gender @xcite @xcite , race @xcite , nationality @xcite , and accent @xcite . dataset documentation is a frequent tool used by mlps to mitigate bias. ## methodology we devise an exploratory study that combines two methods, one focusing on ml practitioners and their experiences creating or consuming voice dataset documentation and the other on dataset documentation artefacts. firstly, we undertake 13 semi-structured interviews with mlps who work with voice or closely adjacent data. we explore their voice dataset document (vdd) approaches across the ml lifecycle. secondly, we turn our attention to existing vdds. vdds represent how mlps generate datasets and release them to the world -they encode practices, beliefs and assumptions @xcite . we select nine vdds for their varied purposes, collection methods and source data. drawing both from our literature review in section 1.1 and from participant data, we develop a rubric for analysis, and assess the vdd artefacts across seven categories. we then triangulate the two methods, showing how vdd practices differ by mlp role, and how vdds may help or hinder mlps in making trade-off decisions. ## findings here, we triangulate our two methods. we characterise the experience of mlps with vdds through the frames of mlp roles and trade-offs the mlp makes, quoting from interview transcripts to highlight key points. at the same time, we corroborate the interview findings by referencing results from the document analysis. this layered approach provides a richer characterisation of vdds. ## righting the docs: towards vdd that help mlps mitigate bias in speech technologies drawing from the gaps in vdd practice uncovered from our exploratory study above, we now propose a program of work to begin to address them. ## limitations additional methods to triangulate findings we recognise the small, although purposive, sample of participants and datasets in our exploratory study. we now intend to administer a questionnaire to a broader group of mlps, to validate or invalidate these initial findings. only publicly knowable datasets were analysed in identifying and selecting datasets for analysis, we recognise that our approach was limited to only publicly knowable datasets; private and/or proprietary datasets used internally by organisations may exhibit very different dataset documentation practices, although this is unlikely based on the work of @xcite and @xcite . ## conclusion here, we have situated voice dataset documentation (vdd) practices conducted by machine learning practitioners (mlps) within broader efforts to reduce bias in ml-enabled speech technologies as they go to scale. we first provided a brief literature review of ml-related dataset documentation work, identifying that vdd practices are understudied. we presented an exploratory study that combined two methods -semi-structured interviews and document analysis -to provide a rich characterisation of practices surrounding vdds. we find that vdds are currently inadequate to meet the needs of mlps who create and consume voice datasets. in particular, they often fail to describe voice dataset contents accurately, if at all, and the range of representation formats used makes it difficult for mlps to combine datasets effectively -as is often required in bias reduction efforts. drawing from these findings, we propose actions that seek to "right the docs", focusing on unified formats for dataset documentation, as well as the need for common taxonomies for data items common to voice datasets.