Studying and Mitigating Biases in Sign Language Understanding Models (2024)

Abstract: Ensuring that the benefits of sign language technologies are distributed equitably among all community members is crucial. Thus, it is important to address potential biases and inequities that may arise from the design or use of these resources. Crowd-sourced sign language datasets, such as the ASL Citizen dataset, are great resources for improving accessibility and preserving linguistic diversity, but they must be used thoughtfully to avoid reinforcing existing biases.In this work, we utilize the rich information about participant demographics and lexical features present in the ASL Citizen dataset to study and document the biases that may result from models trained on crowd-sourced sign datasets. Further, we apply several bias mitigation techniques during model training, and find that these techniques reduce performance disparities without decreasing accuracy. With the publication of this work, we release the demographic information about the participants in the ASL Citizen dataset to encourage future bias mitigation work in this space.

## introduction within the field of natural language processing, sign languages are under-resourced compared to spoken languages, compounded by the fact that most accessible information (e.g. online resources and social media) is written in a spoken language @xcite . datasets like the asl citizen dataset offer significant potential for improving accessibility and preserving the linguistic richness of sign languages, yet their use requires careful consideration to avoid reinforcing existing biases. in this context, our research aims to explore the factors that might influence the performance of models trained on these datasets, particularly when used for dictionary retrieval tasks. because sign languages have comparatively fewer resources than spoken languages, identify-figure 1 : accuracy and gender parity (calculated by dividing accuracy on female participants by accuracy on male participants) of the baseline pose-based islr model released with the asl citizen dataset (left) and our best-performing feature-based debiasing technique (right), in which we resample videos with lower video quality scores at a higher rate. our approach improves both overall model accuracy and the gender parity. in this work, we analyze how signer demographics and more latent sources of bias may impact models trained on the asl citizen dataset for the task of isolated sign language recognition (islr). we first examine the demographic distributions in the asl citizen dataset, and present a linguistic analysis of the dataset based on the asl-lex @xcite annotations for each sign. we then report the prevalence of various linguistic and video-level features among demographics. we examine how demographic features, in conjunction with lexical and video-level features, may impact model results. finally, we experiment with multiple debiasing techniques to reduce performance gaps between genders, and find that we are able to reduce these gaps and improve overall model accuracy (figure 1 ). in summary, we present an analysis of demographics, sign-level features, and video-level features in the asl citizen dataset and address the following research questions: ## related work most readily-available information (i.e. online resources and social media) is written, which may limit accessibility for signers. sign language processing tasks, such as dictionary retrieval, are designed to improve the accessibility of existing systems and resources for deaf and hard-of-hearing (dhh) people. @xcite created the asl citizen dataset to supplement existing dictionary retrieval resources with crowd-sourced videos from signers. the asl citizen dataset was released to 1) address the resource gap between sign and spoken languages, and 2) improve video-based dictionary retrieval for sign language, where signers demonstrate a particular sign and the system returns a list of similar signs, ranked from most to least similar. video-based dictionary retrieval systems can help language learners understand the meaning of a sign, and allow signers to access dictionary resources using sign languages @xcite . as a crowd-sourced dataset with videos of individual signs, the asl citizen dataset also serves to improve documentation of sign languages. this dataset is the first crowd-sourced dataset of videos for isolated signs, and members of deaf communities participated in, and were compensated for, this effort. when supplemented with the sem-lex benchmark @xcite , a crowdsourced islr dataset released shortly after, 174k videos in total can be used for islr. the asl citizen dataset is licensed by microsoft research and is bound by the microsoft research licensing terms foot_2 . the asl citizen dataset is composed of videos of individual signs for isolated sign language recognition (islr). other islr datasets with videos of individual signs have been released, including wl-asl @xcite , purdue rvl-sll @xcite , @xcite , and rwth boston-50 @xcite . the above datasets, however, are not crowd-sourced. the closest dataset to the asl citizen dataset is the sem-lex benchmark @xcite , a crowdsourced islr dataset with over 91k videos. because the sem-lex benchmark does not release demographic information about the participants, we are not able to include it in our bias studies. the asl citizen dataset is made up of crowdsourced videos from asl signers, where each video corresponds to a particular sign. the corpus is composed of videos for 2731 unique signs, all of which are contained in the asl-lex dataset @xcite , a lexical database of signs with annotations including the relative frequency, iconicity, grammatical class, english translations, and phonological properties of the sign. thus, researchers studying this dataset can also take advantage of the asl-lex annotations. as part of the original data collection effort, demographic information about each participant was collected, but it was not released. with the publication of this work, we release the demographic data in this set, and provide a detailed analysis of this data. our analyses of demographics and bias are motivated by evidence in the literature that a signer's demographics may impact their signing. for instance, characteristics of particular spoken languages or dialects have been shown to influence gestures, and in turn sign production @xcite . one example of an asl dialect is black asl, which scholarly evidence has shown to be its own dialect @xcite , and for which documentation of dialectical differences dates back to @xcite . whether an individual speaks black asl is likely heavily influenced by their race or ethnicity. an example of geographic differences is martha's vineyard, an island off the coast of the united states, where an entire sign language emerged due to the high prevalence of dhh individuals in this community. hearing and dhh people alike used this language to communicate until the mid-1900s @xcite . there is also a distinct canadian asl dialect used by signers in english-speaking areas of canada @xcite , which is documented in a dictionary @xcite . age of language acquisition also impacts asl production; delayed first-language acquisition affects syntactic knowledge for asl signers @xcite and late acquisition (compared to native acquisition) was found to impact sensitivity to verb agreement @xcite . previous work also indicates the impact of certain visual and linguistic features on sign language modeling. training an islr model to predict a sign and its phonological characteristics was found to improve model performance by almost 9% @xcite . @xcite find improved performance when using attention to focus on hand movements in sign videos. to our knowledge, there are no existing works that extensively study various sources of model bias on a crowd-sourced dataset of sign videos with labeled participant demographics. with this work, we aim to address this gap with a systematic analysis of the impact of various participant-level, signlevel, and video-level features, and experiment with debiasing techniques to reduce disparities in model performance. ## data the asl citizen dataset is a crowd-sourced dataset containing 83,399 videos of individual signs in asl from 52 different participants. the dataset contains 2731 unique signs that are included in the asl- @xcite dataset, a dataset with detailed lexical annotations for each sign. the authors of the original work report some demographic statistics, but the demographics of individual (de-identified) participants have not been released. here, we provide a detailed report that includes demographic breakdowns and analyses of various linguistic and video features in the dataset, including the breakdown of these features by gen-der. we release the participant demographics with this work. ## methods here, we describe the baselines for our islr experiments, along with the experimental settings we use. ## conclusion in this work, we address a gap in sign language processing research by exploring biases in sign language resources, and experimenting with strategies to mitigate these biases. we focus on the asl citizen dataset in particular, and release demographic information for this dataset to aid future work. we find performance gaps related to skin tone, participant age, and gender. still, we find that video level features, such as the video quality, signing "speed", and video length, appear to be the best predictors of model accuracy. we find that selectively resampling data with video lengths closer to the mean improves overall performance. we also find that doing this resampling strategy for only the group with lower model performance (female, when comparing genders) improves the gender parity for model performance. we find that resampling lower-quality videos at a higher rate achieves the highest top-1 accuracy and gender parity.