ASL STEM W iki: Dataset and Benchmark for Interpreting STEM Articles (2024)

Abstract: Deaf and hard-of-hearing (DHH) students face significant barriers in accessing science, technology, engineering, and mathematics (STEM) education, notably due to the scarcity of STEM resources in signed languages. To help address this, we introduce ASL STEM Wiki: a parallel corpus of 254 Wikipedia articles on STEM topics in English, interpreted into over 300 hours of American Sign Language (ASL). ASL STEM Wiki is the first continuous signing dataset focused on STEM, facilitating the development of AI resources for STEM education in ASL.We identify several use cases of ASL STEM Wiki with human-centered applications. For example, because this dataset highlights the frequent use of fingerspelling for technical concepts, which inhibits DHH students’ ability to learn,we develop models to identify fingerspelled words—which can later be used to query for appropriate ASL signs to suggest to interpreters.

## introduction american sign language (asl) is the primary and the most accessible language for many deaf children in the u.s. despite the crucial importance of accessible education in asl, deaf and hard of hearing (dhh) students often face significant barriers in accessing science, technology, engineering, and mathematics (stem) education @xcite . few general educational resources exist in asl, and even fewer asl resources exist for stem content. the scarcity of stem resources in asl compounds challenges to dhh students with limited english literacy @xcite . in particular, the lack of standardization of stem terminology in asl creates obstacles for deaf education @xcite . however, despite the existence of asl signs for stem con-1 dataset and code: https://www.microsoft. com/en-us/research/project/asl-stem-wiki figure 1 : one use case of asl stem wiki is automatic sign suggestion. given an english sentence and a video of its asl interpretation, the model detects all clips of asl that contains fingerspelling (fs). then, given the detected fs clip and the english sentence, the model identifies which english phrase in the sentence is fingerspelled in the clip. the english phrase can be used to query an asl lexicon and suggest asl signs. to help develop better asl stem resources and ai models, we introduce the asl stem wiki dataset (figure 2 ): a parallel corpus of wikipedia articles on stem topics in english and american sign language (asl). asl stem wiki consists of 254 english wikipedia articles on 5 stemrelated topics, interpreted into asl by 37 certified interpreters, resulting in 64,266 sentences and over 300 hours of asl video. it is the first continuous sign language dataset focused on stem content, which introduces new ai modeling challenges. in §3, we explain how asl stem wiki was collected and present exploratory dataset statistics. we also propose several possible use cases for this dataset informed by its unique characteristics. we present novel baseline models focused on addressing the high rate of fingerspelling in stem content. specifically, we tackle automatic sign suggestion (figure 1 ; §4): given an english sentence and a video of its asl interpretation, a model detects when the interpreter uses fingerspelling and suggests appropriate asl signs to use instead. the motivations behind this challenge are two-fold: first, as described above, this challenge encourages development of tools that can aid human interpreters in producing higher-quality asl interpretations of technical content, which in turn increases access to educational material in asl and promotes the usage of technical asl signs. second, it creates a unique opportunity to drive progress and evalu-ate ai systems on asl understanding. to train our baseline models, we use contrastive learning to leverage the large set of unlabeled asl videos, and find that contrastive learning can mitigate annotation scarcity and improve fingerspelling detection iou score by 47% ( §5). our primary contributions are: ▷ we present the first dataset of continuous stem content in asl (or any other signed language) and the largest dataset with signer consent. the dataset was recorded by certified interpreters, covers english stem wikipedia articles, and includes sentence alignment. ▷ we provide fingerspelling detection and alignment baselines as first steps towards automatic sign suggestion. we also outline several other new modeling challenges introduced by our continuous stem dataset. ▷ we provide the first evidence that selfsupervised contrastive pretraining can improve fingerspelling detection. deaf students often encounter alternative signs for the same term from different teachers or professionals, and signs generated on the fly are often phonologically or morphologically incorrect, which further hinders ease of learning and scientific communication. studies also show that deaf students report sign clarity as the top priority of teacher characteristics @xcite . to address challenges in stem communication in asl, several efforts have proposed mechanisms for signers to share and discuss asl signs online @xcite in an attempt to disseminate and standardize technical asl signs. however, this does not fully address challenges for comprehension in interpreted asl, especially in the educational setting. @xcite find that deaf students score higher on scientific knowledge with direct instruction in asl than with interpreted instruction. interpreted asl is also more prone to influence from english, such as using an english word order that would be unnatural in asl; using the asl sign for an english homonym instead of the sign that semantically matches the source word (e.g. "protein" in nutrition vs. biology), or defaulting to fingerspelling when the interpreter does not know the corresponding asl sign. since fingerspelling invokes an entirely separate language, the overreliance on fingerspelling is unlikely to help deaf students acquire a conceptual understanding of the term @xcite , and deaf students report to prefer scientific concepts to be signed, rather than fingerspelled alone @xcite . ## asl stem wiki dataset we now describe how we collected asl stem wiki. our dataset is published under a license that permits use for research purposes. we also include the datasheet of asl stem wiki in appendix a with further details. ## automatic sign suggestion as an initial step towards exploring our dataset for modeling and downstream applications, we focus on automatic sign suggestion. this task can be broken into two separate subtasks: identifying fingerspelled words, and retrieving videos of those words being signed in a dictionary. we focus on modeling the first aspect-given an english sentence and its asl interpretation, we detect all instances of fingerspelling (fingerspelling detection) and align each detected fingerspelling segment to the word in the english sentence that it spells out (fingerspelling alignment)-and leave the retrieval task to future work. ## models for fingerspelling detection and alignment we now propose baseline systems for fingerspelling detection and alignment: a neural model on asl stem wiki for fingerspelling detection and a heuristic model based on word frequency for fingerspelling alignment. ## results & discussion we report iou scores for fingerspelling detection and alignment in we find that both baseline models with and with-out pre-training have a higher iou than random. in addition, pre-training with contrastive learning improves model performance on detection. nonetheless, our baselines obtain low iou scores on both tasks, which underscores the potential of this task as a good, challenging benchmark for future models in sign language understanding. we believe that data with fingerspelling annotations may be a bottleneck of this challenge. to improve fingerspelling detection and alignment performance, future work can expand dataset size by either collecting and annotating more data, performing data augmentation, or use synthetic data. we also find evidence of contrastive learning improving modeling, therefore we also suggest exploring other unsupervised and semi-supervised objectives for pre-training. ## conclusion we present asl stem wiki: a large dataset of wikipedia articles on stem topics professionally interpreted from english to asl. the first continuous sign language stem dataset, asl stem wiki captures unique characteristics of interpreted stem content and serves as a valuable resource for developing ai tools that enhance the accessibility of stem education for deaf students. we also present baseline models working towards automatic sign suggestion, which highlight the challenge of this new task while also suggesting that contrastive learning may be a promising approach. we invite others to use asl stem wiki to further advance automatic sign suggestion and address other new challenges in human-centered nlp.