O pen H ands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages (2022)

Abstract: AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sentences. We introduce OpenHands, a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition. First, we propose using pose extracted through pretrained models as the standard modality of data in this work to reduce training time and enable efficient inference, and we release standardized pose datasets for different existing sign language datasets. Second, we train and release checkpoints of 4 pose-based isolated sign language recognition models across 6 languages (American, Argentinian, Chinese, Greek, Indian, and Turkish), providing baselines and ready checkpoints for deployment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unlabelled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare different pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslingual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages reproducible and more accessible.

## introduction according to the world federation of the deaf, there are approximately 72 million deaf people worldwide. more than 80% of them live in developing countries. collectively, they use more than 300 different sign languages varying across different nations (un, 2021). loss of hearing severely limits the ability of the deaf to communicate and thereby adversely impacts their quality of life. in the current increasingly digital world, systems to ease digital communication between deaf and hearing people are important accessibility aids. ai has a crucial role to play in enabling this accessibility with automated tools for sign language recognition (slr). specifically, transcription of sign language as complete sentences is referred to as continuous sign language recognition (cslr), while recognition of individual signs is referred to as isolated sign language recognition (islr). there have been various efforts to build datasets and models for islr and clsr tasks @xcite . but these results are often concentrated on a few sign languages (such as the american sign language) and are reported across different research communities with few standardized baselines. when compared against text-and speech-based nlp research, the progress in ai research for sign languages is significantly lagging. this lag has been recently brought to notice of the wider nlp community @xcite . for most sign languages across the world, the amount of labelled data is very low and hence they can be considered low-resource languages. in the nlp literature, many successful templates have been proposed for such low-resource languages. in this work, we adopt and combine many of these ideas from nlp to sign language research. we implement these ideas and release several datasets and models in an open-source library openhands with the following key contributions: 1. standardizing on pose as the modality: we consider using pose-extractor as an encoder, which processes raw rgb videos and extracts the framewise coordinates for few keypoints. pose-extractors are useful across sign languages and also other tasks such as action recognition @xcite , and can be trained to high accuracy. further, as we report, pose as a modality makes both training and inference for slr tasks efficient. we release pose-based versions of existing datasets for 5 sign languages: american, argentinian, greek, indian, and turkish. through these datasets, models, and experiments we make several observations. first, in comparing standardized models across different sign languages, we find that graph-based models working on pose modality define state-of-the-art results on most sign languages. rnn-based models lag on accuracy but are significantly faster and thus appropriate for constrained devices. second, we establish that self-supervised pretraining helps as it improves on equivalent models trained from scratch on labelled islr data. the performance gap is particularly high if the labelled data contains fewer samples per label, i.e., for the many sign languages which have limited resources the value of selfsupervised pretraining is particularly high. third, we establish that self-supervision in one sign language (indian sl) can be crosslingually transferred to improve slr on other sign languages (american, chinese, and argentinian). this is particularly encouraging for the long tail of over 300 sign languages that are used across the globe. fourth, we establish that for real-time applications, posebased modality is preferable over other modalities such as rgb, use of depth sensors, etc. due to reduced infrastructure requirements (only camera), and higher efficiency in self-supervised pretraining, fine-tuning on islr, and inference. we believe such standardization can help accelerate dataset collection and model benchmarking. fifth, we observe that the trained checkpoints of the pose-based models can be directly integrated with pose estimation models to create a pipeline that can provide real-time inference even on cpus. such a pipeline can enable the deployment of these models in realtime video conferencing tools, perhaps even on smartphones. as mentioned all datasets and models are released with permissible licenses in openhands with the intention to make slr research more accessible and standardized. we hope that others contribute datasets and models to the library, especially representing the diversity of sign languages used across the globe. the rest of the paper is organized as follows. in section 2 we present a brief overview of the existing work. in section 3 we describe our efforts in standardizing datasets and models across six different sign languages. in section 4 we explain our pretraining corpus and strategies for self-supervised learning and detail results that establish its effectiveness. in section 5 we describe in brief the functionalities of the openhands library. in section 6, we summarize our work and also list potential follow-up work. ## background and related work significant progress has been made in isolated sign language recognition (islr) due to the release of datasets @xcite @xcite and recent deep learning architectures @xcite . this section reviews this work, with a focus on pose-based models. ## standardized pose-based islr models across sign languages in this section, we describe our efforts to curate standardized pose-based datasets across multiple sign languages and benchmark multiple islr models on them. ## self-supervised learning for islr in this section, we describe our efforts in building the largest corpus for self-supervised pretraining and our experiments in different pretraining strategies. large text corpora such as bookcorpus, wikipedia dumps, oscar, etc. have enabled pretraining of large language models. although there are large amounts of raw sign language videos available on the internet, no existing work has studied how such large volumes of open unlabelled data can be collected and used for slr tasks. to address this, we create a corpus of indian sl data by curating videos, pre-process the videos, and release a standardized pose-based dataset compatible with the models discussed in the previous section. we manually search for freely available major sources of indian sl videos. we restrict our search to a single sign language so as to study the effect of pretraining on same language and crosslingual islr tasks. we sort the sources by the number of hours of videos and choose the top 5 sources for download. all of these 5 sources, as listed in video sources youtube channel 1 youtube channel 2 youtube channel 3 scheduler crawl process1 crawl process2 crawl process3 storage for downloaded videos stage-1: data crawling cpu 1 cpu 2 cpu 3 manual curation of list stage-2: pose extraction scheduler split all videos in chunks of 5mins (9k frames) cpu-1 cpu-2 cpu-3 mediapipe pose estimator thread-1 mediapipe pose estimator thread-2 mediapipe pose estimator thread-3 storage for generated pose data stage-3: data cleaning drop noise and resplit videos for video regions with: 1. no signing activity 2. no signers 3. more than one signer consolidate all chunked clips from all channels hdf5 storage hierarchical channel-wise grouping compressed cloud storage available via library for download and usage pre-training data-loader faster random access of pose sub-regions trainer pytorch ecosystem cpu gpu stage-4: efficient training we pass these videos through a processing pipeline as described in figure 2 . we initially dump the pose data for all videos, then process them to remove those which are noisy or contain either no person or more than 1 person. this resulted in 1,129 hours of indian sl data, as detailed source-wise in ## the openhands library as mentioned in the main paper, we open-source all our contributions through the openhands library. this includes the pose-based datasets for 5 sls, 4 islr models trained on 7 datasets, the pretraining corpus on indian sl with over 1,100 hours of pose data, pretrained models on this corpus using self-supervised learning, and models fine-tuned for 4 different sls on top of the pretrained model. we also provide scripts for efficient deployment using mediapipe pose estimation and our trained islr models. in addition, the library provides utilities that are helpful specifically for pose-based data. this includes methods to normalize keypoints by width and height of frames, to normalize all of the pose data to be in the same scale and reference coordinate system by using a constant feature of body as reference, and to fill missing keypoints. the library also includes utilities to create data augmentations such as sheartransform to displace the joints in a random direction, rotatationtransform to simulate the viewpoint changes of the camera, scaletransform to simulate different scales of the pose data to account for relative zoomed-in or zoomed-out view of signers, poserandomshift to move a significant portion of the video by a time offset so as to make the islr models robust to inaccurate segmentation of real-time video, uniformtemporalsubsample to uniformly sample frames from the video instead of considering only the initial frames, in cases where the number of frames in a video clip exceeds a maximum limit, and randomtemporalsubsample to sample a random fixed contiguous window of required size covering a maximum number of frames. we encourage researchers to contribute datasets, models, and other utilities to make sign language research more accessible. all the aspects of the toolkit are well-documented online foot_1 for anyone to get started easily. ## conclusion in this work, we make several contributions to make sign language research more accessible. we release pose-based datasets and 4 different islr models across 6 sign languages. this evaluation enabled us to identify graph-based methods such as st-gcn as being accurate and efficient. we release the first large corpus of sl data for selfsupervised pretraining. we evaluated different pretraining strategies and found dpc as being effective. we also show that pretraining is effective both for in-language and crosslingual transfer. all our models, datasets, training and deployment scripts are open-sourced in openhands. several directions for future work emerge such as evaluating alternative graph-based models, experimenting with varying sequence lengths of input data, efficiently sampling the data from the raw dataset for pretraining such that the samples are diverse enough, using better pose estimator models and more keypoints, and quantized inference for 2×-4× reduced latency. on the library front, we aim to release updated versions incorporating more sl datasets, better graph-based models, studying the performance on low fps videos (like 2-4 fps), effect of pretraining using other high-resource sl datasets, extending to cslr, and improving deployment features. note that encoder (pose estimation) and decoder (classifier) are parallelized such that the former is a producer of skeletons for window of live frames, and the latter is a consumer which recognizes glosses.