i S ign: A Benchmark for I ndian S ign L anguage Processing (2024)

Abstract: Indian Sign Language has limited resources for developing machine learning and data-driven approaches for automated language processing. Though text/audio-based language processing techniques have shown colossal research interest and tremendous improvements in the last few years, Sign Languages still need to catch up due to the need for more resources. To bridge this gap, in this work, we propose iSign : a benchmark for Indian Sign Language (ISL) Processing. We make three primary contributions to this work. First, we release one of the largest ISL-English datasets with more than video-sentence/phrase pairs. To the best of our knowledge, it is the largest sign language dataset available for ISL. Second, we propose multiple NLP-specific tasks (including SignVideo2Text, SignPose2Text, Text2Pose, Word Prediction, and Sign Semantics) and benchmark them with the baseline models for easier access to the research community. Third, we provide detailed insights into the proposed benchmarks with a few linguistic insights into the working of ISL. We streamline the evaluation of Sign Language processing, addressing the gaps in the NLP research community for Sign Languages. We release the dataset, tasks and models via the following website: https://exploration-lab.github.io/iSign/

## introduction as per the who estimate, about 63 million people belong to the deaf and hard of hearing (dhh) community in india @xcite . consequently, indian sign language (isl) is widely used in the indian subcontinent. @xcite (a reference publication documenting information about living languages of the world), isl is the world's most widely used sign language. however, there is a considerable deficit of sign language interpreters, e.g., according to the government of india organization isign benchmark ## related work recently, the research community has been actively interested in developing tools and techniques for processing sign languages. since sign languages contain both visual, gestural, and language modalities, both the vision @xcite and natural language @xcite research communities have developed techniques. several tasks for sign language processing have been proposed, for example, sign language detection @xcite , identification @xcite , segmentation @xcite , recognition (gloss detection) @xcite ; sincan and keles, 2020), generation @xcite @xcite , and translation @xcite @xcite @xcite @xcite . isolated sign language recognition/gloss recognition: many benchmarks have been proposed for gloss recognition ( §4) in sign languages other than isl @xcite @xcite @xcite @xcite . there are very few datasets for isl like @xcite ; @xcite ; kishore and kumar (2012); @xcite , include dataset @xcite , isl-cslrt dataset @xcite , cislr @xcite , and isltranslate @xcite . sign language generation: the field of sign language production predominantly revolves around the generation of hand movements, with the majority of existing approaches leveraging gan architectures and sometimes in combination with transformer architecture @xcite @xcite @xcite . ## isign benchmark dataset creation: for creating isign, we primarily use three publicly available and authentic resources on youtube: islrtc videos foot_0 (a government of indian initiative), ish news (news channel in isl), foot_1 and phrases from def (deaf enabled foundation, a non-profit working for the dhh community). 3 these youtube channels provide permissions to scrape videos and use them for research. each of these videos contains a single signer communicating information (educational content or news about current affairs) in isl and a corresponding transcript in english. the videos are pre-processed and split to obtain video-sentence ## isign tasks we propose various tasks in isign to evaluate and compare different models developed for isl processing. the tasks are described below. task 1) isl-to-english translation: it is a standard task of translating a sign (source) language to a spoken (target; in text form) language. as done in previous work, we use standard neural machine translation metrics to benchmark the baseline models on this task, including bleu, meteor, and rouge-l. input (modality) to the translation system is a video in the form of a sequence of rgb images or pose-based features. there is a significant difference in performance based on the input modality. hence, we add two subtasks under this task to facilitate the development and comparison of various approaches: islvideoto-english translation and islpose-to-english translation. the former uses image-based features as input @xcite @xcite , and the latter uses pose-based features in the form of body key points @xcite . image-based approaches have shown promising results for sign language translation tasks. however, image-based architectures are compute-heavy and require more time for inference. moreover, regarding large-scale application perspective, including images may result in signer-based biases creeping into the model learning. in contrast, extracting body pose features is fast and easy on edge devices; hence, pose-based approaches are more practical. task 2) english-to-islpose generation: the goal of this task is to transform textual input into a sequence of body poses that correspond to the sign language representation of the input sentence @xcite . hence, this task aims to generate a sign language video. the generated translations are evaluated using the dynamic time warping (dtw) metric @xcite . the dtw algorithm measures the alignment between the generated pose sequence and the ground truth, allowing us to assess how well the generated poses match the expected sign language representation. task 3) word/gloss recognition (isolated sign recognition): given a video of a signer (performing gestures and actions), the task is to predict the corresponding gloss label (word). we follow an existing work, cislr @xcite , which contains a low number of average videos per word (1.5 videos per word), and formulate the gloss recognition task as a one-shot learning task. overall, the cislr task contains 4765 sign video samples, which act as prototypes, and the task is to classify the remaining 2285 videos into one of the 4765 categories. we consider the standard metric of top-1, top-5, and top-10 classification accuracy scores to evaluate this task. ## models, experiments and results baseline models: we experimented with various models for the proposed task as described next (model training details provided in app. b). 1) isl-to-english translation: for this task, we follow camgoz et al. ( 2020 ) and validate the performance for both the sub-tasks. for islvideo-to-english translation, we use spatial embeddings extracted from pre-trained cnns as input for our model. for islpose-to-english translation, we follow @xcite and create a sequence of poses to act as input to the slt (sign language transformer) model @xcite . for pose key points, we use the mediapipe pose estimation pipeline (mediapipe, 2023). results: ## conclusion and future directions we propose isign, a benchmark for indian sign language processing, to bridge the gap between developments in spoken languages and signed languages and provide a standardization medium for accelerating the improvement. we release one of the largest available datasets for isl with 118k video-sentence/phrase pair sentences to facilitate research. we believe the released dataset will not only help improve translation models but also open up various ways for advancing natural language modeling techniques like contextualized representation learning, mask language modeling, capturing semantic similarities, etc., and encourage research in the nlp community. as a part of the benchmark, we incorporate various sign language tasks. in the future, we plan to grow the dataset by adding more samples. we also plan to add more isl-based tasks (e.g., sign sentence retrieval) to the benchmark. additionally, we provide some linguistic insights into the functioning of isl and discuss the open challenges in sign language processing. accordingly, we plan to incorporate linguistic priors into the models.