CISLR : Corpus for I ndian S ign L anguage Recognition (2022)

Abstract: Indian Sign Language, though used by a diverse community, still lacks well-annotated resources for developing systems that would enable sign language processing. In recent years researchers have actively worked for sign languages like American Sign Languages, however, Indian Sign language is still far from data-driven tasks like machine translation. To address this gap, in this paper, we introduce a new dataset CISLR (Corpus for Indian Sign Language Recognition) for word-level recognition in Indian Sign Language using videos. The corpus has a large vocabulary of around 4700 words covering different topics and domains. Further, we propose a baseline model for word recognition from sign language videos. To handle the low resource problem in the Indian Sign Language, the proposed model consists of a prototype-based one-shot learner that leverages resource rich American Sign Language to learn generalized features for improving predictions in Indian Sign Language. Our experiments show that gesture features learned in another sign language can help perform one-shot predictions in CISLR.

## introduction existing works in natural language processing have shown promising improvements in text classification, translation as well as generation in widely used spoken languages. however, sign language, on the other hand, still lacks sufficient resources for developing models using data-driven approaches, leading to low improvements in tasks like translation and generation. one such low-resource sign language includes indian sign language (isl). as per the 2011 indian census, there are about 6 million deaf people in india @xcite . according to ethnologue (a reference publication documenting information about living languages of the world), isl is the most widely used sign language in the world, and it is 151 st most "spoken" language in the world @xcite . however, there is a huge deficit of sign language interpreters e.g., according to the government of india organization indian sign language research and training center (islrtc), there are only 300 certified sign language interpreters in india ( http://islrtc.nic.in/ ). this has resulted not only in the widening of the communication barrier between the deaf community and the rest of the population but also has resulted in the very limited development of educational material for sign languages. on the technology side, there are lack of standard benchmarks for isl resulting in low development and a lack of comparison of machine learning based solutions for isl, e.g., word recognition, translation, etc. in contrast, relatively speaking, other sign languages (e.g., american sign language (asl), deutsche gebärdensprache (dgs)) have a sufficient number of annotated resources for data-driven approaches @xcite @xcite , and recent multimodal approaches have shown significant improvements in terms of applications @xcite @xcite . with an increase in the number of deaf people in india ( > 6 million), it is important to develop technologies that could aid in processing isl and narrow the communication gap between the deaf community and the rest of the population. this paper is aimed towards this goal, and via this paper, we draw the attention of the nlp research community to developing technologies for highly lowresource isl. in this work, we introduce corpus for indian sign language recognition (cislr). the corpus consists of word-level videos of trained isl signers. the corpus consists of 7050 videos covering 4765 words. along with the corpus, we introduce the task of word-level indian sign language recognition. the task is challenging due to the limited number of videos for each word and the variability in the signs for the same word by different signers. for example, as shown in fig. 1 , the same signer represents the word "buddhist" with different hand gestures. another example is the word "command" where the two signers explain the word; however, they use slightly different gestures (app. fig. 3 ). given that we have limited data for training models for word recognition in isl, we propose using relatively high-resource languages like asl (american sign language) to transfer knowledge to isl. recently, wlasl (word level american sign language) @xcite , a largescale word-level sign language dataset, has been released. we try to leverage the wlasl benchmark to extract gesture features. our experiments show that gesture features learned in another sign language can help perform one-shot predictions in indian sign language. overall, we make the following contributions in this work: there has been active interest in the research community in developing tools and techniques for pro-cessing sign languages. since sign languages contain both visual, gestural, and language modalities, both the vision @xcite ) and natural language @xcite research communities have developed techniques. a number of tasks for sign language processing have been proposed, for example, sign language detection @xcite , identification @xcite , segmentation @xcite , recognition (gloss detection) @xcite ; sincan and keles, 2020), translation @xcite @xcite and generation @xcite . in this paper, we focus on the task of sign language word recognition (gloss detection) from videos. a number of benchmarks have been proposed for gloss detection in sign languages other than isl @xcite @xcite . a few of the popular sign language resources include @xcite @xcite . a large number of benchmarks pose the problem of gloss detection using only rgb images. in contrast, few of the resources @xcite provide depth modality facilitating the task of gesture recognition using 3d depth maps. in contrast, there are a very few datasets for the indian sign language. some of the existing datasets include @xcite which consists of 290 static images for 26 alphabets, @xcite contains 600 videos corresponding to 22 classes, and kishore and kumar ( 2012 ) contains 800 videos for 80 different classes. moreover, the unavailability of these datasets publicly remains a problem for data-driven approaches. include dataset @xcite ) contains 263 classes from 15 different word categories in the form of 4287 videos. another resource for continuous indian sign language is isl-cslrt @xcite which captures 100 sentences in the form of 700 sign videos. recently, self-supervised pretraining has attracted attention in the sign language recognition community, @xcite released a large corpus of sign language data for self-supervised pretraining, highlighting the significance of pretraining for both in-language and cross-lingual transfer. another exciting work in american sign language includes wlasl-lex @xcite , which explores modeling the phonological aspects of sign languages. though the community has started exploring a wide variety of nlp techniques for sign languages, the unavailability of annotated data resources and benchmarks remains the primary challenge for data-driven approaches. ## cislr: corpus for indian sign language recognition the major challenge with existing sign language datasets is the small vocabulary size. limited vocabulary makes it harder to apply deep learning methods for real-world use cases. one of the central motivations for creating a new isl benchmark is to have a larger vocabulary size to aid further processing. we create cislr by scraping and curating data from two publicly available internet resources. the first source is the indian sign language research and training center (islrtc) (a government of india initiative: https://islrtc.nic.in/ ), which provides an isl dictionary with the sign language words in the form of videos. our second source is a non-profit organization indiansignlanguage ( www. indiansignlanguage.org ) which offers another collection of indian sign language (isl) signs in the form of videos. we scrape both publicly available videos from youtube ( www.youtube.com ) for both sources. each video is annotated with an english word. ## task formulation the gloss recognition tasks in cislr can be formally defined as follows. given a video of a signer (performing gestures and actions), the task is to predict the corresponding gloss label (word). as the proposed cislr dataset contains a low number of average videos per word (1.5 videos per word), we formulate the gloss recognition task as a one-shot learning task and provide a single sign video of all unique labels in the corpus as prototypes. the remaining samples in the corpus are considered test queries for evaluating gloss recognition. ## baseline model for word recognition the one-shot recognition task in the proposed cislr corpus reflects a practical setting where it is figure 2 shows the pipeline for representation learning in cislr. we use the state-of-the-art model on the wlasl dataset inception3d (i3d) and train it on the 2000 class wlasl dataset. the original i3d network @xcite is trained on imagenet @xcite and fine-tuned on kinetics-400 @xcite . we follow the same strategy as @xcite to fine-tune i3d architecture on the wlasl dataset and obtain the classification scores. we use the penultimate layer of the trained model to generate features corresponding to the prototype videos in the cislr dataset. for each test sample video, we generate the features using the same i3d network and assign the gloss corresponding to the nearest prototype. ## discussion and future directions apart from the low resource availability in indian sign language, there are other challenges. due to the vast diversity of isl users, we observed the use of slightly different gestures for the same gloss. this makes the gloss recognition task more challenging in a low-resource setting. we also observe dialectical variations in isl, i.e., the presence of various demography-specific words which are grounded in the region/cultural-specific concepts. moreover, we also observed that for a few of the words like "author" (explained as a gesture for "book" followed by a gesture for "writing"), the order of gestures is not fixed, and different signers use different order to convey the word "author." in the future, we plan to expand the corpus by including more words and releasing an updated version of the proposed cislr. as the proposed dataset is gloss/word level, it limits the application of linguistic analysis of sign language. a possible future work would be to extend the corpus, including sentence-level translations, which would facilitate pretraining on sentence-level translations for better gloss recognition in isl. moreover, on the gloss/word level, it would be interesting to explore works like wlasl-lex @xcite for the proposed cislr corpus. ## conclusion we introduce corpus for indian sign language recognition (cislr) and propose the task of sign word recognition from videos. the dataset is reflective of a practical low-data setting where it is not possible to have multiple videos for a sign. to address this we propose a transfer learning based technique to use high-resource american sign language for performing classification on isl. the results encourage the exploration of one-shot techniques further in this domain.