ISLT ranslate: Dataset for Translating I ndian S ign L anguage (2023)

Abstract: Sign languages are the primary means of communication for many hard-of-hearing people worldwide. Recently, to bridge the communication gap between the hard-of-hearing community and the rest of the population, several sign language translation datasets have been proposed to enable the development of statistical sign language translation systems. However, there is a dearth of sign language resources for the Indian sign language. This resource paper introduces ISLTranslate, a translation dataset for continuous Indian Sign Language (ISL) consisting of 31k ISL-English sentence/phrase pairs. To the best of our knowledge, it is the largest translation dataset for continuous Indian Sign Language. We provide a detailed analysis of the dataset. To validate the performance of existing end-to-end Sign language to spoken language translation systems, we benchmark the created dataset with a transformer-based model for ISL translation.

## introduction there are about 430 million hard-of-hearing people worldwide 1 of which 63 million are in india 2 . sign language is a primary mode of communication for the hard-of-hearing community. although natural language processing techniques have shown tremendous improvements in the last five years, primarily, due to the availability of annotated resources and large language models @xcite , languages with bodily modalities like sign languages still lack efficient language-processing systems. recently, research in sign languages has started attracting attention in the nlp community @xcite @xcite @xcite @xcite ; joshi 1 https://www.who.int/news-room/fact-sheets/ detail/deafness-and-hearing-loss 2 https://nhm.gov.in/index1.php?@xmath0. the availability of translation datasets has improved the study and development of nlp systems for sign languages like asl (american sign language) @xcite , bsl (british sign language) @xcite , and dgs (deutsche gebärdensprache) @xcite . on the other hand, there is less amount of work focused on indian sign language. the primary reason is the unavailability of large annotated datasets for indian sign language (isl). isl being a communication medium for a large, diverse population in india, still faces the deficiency of certified translators (only 300 certified sign language interpreters in india 3 ), making the gap between spoken and sign language more prominent in india. this paper aims to bridge this gap by curating a new translation dataset for indian sign language: isltranslate, having 31, 222 isl-english pairs. due to fewer certified sign language translators for isl, there is a dearth of educational material for the hard-of-hearing community. many government and non-government organizations in india have recently started bridging this gap by creating standardized educational content in isl. the created content helps build basic vocabulary for hard-of-hearing children and helps people use spoken languages to learn and teach isl to children. considering the standardized representations and simplicity in the vocabulary, we choose these con-tents for curating an isl-english translation dataset. we choose the content specific to education material that is standardized and used across india for primary-level education. consequently, the vocabulary used in the created content covers diverse topics (e.g., maths, science, english) using common daily words. isl is a low-resource language, and the presence of bodily modality for communication makes it more resource hungry from the point of view of training machine learning models. annotating sign languages at the gesture level (grouping similar gestures in different sign sentences) is challenging and not scalable. moreover, in the past, researchers have tried translating signs into gloss representation and gloss to written language translation (sign2gloss2text @xcite ). a gloss is a text label given to a signed gesture. the presence of gloss labels for sign sentences in a dataset helps translation systems to work at a granular level of sign translation. however, generating gloss representation for a signed sentence is an additional challenge for data annotation. for isltranslate, we propose the task of end-to-end isl to english translation. figure 1 shows an example of an isl sign video from isltranslate. the example shows a translation for the sentence "let's discuss", where the signer does the sign for the word "discuss" by circularly moving the hands with a frown face simultaneously followed by palm lifted upwards for conveying "let's." the continuity present in the sign video makes it more challenging when compared to the text-to-text translation task, as building a tokenized representation for the movement is a challenging problem. overall, in this resource paper, we make the following contributions: ## related work in contrast to spoken natural languages, sign languages use bodily modalities, which include hand shapes and locations, head movements (like nodding/shaking), eye gazes, finger-spelling, and facial expressions. as features from hand, eye, head, and facial expressions go in parallel, it becomes richer when compared to spoken languages, where a continuous spoken sentence can be seen as a concatenated version of the sound articulated units. moreover, translating from a continuous movement in 3 dimensions makes sign language translation more challenging and exciting from a linguistic and research perspective. sign language translation datasets: various datasets for sign language translation have been proposed in recent years @xcite . specifically for american sign language (asl), there have been some early works on creating datasets @xcite , where the datasets were collected in the studio by asking native signers to sign content. other datasets have been proposed for chinese sign language @xcite , korean sign language @xcite , swiss german sign language -deutschschweizer gebardensprache (dsgs) and flemish sign language -vlaamse gebarentaal (vgt) @xcite . in this work, we specifically target indian sign language and propose a dataset with isl videos-english translation pairs. end-to-end sign language translation systems: most of the existing approaches for sign language translation @xcite @xcite depend on intermediate gloss labels for translations. as glosses are aligned to video segments, they provide fine one-to-one mapping that facilitates supervised learning in learning effective video representations. previous work @xcite has reported a drop of about 10.0 in bleu-4 scores without gloss labels. however, considering the annotation cost of gloss-level annotations, it becomes imperative to consider gloss-free sing language translation approaches. moreover, the gloss mapping in continuous sign language might remove the grammatical aspects from the sign language. other recent works on sign language translation include voskou et al. (2021); yin and read (2020), which try to remove the requirement for a glossing sequence for training and proposes a transformer-based architecture for end-to-end translations. we also follow a gloss-free approach for isl translation. the videos cover the ncert foot_1 standardized en-glish educational content in isl. as the targeted viewers for these videos are school children and parents, the range of words covered in the videos is beginner-level. hence, it provides a good platform for building communication skills in isl. the videos cover various ncert educational books for subjects like science, social science, and literature. a single video (about 15-30 minutes) usually covers one chapter of a book and simultaneously provides an audio voice-over (in english) conveying the same content. apart from islrtc's educational sign videos which make up a significant portion of isltranslate, we also use another resource from deaf enabled foundations (def) ( https://def.org.in/ ). def videos consist of words with respective descriptions and example sentences for the same words, along with the text transcriptions available in the descriptions foot_2 . we split the def sign videos into multiple segments using visual heuristics for separating segments corresponding to words, descriptions, and examples. in total, isltranslate consists of 2685 videos (8.6%) from def, and the remaining 28537 videos (91.4%) are from islrtc. isltranslate creation: we use the audio voiceover (by analyzing the speech and silence parts) to split the videos into multiple segments. further, these segments are passed through the sota speech-to-text model @xcite to generate the corresponding text. as the generated text is the same as present in the book chapters' text, verifying the generated sample was easy and was done by manually matching them with the textbook. in general, we found automatically transcribed text to be of high quality; nevertheless incorrectly generated text was manually corrected with the help of isltranslate translations isl-signer translations (references) birbal started smiling. when it was his turn, he went near the line. birbal started smiling. he turned towards the drawn line. discuss with your partner what birbal would do. discuss with your partner what birbal would do. birbal drew a longer line. under the drawn line, birbal under the first one. drew a longer line. saw what he drew and said. that's true, the first line is shorter now. one day, akbar drew a line on the floor and ordered. one day, akbar drew a line and ordered make this line shorter. make this line shorter. rita is shorter than radha. rita is short and the other is radha. rajat is taller than raj. rajat is taller and the other is raj. but don't rub out any part of it. but don't rub out any part of it. try to draw rajat's taller than raj. first draw rajat as taller, then draw raj on the right. no one knew what to do. no one knew what to do. each minister looked at the line and was puzzled. each minister looked at the line and was puzzled. no one could think of any way to make it longer. no one could think of any way to make it longer. have you seen the fine wood carving? look at its architecture. most houses had a separate most houses had a separate bathing beding area. separate and some had wells to supply water. many of these cities had covered drains. notice how carefully these were laid out in straight lines. notice how carefully these were laid out in straight lines. comparison with other continuous sign-language datasets: we primarily compare with video-based datasets containing paired continuous signing videos and the corresponding translation in written languages in ## baseline results given a sign video for a sentence, the task of sign language translation is to translate it into a spoken language sentence (english in our case). for benchmarking isltranslate, we create a baseline architecture for isl-to-english translation. we propose an isl-pose to english translation baseline (referred to as pose-slt) inspired by sign lan- we use the mediapipe pose estimation pipeline foot_3 . a similar slt-based pose-to-text approach was used by @xcite , which proposes progressive transformers for end-to-end sign language production and uses slt-based pose-to-text for validating the generated key points via back translation (generated pose key points to text translations). though the pose-based approaches are faster to process, they often perform less than the image-based methods. for the choice of holistic key points, we follow @xcite , which returns the 3d coordinates of 75 key points (excluding the face mesh). further, we normalize every frame's key points by placing the midpoint of shoulder key points to the center and scaling the key points using the distance between the nose key point and the shoulders midpoint. we use standard bleu and rouge scores to evaluate the obtained english translations (model hyperparameter details in app.d). ## conclusion we propose isltranslate, a dataset of 31k isl-english pairs for isl. we provide a detailed insight into the proposed dataset and benchmark them using a sign language transformer-based isl-poseto-english architecture. our experiments highlight the poor performance of the baseline model, pointing towards a significant scope for improvement for end-to-end indian sign language translation systems. we hope that isltranslate will create excitement in the sign language research community and have a significant societal impact.