Open-Domain Sign Language Translation Learned from Online Video (2022)

Abstract: Existing work on sign language translation – that is, translation from sign language videos into sentences in a written language – has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings. In this paper, we introduce OpenASL, a large-scale American Sign Language (ASL) - English dataset collected from online video sites (e.g., YouTube).OpenASL contains 288 hours of ASL videos in multiple domains from over 200 signers and is the largest publicly available ASL translation dataset to date. To tackle the challenges of sign language translation in realistic settings and without glosses, we propose a set of techniques including sign search as a pretext task for pre-training and fusion of mouthing and handshape features. The proposed techniques produce consistent and large improvements in translation quality, over baseline models basedon prior work.

## introduction sign language, a type of visual language that conveys meaning through gestures, is the most widely used form of linguistic communication among deaf and hard of hearing people. according to the world health organization, over 5% of the world's population (∼430 million people) suffer from disabling hearing loss. 2 automatic sign language processing can facilitate the daily activities of deaf people and make artificial intelligence technologies more accessible to deaf users. for example, such techniques would allow deaf users to interact with intelligent virtual assistants using sign language and would support automatic interpretation between sign languages and spoken languages. interest in sign language research has recently been growing in the computer vision (cv) @xcite @xcite and natural language processing (nlp) communities @xcite in this paper, we study sign language translation (slt), foot_2 the task of translating continuous signing video into written language sentences. unlike other sign language processing tasks such as sign spotting @xcite or continuous sign language recognition (sign-to-gloss transcription) @xcite , slt has not been studied until recently @xcite and is still restricted to specific domains (e.g., weather forecasts @xcite , emergency situations @xcite ), characterized by small vocabulary size and lack of visual variability. the lack of large-scale translation datasets in the wild is a central challenge in developing slt technologies serving real-world use cases. in terms of translation modeling, most existing slt approaches @xcite @xcite @xcite rely on glosses, which are a transliteration system for sign language. annotating sign language videos with glosses is expensive and hard to scale up. building effective methods for slt without glosses is an under-studied challenge. in this work, we introduce openasl, a largescale asl-english translation dataset. openasl has 288 hours of real-world asl videos from over 200 signers, making it the largest asl-english translation dataset to date. openasl covers multiple domains drawn from a mix of news and vlogs. to handle challenges in slt modeling without glosses, we propose a set of techniques including pre-training with spotted signs and fusion of multiple local visual features, which improves over existing slt baselines by a large margin. 2 related work ## the openasl dataset our videos are collected from video websites, mainly youtube. a large portion of our data consists of asl news, which come primarily from the youtube channels thedailymoth and sign1news. we download all videos with english captions in these two channels through june 2021. the rest of the dataset is collected from short youtube videos uploaded by the national association of the deaf (nad). those videos are mostly in the form of sign vlogs of various types including announcements, daily tips, and short conversations. the raw video is divided into roughly sentencesized clips based on the associated subtitles. specifically, we split the transcript into sentences with the nltk foot_3 sentence segmentation tool and retrieve the corresponding (video clip, sentence) pairs. this procedure produces 98,417 translation pairs in total, with 33,549 unique words. figure 4 shows the distribution of sentence length in our data. we randomly select 966 and 975 translation pairs from our data as validation and test sets respectively. the annotation of the validation and test sets is manually verified. specifically, the english translation and time boundaries of each video clip are proofread and corrected as needed by professional asl interpreters. each annotator views the video clip and is given the original english sentence from the subtitle for reference. the annotator marks the corrected beginning and end of the sentence, and provides a corrected english translation if needed as well as the corresponding gloss sequence. during translation of each sentence, the annotator has access to the whole video in case the context is needed for accurate translation. figure 2 shows the distribution of several properties in our dataset. note that these are not groundtruth labels, but rather approximate labels as perceived by an annotator. the goal is to give an idea of the degree of diversity in the data, not to provide ground-truth metadata for the dataset. the label "other" covers a variety of categories, including examples where the annotator is unsure of the label and examples that contain multiple signers. one feature of our data is the use of subtitles associated with the video as the english translation, thus saving effort on human annotation. subtitled videos have also been employed in prior work @xcite for constructing sign language datasets. as prior work has mostly focused on interpreted signing videos where content originally in the spoken language is interpreted into sign language, the subtitles used there are naturally aligned to the audio instead of the signing stream. as is shown in @xcite , there exists a large time boundary shift between the two. the videos used in openasl are "self-generated" rather than interpreted, so the english subtitles are already aligned to the video accurately. as can be seen from figure 3 , the sentence alignment in the subtitles is of overall high quality (usually less than 2 second time shifts, although a small percentage (< 5%) are larger). we measure the degree of agreement between the original and corrected translations using bleu score @xcite . the original translation achieves 81.0 bleu-4 score when it is compared against the corrected one. the high agreement in translation, as well as the small alignment error from figure 3 , shows the overall high quality of the subtitles. thus to save annotation effort, we do not proofread the training data. ## model and pre-training for gloss-free translation a translation model maps a sequence of t image frames i 1:t to a sequence of n words w 1:n . in the most recent state-of-the-art approaches @xcite for slt, a visual encoder m v g first maps i 1:t to a visual feature sequence f 1:t ′ , and a transformer-based sequenceto-sequence model decodes f 1:tg into w 1:n . our approach is based on the same overall architecture (see figure 6 ). we further incorporate several techniques for pre-training and local feature modeling, described next. ## conclusion our work advances sign language translation "in the wild" (i.e., directly translating real-world sign language videos into written language) both ( 1 ) by introducing a new large-scale asl-english translation dataset, openasl, and (2) by developing methods for improved translation in the absence of glosses and in the presence of visually challenging data. openasl is the largest publicly available asl translation dataset to date. by using online captioned asl videos, we have been able to collect a large amount of high-quality and wellaligned translation pairs (as verified by professional translators) that represent a wide range of signers, domains, and visual conditions. our translation approach, which combines pre-training via sign spotting and multiple types of local features, outperforms alternative methods from prior work by a large margin. nevertheless, the overall translation quality for sign language videos, in both our work and prior work, is significantly lower than that of machine translation for written languages. there is therefore much room for future improvement, and we hope that openasl will enable additional progress on this task.