Towards Privacy-Aware Sign Language Translation at Scale (2024)

Abstract: A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a two-stage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages self-supervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.

## introduction used by millions worldwide, sign languages play a crucial role in facilitating communication for many d/deaf and hard-of-hearing individuals. visual in nature, these languages make use of the co-articulated features of hands (i.e., finger positioning, shape, movement, palm orientation, etc.), body postures, gaze, mouth gestures, mouthings and facial expressions to convey meaning @xcite . globally, there are more than 300 sign languages, each with their own grammar and vocabulary. despite the prevalence of sign languages, they are still under-served by translation technology. besides under-investment @xcite and the inherent difficulty of slt, 3 another key explanation for this imbalance is the lack of sufficiently large, clean, and labeled parallel corpora. current state-of-the-art slt systems require detailed and time aligned annotations @xcite , which is not scalable, as annotating sign language data is a labour intensive task and can only be done by proficient signers. we argue that a promising solution to slt's data scarcity is to utilize publicly available unannotated sign language data. 4 in other domains of computer vision and nlp, a common practice is to pretrain on large-scale unannotated web datasets and later finetune on curated, task-specific datasets @xcite @xcite @xcite . this practice is largely unexplored in the slt domain and comes with additional challenges. in particular, moving to large-scale sign language processing makes it increasingly difficult to control the composition of the training data. because sign language videos typically feature faces and upper bodies and thus are biometrically identifying, such research may exacerbate privacy risks. hence, developing sign language technologies responsibly requires us to account for these risks and explore techniques to protect privacy. in this work, we study the effectiveness of selfsupervised video pretraining for slt, under consideration of the aforementioned privacy risks. we first propose a generic, scalable and privacyaware two-stage framework for slt, summarized in however, glosses are also considered an incomplete and inaccurate representation of sign language @xcite . furthermore gloss annotation is a labour intensive task. due to these constraints, there is a growing body of research on gloss-free slt. most such approaches incorporate techniques aimed at reducing the modality gap, such as training the visual encoder via sign spotting @xcite , 5 adding inductive bias in the attention mechanism @xcite , using conceptual anchor words @xcite , or performing visual-language pretraining @xcite . @xcite also benefit from a pretrained text model (t5; @xcite . similar to @xcite , we also leverage language supervision to reduce the modality gap, albeit in conjunction with self-supervised video pretraining. ## generic framework we first outline a generic, scalable and privacyaware two-stage transfer learning framework for slt (see in stage i, we train a model, with the goal to learn high-quality continuous sign language representations, via self-supervised learning. the data used at this stage is always anonymized. we make no assumptions on how the data may be anonymized, i.e. face blurring as discussed in §2, or more sophisticated methods such as using synthetic appearances. in stage ii, we finetune the model from the first stage in a supervised manner using a smaller and hand-curated parallel dataset. ideally, the finetuning dataset, being manageable in size, can be unanonymized after gaining explicit consent from signers in the data to minimize information loss. ## method the base implementation of our framework is designed as a two-step approach, termed ssvp-slt. we provide a high-level overview in fig. 1 . ## results and discussion comparison against the state-of-the-art we present our main results in we find that yt data is beneficial both for pretraining and finetuning, which emphasizes the importance of training on large and diverse data and suggests that we can expect further gains from scaling to large public unannotated video. finally, we find that bridging the modality gap via language-supervised pretraining yields a 1 what's the effect of blurring? we isolate the impact of facial obfuscation via blurring on slt performance by training ssvp-slt models on dailymoth-70h with and without blurring during pretraining and slt training. we report the results in is encoding longer clips necessary? increasing clip length is costly due to the poor scaling of global attention, raising the question of whether encoding longer clips is needed. our results indicate that the answer is yes. when finetuning bart and t5, initialized randomly (@xmath0. how to choose the text model? we investigate how the architecture and initialization of the text transformer affects performance. overall, t5 outperforms bart, possibly due to larger capacity, but the gap is small. while it is worse to finetune a randomly initialized t5 model compared to the pretrained one (corroborating findings by @xcite ), for bart we find the opposite result. we conclude that whether text pretraining is helpful needs to be evaluated on a case-by-case basis. it may be worth investigating in the future whether an additional pretraining or finetuning step may lead to better adaptation of the text model to sign language. aug bleu rouge-l bleurt ✗ 14.3 36.9 48.9 ✓ 14.7 37.2 49.0 with and without finetuning augmentation. should we augment data at finetuning? augmentation such as flipping, cropping, and ran-daug may improve generalization, but it comes at a high storage cost at finetuning time, as the video features are extracted offline. is it worth the cost? we compared using 60 epochs of augmented videos (a 60-fold increase in storage) with not using any augmentation. the results in are both pretraining objectives necessary? in §6, we saw that language-supervised video-text pretraining is highly effective when combined with mae. are both necessary? we compared pretraining for 200 epochs with either and both objectives, initializing from a 600-epoch ssvp-slt checkpoint, and also performed 200 epochs of clip-only pretraining from the original pretrained hiera. the results in ## conclusion through controlled experiments, we studied the effectiveness of self-supervised pretraining for slt while considering privacy risks. we introduce ssvp-slt, a novel, scalable, and privacy-aware slt method that leverages masked autoencoding on anonymized video. it achieves state-of-theart asl-to-english translation performance on the how2sign benchmark, outperforming the best previous models in the finetuned and zero-shot setting by over 3 bleu. our results demonstrate the promise of selfsupervised learning to alleviate the data scarcity issue and further scale up sign language processing in the future. we found that video anonymization, even via simple techniques such as face blurring, has relatively little negative impact on downstream performance, further proving that we can build more proficient systems without neglecting important privacy concerns. we hope that this work, alongside the code and data we release, will spur future developments that benefit the d/deaf and hard of hearing communities.