S ign CLIP : Connecting Text and Sign Language by Contrastive Learning (2024)

Abstract: We present SignCLIP, which re-purposes CLIP (Contrastive Language-Image Pretraining) to project spoken language text and sign language videos, two classes of natural languages of distinct modalities, into the same space. SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs, without directly optimizing for a specific task or sign language which is often of limited size.We pretrain SignCLIP on Spreadthesign, a prominent sign language dictionary consisting of ~500 thousand video clips in up to 44 sign languages, and evaluate it with various downstream datasets. SignCLIP discerns in-domain signing with notable text-to-video/video-to-text retrieval accuracy. It also performs competitively for out-of-domain downstream tasks such as isolated sign language recognition upon essential few-shot prompting or fine-tuning.We analyze the latent space formed by the spoken language text and sign language poses, which provides additional linguistic insights. Our code and models are openly available.

## introduction sign(ed) languages are the primary communication means for ∼70 million deaf people worldwide 2 . they use the visual-gestural modality to convey meaning through manual articulations in combination with non-manual elements like the face and body @xcite . sign language processing (slp) @xcite is a subfield of natural language processing (nlp) that is intertwined with computer vision (cv) and sign language linguistics. slp spans the tasks of sign language recognition @xcite , translation (de coster et al., 2023), and production @xcite . typical datasets, equipped with spoken language text and sign language glosses in addition to videos, that support slp research are rwth-phoenix-weather 2014t, in german sign language (dgs), introduced by @xcite ; and csl-daily, in chinese sign language (csl), introduced by @xcite . however, advances on a specific task/dataset/language are often limited and non-transferable to more generic and challenging settings @xcite due to the small, domain-specific vocabulary (1,066 and 2,000 signs, respectively) and data size (11 and 23 hours, respectively). recent sign language corpora with a scale of more than a thousand signing hours have emerged for relatively high-resource sign languages, e.g., @xcite for british sign language (bsl) and youtube-asl @xcite for american sign language (asl), a subset of youtube-sl-25 @xcite . jwsign @xcite is also a notable corpus that consists of 2,530 hours of bible translations in 98 sign languages. in the meanwhile, outside the world of slp, there is great progress in deep pretrained models of different modalities, gpt @xcite ) for text, masked autoencoders @xcite for images, and wav2vec 2.0 @xcite for speech, to name a few. they are commonly pretrained on a huge amount of data (e.g., over 15 trillion tokens for llama 3, @xcite ) with very little or weak supervision, but present striking multilingual and multi-task ability, for zero-shot prediction, fine-tuning, and representation learning. ideally, the visual aspect of sign languages, i.e., lexical similarity due to iconicity @xcite (illustrated in figure 1 ) makes transferring between different sign languages easier than the text of different spoken languages or even scripts. the latter faces unfair tokenization issues @xcite and out-of-vocabulary errors. at the same time, unlike discrete text tokens (see the comparison in since slp tasks and datasets usually involve both the text and visual/signed modalities, we take inspiration from openai's clip model @xcite but use contrastive learning to connect text with sign language videos instead of images. for video understanding, follow-up work such as videoclip @xcite mainly deals with tasks including action recognition @xcite and videoqa @xcite . however, both clip, videoclip, and other existing multimodal models understand visual content on a coarse-grained level and generic domain and do not address the intricacy of sign language. we show the lack of sign language understanding in contemporary ai models both intuitively (figure 3 in appendix a) and empirically ( this work uses sign-language-specific data to train a clip-like model for slp. we first validate the approach's feasibility on fingerspelling, a subsystem of sign language, by a model named finger-clip ( §4), which correctly understands the fingerspelling of individual letters. we then curate the spreadthesign foot_0 dictionary as a large-scale pretraining dataset consisting of ∼500 hours of signing, as well as diverse public downstream task datasets to run comprehensive pretraining, fine-tuning, and evaluation on full-fledged sign languages. by contrastive training on ∼500 thousand video-text pairs, we obtain a multimodal and multilingual model named signclip ( §5), illustrated by figure 1 . sign-clip excels at various slp tasks and datasets and presents a compelling latent space for signed video content aligned with spoken language text ( §6). ## background: sign language representation representation is a key challenge for slp. unlike spoken languages, sign languages have no widely adopted written form. as sign languages are conveyed through the visual-gestural modality, video recording is the most straightforward way to capture them. the final goal of signclip is to represent a sign language video clip by an embedding that aligns with a ubiquitous text encoder like sentence-bert @xcite . generally, end-to-end training on the raw videos is computationally costly, and various intermediate representations alleviate this issue. videoclip and video encoders our work adapts videoclip, which is pretrained by general instructional videos from the howto100m @xcite dataset. we aim at replacing howto100m with domain-specific sign language videos, such as the dataset how2sign @xcite , albeit on a considerably smaller scale. videos are very dense temporally (frame rate) and spatially (video resolution). a 3d-cnn-based video encoder is often used to extract informative features with reduced dimensionalities for downstream tasks. videoclip uses an s3d @xcite model pretrained on howto100m that produces one video token (i.e., a video embedding for the temporal window) per second. for slp, it is possible to use a video encoder pretrained specifically on sign language videos. a prominent one is the i3d @xcite model pretrained on the bsl sign language recognition task @xcite with the bsk-1k dataset @xcite . a more recent approach to simultaneously address temporal and spatial complexity is the video swin transformer proposed by @xcite , and prajwal et al. ( 2022 ) trains one such model for bsl fingerspelling recognition. pose estimation a potentially more interpretable and universal way of extracting sign language-related features from videos is human pose estimation @xcite , for example, using mediapipe holistic @xcite . each video frame is converted into the location (x, y, z) of 543 full-body keypoints in a 3d space. however, the immediate applicability of the pose estimation systems for slp is questionable @xcite , and known issues such as the lack of accurate depth information are presented @xcite . generally, there is a trade-off between userfriendliness and accuracy for pose estimation tools and slp researchers often prefer mediapipe holistic for the former @xcite over others like openpose @xcite and alphapose @xcite . an even more universal alternative approach to track keypoints named tracking everything everywhere all at once is proposed by @xcite . @xcite uses a similar cotracker @xcite model to study sign language prosody. such models produce smoother signals than traditional pose estimation. discrete representation non-standard written forms of sign language, including signwriting @xcite , hamnosys @xcite , and glosses @xcite , offer the possibility to incorporate sign language content into a text-based nlp pipeline @xcite . however, a good segmentation @xcite and transcription model to process raw video input is first required, which is not well-researched and is not part of this paper. recently, vector quantization (vq) approaches @xcite such as signvqnet @xcite demonstrate the ability to convert the continuous signal of videos/poses to discrete tokens similar to spoken language sub-words @xcite , which might be a promising direction to pursue in future work. comparison in this work, we only empirically experiment with the video encoder and pose-based methods since there are not yet mature and open solutions for the others at the time of writing. given a hypothetical 10-second, 30 fps, 480p (640×480), rgb (3 channels) video of 12 consecutive signs, we compare the dimensionalities of the most common representations in representation temporal spatial interpretable original video 10x30 640×480×3 -s3d (howto100m) 10 512 no i3d (bsl-1k) 10 1024 no mediapipe holistic 10×30 543×3 yes vq (e.g., signvqnet) 10 1024* no signwriting/hamnosys/gloss 12 1024* yes ## model architecture we follow the setups in videoclip and reuse their codebase foot_1 . the most essential model architecture with minor modifications to adapt our experiments is described here. we take pairs of video and text samples (v, t) as inputs, where for each video clip, c v is a sequence of continuous video frames and is processed by a video encoder f θve . this is then followed by a trainable mlp projection layer, f θvp , to project the embedding to the same dimension, @xmath0, as the word embedding on the text side: the frozen video encoder f θve is by default a 3d-cnn network but can be replaced by any other visual backbone or a black-box pose estimation system as summarized in we optionally add two linear multimodal projection layers on top of z v and z t , which are missing in videoclip but present in clip (see figure 3 of the clip paper). finally, we employ the in-fonce loss @xcite as the contrastive objective to discern the relationship between the embedded n video-text pairs in each mini-batch and run contrastive training over the whole dataset. ## fingerclip as a proof of concept, we first apply this contrastive training approach to fingerspelling @xcite @xcite , a subsystem of sign languages heavily influenced by the surrounding spoken languages. for concepts that do not (yet) have associated signs (names of people, locations, organizations, etc.), sign language users borrow a word of a spoken language by spelling it letter-by-letter with predefined signs for that language's alphabet. isolated fingerspelling recognition 5 can therefore be considered a toy task similar to the mnist @xcite handwritten digits classification task in cv. we name the model fin-gerclip, a mini-version of signclip ( §5). dataset we start with the rwth german fingerspelling database @xcite , containing ∼1400 videos of 35 dgs fingerspelling and number signs, five of which contain inherent motion. we provide details and an illustration of the dataset in appendix b. we split all examples randomly into training/validation/test sets at the ratio of 8:1:1. ## signclip pretraining to fully realize the power of contrastive learning, we train and evaluate signclip on larger and more diverse sign language datasets. for efficient experimenting, we start exploring datasets consisting of relatively short-duration sign language video examples, e.g., for the task of isolated sign language recognition (islr) instead of machine translation. in ## downstream tasks and analysis in this section, we evaluate signclip on several downstream tasks and datasets, and we discuss the ideas for further enhancement and evaluation in §9. ## related work contrastive learning has been increasingly used in contemporary slp work. we discuss some related work in this session, which aims mainly at improving performance on dedicated tasks and datasets. therefore, their goals are slightly different than signclip, a universal pretrained sign language embedding model aligned to spoken language text. gan et al. (2023 @xcite has a similar sign embedding idea as signclip, namely learnt contrastive concept, that aligns with word embeddings of the linguistic labels for sign video and incorporates this into islr pipelines for the wlasl and bobsl datasets. @xcite , on the other hand, tackles cslr on bobsl by a multi-task setting, including two contrastive losses for sign-level and sentence-level retrieval, respectively. they show that joint training for cslr and sign language retrieval is mutually beneficial. like our pretraining task framing introduced in §4, @xcite explicitly formulates textto-video/video-to-text retrieval as a cross-lingual contrastive learning task. they address the data scarcity issue by combining a domain-agnostic and a domain-aware video encoder and show its effectiveness on how2sign and phoenix-2014t. ## conclusion: where are we for slp? this work involves slp, an interdisciplinary field that suffers from low-resourceness compared to mainstream nlp and cv. to overcome the nongeneralizability of a specific dataset/task/language, we adapt (video-)clip and propose signclip. signclip is trained on spreadthesign, a multilingual, generic sign language dictionary consisting of ∼500 hours of signing in 41 sign languages, and is evaluated extensively for various purposes. signclip demonstrates excellent in-domain performance but falls short of immediate zero-shot prediction on downstream islr tasks. this finding is consistent with previous cv studies @xcite before clip reached its scale. similar models trained on smaller datasets close to an ima-genet scale performed much worse than supervised baselines on common benchmarks. by comprehensively evaluating downstream asl islr performance ( as a middle ground between full zero-shot prediction and full supervision, few-shot learning or fine-tuning is essential to tackle domain shift and is more realistic given the present methodology and data scale. the cross-lingual transfer effect is prospective for sign language thanks to iconicity. with no surprise, under a sensible data condition, the universal pretraining paradigm that is transforming nlp and cv is also a promising research direction for slp.