Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition (2023)

Abstract: Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2sign-U, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at https://github.com/cactuswiththoughts/UnsupSpeech2Sign.gitcactuswiththoughts/UnsupSpeech2Sign.git .

## introduction many hearing-impaired people communicate natively in sign language (sl); for them, sl communication is as effortless as native spoken communication is for normal-hearing people. however, when it comes to a conversation between a hearingimpaired and a normal hearing, tremendous barriers exist for several reasons. first, there is a shortage of people who are bilingual in spoken and sign languages. automatic sign language recognition models exists @xcite but are fully supervised and require a large number of annotated data, which are hard to acquire. as a result, such systems are often limited to a small vocabulary. on the other hand, untranscribed speech audio and sl videos are quite common on the internet, presenting an exciting possibility: given a non-parallel pair of speech and sign language datasets, can we train a model to translate between spoken and sign languages? this task, we called unsupervised speech-to-sign language recognition (ssr-u), is analogous to well-known problems such as unsupervised machine translation (mt-u) @xcite @xcite and unsupervised automatic speech recognition (asr-u) @xcite @xcite , albeit with a few new challenges. first of all, in the case of ssr-u, both modalities are continuous as opposed to at least one of them being discrete in the case of asr-u and mt-u. consequently, the matching process is much more challenging due to higher within and cross-modal variability. further, most sign language and spoken language can only be matched on the word level as opposed to the subword level in the case of asr-u. not only does the space of possible mappings explode combinatorially, but less training data and fewer temporal constraints are also available to recover the correct mapping. in this paper, we develop a neural network-based framework, speech2sign-u, for both character-level (with fingerspelling sequence) and word-level ssr-u. it achieves promising results on datasets with up to around 900 asl signs. ## problem formulation suppose we have a corpus of unlabeled speech recordings sampled from the random process @xmath0, • • • , a t ) and another separately collected corpus of unlabeled sign language videos sampled from the random process both a and v contain the same semantic information but different para-linguistic information such as speaker/signer identity and prosody. in other words, if we filter out the para-linguistic information and retain the semantic information as x := x(a) = (x 1 , • • • , x t ) ∼ p x for the speech and y := y (v ) = (y 1 , • • • , y l ) ∼ p y for the videos, we can find a generator function g : x @xmath1. since the corpora are unpaired, we cannot estimate g directly from samples, and the goal of ssr-u is to "decipher" it using only the relations between the speech-only and video-only distributions, p x and for all sign language unit sequences @xmath2, where g(y|x) = 1 if and only if @xmath3. ## related works sign language recognition one way to bridge between sign language and written/spoken language is to build a sign language recognition (slr) system trained on parallel sign language and text corpora. the earliest attempts tried to recognize fingerspelling gestures using hand-tracking signals from wired gloves @xcite . later works introduced vision to either correct the errors made by the handtracking model, or to serve as a cheaper and lessintrusive alternative @xcite . focusing on the problem of isolated sign recognition and treating it as a classification task, a variety of statistical and deep learning models have been proposed, such as hmm (starner and pentland, 1997), 3d-cnn (huang et al., 2015), twostream inflated 3d (i3d) cnn (carreira and zisserman, 2017; joze and koller, 2019), and transformer (boháček and hrúz, 2022), among others. to handle multi-sign video sequences, @xcite @xcite reformulate the problem as a sequence labeling problem and develop various systems based on 2d-cnn-hmm hybrid models for german sign language recognition. later works improve the alignment mechanism of previous models using soft dtw @xcite , ctc with dtw contraints (pu et al., 2019) or pseudo-labeling refinement @xcite . while some aim to directly use raw rgb images or generic action features like optical flow as inputs @xcite @xcite , others have found domainspecific features like whole-body and hand keypoints to be more reliable and robust @xcite . thanks to the rapid development of the field, there are now many word-level and sentence-level datasets available in different sls, and we refer to (joze and koller, 2019) for a more comprehensive review. unsupervised cross-modal alignment the task of translating between two languages without parallel corpora has been demonstrated between written language pairs (mt-u) and between spokenwritten language pairs (asr-u). @xcite and @xcite are respectively the first to treat word-level and sentence-level mt-u as a distribution matching problem and built the first such systems by training statistical machine translation systems using nonparallel corpora, which are further improved by @xcite . to allow more general source and target distributions, @xcite @xcite instead use neural networks to embed the source and target distributions and match the distributions using either shared denoising autoencoder @xcite , earth-mover distance minimization (zhang et al., 2017b) or a generative adversarial network (gan) with additional regularization losses @xcite @xcite . @xcite @xcite @xcite adapt and perfect the gan-based approach for spoken-written language pairs by leveraging large-scale self-supervised speech representation learning models @xcite as well as iterative self-training techniques @xcite . ## conclusion in this paper, we propose the task of unsupervised speech-to-sign language recognition and a neural network model, speech2sign-u, capable of both character-level and word-level ssr-u. on various unpaired speech and asl datasets, our models consistently outperform previous unsupervised models such as wav2vec-u. further, we found our model reliable to train for a variety of vocabulary sizes and robust against various types of noise in both speech and visual modalities.