Unsupervised Discrete Representations of A merican S ign L anguage (2024)

Abstract: Many modalities are naturally represented as continuous signals, making it difficult to use them with models that expect discrete units, such as LLMs. In this paper, we explore the use of audio compression techniques for the discrete representation of the gestures used in sign language. We train a tokenizer for American Sign Language (ASL) fingerspelling, which discretizes sequences of fingerspelling signs into tokens. We also propose a loss function to improve the interpretability of these tokens such that they preserve both the semantic and the visual information of the signal. We show that the proposed method improves the performance of the discretized sequence on downstream tasks.

## introduction modern nlp models operate on a set of discrete tokens from a fixed vocabulary, which is the most common way to represent textual data. however, integrating modalities that are continuous into llms, such as videos, audio, or gestures, remains an open research question. in recent years a new paradigm has emerged, where an underlying sequence is first discretized ("tokenized"), and then provided to the llm decoder @xcite @xcite @xcite . while this is an intuitive way to incorporate more modalities into llms, it is not well understood how comparable these tokens are with respect to the ones used in text. in this work, with a specific focus on american sign language (asl) fingerspelling, we investigate what kind of information is contained in these tokens, how this information is represented, and whether their intuitive meaning is similar to the subword tokens used for textual data. we focus on fingerspelling asl for several reasons. first, each visual sign in a fingerspelling video corresponds to a single text character and figure 1 : a fingerspelling sign sequence together with the corresponding phrase. some characters (e.g., "t" and "e" here) can be visually similar, which makes the task challenging. we focus on pose-based inputs as a representation of asl, as opposed to learned video embeddings, for two main reasons. first, it is a computationally faster method than processing the entire video, and because of this it can also be effectively run on a cpu. second, flattening pose-based (x,y) coordinates allows us to apply techniques from the audio domain, such as neural compression, to the asl domain. the paper makes the following main contributions. first, we develop and train a residual vector quantized (rvq) hand fingerspelling tokenizer. tokenization applied to this particular modality has not been explored before. we show that such tokenization opens possibilities for a range of applications, such as using a plain seq2seq transformer model for machine translation between sign language and written language, without convolutional architecture. second, given our finding that these rvq tokens are hard to interpret with information spread across several tokens, we propose an additional training objective that stabilizes the training and improves the interpretability of the resulting tokens. specifically, we add during training a connectionist temporal classification (ctc) loss between the predicted characters for each frame and ground truth fingerspelled character phrase, and show that this loss function improves the quality of the tokenizer on downstream tasks. finally, we provide an in-depth analysis of the learned tokens and show that the residual quantizers encode information hierarchically from coarser to finer details. ## related work audio tokenization. a recent paradigm in audio processing is to map continuous audio signals into discrete tokens via self-supervised learning. this is typically done by imposing constraints on a latent space being discrete @xcite and pre-training on a large amount of unlabeled data. for example, @xcite ; @xcite ; @xcite formulate self-supervised tasks as masked language modelling similar to bert @xcite . @xcite and @xcite utilize adversarial training and rvq to learn the discrete codes. language modeling on discrete tokens. discretizing audio allows us to apply the next token prediction objective to the audio domain. previous work has explored this direction, for instance au-diolm @xcite , vall-e @xcite , audiogen @xcite , musicgen @xcite . they show that language modeling is a meaningful task on audio tokens and, combined with powerful vocoders, can lead to state-of-the-art audio generation. latent space discretization for other modalities. in computer vision, stable diffusion @xcite and dalle-e @xcite both discretize and downsample the image space with discrete vae (kingma and welling, 2022), and run the generation tasks in this latent space. @xcite trains a tokenizer of the human poses and adds those tokens to the vocabulary of a language model. @xcite uses bert-like training on the word-level sign language. different from those works, we operate on a character-level sign language. ## dataset we perform all of the experiments using the american sign language fingerspelling dataset (ashley chow, 2023). this is the same dataset used in a kaggle competition for asl fingerspelling. the dataset contains signs for randomly generated addresses, phone numbers, and urls, in total more than more than three million fingerspelled characters produced by over 100 deaf signers. the keypoints were extracted from raw videos with the mediapipe holistic model, 543 different landmarks with (x, y) coordinates. there are total of 63 unique characters in the dataset. we follow the official competition vocabulary, which includes many more characters that do not have phonologically distinct handshapes, for example "!", "&", or "@". the motivation to include those characters is to include urls in the list of fingerspelled phrases. we only use hand landmarks, 21 for each hand, so 42 (x, y) points for each video frame in total. we only use (x, y) coordinates since the z coordinates are noisy according to the dataset description webpage. we discard frames that do not have any hand coordinates on them, normalized the coordinates per video, and filled nans with zeros. we split the dataset into training, validation, and test sets, a total of 38,000 sign sequences for training and 13,000 for validation and test. ## discrete tokenization of asl fingerspelling to create a discrete representation of the continuous signal of asl fingerspelling, we first start by discretizing the individual video frames in the input sequence using an autoencoder with the discrete latent space induced by a residual vector quantizer (rvq) @xcite . our method builds upon @xcite , but without adversarial training and without time-domain compression. we do not compress along the time dimension since the sign language domain is different from the audio signal. for instance,encodec compresses 16khz audio into 50 integers, resulting in 320x compression. however, mono audios include only one channel, while in the asl fingerspelling dataset, we use 42*2 channels, but relatively short duration (97% of videos are less than 262 frames in the video). additionally, omitting compression along the time domain allows for easier visual inspection of individual tokens. therefore we opt out for a 1:1 time mapping between continuous and discrete input and only compress along the channel dimension, so 42 (x, y) float coordinates are encoded into a single integer. autoencoder architecture. we use the modified squeezeformer @xcite adapted for gesture recognition from the first place of the asl kaggle competition foot_0 as an encoder. squeezeformer is a hybrid architecture that combines convolutional and self-attention blocks. it is a smallsized model with a total of 1.329m parameters. a decoder is a mirrored version of the encoder with the same number of parameters. rvq utilizes several codebooks n q to compress the signal hierarchically. the first codebook receives the embeddings from the encoder and compresses embedding for each frame into a single integer. each subsequent codebook operates on the difference between the actual and quantized embeddings from the previous codebook. additionally, @xcite uses quantizer dropout (qdrop), where during the training all codebooks after random value 1 < @xmath0, so the model learns to compress the signal hierarchically. we present all our results with 1024 discrete codes and 4 codebooks, with quantizer dropout. we ran preliminary experiments with fewer discrete codes/codebooks, but we found that the reconstruction quality was worse. this is in line with previous findings on video and audio tokenization @xcite , where researchers found that a large vocabulary size is beneficial for reconstruction. training. we train for 100 epochs in total and select the model with the lowest validation loss. we flip between right-hand and left-hand with 25% probability for each video. we use an adamw schedule-free optimizer with a learning rate of 1e-figure 2 : cosine similarities between sign token embeddings that represent the characters. we define a sign language token to be representative for specific character by calculating the mutual information between the character and all sign tokens, and selecting the one with the highest mutual information. the tokenization model learns to group similar gestures together in the embeddings space, for example sign token embeddings "i" and "j", or "h" and "g". ## conclusion in this work, we developed a tokenizer for sign language, and used several strategies to probe the sign tokens for various properties. we showed that an rvq quantizer can result in minor loss of information, and proposed a method to improve learned representations. our results indicate that these tokens can be used as a drop-in replacement for their continuous counterpart, and they can be useful for architectures that assume discrete representations on continuous multimodal signals. our code is available at https://github.com/michigannlp/ discrete_asl .