Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods (2023)

Abstract: The majority of existing work on sign language recognition encodes signed videos without explicitly acknowledging the phonological attributes of signs. Given that handshape is a vital parameter in sign languages, we explore the potential of handshape-aware sign language recognition. We augment the PHOENIX14T dataset with gloss-level handshape labels, resulting in the new PHOENIX14T-HS dataset. Two unique methods are proposed for handshape-inclusive sign language recognition: a single-encoder network and a dual-encoder network, complemented by a training strategy that simultaneously optimizes both the CTC loss and frame-level cross-entropy loss. The proposed methodology consistently outperforms the baseline performance. The dataset and code can be accessed at: www.anonymous.com.

## introduction sign languages are primarily the languages of deaf people. they are the center of the deaf culture and the daily lives of the deaf community. 1 in the u.s., estimates suggest that between 500,000 to two million people communicate using american sign language (asl), making it the fifth mostused minority language in the country after @xcite . natural sign languages, which develop independently and possess unique grammatical structures distinct from surrounding spoken languages, are just as crucial to include in the field of natural language processing (nlp) as any other language, as @xcite advocates. one direction in sign language processing (slp) is sign language recognition (slr), a task of recognizing and translating signs into glosses, the writ-1 deaf sociolinguist barbara kannapell: "it is our language in every sense of the word. we create it, we keep it alive, and it keeps us and our traditions alive." and further, "to reject asl is to reject the deaf person." ## datasets we have enriched the slr dataset phoenix14t by incorporating handshape labels derived from the signwriting dictionary and manual labeling. in the subsequent sections, we initially present the original phoenix14t dataset (3.1) and the sign-writing dictionary (3.2), followed by a detailed description of the updated phoenix14t dataset (phoenix14t-hs), now featuring handshape labels (3.3). ## methods the task of slr can be defined as follows. given an input sign video @xmath0, ..., v t ) with t frames, the goal is to learn a network to predict a sequence of glosses @xmath1, ..., g l ) with l words, monotonically aligned to t frames, where @xmath2. in this section, we start by describing the vanilla slr network, where handshapes are not provided or learned during training in section 4.1. we then introduce the two handshape-inclusive network architectures employed in this study in section 4.2. specifically, these networks are designed to predict sign glosses and handshapes concurrently. finally, we elaborate on our chosen training and pretraining strategy in section 4.3 and 4.4. ## experiments in this section, we present the performance of our top-performing model (section 5.1) and further conduct ablation study (section 5.2) to analyze the crucial components of our implementations. ## conclusions in this work, we introduce the concept of handshape-aware slr, enriching this area of research by offering a handshape-enriched dataset, phoenix14t-hs, and proposing two distinctive handshape-inclusive slr methods. out methodologies maintain orthogonality with existing slr architectures, delivering top performance among single-modality slr models. our goal is to draw increased attention from the research community toward the integration of sign language's phonological features within slr systems. furthermore, we invite researchers and practitioners in nlp to contribute to the relatively nascent and challenging research area of slp, thus fostering a richer understanding from linguistic and language modeling perspectives. in future work, we would like to explore three primary avenues that show promise for further exploration: (1) extension of multimodal slr models. this involves expanding multi-modality slr models, which use inputs of various modalities like rgb, human body key points, and optical flow, to become handshape-aware. this approach holds potential as different streams capture distinct aspects of sign videos, supplying the system with a richer information set. (2) contrastive learning. rather than using handshape labels as supervision, they can be employed to generate negative examples for contrastive learning. this can be achieved by acquiring the gloss segmentation from the ctc decoder and replacing the sign in the positive examples with its counterpart in the handshape minimal pair. the resulting negative examples would be particularly challenging for the model to distinguish, thereby aiding in the development of better representations. (3) data augmentation. alternatively, to create negative examples for contrastive learning, the data volume could be increased using the same method that generates negative examples for contrastive learning.