Improving Sign Recognition with Phonology (2023)

Abstract: We use insights from research on American Sign Language (ASL) phonology to train models for isolated sign language recognition (ISLR), a step towards automatic sign language understanding. Our key insight is to explicitly recognize the role of phonology in sign production to achieve more accurate ISLR than existing work which does not consider sign language phonology. We train ISLR models that take in pose estimations of a signer producing a single sign to predict not only the sign but additionally its phonological characteristics, such as the handshape. These auxiliary predictions lead to a nearly 9% absolute gain in sign recognition accuracy on the WLASL benchmark, with consistent improvements in ISLR regardless of the underlying prediction model architecture. This work has the potential to accelerate linguistic research in the domain of signed languages and reduce communication barriers between deaf and hearing people.

## introduction when learning to recognize sign language, there is evidence that people rely on breaking signs down into their constituent parts, such as the configuration and location of the hand @xcite . this process is also true of spoken language recognition, where recognizing sound patterns plays a crucial role in one's ability to recognize a word. sometimes, one of these "parts" (phonemes) is the only distinguishing factor between two very different terms, as seen in the signs for difference (palms up) and balance (palms down) in croatian sign language @xcite . thus, the ability to encode and recognize individual phonemes and the relationships among them is essential for sign recognition. as a first step in exploring the practicality of phoneme recognition, we ask: can machine learning models for isolated sign language recognition (islr) benefit from the phonological structure of signs? while some islr models explicitly focus on the signers' hands @xcite or face @xcite , none have leveraged sign language phonology. instead, islr has been treated similarly to gesture recognition, where a "gesture" (such as swinging an imaginary bat or waving a hand) has no underlying structure except for that of the human body itself. this lack of structure might explain why state-of-the-art models like the sign language graph convolution network (sl-gcn, jiang et al. 2021) sometimes predict labels that are visually and phonologically unrelated to the ground truth, as shown in figure 1 . in contrast, we show that models trained to recognize both signs and their phonemes will be more accurate at sign identification than those trained for islr alone. our main contributions are: ## background sign languages are complete and natural languages primarily used by deaf and hard-of-hearing people. there are hundreds of sign languages in the world today collectively used by tens of millions of people @xcite . they rely on the hands, face, and body to communicate meaning according to complex grammars which are independent of any spoken language. sign languages have been and continue to be largely overlooked in natural language processing (nlp) research, necessitating explicit calls for more inclusivity (e.g. @xcite . in this paper, we seek to bridge robust techniques in nlp with insights from theories of sign language phonology. sign language phonology is an abstract system of rules that governs how the structural units of signs (e.g., handshape, location, movement) are combined to create an infinite number of utterances. these manual units play a significant role at the phonological level similarly to place of articulation, manner, and voicing in spoken language. theories of sign language phonology attempt to enumerate the meaningless units or "phonemes" found in a sign language and describe the complex relationships among them. in asl-lex 2.0, @xcite describe 16 types of phonemes, largely guided by brentari's prosodic model @xcite . we provide three examples of these phoneme types here: brentari's prosodic model @xmath0, each of which can be observed during the production of any sign. in asl-lex 2.0, about 70% of signs can be uniquely identified by their phonemes, making them an appealing conduit for learning to recognize signs. we leverage these properties by using them as target labels alongside the target gloss. ## method we combine two datasets for the task of islr, asl-lex 2.0 @xcite and wlasl 2000 @xcite , in order to learn asl phonology ( §3.1). then, we describe how to utilize these data by learning two islr models to predict both the target gloss and the phonemes for any input ( §3.2). finally, we address the questions of how many and which phoneme types are best for islr ( §3.3). the dataset and modified models are released for replication and future work. foot_2 ## results we demonstrate across-the-board improvements on islr when predicting phonemes alongside glosses. we measure model performance via islr accuracy, both top-1 and top-3, as well as mean reciprocal rank (mrr), which ranges from 1 (correct sign given highest prediction score) to 1/2000 (correct sign given lowest prediction score). ## discussion we find that adding auxiliary classification tasks for sign phonemes to islr models statistically significantly improves sign recognition accuracy. representing phonemes during training may enable models to learn a more holistic latent representation of sign videos compared to models that only predict the target gloss. the success of this approach provides evidence that handshape and minor location are not only useful in recognizing signs, but also easy enough to learn with semi-supervision (recall that only 48% of the dataset has handshape and minor location labels). our findings show that both models learn these new labels well (fig 3 ) and as a result, the encodings for all videos contain more relevant information for islr. a secondary finding of this paper is that sl-gcn, when trained to recognize all 16 phoneme types, outperforms prior work by anywhere from 1%-9%. still, there is room for improvement in phoneme recognition, especially for handshape, minor location, and path movement. ## limitations the wlasl benchmark has several notable limitations that must be taken into account by those interested in using it. @xcite show that incorrect labels are pervasive in wlasl, causing lower islr accuracy and, in this work, incorrect phoneme labels. additionally, existing sign language datasets do not provide information about the signers' fluency, dialect, age, or race and therefore may not be representative of those who use asl. finally, we caution those interested in collecting asl data against scraping websites without permission, and we encourage acknowledging the creators of those sources. as a first attempt to model sign language phonology in order to improve sign recognition, we applied our approach to two models and used data for one language pair (asl/english). although many phonemes are shared across signed languages, more language pairs and models should be tested in order to verify our claim that learning phonology improves sign recognition in general. in particular, the two-stream inflated 3d convnet (i3d; carreira and zisserman 2017) model, designed for gesture recognition, has also been shown to do well on islr (e.g. @xcite ) and we look forward to extending our method to this model as well.