Linguistically Motivated Sign Language Segmentation (2023)

Abstract: Sign language segmentation is a crucial task in sign language processing systems. It enables downstream tasks such as sign recognition, transcription, and machine translation. In this work, we consider two kinds of segmentation: segmentation into individual signs and segmentation into phrases , larger units comprising several signs. We propose a novel approach to jointly model these two tasks. Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing. Given that prosody plays a significant role in phrase boundaries, we explore the use of optical flow features. We also provide an extensive analysis of hand shapes and 3D hand normalization. We find that introducing BIO tagging is necessary to model sign boundaries. Explicitly encoding prosody by optical flow improves segmentation in shallow models, but its contribution is negligible in deeper models. Careful tuning of the decoding algorithm atop the models further improves the segmentation quality. We demonstrate that our final models generalize to out-of-domain video content in a different signed language, even under a zero-shot setting. We observe that including optical flow and 3D hand normalization enhances the robustness of the model in this context.

## introduction signed languages are natural languages used by deaf and hard-of-hearing individuals to communicate through a combination of manual and nonmanual elements @xcite . like spoken languages, signed languages have their own distinctive grammar, and vocabulary, that have evolved through natural processes of language development @xcite . sign language transcription and translation systems rely on the accurate temporal segmentation of sign language videos into meaningful units such as signs @xcite or signing sequences corresponding to subtitle units foot_0 @xcite . however, sign language segmentation remains a challenging task due to the difficulties in defining meaningful units in signed languages @xcite . our approach is the first to consider two kinds of units in one model. we simultaneously segment single signs and phrases (larger units) in a unified framework. previous work typically approached segmentation as a binary classification task (including segmentation tasks in audio signal processing and computer vision), where each frame/pixel is predicted to be either part of a segment or not. however, this approach neglects the intricate nuances of continuous signing, where segment boundaries are not strictly binary and often blend in reality. one sign or phrase can immediately follow another, transitioning smoothly, without a frame between them being distinctly outside (figure 1 and §3.1). we propose incorporating linguistically motivated cues to address these challenges and improve sign language segmentation. to cope with contin-figure 2 : the annotation of the first phrase in a video from the test set (dgskorpus_goe_02), in yellow, signing: "why do you smoke?" through the use of three signs: why (+mouthed), to-smoke, and a gesture (+mouthed) towards the other signer. at the top, our phrase segmentation model predicts a single phrase that initiates with a b tag (in green) above the b-threshold (green dashed line), followed by an i (in light blue), and continues until falling below a certain threshold. at the bottom, our sign segmentation model accurately segments the three signs. evaluating on the public dgs corpus @xcite (dgs stands for german sign language), we report enhancements in model performance following specific modifications. we compare our final models after hyperparameter optimization, including parameters for the decoding algorithm, and find that our best architecture using only the poses is comparable to the one that uses optical flow and hand normalization. reassuringly, we find that our model generalizes when evaluated on additional data from different signed languages in a zero-shot approach. we obtain segmentation scores that are competitive with previous work and observe that incorporating optical flow and hand normalization makes the model more robust for out-of-domain data. lastly, we conduct an extensive analysis of posebased hand manipulations for signed languages (appendix c). despite not improving our segmentation model due to noise from current 3d pose estimation models, we emphasize its potential value for future work involving skeletal hand poses. based on this analysis, we propose several measurable directions for improving 3d pose estimation. our code and models are available at https: //github.com/sign-language-processing/ transcription. ## motivating observations to motivate our proposed approach, we make a series of observations regarding the intrinsic nature of sign language expressions. specifically, we highlight the unique challenges posed by the continuous flow of sign language expressions ( §3.1), the role of prosody in determining phrase boundaries ( §3.2), and the influence of hand shape changes in indicating sign boundaries ( §3.3). ## experimental setup in this section, we describe the experimental setup used to evaluate our linguistically motivated approach for sign language segmentation. this includes a description of the public dgs corpus dataset used in the study, the methodology employed to perform sign and phrase segmentation, and the evaluation metrics used to measure the performance of the proposed approach. ## results and discussion we report the mean test evaluation metrics for our experiments in from the non-exhaustive hyperparameter search in the preliminary experiments ( in e2s, we reaffirmed that adding face keypoints does not yield beneficial results, so we exclude face in future experiments. although the face is an essential component to understanding sign language expressions and does play some role in sign and phrase level segmentation, we believe that the 128 face contour points are too dense for the model to learn useful information compared to the 75 body points, and may instead confuse the model. in addition, the benefits of explicitly including optical flow (e3s) fade away with the increased model depth and we speculate that now the model might be able to learn the optical flow features by itself. surprisingly, while adding hand normalization (e4s) still slightly worsens the overall results, it has the best phrase percentage. from e4s we proceeded with the training of e5, an autoregressive model. unexpectedly, counter to our intuition and previous work, e5 underachieves our baseline across all evaluation metrics foot_9 . ## conclusions this work focuses on the automatic segmentation of signed languages. we are the first to formulate the segmentation of individual signs and larger sign phrases as a joint problem. we propose a series of improvements over previous work, linguistically motivated by careful analyses of sign language corpora. recognizing that sign language utterances are typically continuous with minimal pauses, we opted for a bio tagging scheme over io tagging. furthermore, leveraging the fact that phrase boundaries are marked by prosodic cues, we introduce optical flow features as a proxy for prosodic processes. finally, since signs typically employ a limited number of hand shapes, to make it easier for the model to understand handshapes, we attempt 3d hand normalization. our experiments conducted on the public dgs corpus confirmed the efficacy of these modifications for segmentation quality. by comparing to previous work in a zero-shot setting, we demonstrate that our models generalize across signed languages and domains and that including linguistically motivated cues leads to a more robust model in this context. finally, we envision that the proposed model has applications in real-world data collection for signed languages. furthermore, a similar segmentation approach could be leveraged in various other fields such as co-speech gesture recognition @xcite and action segmentation @xcite .