Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of M edia P ipe (2023)

Abstract: Linguists can access movement in the sign language video corpus through manual annotation or computational methods. The first relies on a predefinition of features, and the second requires technical knowledge. Methods like MediaPipe and OpenPose are now more often used in sign language processing. MediaPipe detects a two-dimensional (2D) body pose in a single image with a limited approximation of the depth coordinate. Such 2D projection of a three-dimensional (3D) body pose limits the potential application of the resulting models outside the capturing camera settings and position. 2D pose data does not provide linguists with direct and human-readable access to the collected movement data. We propose our four main contributions: A novel 3D normalization method for MediaPipe’s 2D pose, a novel human-readable way of representing the 3D normalized pose data, an analysis of Japanese Sign Language (JSL) sociolinguistic features using the proposed techniques, where we show how an individual signer can be identified based on unique personal movement patterns suggesting a potential threat to anonymity. Our method outperforms the common 2D normalization on a small, diverse JSL dataset. We demonstrate its benefit for deep learning approaches by significantly outperforming the pose-based state-of-the-art models on the open sign language recognition benchmark.

## introduction our research aims to find a movement representation that allows processing of sign language movement directly, without relying on annotations or systems of predefined movement features, such as the hamburg notation system @xcite . and help to overcome the camera settings constraints of the available datasets. ## methodology to achieve our goal of directly processing sign language, we set the following requirements for the desired movement representation: the adherence to the triangle inequality, the capability of movement synthesis, being intuitive and understandable to humans, and being low-dimensional. the adherence triangle inequality is essential for automated data processing techniques like clustering, machine learning, and indexing. movement representation data must be distributed across a space compatible with the notion of distance and similarity. sampling from such a space should return the corresponding pose, and moving through it should produce movement to meet the movement synthesis requirement. the space should not have any latent features, and its dimensions must be perceivable by a human. to promote readability and facilitate processing, the space must be as low-dimensional as possible to eliminate unnecessary information from representations. normalization must transform pose data into the desired space, and encoding must represent it suitably for human perception. to determine the degree of adherence to these requirements and compare our method to the basic normalization method, mentioned in section 1.1, we conducted experiments on two types of sign language data using standard machine learning and deep-learning techniques. ## experimental setup the proposed normalization was explicitly developed to process data with high variance, which is typical of data captured in real life. the decomposition property of our approach allows for comparing pure movement data on a joint-by-joint basis. in this section, we compare the performance of our method to the basic normalization method, mentioned in section 1.1, on a dataset composed of continuous samples -the jsl dataset and a public benchmark dataset collected of isolated samples -the wlasl-100 dataset. the jsl dataset has a variation in the camera angle and includes coding for various sociolinguistic features. however, it is a small and very diverse dataset; therefore, it will be used for feature exploration and camera settings boundary testing using standard machine learning algorithms. the isolated wlasl-100 is more suitable for deep-learning testing since it is an established public sign language recognition benchmark. ## discussion and conclusions the proposed methods allow linguists and engineers to directly access the movement captured in the sign language corpus. before, they had to use human annotation or recognition methods, which both relied on a predefinition of the features and were effectively limited by it. sign language movement can now be represented and stored in human-readable form with the proposed encoding method, allowing researchers to observe and comprehend it visually. normalized pose data are distributed over a joint-based, low-dimensional feature space with distinct and fully explainable dimensions. machine learning methods can also process it directly since it complies with the distance notion and the triangular inequality. the embedding results showed the presence of stylistic movement features that correspond to known sociolinguistic features of jsl, similar to predictions of the speaker's country of origin based on their accent. linguists and sign language ex our method performs consistently well across all data sets. we satisfied the initial requirements outlined in section 2 and addressed the generalization issue discussed in section 1. the proposed methods are suitable for any sign language, and multiple sign languages can be encoded into one signing space, thus facilitating cross-language studies in future research.