Analyzing the Potential of Linguistic Features for Sign Spotting: A Look at Approximative Features (2023)

Abstract: Sign language processing is the field of research that aims to recognize, retrieve, and spot signs in videos. Various approaches have been developed, varying in whether they use linguistic features and whether they use landmark detection tools or not. Incorporating linguistics holds promise for improving sign language processing in terms of performance, generalizability, and explainability. This paper focuses on the task of sign spotting and aims to expand on the approximative linguistic features that have been used in previous work, and to understand when linguistic features deliver an improvement over landmark features. We detect landmarks with Mediapipe and extract linguistically relevant features from them, including handshape, orientation, location, and movement. We compare a sign spotting model using linguistic features with a model operating on landmarks directly, finding that the approximate linguistic features tested in this paper capture some aspects of signs better than the landmark features, while they are worse for others.

## introduction sign language processing (slp) @xcite is the field of research that studies how signs and signed phrases can be recognized, retrieved and spotted in videos. key approaches differ with respect to whether they attempt to leverage linguistics/phonology, and the way in which they do it, as shown in figure 1 . some recent work used pixel information as input (approach a) without explicitly considering linguistic features that are relevant for sign language (e.g. handshape and orientation of the hand) @xcite . on the other hand, earlier work in sign language recognition proposed methods to extract phonological properties of signs from pixel information @xcite @xcite (approach b). other approaches have applied a landmark detection tool, such as openpose, to obtain the location of landmarks in the body from the pixel input and used them to train a model @xcite (approach c). angle and distance features which approximate the phonological properties of a sign have also been extracted from these landmarks @xcite @xcite (approach d). slp research seeks to use approximative features where possible to avoid the computational overhead of calculating features that reflect linguistic properties exactly. an approximation is considered sufficiently good if it contributes to the performance of an slp system. incorporating linguistic features holds great promise for improving slp in terms of generalizability and explainability. the drawback of incorporating linguistic features based on pixel information, as in approach b, is that this method is sensitive to particular properties of the training data, such as the lighting conditions, the skin colour of the signer, the color and shape of the signer's clothes, and the recording background. approach d, which we pursue here, improves on this by implementing a modular approach which is potentially more robust because linguistic features are extracted from landmarks rather than pixel input. the purpose of this paper is to move research adopting approach d beyond the current state of the art. we make two contributions. first, we expand the inventory of approximative linguistic features that are used for slp. second, we seek to understand when linguistic features deliver an improvement over the landmark features from which they are produced. in contrast, previous work solely focused on the ability of linguistic features to improve performance of slp systems and did not examine what makes these features important. the reason why linguistic features extracted from landmarks can be anticipated to be more robust is that the modularity of this approach makes it possible to use existing tools for landmark detection, such as mediapipe @xcite , openpose @xcite , or mmpose @xcite . mediapipe, for instance, has been trained on a large-scale, in-the-wild dataset (as well as curated and synthetic data) with high variability in background, lighting conditions, the skin colour of subjects, and other visual artifacts. in addition, the modular nature of the approach makes it straightforward to incorporate future improvements of landmark detection technologies as they become available. this paper focuses on the task of sign spotting, which has as its goal to determine when a given target sign occurs in a video of continuous signing. sign spotting is distinct from sign recognition, because we need to establish when a sign occurs in a given video. recognition only uses video segments in which one isolated sign is performed. building on the aforementioned work in sign language recognition (slr) @xcite @xcite , we first detect landmarks with mediapipe and then extract linguistically relevant features from these landmarks. in the extraction phase, we expand on previous work in that we do not only extract features that serve as an approximate representation of the handshape of the signer, but also features that correspond to other relevant properties, such as the orientation of the hand, its location relative to the body, and its movement through space. we compare a sign spotting model which makes use of these approximative linguistic features with one that operates on landmarks directly (approach d and c, respectively). we find that the approximate linguistic features tested in this paper capture some aspects of signs better than the landmark features, while they are worse for others. our code is made available on github foot_0 . ## results the results of our evaluation are shown in ## conclusion in this paper, we investigated how linguistic features, extracted from landmarks of the hands and body of a signer, can be used in the context of sign spotting. we built on recent work in sign language recognition which derived an approximate representation of the handshape of a sign from mediapipe landmarks, and developed our own features to capture the orientation, location and movement of the hands. we compared a sign spotting model that uses these approximate linguistic features with a model that incorporates landmarks directly as training input. our results show that the model using approximate linguistic features captures some aspects of signs better than the landmark model. in future work, our approach to extracting linguistic features could be further improved. for example, the trajectory and repetition of movements may be better captured by including additional features besides wrist velocity. furthermore, it could be interesting to train a model using a combination of landmark coordinates and linguistic features. our general approach will also benefit from further improved landmark detection technologies. current technologies only reliably deliver 2d landmark coordinates. if an accurate estimation of the z coordinate were available, we could work with 3d representations of the hands and bodies of signers. based on such 3d representations, linguistic features could be extracted in a more robust way. for instance, the left and right image in figure 5 depict the same handshape viewed from two different angles. based on 2d landmark coordinates, it would be possible to derive the curvature of the index finger under the perspective on the left (side view), but not under the perspective on the right (front view). from 3d landmark coordinates, the curvature could be derived precisely and reliably. another limitation of currently available landmark detection technologies, such as mediapipe, is that they are not explicitly trained on sign language data. certain handshapes are frequent in sign languages but may not be as frequent in generalpurpose datasets. as a result, the current performance of mediapipe and similar tools may be limited for such handshapes. that said, an important advantage of the modular approach we adopted is that it allows for the direct incorporation of future improvements of landmark detection technologies. finally, while not the focus of this work, we note that the model chosen to demonstrate the performance of the two types of features can be improved. a more sophisticated model architecture may result in better sign spotting performance.