Extracting Sign Language Articulation from Videos with M edia P ipe (2023)

Abstract: This paper concerns evaluating methods for extracting phonological information of Swedish Sign Language signs from video data with MediaPipe’s pose estimation. The methods involve estimating i) the articulation phase, ii) hand dominance (left vs. right), iii) the number of hands articulating (one- vs. two-handed signs) and iv) the sign’s place of articulation. The results show that MediaPipe’s tracking of the hands’ location and movement in videos can be used to estimate the articulation phase of signs. Whereas the inclusion of transport movements improves the accuracy for the estimation of hand dominance and number of hands, removing transport movements is crucial for estimating a sign’s place of articulation.

## introduction sign languages -or, signed languages -are languages produced with gestures articulated in space and perceived visually or tactilely. over 200 sign languages have been documented around the globe @xcite but they are minoritized and under-researched. one challenge for quantitative research on sign languages is that they generally lack a conventionalized representation in a machine-readable form, such as phonetic transcription or orthography (see e.g., @xcite @xcite . following technological advances in computer vision, methods have emerged that allow a degree of formbased analysis of body movements, such as gesturing and signing, through human body pose estimation tracking of either real-time or pre-recorded video data @xcite . whereas most body pose tracking utilized in sign/gesture research used to involve either wearable devices (e.g., motion capture sensors) @xcite or 3d cameras (e.g., kinect) @xcite , thus requiring designated hardware, there are now pre-trained models that do human body pose estimation either real-time through a regular video camera or on pre-recorded video data, providing a cost-efficient alternative that has proven to be reliable in estimating human gesturing @xcite . a popular tool for such analysis is openpose @xcite , which has been successfully applied in research on both sign language and gesture @xcite @xcite @xcite . a tool that has become available more recently is google's mediapipe @xcite , which similarly performs human body pose estimation of video data and outputs coordinates of landmarks (joints and anchor points such as eyes, nose and eyebrows). ## conclusions in this paper, i have shown initial explorations of methods to extract basic information about articulation and sign form from sign language video data using mediapipe. the first step of estimating an approximate articulation phase of the sign proved to be possible for most sign videos in the data set, which turned out to be a fruitful endeavor in order to then accurately estimate the place of articulation across signs. for the purpose of estimating hand positions corresponding to a phonological place of articulation, estimating the articulation phase is crucial, since the signal is otherwise disrupted by noise from rest positions and transport movements. being able to automatically segment the articulation phase of signs would have other obvious applications, when extracting phonological information about the actual sign (articulation) rather than contextual noise (transport and rest). however, when estimating hand dominance and number of signs articulating, the full method, which included data from all frames in the sign video, consistently outperformed the short method, for which the data only included frames within the estimated articulation phase. it seems as though the crude method of comparing the relative distance traveled between the two hand benefits from more data than the short articulation phase provides, and that the transport movements to and from the articulation phase are in fact quite useful for magnifying the differences in distance traveled between the two hands. this method works quite well with dictionary data here, with each video containing a single (non-compound) sign. if applied to complex/compound signs or stretches of multiple signs in succession, as in conversational data, transport movements may not be as distinct and more elaborate methods to estimate articulation phases would be necessary. the results of this preliminary and exploratory study has demonstrated some possibilities in ex-tracting sign language articulation from videos with mediapipe, which can be used as a fast and cost-efficient way to analyze pre-recorded but unannotated sign language data in substantially larger quantities than would be feasible with manual annotation.