Meaningful Pose-Based Sign Language Evaluation (2025)

Abstract: We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through (1) automatic meta-evaluation of sign-level retrieval, and (2) a human correlation study of text-to-pose translation across different sign languages. Our findings, along with the open-source pose-evaluation toolkit, provide a practical and reproducible approach for developing and evaluating sign language translation or generation systems.

## introduction automatic evaluation metrics are essential for assessing the quality of automatically generated language content and tracking progress over time. for instance, machine translation (mt) studies rely heavily on bleu @xcite , even though newer metrics have shown stronger correlation with human judgment @xcite . this trend continues in sign language processing (slp; @xcite ; @xcite ), an interdisciplinary subfield of natural language processing and computer vision. sign language translation (slt; @xcite ; de coster et al. (2023)), denoting the part of slp concerned with translating sign language videos into spoken language text, reuses text-based metrics. @xcite puts forward concrete suggestions on evaluating generated text (especially glosses) in a sign language context. they suggest always computing metrics with standardized tools (e.g., sacrebleu @xcite for bleu) and reporting the metric signatures for reproducibility and fair comparison with other work. the * equally contributed as co-second authors. opposite direction-generating or translating into sign language utterances (usually from source text)-presents additional challenges for evaluation. namely, standardized metrics, tooling, and correlation with human evaluation are lacking. in this work, we systematically examine the metrics employed for evaluating sign language output, especially formatted as human skeletal poses @xcite that contain motion of signing (e.g., mediapipe holistic; @xcite ; @xcite ). we start by a literature review of current research practices in §2, and summarize two major families of metrics: (a) distance-based metrics ( §3.1) informed by human motion generation ( §2.3) and sign language assessment (sla; §2.4), assuming the access to reference poses and then computing the distance from the predicted poses to the reference poses, either in the raw 2d/3d keypoint space or an embedding space; (b) back-translation-based metrics ( §3.3) borrowed from mt @xcite and speech translation @xcite , assuming the pre-existence of a pose-to-text translation model. after the initial conceptual review, we select, implement, and meta-evaluate typical metrics along with additional innovative ones proposed by us (as summarised in figure 1 ), through two empirical approaches: automatic meta-evaluation with a signlevel retrieval task ( §4); and a sentence-level correlation study between metrics of interest versus deaf evaluator ratings on three text-to-pose mt systems in three spoken-sign language pairs ( §5). we find that keypoint distance-based metrics, when carefully tuned, can rival more advanced approaches for sign retrieval and human-judgment correlation. on the other hand, embedding-based metrics, including those borrowed from sla, excel in their own domain but struggle at the sentence level across different systems. back-translation likelihood emerges as the most consistent metric, highlighting the need for open, standardized poseto-text models alongside human evaluation. the source code of the suggested evaluation metrics and the proposed meta-evaluation protocols in §4 are openly maintained in pose-evaluation, a public github repository. the human correlation data and evaluation scripts in §5 are also released in a separate text2pose-human-eval repository to encourage future research. ## related work we discuss four related fields in this section with a special emphasis on the evaluation methodology, and outline recent work in sign language generation (slg; §2.2) in ## evaluation metrics in this section, we formally define common evaluation metrics mentioned in related work ( §2) and implement them, reusing open-source code where available, to prepare for the upcoming empirical study on pose evaluation in §4 and §5. ## automatic meta-evaluation in this section, we explore methods to automatically (meta-)evaluate proposed metrics, especially when there are many variants as seen in §3.1. we adopt the retrieval-based evaluation protocol from @xcite to assess how well different metrics capture meaningful distinctions between signs. each pose sequence is treated as a query, and the goal is to retrieve other samples of the same sign (targets) from a pool that includes unrelated signs (distractors). we focus primarily on the distance-based metric variants introduced in §3.1, and compare them against embedding-based alternatives such as signclip score ( §3.2.2). evaluation is conducted on a combined asl dataset of asl citizen @xcite , sem-lex @xcite , and popsign asl @xcite . for each sign/gloss, we use all available samples as targets and sample four times as many distractors, yielding a 1:4 target-to-distractor ratio. for instance, for the sign house with 40 samples (11 from asl citizen, 29 from sem-lex), we add 160 distractors and compute pairwise dis- base (f) fill (e) trim (b) norm. (a) padding (f) keypoints (c) map↑ p@10↑ ham2pose nape 0* ✗ ✔ zero reduced 26% 14% ham2pose ndtw(-mje) unspecified ✗ ✔ / reduced 27% 14% ape 10 ✗ ✗ zero upper body 33% 27% ape 10 ✗ ✗ first-frame upper body 34% 29% ape 10 ✔ ✗ zero upper body 35% 30% ape 10 ✗ ✔ zero reduced 36% 32% ape 10 ✗ ✗ first-frame reduced 37% 32% ape 10 ✗ ✗ first-frame yt-asl 39% 36% ape 10 ✔ ✗ first-frame reduced 40% 36% ape 10 ✔ ✔ zero upper body 41% 37% ape 10 ✗ ✗ zero hands 42% 38% ape 10 ✔ ✔ first-frame yt-asl 43% 39% ape 10 ✔ ✗ zero hands 45% 41% dtw 10 ✗ ✗ / upper body 36% 32% dtw 10 ✔ ✗ / upper body 37% 33% dtw 10 ✗ ✗ / reduced 42% 40% dtw 10 ✗ ✔ / upper body 43% 41% dtw 10 ✔ ✔ / upper body 43% 41% dtw 10 ✗ ✔ / reduced 44% 41% dtw 10 ✗ ✔ / hands 45% 41% dtw 10 ✗ ✗ / yt-asl 48% 47% dtw 10 ✔ ✗ / yt-asl 49% 48% dtw 10 ✗ ✗ / hands 53% 52% dtw ‡ 10 ✔ ✗ / hands 53% 52% dtw † 1 ✗ ✔ / hands 55% 53% signclipscore p-p (multilingual) 50% 48% signclipscore p-p (asl finetuned) 91% 92% the results show that, as expected, dtw-based metrics outperform padding-based ape baselines. while selecting hands-only keypoints appears to yield the best results, a more sophisticated selection that includes non-manuals might still be desirable. embedding-based methods, particularly signclip models fine-tuned on in-domain asl data, achieve the strongest retrieval scores. we mark the two best dtw-based metrics by ‡ and † , and rename them dtwp and ndtwp for use in the rest of the paper. ## text-to-pose translation study with human evaluation this section shifts our evaluation focus from automatic sign-level tasks to a sentence-level text-topose sign language machine translation scenario. due to their subjective and diverse nature, open-ended text or utterance generation tasks inherently lack a single "correct"/"ground-truth" answer. consequently, automatic evaluation metrics are only meaningful if they correlate closely with human judgments @xcite . ## discussion and recommendations distance-based metrics are efficient defaults, but the devil is in the implementation details. although seemingly straightforward to implement, distance-based metrics involve many design choices, including pose format and keypoint selection. we empirically demonstrate the effectiveness of correcting these choices through a random parameter search, following our established meta-evaluation protocols in §4. we recommend using the tuned versions-dtwp and ndtwp-in our pose-evaluation library, or tuning your distance-based metrics when necessary. upon successful tuning, a distance-based metric achieves decent sign retrieval and correlation with humans in text-to-pose translation. our tuned metrics can even be used as a distance function for a nearest neighbor classifier 10 , and reach close performance as the signclip model pretrained on multilingual sign language data; still, it lags behind a signclip model fine-tuned on in-domain data ( sla metrics correlate with humans on the segment level but are confused on the system level. while verified to align with human ratings for their tasks on evaluating human-produced signing (usually involving fixed individual signs), text-to-pose translation is more lengthy and open-ended, which hinders direct transferability. a proper length normalization (as seen in the case of svae n vs. svae) might help on the system level at the price of losing precision on the segment level. signclip, used as a multilingual embedding device, excels on the sign level, but falls short for sentence-level translation evaluation. we speculate that using a single embedding to summarize a long-duration (> 10 seconds) signing video is inherently limited, especially for dsgs, a language unseen during signclip's pretraining. nevertheless, the reference-free variant exhibits a moderate correlation at the system level, and we observe a similar tradeoff (p-p vs. p-t) between segment and system level correlations, as seen in the sla metrics. back-translation-based approaches correlate properly with human judgment; a gap remains compared to inter-human correlation. in addition to the standard practices suggested by @xcite on computing text-based metrics, we call for open, standardized pose-to-text translation models that include both the model weights and the source code. yet, as noted in when using back translation, likelihood is consistent and more reliable than text metrics. bleu, chrf, and bleurt show weaker or unstable correlations with humans in ## conclusion this work presents a unified framework and an open-source pose-evaluation toolkit for systematically assessing (generated) sign language utterances based on human skeletal poses. we implemented and compared a wide range of metrics ( §3)-distance-based, embedding-based, and backtranslation-based-via automatic meta-evaluation on sign retrieval ( §4) and a comprehensive human correlation study across three sign languages ( §5). our results demonstrate that carefully tuned distance metrics, namely dtwp and ndtwp, and back-translation likelihoods yield the strongest agreement with native signer judgments. we release our code, evaluation protocols, and human ratings to foster reproducible and fair comparisons in computational sign language research.