Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation (2025)

Abstract: Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL—three leading, publicly available, gloss-free SLT models—on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train–test sentence overlap to improve transparency and comparability.

## introduction sign language translation (slt) aims to convert sign language videos into spoken or written language, enabling communication between deaf and hearing communities. a key challenge in slt is signer dependence, where models are often trained and evaluated on splits where the same signers appear in both training and test sets. this setup risks overestimating model performance, as models may exploit signer-specific patterns rather than learning to generalise to unseen individuals @xcite @xcite . the most widely used slt benchmark, rwth-phoenix-weather-2014t (phoenix14t) @xcite foot_0 , exemplifies this problem. its default split includes overlapping signers across train, development, and test sets. although it provides high-quality gloss and translation annotations across 8,000 weather forecast videos featuring nine signers, evaluations on this split do not reflect signer-independent performance. a similar issue exists in csl-daily @xcite foot_1 , a largescale chinese slt dataset covering daily topics with 10 signers, which has also become a standard benchmark @xcite @xcite . many recent slt models-including both glossbased @xcite and gloss-free approaches @xcite @xcite @xcite -have reported impressive gains. however, these gains are typically measured on signer-overlapping splits, leaving open the question of how well such models generalise to unseen signers. to address this, we perform signer-fold crossvalidation on phoenix14t and csl-daily, evaluating three state-of-the-art gloss-free slt models: gfslt-vlp @xcite , gaslt @xcite , and signcl @xcite . each fold withholds a signer entirely from training and development. our results (section 3.2) show that, on average, translation performance drops under signerindependent evaluation compared to default-split baselines. this highlights how signer-dependent evaluations can obscure the true generalisation ability of slt models. this study provides the first systematic analysis of signer-independent slt performance using signer-fold cross-validation. we advocate for future slt research to adopt signer-independent protocols to ensure robust and fair evaluation. the remainder of this paper reviews related work, describes the experimental setup, presents results and qualitative analyses, and discusses implications for future slt development. ## related works signer independence remains a critical challenge in sign language recognition (slr) and sign language translation (slt), where models often overfit to signer-specific traits such as hand shape, articulation, or signing speed. while this issue has been well-documented in slr since early work in 2013 @xcite , it has received comparatively little attention in slt. most existing slt research-whether glossbased or gloss-free-continues to evaluate models using signer-overlapping splits. gloss-based pipelines, including gloss-to-text (g2t), sign-to-@xmath0, and related variants @xcite @xcite , have shown steady performance improvements, but do not assess generalisation to unseen signers. this limitation persists in recent models such as ip-slt @xcite and cv-slt @xcite , which continue to rely on signer-dependent splits. gloss-free slt approaches, which bypass intermediate gloss representations, have gained traction for their scalability in low-resource settings. early models such as s2t @xcite , nslt @xcite , and tspnet @xcite laid the groundwork, but similarly evaluated only on signer-overlapping data. recent visionlanguage pretrained models-gfslt-vlp @xcite , gaslt @xcite , signcl @xcite , sign2gpt @xcite , and signllm @xcite -have reported strong results on phoenix14t and csl-daily. however, these evaluations remain constrained to the default splits, which do not isolate signer effects. although signer-dependent training has been shown to inflate performance in slr @xcite , slt studies have yet to investigate this systematically. no prior work, to our knowledge, has performed signer-fold cross-validation in slt to evaluate robustness across all signers. our study addresses this gap by applying signer-fold evaluation to three strong, publicly available gloss-free models: gfslt-vlp, gaslt, and signcl. this allows us to quantify how signer overlap in standard protocols may obscure the true generalisation ability of slt systems. ## conclusion this study highlights limitations in current slt evaluation, particularly the field's reliance on signer-dependent protocols. across folds and datasets, evaluation on unseen signers often leads to lower scores, although the magnitude and pattern of effects differ between phoenix14t and csl-daily. to strengthen evaluation and address dataset-specific confounds such as sentence repetition, we recommend: adopting signer-independent evaluation as a default; enforcing sentence-level disjointness when constructing splits (e.g., split over unique sentences first and then map to clips); and exploring signer-agnostic representations and training strategies that reduce template exploitation (e.g., frequency-aware sampling/downweighting of repeated targets). by addressing both signer dependence and sentence-level duplication, the field can move toward more generalisable and practically deployable slt systems.