Considerations for meaningful sign language machine translation based on glosses (2023)

Abstract: Automatic sign language processing is gaining popularity in Natural Language Processing (NLP) research (Yin et al., 2021). In machine translation (MT) in particular, sign language translation based on glosses is a prominent approach. In this paper, we review recent works on neural gloss translation. We find that limitations of glosses in general and limitations of specific datasets are not discussed in a transparent manner and that there is no common standard for evaluation. To address these issues, we put forward concrete recommendations for future research on gloss translation. Our suggestions advocate awareness of the inherent limitations of gloss-based approaches, realistic datasets, stronger baselines and convincing evaluation.

## introduction automatic sign language processing is becoming more popular in nlp research @xcite . in machine translation (mt) in particular, many recent publications have proposed sign language translation (slt) based on glosses. glosses provide semantic labels for individual signs. they typically consist of the base form of a word in the surrounding spoken language written in capital letters (see in this paper, we review recent works on neural gloss translation. we find that limitations of gloss-based approaches in general and limitations of specific datasets are not transparently discussed as inherent shortcomings. furthermore, among gloss translation papers there is no common standard for evaluation, especially regarding the exact method to compute bleu scores. ## related work for a general, interdisciplinary introduction to sign language processing see @xcite . yin et al. (2021); moryossef and goldberg (2021) and de coster et al. (2022) for a comprehensive survey l datasets translation directions code evaluation metrics bleu tool p o @xmath0. (2018) -✔ -✔ --✔ ✔ ✔ ✔ -tensorflow stoll et al. (2018) -✔ --✔ --✔ ✔ ✔ -(unclear) camgöz et al. (2020b) ✔ ✔ -✔ --✔ ✔ ✔ -wer (unclear) camgöz et al. (2020a) ✔ ✔ -✔ ----✔ ✔ -(unclear) yin and read (2020) ✔ ✔ aslg-pc12 ✔ -@xmath1. (2020) ✔ ✔ --✔ -✔ ✔ ✔ ✔ -(unclear) stoll et al. (2020) ✔ ✔ --✔ --✔ ✔ ✔ wer (unclear) orbay and akarun (2020) -✔ -✔ ---✔ ✔ ✔ -(unclear) moryossef et al. (2021) -✔ ncslgr ✔ -@xmath2. (2021) -✔ --✔ -✔ -✔ ✔ meteor, ter sacrebleu saunders et al. (2022) -✔ dgs corpus -✔ ---✔ ✔ -(unclear) angelova et al. (2022) ✔ ✔ dgs corpus ✔ --✔ -✔ --sacrebleu walsh et al. (2022) -✔ dgs corpus -✔ --✔ ✔ ✔ -(unclear) ## recommendations for gloss translation based on our review of recent works on gloss translation, we make the following recommendations for future research: if bleu is used, compute it with sacrebleu, report metric signatures and disable internal tokenization for gloss outputs. do not compare to scores produced with a different or unknown evaluation procedure ( §2.3). we also believe that publishing reproducible code makes works on gloss translation more valuable to the scientific community. ## alternatives to gloss translation in previous sections we have established that glosses are a lossy representation of sign language. we also argued that the most prominent benchmark corpus for gloss translation (phoenix) is inadequate, but other, preferable corpora do not contain glosses. this begs the question: if not gloss translation, what other approach should be pursued? representing sign language alternatives include translation models that extract features directly from video, generate video directly or use pose estimation data as a sign language representation @xcite . a distinct advantage of such systems is that they produce a sign language output that is immediately useful to a user, whereas glosses are only an intermediate output that are not intelligible by themselves. if a system generates a continuous output such as a video, then evaluating translation quality with an automatic metric is largely an unsolved problem. even though there are recent proposals for metrics (e.g. @xcite , more fundamental research in this direction is still required. ## conclusion in this paper we have shown that some recent works on gloss translation lack awareness of the inherent limitations of glosses and common datasets, as well as a standardized evaluation method ( §2). in order to make future research on gloss translation more meaningful, we make practical recommendations for the field ( §3). we urge researchers to spell out limitations of gloss translation approaches, e.g. in the now mandatory limitation sections of *acl papers, and to strengthen their findings by implementing existing best practices in mt. finally, we also caution that researchers should consider whether gloss translation is worthwhile, and if time and effort would be better spent on basic linguistic tools (such as segmentation, alignment or coreference resolution), creating training corpora or translation methods that do not rely on glosses.