An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation (2023)

Abstract: Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.

## introduction sign language plays a crucial role in communication for many deaf foot_0 individuals worldwide. however, producing sign language content is often a challenging, laborious, and time-consuming process, requiring skilled translators/interpreters for effective communication. recent technological advancements have led to the development of automated sign language translation systems, which have the potential to increase accessibility for the deaf community and enhance communication. one of the critical issues in this field is the lack of a reproducible and reliable baseline for sign language translation systems. without a baseline, it is challenging to measure the progress and effectiveness of new methods and systems. additionally, the absence of such a baseline makes it difficult for new researchers to enter the field, hampers comparative evaluation, and discourages innovation. addressing this gap, this paper presents an open-source implementation of a text-to-gloss-topose-to-video pipeline approach for sign language translation, extending the work of @xcite . our main contribution is the development of an open-source, reproducible baseline that can aid in making sign language translation systems more available and accessible, particularly in resource-limited settings. this opensource approach allows the community to identify issues, work together on improving these systems, and facilitates research into novel techniques and strategies for sign language translation our approach involves three alternatives for text-to-gloss translation, including a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation (nmt) system. for gloss-to-pose conversion, we use lexiconacquired data for three signed languages, including swiss german sign language (dsgs), swiss french sign language (lsf-ch), and swiss italian sign language (lis-ch). we extract skeletal poses using a state-of-the-art pose estimation framework, and apply a series of improvements to the poses, including cropping, concatenation, and smoothing, before applying a smoothing filter. suchen sie eine ärztin auf, wenn sie auskünfte oder hilfe benötigen. sie-ix auskünft+ oder hilfe benötigen sie-ix ärztin aufsuchen sie auskunft+ oder hilfe (missing) sie (missing) suchen text-to-gloss gloss lookup gloss-to-pose pose-to-video ## background sign language translation can be accomplished in various ways. in this section, we focus on the pipeline approach that involves text-to-gloss, gloss-to-pose, and, optionally, pose-to-video techniques. the text-to-gloss technique translates spoken language text into sign language glosses, which are then converted into a sequence of poses by gloss-to-pose techniques, and into a photorealistic video using pose-to-video techniques. this pipeline offers the benefit of preserving the content of the sentence, while exhibiting a tendency for verbosity and a lower degree of fluency. in this section, we explore each of the pipeline components comprehensively and examine recent progress in sign language translation utilizing these methods. ## method in this section, we provide an overview of our textto-gloss-to-pose-to-video pipeline, detailing the components and how they work together to convert input spoken language text into a sign language video. the pipeline consists of three main components: text-to-gloss translation, gloss-to-pose conversion, and pose-to-video animation. for text-togloss translation, we provide three different alternatives: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. figure 1 illustrates the entire pipeline and its components. ## text-to-gloss we explore three different components as part of text-to-gloss translation, including a lemmatizer ( §4.1), a rule-based word reordering and dropping component ( §4.2), and a neural machine translation (nmt) system ( §4.3). ## gloss-to-pose gloss-to-pose translation involves converting sign language glosses into a sequence of poses that adequately represent a sequence of signs. we use the signsuisse dataset (schweizerischer gehörlosenbund sgb-fss, 2023), which consists of sign language videos in three different lan-guages. we extract skeletal poses from these videos using mediapipe holistic @xcite , a state-of-the-art pose estimation framework that estimates 3d coordinates of various landmarks on the human body, including the face, hands, and body. we preprocess the poses by ensuring that the body wrists are in the same location as the hand wrists, removing the legs, hands, and face from the body pose, and cropping the videos in the beginning and end to avoid returning to a neutral body position. we concatenate the poses for each gloss by finding the best 'stitching' point that minimizes l2 distance. we then concatenate these poses, adding 0.2 seconds of 'padding' in between, before applying cubic smoothing on each joint to ensure smooth transitions between signs, and filling in missing keypoints. finally, we apply a savitzky-golay motion transition smoothing filter @xcite , similar to @xcite , to reduce unnatural motion. ## pose-to-video we use a semi-realistic human-like avatar system to animate the poses generated by our approach. the avatar system is a pix2pix model @xcite adjusted to operate on pose sequences, not individual images. with her permission, we use the likeness of maayan gazuli foot_2 . we use opencv @xcite to render the poses as images and feed them into the pix2pix model to generate realistic-looking video frames. the avatar system can run in real-time on supported devices and is integrated into https://sign.mt @xcite . this system is far from the state of the art, however, we believe that the open-source nature of it will bring rapid improvements, like faster inference speed, and higher animation quality. ## future work here we include several future work directions that we believe have the potential to further enhance the performance and user experience of our system for text-to-gloss-to-pose-to-video generation, and we look forward to exploring these possibilities in the future, together with the open-source community. ## conclusions we presented an implementation of a text-to-glossto-pose-to-video pipeline for sign language translation, focusing on swiss german sign language, swiss french sign language, and swiss italian sign language. our approach comprises three main components: text-to-gloss translation, glossto-pose conversion, and pose-to-video animation. we explained the structure of our system and discussed its limitations, as well as future work directions to address them. these directions have the potential to improve our system, and we look forward to exploring them in collaboration with the open-source community. the main contribution of this paper is the creation of a reproducible baseline for spoken to signed language translation. the system should serve as a baseline for comparison with more sophisticated sign language translation systems and can be improved upon by the community. you can try our system for the three signed languages of switzerland on https://sign.mt .