AVASAG : A G erman S ign L anguage Translation System for Public Services (short paper) (2021)

Abstract: This paper presents an overview of AVASAG; an ongoing applied-research project developing a text-to-sign-language translation system for public services. We describe the scientific innovation points (geometry-based SL-description, 3D animation and video corpus, simplified annotation scheme, motion capture strategy) and the overall translation pipeline.

## introduction the development of software solutions able to translate (bi-directionally) from spoken language to sign-language (sl) has received a lot of attention during the last years. in europe, the involvement of the public institutions in such line of research culminated with the funding, under the h2020 program, of two 3-year long research projects, namely easier @xcite and signon @xcite . in this paper, we present the architecture of project avasag @xcite (avatar-basierter sprachassistent zur automatisierten @xmath0, which is a project funded by the german ministry for education and research (bmbf) aiming at deploying a commercial system able to automatically translate text to sign language in various domains of public services (e.g., announcements for railway stations, airports, harbors, and hygiene warnings). the implementation choices are driven by the following requirements and constraints: from a scientific point of view, the project aims at the following innovation points. first, the project is developing a translation system that goes beyond classic symbolic representation of sl. existing sl-descriptions range from mere un-contextualized glosses, to more sophisticated formats specifying hand (shape, orientation, location, trajectory) and facial movements (e.g., stokoe @xcite , hamnosys @xcite ). this gives the opportunity to human operators for the corrections of sentences when the text-to-sl-description fails. however, compared to data-driven approaches, animations driven by sl symbolic descriptors are generally judged as generating non-believable unnatural animations (see @xcite for an overview). on the other hand, recent end-to-end data-driven approaches are moving towards the generation of 3d sign pose sequences @xcite that could be used to animate an avatar from a kinematic level, but hinders the possibility of a manual correction of the translation result. in this project, we try to merge the advantages of data-driven animation, which leads to more natural looking results, while leaving the capability of post-translation manual correction. given a vocabulary of motion captured signs, the inflection of signs within the context of a sentence will be realized by transforming a sign data using: i) non-rigid 3d transformation (translation, rotation, scaling, shearing) of the hand trajectories and torso movement, ii) corrective blend-shapes on the facial animation, and iii) time-warping functions controlling the dynamic of the execution. all of those inflection transformations are driven by numerical parameters that can be manipulated by a human through the use of "3d gizmos" in a dedicated editing gui. as an advantage with respect to performing end-to-end translation (directly from text to avatar animation data), predicting only inflection parameters significantly reduces the size of the target output, thus likely allowing for the preparation of accurate models with much less training material. second, the project will deliver to the scientific community, with public unrestricted access, a corpus of sentences with parallel data whose entries are composed of: in addition, the corpus will be paired with a vocabulary of signs (aka signary), indexed by gloss-ids, where each sign will contain: the mocap data and video of non-inflected signs executed in the same settings as for the sentences, syntactic information of the sign, such as corpus: animation data nova vocabulary animation analyzer corpus animation analyzer text2mms (training) vocabulary animation data vocabulary interpreter video and motion capture corpus: training data (mms) translation model computes geometric info adds new signs corpus: video annotations avatar player input sentence text2mms (inference) mms instance mms sequencer animation synthesis parameterized signs figure 1: overview of the off-line training (top) and real-time translation (bottom) pipelines. third, the annotation process will follow an innovative "boolean-based" simplified annotation scheme, where annotators on each tier must check a flag only if the execution of a sign in the sentence shows meaningful differences with respect to its lexicalized form in the vocabulary. here, by "meaningful", we mean deliberate inflections (such as sign relocation, eyebrows movement, body shifts, head movement, facial expressions, and the like) of the sign in order to convey additional meaning. the extraction of the exact values of those differences, i.e., the magnitude of the inflection parameters, is delegated to a procedural analysis of the 3d animation data. such a simplified annotation strategy is supposedly faster than existing schemes, where annotators must select values from closed lists or insert free text. fourth, we are employing a mocap system that combines different data streams together, like real-time streamed point clouds and multiple (depth) cameras. these different data sources are then combined and processed inclusively to create a matching animation. ## system overview figure 1 shows a diagram of the offline and realtime phases of the proposed architecture. ## current status and future work at the moment of writing, the project completed its initial investigation stage and it is at the beginning of its development stage. the corpus structure has been finalized. the mocap environment has been tested, finalized, and was used to capture the first sentences of the corpus. the annotation tool has been configured and sign language experts are using it. scripts to automatize the processing of the corpus (e.g., extraction of facial animation data from videos, consistency check) are under development. the avatar animation engine can playback motion captured sentences and non-inflected signs with body and hands. as soon as facial animation is supported, the avatar will undergo the first user evaluation. tools for the analysis of the animation data (such as trajectory transformation and mesh registration) are under investigation. as a first goal, the system shall enable translations for public services, which is characterized by a formal communication register. in the future, the system will be extended to be applied in different contexts, where more complex sign language features, such as iconicity, pose higher challenges for the whole translation pipeline.