Automatic generation of a 3 D sign language avatar on AR glasses given 2 D videos of human signers (2021)

Abstract: In this paper we present a prototypical implementation of a pipeline that allows the automatic generation of a German Sign Language avatar from 2D video material. The presentation is accompanied by the source code. We record human pose movements during signing with computer vision models. The joint coordinates of hands and arms are imported as landmarks to control the skeleton of our avatar. From the anatomically independent landmarks, we create another skeleton based on the avatar’s skeletal bone architecture to calculate the bone rotation data. This data is then used to control our human 3D avatar. The avatar is displayed on AR glasses and can be placed virtually in the room, in a way that it can be perceived simultaneously to the verbal speaker. In further work it is aimed to be enhanced with speech recognition and machine translation methods for serving as a sign language interpreter. The prototype has been shown to people of the deaf and hard-of-hearing community for assessing its comprehensibility. Problems emerged with the transferred hand rotations, hand gestures were hard to recognize on the avatar due to deformations like twisted finger meshes.

## introduction about one million people in europe are deaf and mainly communicate using sign language (sl), which consists of gestures of hands, arms and facial expressions. outside their daily routines, deaf or hard-of-hearing (dhh) people face barriers for independently participating in society. the communication between signers and hearing speakers is often supported by trained sl interpreters. beyond their respective communities, the availability of these services is often lacking or costly. new technological solutions may provide democratic alternatives to expensive and scarce translation resources and enable an independent communication between hearing and dhh people. our approach is based on the assumption that a simultaneous translation from spoken language to sl can enable more direct communication between deaf and hearing people. through interviews with members of the dhh community, it was apparent that translations are understood even better when the interpreter's signs can be perceived with the speaker's mouth and gestures at the same time. if the person has a good picture of the mouth movement, around 30% of the speech can be read from the lips (deutscher gehörlosen-bund e.v., 2021). several phases of the development were conducted with the support of members from the dhh community through a co-operation with the center for culture and visual communication of the deaf in berlin and brandenburg (zfk) a as described in our previous publication @xcite . in the discovery phase, qualitative interviews were conducted to provide user insights and lead the basic design decisions, including the choice of developing an avatar on ar glasses, as mentioned. in the early implementation phase, a professional deaf sl interpreter was video-recorded to provide material for the sl animation. in the evaluation phase, the efficacy of our proposed solution was evaluated via a wizard-of-oz experiment, where a prototypical avatar was displayed in a way that it appeared as a fully developed automatic interpreter. in this paper we focus on the technical implementation of this avatar. the 2d videos of the sl interpreter were analyzed with computer vision methods to extract human joint coordinates (landmarks). after mapping these landmarks on a pre-built avatar model, the virtual interpreter was displayed on a the ar glasses in the deaf person's field of view. in the next chapter we give an overview of the related work. chapter 3 provides the details about the implementation. in chapter 4 the evaluation process is described, whereas in chapter 5 we give a conclusion and indications for further research. ## related work in the last 30 years, diverse approaches to the automatic generation of sl have emerged. tokuda and okumura (1998) presented a prototype system for a word-to-word translation of japanese to japanese sign language (jsl) by finger spelling. by the beginning of the century, @xcite specified the first framework for producing avatar based sl from text. they proposed the signing gesture markup language (sigml; @xcite , an xml-formatted sl sequence description to drive the animation of an avatar in a web browser. it is built on the hamburg notation system @xcite ) that allows phonetic representations of signs. sigml is still applied in recent sign generation concepts @xcite @xcite . it is used as input language for the sl animation system java avatar signing (jasigning) @xcite , the successor of sigmlsigning @xcite . this was applied by @xcite for their open online sl translation application development platform and @xcite , who developed a system that produces indian sign language (isl) from english text while considering isl grammar. for the correct grammar they created a hamnosys database before converting the representations to sigml. another architecture was introduced by @xcite . they presented the embodied agents behavior realizer (embr) engine for robust real-time avatar animation. it is controlled by the embrscript which defines key poses for animation sequences. the embr system was later extended by @xcite to build a sign animation tool based on a gloss database. more similar to our approach, there is recent work that builds upon open source machine learning solutions which track human body keypoints from 2d video material. mcconnell et al. (2020) animated a two-dimensional virtual human that is able to sign the british sign language (bsl) alphabet, based on body landmarks estimated by the openpose library @xcite . although we also focus on a method requiring no special hardware or costly computing resources, we produce translations on a sentence level and we animate an avatar on all three dimensions. our work uses the three-dimensional landmark prediction of the lightweight and fast me-diapipe framework @xcite . this was also considered for the automatic recognition of sl @xcite @xcite @xcite in previous work, but not for producing sl. mediapipe is compatible a zentrum für kultur und visuelle kommunikation gehörloser in berlin & brandenburg e.v., potsdam, germany to more hardware systems and has less requirements than openpose. with the 2d skeleton generated by openpose and generative adversarial networks, stoll et al. (2020) and ventura et al. (2020) elaborated novel approaches to generate signing videos with photo realistic humans. regarding the animation of 3d avatars based on 2d recordings, @xcite developed a pipeline that creates skeletal data from sign videos with the help of multiple recurrent neural networks, proposing a new pose estimation framework specially designed for sl. the authors also compare the features of different available architectures, including openpose and mediapipe. they provide a solution that covers all compared capabilities. to drive their avatar, angular and positional joint displacements are calculated with inverse kinematics, the results are then saved in a bvh file. user experiments showed that the synthesized avatar signing is comprehensible. for this paper we implemented as a first step a more simplistic method, using an existing machine learning architecture and driving a virtual avatar by raw bone rotation data calculated in a 3d graphic suite. we are aiming to reach likewise comprehensibility results with a more refined animation approach in the future. ## implementation the goal of the implementation was to create a sl avatar that is displayed in a hololens 1 b and can be used for a proof of concept where we access the acceptability and comprehensibility of a simulated real-time translation on ar glasses among people from the dhh community. converting 2d sl video to a 3d avatar has multiple benefits @xcite . in a simple use case, one can reside to this kind of conversion to create avatars out of pre-recorded sl speeches, while preserving the anonymity of the speakers. in a more advanced use case, such as the one of the automatic sl interpretation that we aim at, a big amount of sequences of sl gestures recorded from human speakers are required as graphical linguistic units. these will be used by a unified pipeline that generates full sl sentences, including methods of speech recognition and machine translation, to be implemented in further work. collecting bigger amounts of human sl speaker recordings is aided if 2d cameras are used, as this is straightforward, does not require advanced equipment and allows using videos of different signers. our process of creating a sl avatar for ar glasses consists of four steps. first, we collected video footage of the predefined phrases by a professional interpreter (section 3.1). then, based on the collected video footage, motion capture was used to convert the gestures and facial expressions into tracking points, using a motion tracking model that analyzes images and extracts body landmark positions (section 3.2). based on the motion tracking points, the animations were transferred to the skeleton of the avatar after calculating rotation vectors (section 3.3). finally, an application was developed to display the animated avatar on the ar glasses (section 3.4). the application, scripts and other material are published as open source. cd ## conclusion we developed a method to generate sl from 2d video recordings with the help of machine learning methods of computer vision and a 3d creation suite. a skeleton was built from the detected body landmarks by creating bones between pairs of specified landmarks. this was then exported as bvh data and applied to a rigged avatar. the avatar's bone constraints were set to copy the rotations of the matching bones from the bvh skeleton. some problems became apparent after mapping the bone rotations to the avatar. besides modified distances between hands or arms to the body, the most severe issue was the twisting of the finger meshes which altered their appearance significantly. the user study with our prototype system showed that the avatar's hand gestures were hard to recognize for most participants. this led to poorer comprehensibility for sentences where the attention had to be paid predominantly to the finger movements, while sentences with discernible arm movements were recognized by the majority. as it has been noted, the animation approach needs to be optimized to achieve more comprehensible results. facial expressions and if possible lip movements should be enabled in future implementations. still, a high acceptance level could be observed for the concept of displaying a sl avatar in ar glasses. we believe that a well developed automated speech to sl system could enable more freedom and flexibility for deaf people in situations where an interpretation can enhance communication significantly, but would be normally not affordable due to the scarce availability of interpreters. even if our prototype system for the experiment is static through the predefined, manually triggered questions, it opens the door for several use cases where content and vocabulary are restricted, e.g. a museum tour. moreover, it was important for us to create an animation solution that requires no special hardware and enables other researchers as well as non-professionals an open-source tool for producing three-dimensional virtually performed sl with an avatar. our vision is to create the basis for an animation data-set which can grow with the addition of more scenarios and can be used for the further development of a system allowing the real-time translation of arbitrary conversations, in conjunction with methods from speech recognition and machine translation.