Sign Language Translation in a Healthcare Setting (2021)

Abstract: Communication between healthcare professionals and deaf patients is challenging, and the current COVID-19 pandemic makes this issue even more acute. Sign language interpreters can often not enter hospitals and face masks make lipreading impossible. To address this urgent problem, we developed a system which allows healthcare professionals to translate sentences that are frequently used in the diagnosis and treatment of COVID-19 into Sign Language of the Netherlands (NGT). Translations are displayed by means of videos and avatar animations. The architecture of the system is such that it could be extended to other applications and other sign languages in a relatively straightforward way.

## introduction communication between healthcare professionals and deaf patients is challenging @xcite , especially during the current covid-19 pandemic @xcite . sign language interpreters can often not enter hospitals and clinics, interpreting via video relay is not always viable, and face masks make lipreading impossible @xcite . we conducted an online questionnaire to understand how these difficulties are perceived by deaf people in the netherlands. questions were video recorded in sign language of the netherlands (ngt) and also presented in written dutch. 179 people participated, aged 20 to 84. the results-which will be presented in detail elsewhere-show that the general inability of healthcare professionals to communicate in sign language is indeed perceived as a very significant threat. specifically, 88% of participants stated that they were worried about communication barriers should they need to be hospitalised with covid-19, while, for comparison, only 33% stated that they were worried about the fact that friends and relatives would not be allowed to visit them in the hospital. to address this problem, we have developed a modular system which allows healthcare professionals to translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to covid-19, from dutch or english into ngt. for a limited number of sentences, translations are displayed by means of pre-recorded videos. in addition, the system is able to generate translations that are displayed by means of a signing avatar. the present paper focuses on describing the latter part of the system. although we have concentrated on ngt as the target sign language, the problem we aim to address manifests itself worldwide. 1 therefore, in contrast to most existing text-to-sign translation systems (which are tailor-made for a specific target sign language and not easily portable to other languages, see sections 3 and 4), we have aimed to design the system in such a way that it could be extended to include other source and target languages in a relatively straightforward way. we should emphasise that a qualified human sign language interpreter should, whenever available, always be preferred over a machine translation system. still, it is worth investigating the extent to which a machine translation system can be of help in situations in which a human interpreter cannot be employed, especially in the medical setting where effective, instantaneous communication between healthcare professionals and patients can be of critical importance. ## brief background on sign languages evidently, we cannot provide a comprehensive overview here of the linguistic properties of sign languages in general (see, e.g., @xcite ), nor of ngt in particular (see @xcite ). we will, however, highlight some important features which any textto-sign translation system needs to take into account. first of all, sign languages have naturally evolved in deaf communities around the world. this means that, contrary to a rather common misconception, there is not a single, universal sign language used by all deaf people worldwide, but many different sign languages, just as there are many different spoken languages. second, deaf people often have great difficulties processing spoken language even if presented in written form. the median reading level of deaf adolescents when finishing high-school is comparable to that of 8-year-old hearing children @xcite . this may be surprising at first sight but not so much upon reflection. imagine what it would be like as a native speaker of, say, english, to learn hebrew or thai just by looking at the characters, without being told how these characters are pronounced. thus, for healthcare professionals to communicate with deaf patients exclusively through written text would not be satisfactory. this is especially true in a medical setting, where it is critical to avoid miscom-munication, to obtain reliable informed consent for interventions, and to foster an environment in which patients feel maximally safe. third, there is generally no direct correspondence between the sign language used in a given country and the spoken language used in that same country. for instance, while english is the mainstream spoken language both in the us and in the uk, american sign language (asl) and british sign language (bsl) differ considerably from each other, as well as from spoken english. such differences do not only pertain to the lexicon, but also to grammatical features such as word order. this means in particular that, to translate a sentence from english to asl or bsl it does not suffice to translate every word in the sentence into the corresponding sign in asl/bsl and then put these signs together in the same order as the words in the english sentence. fourth, signs are generally not just articulated with the hands, but often also involve facial expressions and/or movements of the head, mouth, shoulders, or upper body. these are referred to as the non-manual components of a sign. a text-to-sign translation system has to take both manual and non-manual components of signs into account. fifth, related to the previous point, non-manual elements are not only part of the lexical make-up of many signs, but are also often used to convey certain grammatical information (comparable to intonation in spoken languages). for instance, raised eyebrows may indicate that a given sentence is a question rather than a statement, and a head shake expresses negation. such non-manual grammatical markers are typically 'supra-segmental', meaning that they do not co-occur with a single lexical sign but rather span across a sequence of signs in a sentence. sign language linguists use so-called glosses to represent sign language utterances. for instance, the gloss in (1) represents the ngt translation of the question are you going on holiday?. a crucial prerequisite for text-to-sign translation is sign synthesis: the ability to create sign language avatar animations. broadly speaking there are two ways to achieve this: key-frame animation (e.g., @xcite ) and motion capture (e.g., @xcite ). while motion capture makes it possible to obtain a library of high-quality animations for lexical signs, a disadvantage of this technique is that animations for lexical signs obtained in this way are difficult to modify so as to incorporate non-manual grammatical markers @xcite . in principle, the same problem also applies to libraries of lexical signs obtained by means of key-frame animation. however, in this case, there is a promising strategy to overcome the problem. namely, rather than directly animating each lexical sign, it is possible to generate keyframe animations of lexical signs procedurally from structured specifications of the phonetic properties of these signs @xcite . such phonetic properties include (but are not limited to) the initial location, shape and orientation of the hands, possibly movements of the hands and other body parts, and facial expressions. several formalisms have been developed to specify the phonetic properties of signs in a structured, computer-readable fashion (see @xcite for an overview). arguably the most extensively developed and most widely used formalism is the sign gesture markup language (sigml) @xcite , which is based on the hamnosys notation originally developed for the annotation of sign language corpora @xcite . for illustration, our sigml encoding of the ngt sign what is given in figure 1 . as can be seen in the figure, both manual components (handshape, location, movement) and non-manual features (mouth, face, head) are encoded. <@xmath0.8"/> <eye_lids @xmath1.8"/> <eye_gaze @xmath2. sigml specifications can be converted into key-frame animations by the jasigning avatar engine @xcite @xcite . this approach makes it possible, in principle, to integrate non-manual grammatical markers with the lexical signs that make up a sentence, although such functionality has not yet been thoroughly implemented in systems based on sigml and jasigning to our knowledge. given these considerations, we opted to use sigml and jasigning as a basis for sign language synthesis, and to implement a new functionality to automate the integration of non-manual grammatical markers with lexical signs. a basic library of sigml specifications of around 2000 lexical signs in ngt was already compiled in the course of previous projects ( @xcite , see also @xcite @xcite ). while we have had to extend this library with healthcare-related as well as some generalpurpose signs, the availability of an initial repertoire of signs encoded in sigml was essential for a timely development of the system. ## text-to-sign translation: a modular approach in text-to-sign translation, two general approaches can be distinguished, differing mainly in the type of intermediate representation that is employed in going from text to sign. in the first approach, which we will refer to as the gloss approach, a given input sentence is transformed into a gloss of the corresponding sign language utterance. next, based on this gloss representation, an avatar animation is generated. gloss approach: @xmath3, for instance, by handtalk, a brazilian company that provides an automated text-to-sign translation service with brazilian portuguese and english as possible source languages, and asl as well as brazilian sign language as possible target languages. handtalk uses machine learning techniques to map input texts to the corresponding glosses, and a combination of key-frame animation and motion capture techniques to generate animations based on a given gloss. in the second approach, which we refer to as the phonetic approach, the given input sentence is transformed into a sequence of phonetic representations of signs. next, based on these phonetic representations, an avatar animation is generated. advantages of the phonetic approach are that (i) grammatical non-manual features can in principle be integrated with lexical signs (though this possibility remains largely unexplored) and (ii) part of the system, namely the software that generates avatar animations based on phonetic representations (i.e., jasigning or a similar avatar engine) is not language-specific and can be used for any target sign language. the main disadvantage is that the initial step from text to phonetic representations involves a lot of manual work. given these considerations, we propose a modular approach, which employs both a gloss representation and a phonetic representation in going from a given input text to an avatar animation of the corresponding sign language utterance. as depicted in figure 2 , our modular approach breaks the translation process up into three steps: ## implementation the implementation choices we have made were driven by the specific objective to address the urgent need for a translation tool to aid healthcare professionals in communicating with deaf patients, ensuing from the current pandemic. two requirements follow from this objective: (i) the system had to be developed within a short time-frame, and (ii) high accuracy of the delivered translations was more important than broad approximate coverage. our aim has therefore not been to automate the entire translation process. in particular, automating the process of mapping input sentences to the corresponding ngt glosses using machine learning techniques would not have been feasible within a short time-frame, and would, even in the somewhat longer term, most likely result in an unacceptably low accuracy rate for use in a healthcare setting. 2 we therefore mainly focused on automating the phonetic encoding step, something that significantly reduces the manual labor needed in the overall translation pipeline. automating the mapping from glosses to phonetic representations has not been done in previous work on ngt @xcite and, to the best of our knowledge, not in work on other sign languages either. ## discussion as a first step in evaluating the system we have consulted extensively with a prominent member of the deaf community in the netherlands who has years of experience in advising organisations (especially museums and hospitals) on how to make their services more accessible to deaf people. based on these consultations and our own experiences in developing the system, we believe that the following considerations will be helpful in guiding further work in this direction. the main advantage of avatar technology over video translation is that it provides flexibility and scales up more easily. once a library of animated signs has been created, and a procedure to integrate non-manual grammatical markers has been implemented, translations for many sentences can be generated. this makes it particularly straightforward to provide translations for sentences that differ only slightly from each other (e.g., in a phrase indicating the time of day). a disadvantage, however, of avatar translations is that they can be less natural and more difficult to comprehend. while several empirical studies have reported promising comprehension rates for jasigning avatars (see, e.g., @xcite ), our consultant indicates that certain avatar translations offered by our system may be difficult to understand for some users. certain signs differ from each other only in rather subtle ways, and may be indistinguishable when produced by the jasigning avatar. certain facial expressions and body movements of the avatar are quite unnatural, which can add to the difficulty of understanding translations. certainly, the avatar's ability to display emotional empathy is very limited. this makes it undesirable to use avatar translations in situations where such empathy is required, as is often the case in medical settings. video translations, on the other hand, have their own benefits and drawbacks. they are better than avatar translations in terms of naturalness and comprehensibility, especially in the case of complex sentences. moreover, our consultant indicates that patients are likely to feel more comfortable watching a video of a human signer rather than an animated avatar in a situation in which their physical well-being is at stake. the main disadvantage of a video translation system is its inability to scale up efficiently. all translations have to be recorded separately, even ones that are almost identical. cutting and pasting video fragments of individual signs to create new sentences does not yield satisfactory results. a general advantage that a machine translation system (using either prerecorded videos, or an avatar, or both) may sometimes have over a human interpreter, especially in the healthcare domain, concerns privacy (see also @xcite ). a patient may receive sensitive information, and may not want this information to be known to anyone else (the deaf community in the netherlands is relatively small, which makes it relatively likely that a patient and an interpreter are personally acquainted). in this case, employing a human interpreter has a disadvantage (though this may of course be outweighed by the higher level of translation accuracy and empathy that can be provided by a human interpreter). it is important to emphasise that constructing sign language translations in either format is a time-consuming affair, though for different reasons. building a corpus of video translations is time intensive because every translation has to be recorded separately. for avatar translations, it takes time to encode individual signs. the latter are reusable, however, which becomes especially attractive as the number of required translations grows. however, the overall preference for one method over another is context-dependent: pros and cons should be carefully weighed in each specific context. finally, we note that one clear limitation of the current system is that it only translates text into sign language, not the other way around. this means, for instance, that if a doctor uses the system to ask a deaf patient an open-ended question such as how do you feel?, and if the patient gives an elaborate answer in ngt, the doctor will most likely not be able to understand the answer and our system will not be of help in this case. overcoming this limitation would require incorporating sign recognition technology (see, e.g., @xcite ), which has been beyond the scope of our project so far. note, however, that if a doctor uses our system to ask a more specific yes/no question such as do you feel dizzy?, then the answer in ngt-involving a head nod in the case of yes and a head shake in the case of no-will most likely be perfectly clear for the doctor even without a general understanding of ngt. thus, the current system is able to support relatively simple dialogues, but it is limited in scope and certainly does not (yet) offer a full-fledged dialogue system. we view it as a first, but critical step toward a more comprehensive solution. ## conclusion and future work we have investigated the potential of automated text-to-sign translation to address the challenges that the current pandemic imposes on the communication between healthcare professionals and deaf patients. we have motivated a modular approach to automated text-to-sign translation, and have built a first prototype system following this approach. we have discussed various prospects and limitations of the system. in future work, we intend to evaluate the system more thoroughly and to employ the modular approach motivated here to develop text-to-sign translation systems for different domains, e.g., for announcements at airports or railway stations, a use case which has already been explored to some extent for other sign languages @xcite . in addition, we also intend to improve the avatar visualisation.