Online Evaluation of Text-to-sign Translation by Deaf End Users: Some Methodological Recommendations (short paper) (2021)

Abstract: We present a number of methodological recommendations concerning the online evaluation of avatars for text-to-sign translation, focusing on the structure, format and length of the questionnaire, as well as methods for eliciting and faithfully transcribing responses

## introduction there is no generally accepted methodology for evaluating the comprehensibility of avatars for text-to-sign translation, let alone for doing so online. evaluation procedures designed in previous work generally involve on-site interaction between experimenters and participants @xcite @xcite @xcite , though see @xcite for exceptions). the covid-19 pandemic has made it necessary to turn to online procedures, which come with additional methodological challenges. on the bright side, such online procedures, if effective, may also have benefits in a post-covid-19 world. we report work in progress on the evaluation of a recently developed prototype system for translating sentences that frequently occur in a healthcare setting, particularly ones that are used in the diagnosis and treatment of covid-19, from dutch into dutch sign language (ngt). the system itself is described in some detail in @xcite . here, we share some of the lessons we have learned in designing a methodology for evaluating this system online. some of these lessons specifically concern the online nature of the evaluation procedure, but others are more general and would apply to on-site evaluation as well. in the process of designing our methodology, we held a feedback session with seven deaf researchers at various career stages, all users of ngt and familiar with (socio-)linguistic experimental methodologies, in which we discussed a preliminary setup of the evaluation procedure. after incorporating feedback from this session we carried out a pilot study with five participants (all consider ngt (one of) their mother tongue(s)). while the feedback session had already led to important improvements of the design, the pilot study brought out a number of further methodological issues, serious enough to render the results essentially uninterpretable. to address these issues, we have further adapted the design of the evaluation procedure, which is described in more detail in section 4. the adapted procedure is already in use. although it is too early to present quantified results, it is clear that the methodological adjustments we made are effective, as the issues experienced in the pilot study are no longer present. by sharing the lessons we have learned from the feedback session and the pilot study, we hope that other researchers evaluating avatars for text-to-sign translation in the future, be it online or on-site, will be able to avoid making the same mistakes we did initially and arrive at a suitable evaluation procedure more directly. the extended abstract is organised as follows: section 2 outlines the goals of our evaluation procedure, section 3 discusses the design of the questionnaire, section 4 turns to issues concerning elicitation and transcription of participants' responses, and section 5 concludes. ## evaluation goals as mentioned above, the system we are evaluating translates sentences that frequently occur in a healthcare setting, especially in the diagnosis and treatment of covid-19, from dutch to ngt. for instance, a healthcare professional may enter the sentence 'gebruikt u medicijnen?' ('do you use any medications?') and the system will produce a translation in ngt. some translations have been pre-recorded on video, others are displayed by means of an avatar, making use of the jasigning avatar software @xcite . we are mainly interested at this point in evaluating the comprehensibility of these avatar translations. more specifically, our primary goal currently is to answer the following three questions: measuring individual sign recognition alongside sentence comprehension provides us with additional insights as to why a sentence is (mis)understood, and highlights specific areas for improvement. for example, some participants may recognise individual signs yet misidentify the meaning of a sentence (or vice versa). a secondary goal (equally important in general, but less central in the present study) is to find out how members of the deaf community in the netherlands view avatar technology for sign language translation, and the potential application of such technology in various domains (cf., @xcite @xcite . ## design of the questionnaire we will comment on three design features of the questionnaire: its structure, format, and length. structure in evaluating the comprehensibility of avatar translations, it is crucial to have a standard of comparison. suppose, for instance, that we find that users correctly recognise 75% of the individual signs that the avatar produces. this information in itself does not tell us much. is this a positive result, or a negative one? we cannot tell as long as we do not have a baseline. this concern is particularly relevant here for two reasons. first, some of the translations involve medical terms (e.g., 'intravenous drip') which may not be familiar to all participants and therefore poorly recognised even if they are signed correctly by the avatar. second, there is considerable regional and intergenerational variation in ngt, which means that certain signs may be familiar to ngt users from one region/generation, but not to users from another. to address this issue, we compare the comprehensibility of avatar translations to that of video recordings of the same sentences signed by a deaf signer. the core of the questionnaire, then, consists of two parts: one that assesses the comprehensibility of avatar translations, and one that does the same for baseline videos signed by a deaf signer. the avatar part precedes the baseline part, to avoid a learning effect when assessing the avatar. after these two core comprehension parts, the questionnaire inquires about the participant's general perception of avatar technology for sign language translation, and their views on the potential application of such technology in various domains. in addition, the questionnaire includes an introductory part (informed consent, information about the structure of the questionnaire, and some questions about the language background of the participants) and a closing part which checks whether the questions in the questionnaire were clearly posed and could be responded to in a satisfactory way. format all questions and instructions in the questionnaire are presented both in ngt (by means of pre-recorded videos) and in written dutch. the person giving instructions and asking questions in the videos is a deaf ngt user, distinct from the signer in the baseline video translations discussed above. participants are given a choice as to whether they want to watch the questions/instructions in ngt, read the dutch text, or both. most participants preferred the videos, but some chose to read. several participants explicitly commented that they appreciated having this choice. some explicitly commented that they found it pleasant that the person in the video was a deaf signer. all participants reported that the questions and instructions were clear. length we aim to keep sessions under 45 minutes to avoid concentration difficulties. this seems to work well-participants appear to be focused all the way through. what we have learned, however, is that this means that the number of test sentences has to be kept quite low. our initial plan was to present 24 avatar translations and 24 corresponding baseline videos, but this turned out not to be feasible at all. we now present 12 avatar translations and 12 baseline videos, and this generally fits the 45 minute window. another lesson we learned is that, in order to measure the extent to which the individual signs in a sentence are correctly recognised, the length of test sentences should be restricted to around 7 signs. it is well-known that most adults cannot store more than 7 items in their shortterm memory @xcite . indeed, when we presented longer sentences in our pilot study and asked participants to list the individual signs in these sentences, they had great trouble reproducing the right sequence even if they had fully understood the meaning of the sentence as a whole. since our aim here is not to test participants' short term memory capacity but just comprehension, we have decided to keep all test sentences relatively short (4-7 signs). in the evaluation sessions we are currently running this appears to work well. ## eliciting and transcribing responses for a proper evaluation procedure (ensuring that responses are correctly understood by all parties), the responses that participants provide in ngt have to be simultaneously interpreted into dutch. this is not straightforward if, as in our case, the experimenters are not fluent signers: one is a new signer using ngt on a daily basis and the other has taken a number of ngt courses but does not use the language daily. the online setting makes this issue even more acute. we are addressing this issue as follows. during an evaluation session, the participant does not open the questionnaire on their own computer. rather, one of the experimenters opens the questionnaire on their computer and shares their screen. an experienced sign language interpreter, with high awareness of regional and generational variation, is present as well. before getting started, we make sure that both the questionnaire and the sign language interpreter are visible for the participant. participants answer questions in ngt, i.e., they do not need to type anything themselves. figure 1 : an example of an item in the questionnaire assessing comprehension of the avatar. for illustration, the questions are formulated in english here; in reality they are in dutch and accompanied by instructions in ngt. the interpreter interprets the answers into dutch, one of the experimenters types the verbatim interpretation visible for the participants, so that they can check that their responses are properly interpreted. typically, participants correct interpretations a few times each session and the transcript is then changed accordingly. in other cases, participants typically indicate explicitly that the interpretation is correct (usually with a confirming head nod after the transcription appears on the screen). finally, we turn to the issue of how to properly assess the extent to which participants recognise the individual signs in the avatar translations. this issue is more specific than the ones discussed above, but needs to be carefully addressed in any study that evaluates the comprehension of signing avatars. indeed, the data obtained in our pilot study was uninterpretable mostly because we had not addressed this issue carefully enough. in the pilot study, we gave participants instructions (both in ngt and in written dutch) that they would be shown a video of an avatar signing a sentence and would then be asked three questions (i) what are the individual signs in the sentence? (ii) what is the meaning of the sentence as a whole? and (iii) how clearly was the sentence signed? next, we showed participants a video, and then questions (i)-(iii), in dutch. responses to the first two questions (individual signs and sentence meaning) had to be entered in a textfield, while responses to the third question (clarity) had to be given on a scale from 0 to 10. the problem was that participants generally (with very few exceptions in fact) immediately started answering the second question. it was not sufficiently clear what was intended with the first question. we took two measures to address this issue. first, rather than a single textfield for listing the individual signs in the sentence, we now present a separate textfield for each sign and label these textfields as 'sign 1', 'sign 2', etc (see figure 1 ). second, when giving instructions beforehand we now present two examples: one of an avatar translation with 'gloss subtitles', where the item in the gloss that corresponds to the current sign gets highlighted in yellow, and a second example of an avatar translation with question marks in the subtitles (see figure 1 ). during the first sign the first question mark is highlighted, during the second sign the second question mark etc. together with this second example video we also show the first two questions (concerning individual signs and sentence meaning, respectively), and exemplify what a possible response could look like. the question mark subtitles are also included in the actual test items. these two revisions of the design appear to achieve the intended effect: in the evaluation procedure we are currently running participants so far respond to all questions as intended. ## conclusion in this extended abstract, we have shared a number of methodological lessons we have learned in designing and piloting an online procedure to evaluate the comprehensibility of an avatar for text-to-sign translation. we hope that the recommendations we have made concerning the structure, format, and length of the questionnaire and test items, as well as the elicitation and transcription of responses will be helpful for other researchers in designing their evaluation procedures. in the long run, we hope that they contribute to the development of more standardised methodologies and best practices for the evaluation of sign language technology.