Translation Quality Evaluation of Sign Language Avatar (2024)

Abstract: “Sign Language Avatar technology aims to create virtual agents capable of communicating with deaf individuals through sign language, similar to the text dialogue agent ChatGPT but focusing on sign language communication. Challenges in sign language production include limited dataset sizes, information loss due to reliance on intermediate representations, and insufficient realism in generated actions. In this event, we particularly focus on the ability of the Sign Language Avatar to translate spoken language text into sign language that is easily understood by deaf individuals. As the first sign language avatar event held by the China National Conference on Computational Linguistics(CCL), this event attracted wide attention from both industry and academia, with 14 teams registering and 10 of them submitting their system interfaces on time. We provided a dataset consisting of 1074 text-video parallel sentence pairs for training, and the evaluation team comprised proficient Chinese sign language users and professional sign language translators. The scoring method employed a comprehensive evaluation based on multiple metrics, focusing primarily on sign language grammar accuracy, naturalness, readability, and cultural adaptability. The final scores were determined by considering performance across these four aspects. The final scores, taking into account these four aspects, showed that four teams demonstrated good readability, with Vivo Mobile Communication Co., Ltd. ranking first with a score of 3.513 (out of a full score of 5), leading the baseline model by 1.394 points. According to the analysis of the results, most teams used the traditional method of converting text into Gloss sequences before generating sign language. Additionally, some teams experimented with emerging methods, including gloss-free end-to-end training and Large Language Model(LLMs) prompt learning, which also achieved promising results. We anticipate that this event will promote the development of sign language avatar technology and provide higher-quality communication tools for the deaf community. For more information on this task, please visit the website of the CCL24-Eval: Translation Quality Evaluation of Sign Language Avatar Task.”

## introduction sign language, a rich and complex form of communication with its own unique vocabulary and grammar, is used by over 70 million deaf and hard of hearing people worldwide. unlike spoken languages, sign language emphasizes body language, incorporating hand shapes, movements, positions, and palm orientations, as well as non-manual elements like body posture and facial expressions @xcite . despite its widespread use and significance, the distinct differences and unique modes of expression in sign language pose challenges to its dissemination and understanding. the purpose of this evaluation competition is to assess sign language avatars that can translate spoken text into sign language, enhancing comprehension for deaf or hard of hearing individuals who use sign language. recently, sign language research has become an active area within computer vision (cv) and natural language processing (nlp) @xcite @xcite , particularly in sign language recognition (slr) @xcite @xcite and translation (slt) @xcite @xcite . however, research publications in sign language production (slp), which are closely related to this evaluation task, are scarce @xcite . the goal of slp is to translate spoken or written content into sign language, making it accessible for the deaf. slp faces challenges such as capturing detailed hand and body movements and dealing with unique visual semantics and grammar. additionally, limited datasets, difficulties in simulating sign details, and technological constraints in producing contextually accurate signs complicate slp tasks @xcite . early sign language processing (slp) methods relied on one-to-one gloss-sign correspondences, producing only isolated sign words @xcite . subsequent attempts employed machine translation techniques to translate spoken text into continuous gloss sequences, which were then converted to sign language gestures @xcite . @xcite . however, this approach often failed to capture contextual information, leading to semantic losses. a more recent development is the end-to-end slp method, which directly translates text into sign language videos without intermediate gloss, showing significant improvements with larger data volumes @xcite . regarding avatar technologies, initial methods employed 2d or 3d skeletal models extracted from videos @xcite . recent advancements have focused on generating sign language from these models through rendering techniques @xcite @xcite . the use of pretrained avatar models like smpl-x has also been explored for enhanced sign language representation @xcite . in china, sign language avatars are gaining traction. initiatives like zhipu's "ai sign language classmate" and vivo's "sign language translator" highlight a deep understanding and proactive response to the deaf community's communication needs. these avatars are now visible at events such as sports games and news programs. with ongoing technological progress, sign language avatars are expected to play a vital role, enhancing communication for the deaf community. this evaluation assesses the effectiveness of sign language avatars by using recorded videos and multiple-choice questions for feedback. we expect this round to enhance the avatar team's understanding of the deaf community's needs and pinpoint improvement areas. ## task description the purpose of this evaluation is to assess the naturalness and accuracy of sign language avatars translating chinese into chinese sign language(csl), ensuring that the translations adhere to sign language grammatical rules and are understandable and acceptable to the deaf community. during the evaluation, we collect and construct a rich corpus covering various common scenarios for teams to train on. additionally, we design a series of test sentences, four in total, each accompanied by the participating team's sign language avatar videos, along with four multiple-choice questions and optional evaluations. each multiple-choice question provides four options, allowing evaluators to choose during the evaluation process to determine which option best matches the performance of the sign language avatar. ## evaluation method in the fields of slr and slt, common metrics such as accuracy, recall, word error rate (wer), and translation-specific indicators like bleu, cider, rouge, meteor often require comparison with ground truth videos, which are difficult to obtain @xcite . therefore, automated machine evaluation is not used. additionally, the diversity in styles of sign language avatars complicates the use of a standardized automated evaluation method. instead, human evaluation is preferred due to its flexibility, accuracy, and thorough analytical capabilities, making it the method of choice for assessing the naturalness and fluency of sign language translations. ## registration and evaluation results in this evaluation event, 14 teams, including 9 from academia and 5 from industry, registered and applied for the corpus, highlighting widespread interest in sign language avatar technology. by the submission deadline, 10 teams had submitted their interfaces. the event was supported by 18 professional evaluators from the chinese association of the deaf's committee for sign language research and promotion, split into an expert group and a collection group. the expert group, proficient in sign language, established benchmarks and facilitated comparisons across different groups. meanwhile, the collection group, composed of members from 12 different regions, was responsible for identifying regional variations in sign language. the event also included 9 general evaluators-comprising university students, working adults, and retirees. a notable focus was placed on 24 experienced sign language users over the age of 35, 18 of whom are native users. their extensive use and deep understanding of sign language provided crucial insights into its natural fluency, accuracy, and cultural nuances, thereby ensuring the evaluation's reliability and enhancing the assessment of the avatars' capability to capture nuanced and emotional expressions in sign language. after preliminary screening by the evaluation team, we found that the system interfaces submitted by 4 of the teams listed in vivo mobile communication co., ltd. (vivo) topped the competition with a score of 3.513, outperforming bjtt by 1.394 points. vivo's success was credited to its pre-trained multilingual model fine-tuning, back-translation for data augmentation, and strategies to smooth animation, leading to high marks in accuracy, naturalness, and readability from general users.gobetter accesstech (suzhou) co., ltd. (gbat) placed second with a score of 2.447, performing slightly above the baseline and noted for its accuracy and naturalness by general users. qingdao heshi digital technology co., ltd. (qdhdt) ranked fourth, facing difficulties in accuracy and readability in the collection group. in the evaluation, vivo excelled, while gbat, bjtt, and qdhdt showed potential for improvement in accuracy and cultural adaptability. future efforts should aim at enhancing sign language avatar performance and user satisfaction. to gather specific feedback, optional evaluations were added to the questionnaire for each avatar. vivo's avatar was praised for its clear, natural movements and coordinated expressions, though it could improve in expression richness and contextual adaptability. gbat's avatar stood out for clear hand-c c l 2 0 2 4 shapes but needs better movement naturalness, expression richness, and sign language fluency. bjtt's avatar was noted for clear hand movements and expressions but required improvements in accuracy and naturalness. qdhdt's avatar was recognized for good visual design and movement coordination but needs better vocabulary accuracy, expression fluency, and non-manual element depiction. in summary, each team has made certain progress in the development of sign language avatars, but they also face challenges in accuracy, naturalness, and cultural adaptability. future research and development should focus on these challenges to continuously optimize the technology, allowing sign language avatar to naturally display sign language translations. ## overview of methods used by participating teams currently, the participating teams generally adopt the text-gloss-video technology route. this method first uses a translation model to convert spoken text into gloss. then, it retrieves corresponding videos from a gloss-video database and uses smoothing techniques to produce fluid, continuous sign language videos. figure 1 shows the basic process of generating avatar for sign language translation below, we will discuss the methods from two aspects: text2gloss translation and avatar synthesis. ## conclusion and future prospects this evaluation involved 14 teams-9 from academia and 5 from industry-highlighting significant interest in sign language avatar technology. by the submission deadline, 10 teams had successfully submitted their interfaces, with 4 advancing past preliminary screening due to their effectiveness and readability. vivo stood out by ranking first, excelling across expert, collection, and general groups. despite all teams following a text-gloss-video route, varied approaches were seen in text2gloss translation and avatar synthesis, including fine-tuning pre-trained models, utilizing llm prompts, ensuring smooth transitions, and implementing end-to-end synthesis. evaluators consistently found that while some avatars showcased fluent, clear, and accurate sign language expressions, many still had issues like missing information, inaccurate gestures, and stiff movements. these shortcomings impact the accuracy, readability, and overall viewing experience. future work should concentrate on refining these technologies to better serve sign language users, aiming for continuous improvement and enhanced user satisfaction. we look forward to sign language avatars serving the deaf community better in the future and providing them with a more convenient and friendly communication experience. tong sun, @xcite . adaptive simultaneous sign language translation with confident translation length estimation. in proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pages 372-384.