Findings of the Second WMT Shared Task on Sign Language Translation ( WMT - SLT 23) (2023)

Abstract: This paper presents the results of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23; https://www.wmt-slt.com/). This shared task is concerned with automatic translation between signed and spoken languages. The task is unusual in the sense that it requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation (MT). The task offers four tracks involving the following languages: Swiss German Sign Language (DSGS), French Sign Language of Switzerland (LSF-CH), Italian Sign Language of Switzerland (LIS-CH), German, French and Italian. Four teams (including one working on a baseline submission) participated in this second edition of the task, all submitting to the DSGS-to-German track. Besides a system ranking and system papers describing state-of-the-art techniques, this shared task makes the following scientific contributions: novel corpora and reproducible baseline systems. Finally, the task also resulted in publicly available sets of system outputs and more human evaluation scores for sign language translation.

## introduction this paper presents the outcome of the second wmt shared task on sign language translation (wmt-slt23). this shared task focuses on automatic translation between signed and spoken languages. our main goal is working towards including signed languages in nlp research @xcite . sign language translation requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation (mt). as a consequence, viable solutions need to consider a combination of natural language processing (nlp), computer vision (cv), computer graphics and animation techniques. we build on and extend the work done for the first shared task on sign language translation (wmt-slt22; @xcite . compared to the first edition, we • extended our competition to more languages (three language pairs instead of one), in this second edition of the shared task, we considered the following languages: swiss german sign language (dsgs), french sign language of switzerland (lsf-ch), italian sign language of switzerland (lis-ch), german, french, and italian. we offered four tracks: dsgs-to-german translation, german-to-dsgs translation, frenchto-lsf translation, and italian-to-lis translation. four teams participated in the task, which we consider a success. all teams submitted to the dsgs-to-german track, while there were no submissions to any of the tracks where a sign language is the target language. the remainder of this paper is organized as follows: ## background in recent years, sign language processing (slp) has emerged as a sub-area of natural language processing (nlp). within this field, automatic sign language translation (slt; or sign language machine translation, slmt) represents a more specialized discipline, aiming to develop technology that facilitates translation between sign languages and spoken or written languages, but also between sign and sign languages. however, the challenges related to slp and slt differ from those of nlp and mt for spoken languages in both range and complexity. due to the different modality, lack of structured, high-quality, high-quantity data, and the lack of nlp tools, joint efforts from the fields of sign linguistics and computational linguistics, computer science, machine learning, computer vision, 3d animation and others are needed in order to advance this field. in this section we give an introduction to sign languages ( §2.1) and describe the societal and academic relevance of slp ( §2.2). then we give an overview of slp in general ( §2.3) and of slt in particular ( §2.4) for a general motivation for a shared task involving sign languages see @xcite . ## tracks and submission procedure we offered four translation directions ("tracks"): translation from dsgs to german and vice versa, french to lsf-ch, and italian to lis-ch. for dsgs to german, submitted systems were ranked on a leaderboard. for all other directions, no automatic ranking was shown since automatic metrics of translation quality do not exist for sign languages as the target language. we provided baseline systems for both translation scenarios (translating from or to a sign language). we were prepared to provide human evaluation for all submitted systems, regardless of the translation direction or language pair. we deliberately did not limit the shared task to any particular kind of sl representation as input or output of an mt system. for dsgs-to-german translation, participants were free to use video frames, pose estimation, or something else. for german-to-dsgs participants were free to submit a video showing pose estimation output, an avatar, or a photo-realistic signer. participants had to submit their translation outputs on the ocelot platform foot_4 which displayed an unofficial public leaderboard based on automatic metrics. participants were allowed to make up to seven submissions and were asked to mark one of them as their primary submission. main outcome four teams (including one from northeastern university whose submission we consider a baseline) participated in our task. all of them submitted to the dsgs-to-german track, while there were no submissions for other translation directions. ## data for this task we provided separate training, development and test data. while the training data was available from the beginning, the test data has been released in two stages, starting with a release of the test sources only. ## data preprocessing for each data set described in §4 we provided videos and corresponding text in a spoken language. in addition, we included pose estimates (location of body keypoints in each frame) as a convenience. ## baselines and submitted systems in this section we describe the submissions to our shared task. in case there are substantial differences between the primary and secondary submissions of a team we opted to describe the primary submission here. at the time of writing this overview paper three out of four teams have given us detailed information about their submissions. the submissions are summarized in overall, the participating teams have diverse academic backgrounds, but their expertise is leaning towards nlp more than computer vision. all submitted systems are sequence-to-sequence models based on transformers @xcite . participants mostly chose to represent sign language data as video frames (using a visual feature extractor on the encoder side). only the baseline system opted for mediapipe pose features instead. two systems, by knowcomp and ttic, are unconstrained because their visual or spoken text components are pretrained on other datasets. their approaches are best summarized as a combination of visual embeddings and pre-trained language models. ttic used additional monolingual video data from openasl for pretraining, and no submission used monolingual text in a spoken language. two teams have published their code, with another team planning to do so in the future. ## evaluation protocols we performed both a human ( §7.1) and an automatic ( §7.2) evaluation of translation quality. our final system ranking is based on the human evaluation only. ## conclusion and future directions in this paper we present the second wmt shared task on sign language translation (wmt-slt23). we consider automatic sign language translation, and sign language processing in general, to be of wide public interest and to have a high potential impact in a societal and academic sense ( §2). compared to last year we ran our shared task for three language pairs instead of one, we distributed considerably more training data (albeit with a higher amount of noise) and we put more emphasis on scenarios where sign languages are the target language. four teams participated in the second edition of the shared task. overall, we observed low system performance with an average human evaluation score of about 1 out of 100 (for the best-performing system), which is not usable in practice. the main reasons for this outcome are a lack of usable training data, a modality gap (considering that most existing work in mt is based on text) and a lack of basic nlp tools specifically for sign languages. future of the shared task after two successful iterations the shared task is now well established, in the sense that suitable protocols are in place for human and automatic evaluation, reasonable baseline systems exist, as well as several training corpora and official wmt test sets. so far our shared tasks have certainly helped to paint a more realistic picture of the translation quality of state-of-the-art systems, but they have not led to any major technical innovation. this may be because technologies more fundamental than machine translation do not exist for sign languages, or are not reliable enough. for this reason we will consider running shared tasks on more fundamental problems in slp such as alignment, segmentation, or automatic filtering of parallel corpora. in the future we could also try to shift the focus away from interpreted news broadcast material as the basis for training and test data. a major challenge to overcome is that interpreted material is available in larger amounts, while signing produced by conventional, off-line translation or produced by native signers is harder to come by. nevertheless, using non-interpreted material largely avoids alignment shifts in the training data and leads to higher scores for the human translations of the test data, among other advantages.