Findings of the First WMT Shared Task on Sign Language Translation ( WMT - SLT 22) (2022)

Abstract: This paper presents the results of the First WMT Shared Task on Sign Language Translation (WMT-SLT22).This shared task is concerned with automatic translation between signed and spoken languages. The task is novel in the sense that it requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation (MT).The task featured two tracks, translating from Swiss German Sign Language (DSGS) to German and vice versa. Seven teams participated in this first edition of the task, all submitting to the DSGS-to-German track. Besides a system ranking and system papers describing state-of-the-art techniques, this shared task makes the following scientific contributions: novel corpora, reproducible baseline systems and new protocols and software for human evaluation. Finally, the task also resulted in the first publicly available set of system outputs and human evaluation scores for sign language translation.

## introduction this paper presents the outcome of the first wmt shared task on sign language translation (wmt-slt22). the focus of this shared task is automatic translation between signed and 1 https://www.wmt-slt.com/ 2 in this paper we use the word "spoken" to refer to any language that is not signed, no matter whether it is represented as text or audio, and no matter whether the discourse is formal (e.g. writing) or informal (e.g. dialogue). sign language translation requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation (mt). as a consequence, solutions need to consider a combination of natural language processing (nlp) and computer vision (cv) techniques. in the field of sign language mt there is a general lack of suitable and freely available datasets and code. for this reason it was necessary for us to build and distribute novel training corpora and we also published reproducible baseline code. likewise, existing protocols and toolkits for human evaluation had to be adapted to support sign languages. in this first edition of the shared task we considered one language pair: swiss german sign language (dsgs) and german. we offered two tracks: dsgs-to-german translation and germanto-dsgs translation. seven teams participated in the task, which we consider a success. all teams submitted to the dsgs-to-german track, while there were no submissions to the german-to-dsgs track. the remainder of this paper is organized as follows: ## background we consider sign language processing (slp) a sub-area of natural language processing (nlp), and automatic sign language translation (slt), a more narrowly focused discipline within slp. we first give an introduction to sign languages ( §2.1) and describe the societal and academic relevance of slp ( §2.2). then we give an overview of slp in general ( §2.3), of slt in particular ( §2.4) and finally motivate this shared task ( §2.5). ## tracks and submission procedure we offered two translation directions ("tracks"): translation from swiss german sign language (dsgs) to german and vice versa. translation from dsgs to german was our primary translation direction in the sense that submitted systems were ranked on a leaderboard and we provided baseline systems. systems translating from german to dsgs were not ranked on the leaderboard while the task was running, but we still encouraged participants to submit such systems. we were prepared to provide human evaluation for all submitted systems, regardless of the translation direction. we deliberately did not limit the shared task to any particular kind of sl representation as input or output of an mt system. for dsgs-to-german translation participants were free to use video frames, pose estimation or something else. for german-to-dsgs participants were free to submit a video showing pose estimation output, an avatar or a photo-realistic signer. participants submitted translations on the ocelot platform foot_2 which has a public leaderboard. we modified ocelot slightly in order to disable automatic metrics on the leaderboard for german-to-dsgs, since currently no automatic metrics exist for sl output. participants were allowed to make up to seven submissions, one of them the primary submission. main outcome seven teams (including one from the university of zurich whose submission we consider a baseline) participated in our task. all of them submitted to the dsgs-to-german track, while there were no submissions for the second translation direction. srf focusnews total direction episodes segments episodes segments segments training (both) 29 7071 197 10136 17207 development (both) 1 287 3 133 420 test dsgs-to-german 1 242 5 246 488 german-to-dsgs 1 183 5 228 411 ## data for this task we provided separate training, development and test data, where the training data was available from the beginning while the development and test data were released in several stages. necessity of creating training data the data we provided are new corpora that we built and published. this was necessary because existing datasets for sl machine translation did not meet our requirements. existing datasets either have a license that is too restrictive, are not parallel enough in the sense of being only "comparable corpora", are too small or have a very limited linguistic domain. for example, the most widely used dataset in sl machine translation research, phoenix (introduced in §2.4), has a size of 8k sentence pairs and contains only weather reports. following the long history of wmt shared tasks for spoken language machine translation @xcite , we opted for data that contains general news, hence a more open domain. ## data preprocessing for each data set described in §4 we provided videos and corresponding subtitles. in addition, we included pose estimates (location of body keypoints in each frame) as a convenience. ## baseline and submitted systems in this section we describe all submissions to our shared task. in case there are substantial differences between the primary and secondary submissions of a team we opted to describe the primary submission here. at the time of writing this overview paper six out of seven teams have given us detailed information about their submissions. the submissions are summarized in overall, the participating teams have diverse academic backgrounds, most of them combine computer vision and nlp expertise. all submitted systems are sequence-to-sequence models based on transformers @xcite . participants chose to represent sign language data as either video frames (using a visual feature extractor on the encoder side) or pose features, with no clear majority in this regard. two systems, by lattic and msmunich, are unconstrained because their visual encoder component is pretrained on wsasl @xcite or is an existing model taken from varol ## evaluation protocols we performed both a human ( §7.1) and an automatic ( §7.2) evaluation of translation quality. our final system ranking is based on the human evaluation only. ## conclusion and future directions in this paper we present the first wmt shared task on sign language translation (wmt-slt22). we consider automatic sign language translation, and sign language processing in general, to be of wide public interest and to have a high potential impact ( §2). seven teams participated in this first edition of the shared task. overall, we observed low system performance with an average human evaluation score of about 4 out of 100 (for the bestperforming system), which is not usable in practice. the main reasons for this outcome are a lack of usable training data, a modality gap (considering that most existing work in mt is based on text) and a lack of basic nlp tools specifically for sign languages. future of the shared task future iterations of the shared task could introduce more language pairs and larger training data. since this year all submissions are signed-to-spoken systems, the shared task could also focus more on sign language generation going forward. furthermore, we will consider introducing additional mt-related tasks such as a sign language version of the metrics task. this perhaps requires a better distribution of human evaluation scores, as our current set of scores very much focuses on both ends of the score spectrum (we do not have many mid-range scores). finally, future human evaluation experiments for spoken-to-signed translation could be run differently than explained in this paper. namely, for campaigns where a sign language is the target language the evaluation could be reference-based instead of source-based. the advantage of this change would be that deaf evaluators can perform this evaluation, instead of hearing interpreters for whom in this case the target language is not their first language.