Better Sign Language Translation with STMC -Transformer (2020)

Abstract: Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to extract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language glosses. This paper focuses on the translation system and introduces the STMC-Transformer which improves on the current state-of-the-art by over 5 and 7 BLEU respectively on gloss-to-text and video-to-text translation of the PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase of over 16 BLEU. We also demonstrate the problem in current methods that rely on gloss supervision. The video-to-text translation of our STMC-Transformer outperforms translation of GT glosses. This contradicts previous claims that GT gloss translation acts as an upper bound for SLT performance and reveals that glosses are an inefficient representation of sign language. For future SLT research, we therefore suggest an end-to-end training of the recognition and translation models, or using a different sign language annotation scheme.

## introduction communication holds a central position in our daily lives and social interactions. yet, in a predominantly aural society, sign language users are often deprived of effective communication. deaf people face daily issues of social isolation and miscommunication to this day @xcite . this paper is motivated to provide assistive technology that allow deaf people to communicate in their own language. in general, sign languages developed independently of spoken language and do not share the grammar of their spoken counterparts @xcite . for this, sign language recognition (slr) systems on their own cannot capture the underlying grammar and complexities of sign language, and sign language translation (slt) faces the additional challenge of taking into account the unique linguistic features during translation. figure 1 : sign language translation pipeline 1 . as shown in figure 1 , current slt approaches involve two steps. first, a tokenization system generates glosses from sign language videos. then, a translation system translates the recognized glosses into spoken language. recent work @xcite has addressed the first step, but there has been none improving the translation system. this paper aims to fill this research gap by leveraging recent success in neural machine translation (nmt), namely transformers. another limit to current slt models is that they use glosses as an intermediate representation of sign language. we show that having a perfect continuous slr system will not necessarily improve slt results. we introduce the stmc-transformer model performing video-to-text translation that surpasses translation of ground truth glosses, which reveals that glosses are a flawed representation of sign language. the contributions of this paper can be summarized as: ## methods despite considerable advancements made in machine translation (mt) between spoken languages, sign language processing falls behind for many reasons. unlike spoken language, sign language is a multidimensional form of communication that relies on both manual and non-manual cues which presents additional computer vision challenges @xcite . these cues may occur simultaneously whereas spoken language follows a linear pattern where words are processed one at a time. signs also vary in both space and time and the number of video frames associated to a single sign is not fixed either. ## model architecture for translation from videos to text, we propose the stmc-transformer network illustrated in figure 2 . ## experiments and discussions our models are built using pytorch @xcite and open-nmt @xcite . we configure transformers with word embedding size 512, gloss level tokenization, sinusoidal positional encoding, 2,048 hidden units and 8 heads. for optimization, we use adam @xcite with β 1 = 0.9, β 2 = 0.998, noam learning rate schedule, 0.1 dropout, and 0.1 label smoothing. we evaluate on the dev set each half-epoch and employ early stopping with patience 5. during decoding, generated unk tokens are replaced by the source token having the highest attention weight. this is useful when unk symbols correspond to proper nouns that can be directly transposed between languages @xcite . we perform a series of experiments to find the optimal setup for this novel application. we equally experiment with various techniques often used in classic nmt to slt such as transfer learning, weight tying and ensembling to improve model performance. for evaluation we use bleu @xcite , rouge @xcite and meteor @xcite . for bleu, we report bleu-1,2,3,4 scores and as rouge score we report the rouge-l f1 score. these metrics allow us to directly compare directly with previous works. meteor is calculated in addition as it demonstrates higher correlation with human evaluation than bleu on several mt tasks. all reported results unless otherwise specified are averaged over 10 runs with different random seeds. we organize our experiments into two groups: ## qualitative comparison example outputs of the g2t and s2g2t models ( we also provide sample g2t outputs on the aslg-pc12 corpus in appendix a.2. ## conclusions and future work in this paper, we proposed transformers for slt, notably the stmc-transformer. our experiments demonstrate how transformers obtain better slt performance than previous rnn-based networks. we also achieve new state-of-the-art results on different translation tasks on the phoenix-weather 2014t and aslg-pc12 datasets. a key finding is we obtain better performance by using a stmc network for tokenization instead of translating gt glosses. this calls into question current methods that use glosses as an intermediate representation, since reference glosses themselves are suboptimal. end-to-end training without gloss supervision is one promising step, though @xcite 's end-to-end model does not yet surpass their joint training model. as future work, we suggest continuing work on end-to-end training of the recognition and translation models, so the recognition model learns an intermediate representation that optimizes translation, or using a different sign language annotation scheme that has less information loss. bleu-4 ref: ähnliches wetter auch am donnerstag . (similar weather on thursday .) g2t: gleich wetter auch donnerstag (same weather on thursday) ähnliches wetter auch am donnerstag . 100.00 (similar weather on thursday .) s2g2t: gleich wetter donnerstag (same weather thursday) ähnliches wetter dann auch am donnerstag . 48.89 (similar weather then on thursday .) ref: der wind weht meist schwach aus unterschiedlichen richtungen . (the wind usually blows weakly from different directions .) g2t: wind schwach unterschied kommen (wind weak difference come) der wind weht meist nur schwach aus unterschiedlichen richtungen . 65.80 (the wind usually blows only weakly from different directions .) s2g2t: wind schwach unterschied (wind weak difference) der wind weht schwach aus unterschiedlichen richtungen . 61.02 (the wind is blowing weakly from different directions .) ref: sonnig geht es auch ins wochenende samstag ein herrlicher tag mit temperaturen bis siebzehn grad hier im westen . (the weekend is also sunny and saturday is a wonderful day with temperatures up to seventeen degrees here in the west .) g2t: wochenende sonne samstag schoen temperatur bis siebzehn grad region (weekend sun saturday nice temperature until seventeen degree region) und am wochenende da scheint die sonne bei temperaturen bis siebzehn grad . 13.49 (and on the weekend the sun shines at temperatures up to seventeen degrees .) s2g2t: wochenende sonne samstag temperatur bis siebzehn grad region (weekend sun saturday temperature until seventeen degree region) am wochenende scheint die sonne bei temperaturen bis siebzehn grad . 12.55 (on the weekend sun shines at temperatures up to seventeen degrees .) ref: es gelten entsprechende warnungen des deutschen wetterdienstes . (appropriate warnings from the german weather service apply .) g2t: ix schon warnung deutsch wetter dienst sturm koennen (ix already warning german weather service storm can) es bestehen entsprechende unwetterwarnungen des deutschen wetterdienstes . 38.26 (severe weather warnings from the german weather service exist .) s2g2t: dann ix schon warnung deutsch wetter sturm koennen es gelten entsprechende warnungen des deutschen wetterdienstes . 100.00 (then ix already warning german weather storm can) (appropriate warnings from the german weather service apply .) ref: richtung osten ist es meist sonnig . (it is mostly sunny towards the east .) g2t: ost meistens sonne (most east sun) im osten bleibt es meist sonnig . 43.47 (in the east it mostly stays sunny .) s2g2t: ost region meistens sonne im osten ist es meist sonnig . 80.91 (most region east sun) (in the east it is mostly sunny .) ref: am tag elf grad im vogtland und einundzwanzig grad am oberrhein . (during the day eleven degrees in vogtland and twenty one degrees in upper rhine .) g2t: am-tag elf vogel land (in-the-day eleven bird land) elf grad am oberrhein . 18.74 (eleven degrees in upper rhine .) s2g2t: elf vogel zwanzig (eleven bird twenty) am tag elf grad im vogtland und zwanzig grad im vogtland . 54.91 (during the day eleven degrees in vogtland and twenty degrees in vogtland .)