Gloss-Free End-to-End Sign Language Translation (2023)

Abstract: In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations. Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain coverage of translation datasets, thus handicapping real-world applications. To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.

## introduction sign language is a type of visual language mainly used by the community of deaf and hard of hearing. it uses a combination of hand gestures, facial expressions, and body movements to convey the message of the signer. sign languages are not simple transcripts of the corresponding spoken languages. they possess unique grammar structures and have their own linguistic properties. according to the world federation of the deaf, there are over 70 million deaf people around the world. the study of automated sign language processing can facilitate their day-to-day life. in this paper, we study the task of sign language translation (slt), which translates the sign videos into the corresponding spoken language. glosses are the transliteration system for sign language. they serve as an intermediate representation of the signs. however, the vocabulary of gloss does not align with the spoken language nor does the order of the glosses. unlike translation between two spoken languages, the number of frames in a sign video is much larger than the number of words in the spoken translation. this imposes a unique challenge for slt. models need to learn a clustering of the frames into gloss-level representation before they can translate the tokens. previous methods solve this problem in two major ways, i.e., pre-train the visual backbone with gloss @xcite or jointly train on both translation and continuous recognition task @xcite with an additional ctc loss @xcite . these methods have been proven effective, but the reliance on gloss annotations makes them hard to apply to more realistic scenarios. as gloss annotations require expert knowledge to make and often are limited in quantity or coverage of domains. like the most frequently used phoenix14t dataset @xcite that focuses on weather reports or the keti dataset @xcite that dedicates to emergencies. datasets like openasl @xcite and how2sign @xcite provide more samples but there are no gloss annotations for training. motivated by these observations and the availability of large-scale slt datasets, we designed a new framework that is gloss-free throughout the entire process and train the visual backbone jointly in an end-to-end manner. the core idea of our method is illustrated in figure 1 , we extract conceptual words from the ground truth spoken translation to be used as a weak form of intermediate representations. this exploits the shared semantics between signs and text. though the extracted words might be different from the glosses, the concept expressed by these words should exist in both sign and text. we treat these words as conceptual anchors (ca) between the two modalities. specifically, we use pre-trained glove embeddings @xcite as the initialization of these anchors. then they are treated as the query of cross attention against the encoded visual features. as illustrated in figure 1 , the query attend to each visual feature across the temporal dimension to calculate the similarity between the query and the visual features. with these similarities as weights of pooling, we get the attended visual features. the order of the most relevant features from the signing video does not match the order of the queries in the translation, so ctc is not viable in this situation. instead, we impose the conceptual constraints in a contrastive manner. for each anchor word, we treated samples containing such words as positive and vice versus. for example, for the word identities in figure 1 sample b is positive and sample a is negative. query results for these positive and negative pairs along with the anchor word form a triplet, among which we conduct a hinge-based triplet loss. this process forces the visual2text encoder to learn the relation between different frames that is part of one sign. in all, our contribution can be summarized as: ## method given a sign video @xmath0, f 2 , . . . , f t } of t frames, our objective is to generate a spoken language sentence @xmath1, w 2 , . . . , w l } of length l under the conditional probability p(y |x). generally speaking, it holds that t ≫ l. this trait makes the task of sign language translation harder compared to the translation task between different spoken languages. past methods mostly use gloss supervision via ctc loss to impose an indirect clustering on the processed visual tokens. gloss annotation provides the relative order and type of the signed word, not including the boundary between sign words. however, the making process of gloss annotations is labor-intensive, thus often in limited quantities. this restricts the scale of slt datasets with gloss annotations. to this end, we are motivated to design a framework that can be trained only on sign video and translation pairs. to reduce processing load and translate longer sign vidoes, we extract pose landmarks x @xmath2, p 2 , . . . , p t } offline from x and use it as the input of our framework. in this section, we first give an overview of the proposed gloss-free end-to-end sign language translation framework, with details about each component. then we elaborate on our approach aims to provide similar supervision to gloss in a self-supervised manner. ## experiments in this section we provide details on the datasets and translation protocol we follow. along with quantitative and qualitative results on different benchmarks. we also give a deep analysis about the design components of our method. ## conclusion in this paper, we propose a novel gloss-free end-toend framework for sign language translation. design an intermediate representation that can act as a fill-in when gloss annotation is not available. we exploit the shared semantics between sign and text, by extracting common conceptual words from the spoken translation. the model is trained end-toend including the visual backbone, no gloss is used in training or pre-training, and achieves state-ofthe-art performance on the largest sign languages translation dataset publicly available.