Paper: Unsupervised Sign Language Translation and Generation (ICLR-2024)

Reviewer comment (Experiments):
The performance is **too bad**. Although it may not be the authors' fault (maybe the dataset is too difficult), the poor performance make the comparison less convincing. Experiments on other widely-adopted benchmarks, e.g., Phoenix-2014T and CSL-Daily, shall be considered.

Author response:
1.2 Other dataset results
For the results on other dataset. Thank you for suggestion. Given that the amount of PHOENIX-2014T and CSL-Daily is too small for training unsupervised model, we do not adopt this results. In the future, we will explore and experiment USLNet on other large sign language dataset, such as OpenASL.

Reviewer comment (Theory):
It seems that there is a factual error in the sliding window based aligner. The text and video are not monotonically aligned. In fact, only video and glosses are monotonically aligned, e.g., 1-10 frames for the first gloss, and 11-20 frames for the second gloss. Thus, it is questionable for the design of the sliding window-based aligner.

Author response:
Q2： Align problem in slide window aligner.  
Yes, video and glosses are monotonically aligned. However, because BOBSL does not have human-evaluated sentence-level glosses annotations, we suggest that video and text are roughly aligned and align video with text. Morepver, we incorporate validation of BOBSL that video and text are roughly aligned. 

To address this issue, we must first obtain the golden sign order. In the sign language domain, text-based interpretations of signs are referred to as glosses[1]. However, BOBSL does not provide sentence-level human-annotated glosses.Therefore, we utilized the automatic gloss annotation released in [2]. This gloss annotation consists of word-level annotations, presented as [video name, global time, gloss, source, confidence]. We converted these gloss annotations into sentence-level annotations and assessed the consistency between the gloss (sign) and text orders.

|                                                | Consistency |
|----------------------------------------------------|-------------|
| **Strictly Consistent**                            | 0.83        |
| **Majority Consistent with two gloss in disorder** | 0.87        |
| **Main Consistent with three gloss in disorder**   | 0.91        |

We will add the experimental results in the revisd version. 

[1] Núñez-Marcos A, Perez-de-Viñaspre O, Labaka G. A survey on Sign Language machine translation[J]. Expert Systems with Applications, 2023, 213: 118993.  
[2] Momeni L, Bull H, Prajwal K R, et al. Automatic dense annotation of large-vocabulary sign language videos[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 671-690.

Reviewer comment (Writing):
The descriptions for the process of the aligner should be more clear. The current form is a bit difficult to understand.

Author response:
Q3: Descriptions problem for the process of the aligner.   
Thank you for your advice. We will simplify our illustration of the slide window aligner in the revised version. We will add a discussion of dual learning in the related works.

Reviewer comment (Evaluation):
Although I like the idea of using pretrained large-scale models and unsupervised learning, I'd expect quantitative results to back up the benefits of employing these ideas. Sadly, the presented results does not suggest the presented approach to be "working" (~0.2 BLEU-4 score on BOBSL, while the state of the art is above 2 https://openaccess.thecvf.com/content/ICCV2023W/ACVR/papers/Sincan_Is_Context_all_you_Need_Scaling_Neural_Sign_Language_Translation_ICCVW_2023_paper.pdf).

Author response:
Q1：Other results on BOBSL and acknowledge that the challenging of BOBSL.  
We greatly appreciate your contribution of state-of-the-art (SOTA) results and insights emphasizing the challenges associated with BOBSL. For the work[1], the best model using the context and spotting information achieves 2.88 B@4 points. However, the model can not obtain context and spotting information in our setting. To make a fair comparison, the appropriate baseline for comparison should be B @ 4=1.27 for video-to-text, as depicted in Table 4 in [1], given the fact that USLNet also do not have extra knowledge.  
[1] Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

Reviewer comment (Experiments):
Experiment on different datasets, such as Phoenix-2014T, or the larger OpenASL and YoutubeASL, which have more state-of-the-art results, hence more data points to gauge the performance/benefits of the proposed approach.

Author response:
Q2：Experiment on different datasets.  
Thank you for suggestion. We follow your suggestion to conduct the experiments and will report the results of Open-ASL in the next few days, given that pretraining and unsupervised training cost lots of time. By the way, because the amount of PHOENIX-2014T is too small for training unsupervised model, we do not adopt this results.

Reviewer comment (Experiments):
Frame the approach as a pretraining method, and do a final supervised finetuning step (with varying amounts of data). One would expect the unsupervised pretraining on unaligned data to yield better performance than straightforward supervised translation approach, which would have strengthened the utility of the proposed method.

Author response:
Q3：Unsupervised + Supervised FT.  
Supervised Ft Results: unsupervised way to obtain more knowledge representation:  

| **Models**                                     | B@1   | B@4      |
|------------------------------------------------|-------|----------|
| **USLNet(unsupervised )**                      | 21.10 | 0.10     |
| **Albanie(supervised)**                        | 12.78 | 1.00     |
| **USLNet(supervised )**                        | 15.50 | 1.00     |
| **Video-text (supervised) [1]**                | 17.71 | 1.27     |
| **USLNet(unsupervised + supervised Finetune)** | 27.00 | **1.40** |

From the above table, we can use unuspervised training way can provide one good representatio and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

Reviewer comment (Evaluation):
Having some qualitative results and failure analysis for translation/generation would have helped the paper immensely. Relying solely on b1 and b4 results does not give enough insights to the reader, and possibly is not doing the proposed approach justice.

Author response:
Q4: Qualitative results and failure analysis.  
Thanks for your suggestions. We completely agree with your opinion that we should dig deep into “why” and “how”. Moreover, it is especially significant for this topic. In the next paragraph, we will give our analysis and possible paths to success.  
1 **For the reason (“why”) part**, we will divide it into results analysis and dataset analysis. first give the analysis about results to tell which we can do relatively better and show comparison between our output and supervised output.
1.1 Case study which can be research on good case, bad case and case for comparison:  
Good cases:  
| **Good Cases**    | Case one                                                  | Case two                            |
|-------------------|-----------------------------------------------------------|-------------------------------------|
| **Reference**     | It’s quite a journey **especially** if I **get the bus**. | It’s hell of a **difference** yeah. |
| **USLNet Output** | It’s **especially** long if I **get the bus**.            | It’s **different completely**.      |



Bad cases:  
| **Bad Cases**     | Case one                                 | Case two                                                                                                   |
|-------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------|
| **Reference**     | Oh, **Ma Effanga** is going to be green. | They started challenging the sultan in a very important aspect, **which is that he is not Muslim enough.** |
| **USLNet Output** | It’s not going to be green.              | This is a very important aspect.                                                                           |



From digging into our results, we find that we can do relatively better in Main ingredients(eg: bus, I, anything), but always fail in other detail, such as proper noun(eg: Ma Effanga), and complex sentence(which is that).  

Comparison cases:  
|                     | Case one                                                  | Case two                                             |
|-----------------------------|-----------------------------------------------------------|------------------------------------------------------|
| **Reference**               | It’s quite a journey **especially** if I **get the bus**. | It’s hell of a **difference** yeah.                  |
| **USLNet Output**           | It’s **especially** long if I **get the bus**.            | It’s **different completely**.                       |
| **Supervised Model Output** | How long have you been in the **bus** now.                | It was like trying to be **different** to the world. |

From results comparison, we find that we can find that our results are competitive to supervised method. Moreover, in some ways, we can obtain more accurate output(for example, especially). 

1.2 Dataset Analysis    
As [1] said, the BOBSL dataset is challenging, and that we totally agree with you that it need several breakthroughs to progress in large scale SL translation/generation.

[1]Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

2 **For the “how” part**, we propose a two-fold approach. First, we suggest allowing unsupervised learning to serve as a representation learning stage. Second, we recommend enhancing USLNet by focusing on improvements in both the pretraining and aligner components.

2.1 Supervised Finetuning Results: unsupervised way to obtain more knowledge representation.    

| **Models**                                     | B@1   | B@4      |
|------------------------------------------------|-------|----------|
| **Albanie**                                    | 12.78 | 1.00     |
| **USLNet(unsupervised )**                      | 21.10 | 0.10     |
| **USLNet(supervised )**                        | 15.50 | 1.00     |
| **USLNet(unsupervised + supervised Finetune)** | 27.00 | **1.40** |

From the above table, we can use unuspervised training way can provide one good representation and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

2.2 Paths to success:  

USLNet can be divided into two primary components: the pretraining module (comprising the text pre-training module and the video pre-training module) and the mapper part (slide window aligner). Consequently, the paths to success can be categorized into two aspects. The first aspect involves pre-training, where we can adapt our method using multi-modal models, such as Video-Llama. The second aspect focuses on designing an effective mapper. We will add the discussion in the revised version.

Reviewer comment (Experiments):
As is, I do not think the reviewer/reader has enough signals to evaluate the benefits of the proposed approach.

Author response:
Q4: Qualitative results and failure analysis.  
Thanks for your suggestions. We completely agree with your opinion that we should dig deep into “why” and “how”. Moreover, it is especially significant for this topic. In the next paragraph, we will give our analysis and possible paths to success.  
1 **For the reason (“why”) part**, we will divide it into results analysis and dataset analysis. first give the analysis about results to tell which we can do relatively better and show comparison between our output and supervised output.
1.1 Case study which can be research on good case, bad case and case for comparison:  
Good cases:  
| **Good Cases**    | Case one                                                  | Case two                            |
|-------------------|-----------------------------------------------------------|-------------------------------------|
| **Reference**     | It’s quite a journey **especially** if I **get the bus**. | It’s hell of a **difference** yeah. |
| **USLNet Output** | It’s **especially** long if I **get the bus**.            | It’s **different completely**.      |



Bad cases:  
| **Bad Cases**     | Case one                                 | Case two                                                                                                   |
|-------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------|
| **Reference**     | Oh, **Ma Effanga** is going to be green. | They started challenging the sultan in a very important aspect, **which is that he is not Muslim enough.** |
| **USLNet Output** | It’s not going to be green.              | This is a very important aspect.                                                                           |



From digging into our results, we find that we can do relatively better in Main ingredients(eg: bus, I, anything), but always fail in other detail, such as proper noun(eg: Ma Effanga), and complex sentence(which is that).  

Comparison cases:  
|                     | Case one                                                  | Case two                                             |
|-----------------------------|-----------------------------------------------------------|------------------------------------------------------|
| **Reference**               | It’s quite a journey **especially** if I **get the bus**. | It’s hell of a **difference** yeah.                  |
| **USLNet Output**           | It’s **especially** long if I **get the bus**.            | It’s **different completely**.                       |
| **Supervised Model Output** | How long have you been in the **bus** now.                | It was like trying to be **different** to the world. |

From results comparison, we find that we can find that our results are competitive to supervised method. Moreover, in some ways, we can obtain more accurate output(for example, especially). 

1.2 Dataset Analysis    
As [1] said, the BOBSL dataset is challenging, and that we totally agree with you that it need several breakthroughs to progress in large scale SL translation/generation.

[1]Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

2 **For the “how” part**, we propose a two-fold approach. First, we suggest allowing unsupervised learning to serve as a representation learning stage. Second, we recommend enhancing USLNet by focusing on improvements in both the pretraining and aligner components.

2.1 Supervised Finetuning Results: unsupervised way to obtain more knowledge representation.    

| **Models**                                     | B@1   | B@4      |
|------------------------------------------------|-------|----------|
| **Albanie**                                    | 12.78 | 1.00     |
| **USLNet(unsupervised )**                      | 21.10 | 0.10     |
| **USLNet(supervised )**                        | 15.50 | 1.00     |
| **USLNet(unsupervised + supervised Finetune)** | 27.00 | **1.40** |

From the above table, we can use unuspervised training way can provide one good representation and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

2.2 Paths to success:  

USLNet can be divided into two primary components: the pretraining module (comprising the text pre-training module and the video pre-training module) and the mapper part (slide window aligner). Consequently, the paths to success can be categorized into two aspects. The first aspect involves pre-training, where we can adapt our method using multi-modal models, such as Video-Llama. The second aspect focuses on designing an effective mapper. We will add the discussion in the revised version.

Reviewer comment (Novelty):
Discussion about existing text-to-video aligner algorithms is not sufficient. For example, although text2video[1] is a text-based talking face generation model, it uses an aligner for phoneme-to-pose.

Author response:
Q1：Discussion text-to-video aligner algorithms.  
We appreciate your suggestions regarding the text-to-video aligner algorithms in our work. Following your recommendation, we will include a subsection discussing text-to-video aligners in the revised version.

Here, we provide a brief outline of the proposed subsection, which will cover the following points:  
a) Text-to-video aligners in sign language domain can be broadly classified into two main categories. The first category involves the use of animated avatars to generate sign language, relying on a predefined text-sign dictionary that converts text phrases into sign pose sequences [1-3]. The second category encompasses deep learning approaches applied to text-video mapping.  [4-5] adapt the transformer architecture to the text-video domain and employ a linear embedding layer to map the visual embedding into the corresponding space. Unlike these methods, which can only decode pose images, our Unsupervised Sequence Learning Network (USLNet) is capable of generating videos. We address the length and dimension mismatch issues by utilizing a simple sliding window aligner.

b) Text-to-video aligners in other domains have also been proposed.  [6] introduced a method for automatic redubbing of videos by leveraging the many-to-many mapping of phoneme sequences to lip movements, modeled as dynamic visemes. The Text2Video approach [7] employs a phoneme-to-pose dictionary to generate key poses and high-quality videos from phoneme-poses. This phoneme-pose dictionary can be considered as a token-token mapper. Analogously, USLNet quantizes discrete videos and extracts video tokens, a standard technique in the audio domain [8-10]. Consequently, the sliding window aligner also serves as a token-token aligner. However, unlike the Text2Video method, which performs a lookup action to obtain target tokens, our approach decodes the target token using all source tokens.

[1]Glauert, J., Elliott, R., Cox, S., Tryggvason, J., Sheard, M.: VANESSA: A System for Communication between Deaf and Hearing People. Technology and Disability (2006)  
[2]Karpouzis, K., Caridakis, G., Fotinea, S.E., Efthimiou, E.: Educational Resources and Implementation of a Greek Sign Language Synthesis Architecture. Computers & Education (2007)   
[3]McDonald, J., Wolfe, R., Schnepp, J., Hochgesang, J., Jamrozik, D.G., Stumbo, M., Berke, L., Bialek, M., Thomas, F.: Automated Technique for Real-Time Production of Lifelike Animations of American Sign Language. Universal Access in the Information Society (UAIS) (2016)   
[4]Saunders B, Camgoz N C, Bowden R. Progressive transformers for end-to-end sign language production[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020: 687-705.  
[5] Saunders B, Camgoz N C, Bowden R. Adversarial training for multi-channel sign language production[J]. arXiv preprint arXiv:2008.12405, 2020.   
[6]Sarah L Taylor, Moshe Mahler, Barry-John Theobald, and Iain Matthews, “Dynamic units of visual speech,” in Proceedings of the 11th ACM SIGGRAPH/Eurographics conference on Computer Animation, 2012, pp. 275–284.  
[7]Zhang, Sibo, Jiahong Yuan, Miao Liao, and Liangjun Zhang. "Text2video: Text-driven talking-head video synthesis with personalized phoneme-pose dictionary." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2659-2663. IEEE, 2022.  
[8]Hsu W N, Bolte B, Tsai Y H H, et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451-3460.  
[9]Wang C, Chen S, Wu Y, et al. Neural codec language models are zero-shot text to speech synthesizers[J]. arXiv preprint arXiv:2301.02111, 2023.  
[10]Borsos Z, Marinier R, Vincent D, et al. Audiolm: a language modeling approach to audio generation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.

Reviewer comment (Novelty):
It seems back translations are highly similar to reconstruction loss that is used in image generation, especially in unpaired I2I tasks for cycle consistency. So you might consider elaborating this in the manuscript.

Author response:
Q2: Similarities and differences between back-translation and reconstruction loss.  
Yes, the loss functions employed in our UNSLT exhibit a high degree of similarity. The distinctions between these two losses stem from the modality of the intermediate representations. In the case of back translation (video-text-video), the intermediate representation is a cross-modal (text) representation, which serves to align the cross-modal representations. Conversely, the intermediate representation for video reconstruction is an unimodal (video) representation, primarily utilized for learning the reconstruction of video sequences.

Reviewer comment (Experiments):
There are no visual results on the manuscript and limited visual results on the supplementary materials. I think it needs to be more convincing that the model is capable of generating sign language videos with high quality.

Author response:
Q3: Evaluation for the fidelity of the generated videos and visual results is limited.  

We appreciate your suggestions regarding the evaluation of video generation. In our study, we have adopted the Frechet Video Distance (FVD) metric [11] to assess generated video quality.

The FVD evaluation is conceptually analogous to the Frechet Inception Distance (FID) metric; however, it replaces the feature extraction network for images with a video extraction network, such as the Inflated 3D ConvNet (I3D). The evaluation results are presented below, demonstrating that joint training with USLNet significantly enhances the video generation capability.
|                            | FVD ↓ |
|----------------------------|-------|
| USLNet w/o. joint training | 872.7 |
| USLNet w.joint training    | 389.2 |

Furthermore, we will include additional visual results in the appendix of the revised version.

[11] Unterthiner T, van Steenkiste S, Kurach K, et al. Towards Accurate Generative Models of Video: A New Metric & Challenges[J]. 2018.

Reviewer comment (Evaluation):
Why there is no evaluation for the fidelity of the generated videos in terms of well-known metrics such as FID, LPIPS, etc.

Author response:
Q3: Evaluation for the fidelity of the generated videos and visual results is limited.  

We appreciate your suggestions regarding the evaluation of video generation. In our study, we have adopted the Frechet Video Distance (FVD) metric [11] to assess generated video quality.

The FVD evaluation is conceptually analogous to the Frechet Inception Distance (FID) metric; however, it replaces the feature extraction network for images with a video extraction network, such as the Inflated 3D ConvNet (I3D). The evaluation results are presented below, demonstrating that joint training with USLNet significantly enhances the video generation capability.
|                            | FVD ↓ |
|----------------------------|-------|
| USLNet w/o. joint training | 872.7 |
| USLNet w.joint training    | 389.2 |

Furthermore, we will include additional visual results in the appendix of the revised version.

[11] Unterthiner T, van Steenkiste S, Kurach K, et al. Towards Accurate Generative Models of Video: A New Metric & Challenges[J]. 2018.

Reviewer comment (Novelty):
Why there is no discussion and explanation of the methods proposed in Albanie 2021 in detail as it is the only method that you make a quantitative comparison? I think it needs to be presented more and more importantly the differences and similarities between this and the proposed methods should be highlighted more.

Author response:
Q5: Discussion about Albanie 2021.  
Thank you for your suggestions regarding the supervised method. We will include a detailed introduction to the Albanie 2021 method, as outlined below: In terms of model architecture, both Albanie 2021 and USLNet employ a standard transformer encoder-decoder structure. In the Albanie method, the encoder and decoder comprise two attention layers, each with two heads. Conversely, USLNet adopts a large model architecture, setting the encoder and decoder layers to six. Regarding methodology, Albanie 2021 utilizes a supervised approach for learning sign language translation. In contrast, USLNet employs an unsupervised method, leveraging an abundant text corpus to learn text generation capabilities and employing video-text-video back-translation to acquire cross-modality skills. Concerning model output, Albanie 2021 has released several qualitative examples. We have compared these with the results from USLNet, which demonstrate that USLNet achieves competitive outcomes in comparison to the supervised method.

|                     | Case one                                                  | Case two                                             |
|-----------------------------|-----------------------------------------------------------|------------------------------------------------------|
| **Reference**               | It’s quite a journey **especially** if I **get the bus**. | It’s hell of a **difference** yeah.                  |
| **USLNet Output**           | It’s **especially** long if I **get the bus**.            | It’s **different completely**.                       |
| **Supervised Model Output** | How long have you been in the **bus** now.                | It was like trying to be **different** to the world. |