Paper: Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR-2024)

Reviewer comment (Novelty):
Using pretrained language model to boost sign language translation (SLT) is not surprisingly novel. Several works [1,2] have already verified that pretrained language model can boost (gloss-free) SLT.

Author response:
While we acknowledge previous research on leveraging pretrained language models for sign language translation, our approach offers several advantages over existing methods. 

Chen et al. [1] primarily focus on utilizing manually annotated sign-ordered gloss supervision, relying on Gloss2Text for progressive pretraining on the encoder-decoder language model. In contrast, our approach does not rely on manually annotated glosses. Instead, we concentrate on developing a method for gloss-free sign representation, utilizing a decoder-only language model without the need for progressive pretraining.

Similarly, Zhou et al. [2] have explored gloss-free pretraining, but their approach involves updating the visual encoder and text decoder simultaneously. In contrast, our pseudo-gloss pretraining is independent of the language model. We keep the pretrained language model frozen by leveraging adapters, allowing the use of large-scale pretrained language models. Our architectural design outperforms their approach, as demonstrated in our results.

Reviewer comment (Writing):
The name of pseudo-gloss pretraining is a bit confusing, although the method itself is sound. Because the pseudo-glosses are in spoken language order, it is not quite appropriate to call them "glosses".

Author response:
We termed them pseudo-glosses as they are potential glosses that are signed. However, while we extract them from the spoken language sentence our proposed pretraining is invariant to the order of the pseudo-glosses. As demonstrated in Figures 4, 5, and 6 the pretraining is also able to localize the pseudo-glosses and thus provide them in sign order.
We are happy to change them to something more appropriate and are open to suggestions.

Reviewer comment (Experiments):
I cannot find an ablation study on the pseudo-gloss pretraining. What is the performance if removing it?

Author response:
The impact of the pseudo-gloss pretraining was shown in Tables 2 and 3 in the paper, but as you have pointed out the notation may have been confusing. The updated tables above should now clearly show the impact of the pretraining.

Reviewer comment (Reproducibility):
In Table 4, removing sinusoidal positional encoding leads to the best performance. Then what poistional encoding is used by default?

Author response:
Table 4 indicates that the inclusion of sinusoidal positional encoding yields superior performance. It's important to clarify that the ablation study focuses not on positional encoding at the start of the sign encoder but rather after the sign features $Z$. The improved performance can be attributed to the model's ability, during pretraining, to align sign features with pseudo-gloss prototypes if present in the sequence. Consequently, this leads to the learned features in pretraining discarding positional information in the output representation $Z$. The reintroduction of sinusoidal positional encoding allows the LLM to incorporate positional information in $Z$ for sign translation.

Reviewer comment (Experiments):
Analysis regarding the sign encoder and pseudo-gloss pretraining is insufficient.

Author response:
Thank you for your suggestion about quantitivate analysis for pretraining we will also include the below table of the precision, recall and F1-score for Phoenix14T and CSL-Daily pseudo-gloss.

|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
Phoenix14T  | 0.52   |  0.39 |  0.44   |
CSL-Daily      |0.38 |   0.34  |   0.36 |

To address the question about needing a sign encoder we have conducted the ablation study showing the pretraining precision, recall and F1-scores on the extracted pseudo-glosses shown in the table below. The role of the sign encoder forms an important part of the model as it learns temporal features which is reflected by the significant improvement in F1-score.
|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
No Sign Encoder  | 0.50    |   0.25   |   0.33   |
Sign Encoder      |**0.52** |  **0.39** | **0.44** |

Reviewer comment (Experiments):
It's great that the proposed method outperforms GFSLT-VLP. However, GFSLT-VLP is based on MBart with ~600M parameters, and this study adopts XGLM with ~1.7B parameters, making the fairness of the comparison questionable. It's unclear whether the improvements are really from the proposed modeling and pretraining strategy. Could you please add further ablations regarding the size of GPT models?

Author response:
While we acknowledge the substantial difference in parameter count, a key strength of our approach lies in the frozen pretrained language and vision model, with weight updates facilitated through adapters. This results in a total trainable parameter count of under 17 million, significantly less than GFSLT-VLP's ~600M parameters.

To address the concern about the total parameter count, we conducted an ablation study using a smaller 564M parameter XGLM model. The results, presented in the tables below, reveal a marginal performance reduction on the test set of the Phoenix14T dataset. 
Even with similar total parameter counts, these results outperform GFSLT-VLP.

|   | BLEU4 | ROUGE | 
|-----------|-----------|-----------|
| GFSLT-VLP | 21.44  |42.49|
| Sign2GPT (XGLM-564M) | 22.29  |48.21|
| Sign2GPT (XGLM-1.7B) |  **22.52**|**48.90**|

Reviewer comment (Experiments):
The sign encoder and pseudo-gloss pretraining take a crucial role in the proposed method, but analysis and ablation regarding them are insufficient.

Author response:
Thank you for your suggestion about quantitivate analysis for pretraining we will also include the below table of the precision, recall and F1-score for Phoenix14T and CSL-Daily pseudo-gloss.

|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
Phoenix14T  | 0.52   |  0.39 |  0.44   |
CSL-Daily      |0.38 |   0.34  |   0.36 |

To address the question about needing a sign encoder we have conducted the ablation study showing the pretraining precision, recall and F1-scores on the extracted pseudo-glosses shown in the table below. The role of the sign encoder forms an important part of the model as it learns temporal features which is reflected by the significant improvement in F1-score.
|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
No Sign Encoder  | 0.50    |   0.25   |   0.33   |
Sign Encoder      |**0.52** |  **0.39** | **0.44** |

Reviewer comment (Experiments):
Do we need the sign encoder? What if dropping it?

Author response:
Thank you for your suggestion about quantitivate analysis for pretraining we will also include the below table of the precision, recall and F1-score for Phoenix14T and CSL-Daily pseudo-gloss.

|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
Phoenix14T  | 0.52   |  0.39 |  0.44   |
CSL-Daily      |0.38 |   0.34  |   0.36 |

To address the question about needing a sign encoder we have conducted the ablation study showing the pretraining precision, recall and F1-scores on the extracted pseudo-glosses shown in the table below. The role of the sign encoder forms an important part of the model as it learns temporal features which is reflected by the significant improvement in F1-score.
|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
No Sign Encoder  | 0.50    |   0.25   |   0.33   |
Sign Encoder      |**0.52** |  **0.39** | **0.44** |

Reviewer comment (Evaluation):
Apart from the visual analysis, what about the top-N prediction accuracy and recall?

Author response:
Thank you for your suggestion about quantitivate analysis for pretraining we will also include the below table of the precision, recall and F1-score for Phoenix14T and CSL-Daily pseudo-gloss.

|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
Phoenix14T  | 0.52   |  0.39 |  0.44   |
CSL-Daily      |0.38 |   0.34  |   0.36 |

To address the question about needing a sign encoder we have conducted the ablation study showing the pretraining precision, recall and F1-scores on the extracted pseudo-glosses shown in the table below. The role of the sign encoder forms an important part of the model as it learns temporal features which is reflected by the significant improvement in F1-score.
|  | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
No Sign Encoder  | 0.50    |   0.25   |   0.33   |
Sign Encoder      |**0.52** |  **0.39** | **0.44** |

Reviewer comment (Theory):
In Eq (1), you mentioned that V represents keys from textual features while K originates from sign features. Shouldn't they both come from sign features?

Author response:
Thanks for pointing that out, you are correct that V and K both originate from sign features, we will clarify this in the manuscript.

Reviewer comment (Experiments):
How did you set the rank in LoRA?

Author response:
During our experiments, we set the LoRA rank and alpha values both as 4.

Reviewer comment (Experiments):
While Phoenix14T and CSL-Daily are popular, they are less significant due to limited vocabulary and data size. Please consider adding results for DGS3-T (Zhang et al., 2023).

Author response:
Current benchmarks for state-of-the-art gloss-free results primarily focus on Phoenix14T and CSL-Daily. We appreciate the suggestion to include results for DGS3-T. However, we encountered challenges in obtaining the dataset as the GitHub instructions for downloading are currently non-functional.

To address this, we have contacted the authors of DGS3-T to explore the feasibility of obtaining the dataset. If successful, we intend to include the results as baselines for gloss-free sign translation, in the appendices in the CRC.

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.

Reviewer comment (Novelty):
The method is mostly based on existing models such as GPT, Dino-V2 and LoRA, so there is not much novelty from the architectural standpoint.

Author response:
While we have used existing models, we introduce a novel technique and model architecture to break the reliance on gloss which is something that everyone working in sign translation would be interested in. 
Moreover, our paper highlights technical considerations inherent to sign language translation, including the handling of numerous frames leading to memory constraints and the challenge posed by limited dataset size which we address with our architectural design of using frozen pretrained models with adapters.
We also introduce a pretraining strategy specific to our architecture, enabling the learning of pseudo-gloss localization which serves as a crucial representation for the LLM.

Reviewer comment (Experiments):
There is no ablations to demonstrate if the choice to use parts-of-speech tagging is effective.

Author response:
The downstream translation effectively learns to replace the missing words as demonstrated by our qualitative translation examples in Tables 5 and 6. This is a key benefit of a strong language model. 

Regarding the ablation of parts-of-speech (POS) tagging, we deliberately excluded certain parts of speech as we know they are not represented in sign, e.g. connected words such as 'and' and articles such as 'the' lack a functional sign equivalent. In the table below, we illustrate the impact of utilizing POS on precision, recall, and F1-score for our pretraining results on Phoenix14T with a threshold of 0.2. Notably, when all words are used as tokens, recall significantly decreases from 0.39 to 0.28. This result validates our assertion that not all words have corresponding signs in sign language.
As outlined in the paper, this pretraining strategy opens avenues for future research, particularly in areas like sign spotting.

| Tokens | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
| All Words | **0.55** |0.28|0.37|
| Pseudo-Glosses | 0.52| **0.39**|**0.44**|

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.

Reviewer comment (Experiments):
Can the downstream translation learn to generate meaningful words that are missing in the parts-of-speech tagging?

Author response:
The downstream translation effectively learns to replace the missing words as demonstrated by our qualitative translation examples in Tables 5 and 6. This is a key benefit of a strong language model. 

Regarding the ablation of parts-of-speech (POS) tagging, we deliberately excluded certain parts of speech as we know they are not represented in sign, e.g. connected words such as 'and' and articles such as 'the' lack a functional sign equivalent. In the table below, we illustrate the impact of utilizing POS on precision, recall, and F1-score for our pretraining results on Phoenix14T with a threshold of 0.2. Notably, when all words are used as tokens, recall significantly decreases from 0.39 to 0.28. This result validates our assertion that not all words have corresponding signs in sign language.
As outlined in the paper, this pretraining strategy opens avenues for future research, particularly in areas like sign spotting.

| Tokens | Precision | Recall | F1-Score | 
|-----------|-----------|-----------|-----------|
| All Words | **0.55** |0.28|0.37|
| Pseudo-Glosses | 0.52| **0.39**|**0.44**|

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.

Reviewer comment (Writing):
The introduction part seems inconsistent with the title. The introduction mentions both large-scale vision and language model, while the title only mentions the language one. What do the authors want to emphasize?

Author response:
We appreciate the reviewer's careful examination of the introduction and title. Our emphasis on 'leveraging large language models' in the title is intentional. Our goal is to create a sign model/representation that is passed to / leveraged by the GPT model for sign language translation, hence the name Sign2GPT. 
The 'Sign' component that is leveraged by the GPT model includes the use of a large vision model and our novel pretraining strategy.
However, we are more than happy to change the title to address this issue.

Reviewer comment (Novelty):
The pseudo-gloss pretraining technique shares a similar core idea with CSGCR. The authors should discuss the difference.

Author response:
We appreciate the reviewer's insights into CSGCR [1] and observe these distinctions between our proposed pseudo-gloss pretraining technique and CSGCR:
- *Training Approach* - CSGCR employs separate transformer encoders for each vocabulary item, necessitating positive and negative samples during training. In contrast, our method utilizes a single transformer encoder for all pseudo-glosses, ensuring scalability for larger vocabularies.
- *Localization Capabilities* - Our pretraining strategy exhibits inherent explicit localization capabilities, as presented in Figures 4, 5, and 6.
- *Translation Process* - Unlike CSGCR's two-step process involving Video2Text (Word Existence Verification) and then Text2Text (Sentence Generation), our approach directly utilizes the pretrained model for Video2Text using the GPT model.

We will update the paper to discuss these differences in the related work.

Reviewer comment (Novelty):
Utilization of pretrained model is not new. The author should cite the following work and discuss the difference with it. Chen Y, Wei F, Sun X, et al. A simple multi-modality transfer learning baseline for sign language translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 5120-5130.

Author response:
Thanks for highlighting the reference. We will incorporate a discussion of its relevance in the related work section. To outline the distinctions, Chen et al. [2] focus on utilizing manually annotated sign-ordered gloss supervision, relying on Gloss2Text for progressive pretraining on the encoder-decoder language model. In contrast, our approach has a significant advantage of not depending on manually annotated glosses. Instead, we concentrate on developing a method for gloss-free sign representation, leveraging a decoder-only language model without the need for progressive pretraining of the language model.

Reviewer comment (Experiments):
What are the performance gains derived from the utilization of large-scale vision and language models? The ablation part should demonstrate it.

Author response:
In the table below, we present the results of an ablation experiment on CSL-Daily, comparing the performance of a ResNet18 backbone used in GFSLT [3] with our Adapted DinoV2 approach within our Sign2GPT architecture. Note this ablation study does not involve pseudo-gloss pretraining. The findings indicate that the model trained with DinoV2 pretraining, incorporating adapters, exhibits slightly better performance than the ResNet18 backbone utilized in the previous state-of-the-art GFSLT approach.

| Spatial Backbone | BLEU4 | ROUGE | 
|-|-|-|
| ResNet18 | 12.28 |38.31|
| DinoV2 (ViT-S/14)  |**12.96**| **41.12**|

A noteworthy advantage of our approach lies in the application of LoRA instead of fine-tuning all parameters. Consequently, the proposed backbone features under three hundred thousand trainable parameters, a significant reduction compared to the 11 million trainable parameters of the ResNet18 model.

Additionally, we commit to updating Tables 2 and 3 in the paper to include GFSLT approach without their vision-language pretraining. The tables below demonstrate that our architecture even without pseudo-gloss pretraining (w/o PGP) can outperform it. This sufficiently demonstrates that our architectural design also shows performance improvements compared to the previous attempts at the utilization of pretrained language models.

*Phoenix14T Translation Results:*
Method|BLEU1|BLEU2|BLEU3|BLEU4|ROUGE|
|-|-|-|-|-|-|
|*No Sign Prior Pretraining*| |||||
GFSLT|41.39|31.00|24.20|**19.66**|40.93|
Sign2GPT (w/o PGP)|**45.43**|**32.03**|**24.23**|19.42|**45.23**|
|*With Sign Prior Pretraining*| |||||
GFSLT-VLP|43.71|33.18|26.11|21.44|42.49|
Sign2GPT(w/ PGP)|**49.54**|**35.96**|**28.83**|**22.52**|**48.90**|

*CSL-Daily Translation Results:*
Method|BLEU1|BLEU2|BLEU3|BLEU4|ROUGE|
|-|-|-|-|-|-|
|*No Sign Prior Pretraining*| |||||
GFSLT|**37.69**|23.28|14.93|9.88|35.16|
Sign2GPT (w/o PGP)|34.80|**24.00**|**17.27**|**12.96**|**41.12**|
|*With Sign Prior Pretraining*| |||||
GFSLT-VLP|39.37|24.93|16.26|11.00|36.44|
Sign2GPT (w/ PGP)|**41.75**|**28.73**|**20.60**|**15.40**|**42.36**|

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.