Paper: Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR-2024)

Reviewer comment (Presentation):
The statement "we place the LLaMA transformer behind the last self-attention block in ViT" is inconsistent with Figure 1. It is unclear why the LLaMA transformer should be placed behind the last self-attention block instead of the last layer (i.e., behind the last FFN block) in ViT. Could the authors provide some insights?

Author response:
We apologize for the confusion and thank the reviewer for pointing this out. Our method is consistent with Figure 1 (Page 2), i.e., the LLaMA transformer is behind the last FFN block in ViT. We will revise the statement to “behind the last transformer block in ViT” to avoid confusion.

Reviewer comment (Experiments):
Some baselines are missing. For example, it is not persuasive enough to only compare with "ViT-MLP" in Table 6, whose number of trainable parameters remains unchanged, but the total number of the parameters is significantly reduced, especially when layers of LLMs have notably more parameters than layers of ViT-Base/-Small. Therefore, an important baseline that should be added is to append frozen ViT layers behind the original ViT to keep the total number of parameters the same as "ViT-S-LLaMA."

Author response:
We agree with the reviewer that baselines in addition to ViT-S-MLP will strengthen our paper. However, we are not sure about the suggestion to “append frozen ViT layers behind the original ViT,” because the LLaMA transformers have different parameter numbers compared to ViT transformers. Inferring from the mention of “to keep the total number of parameters the same,” we presume the reviewer is suggesting baselines with equal parameter counts to ViT-S-LLaMA. Therefore, we add the following two sets of experiments and hope they align with what the reviewer suggests. If our interpretation is incorrect, we kindly request the reviewer to provide further clarification, and we are more than willing to address any concerns. 

- We first add the experiment of **training from a randomly initialized LLaMA block**, which has the same number of parameters but more trainable parameters. The tables below cover all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. We will include these results in the revision. The results in the tables show two main findings which validate that the performance gains truly stem from our method: (1) **Our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**. (2) **Naively adding a large transformer block can be detrimental for many tasks**, resulting in inferior performance even compared to the plain baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image classification   | Acc Top1     | Acc Top5     |
| ---------------------- | -------- | -------- |
| ViT-S                  | 80.1     | 95.0     |
| + LLaMA (Frozen, Ours) | **80.7** | **95.4** |
| + LLaMA (Random)       | 76.9     | 91.9     |

| Point cloud classification | BG       | OBJ      | T50 (Hardest) |
| -------------------------- | -------- | -------- | ------------- |
| PointBERT                  | 87.4     | 88.1     | 83.1          |
| + LLaMA (Frozen, Ours)     | **88.0** | **88.5** | **83.8**      |
| + LLaMA (Random)           | 87.2     | 88.0     | 82.6          |

| Action recognition     | Acc Top1     | Acc Top5     |
| ---------------------- | -------- | -------- |
| ViT-S                  | 64.7     | 89.2     |
| + LLaMA (Frozen, Ours) | **65.9** | **89.9** |
| + LLaMA (Random)       | 64.1     | 88.8     |

| Motion Forecasting     | ADE↓     | FDE↓     | MR↓      |
| ---------------------- | -------- | -------- | -------- |
| MMTransformer          | 0.72     | 1.10     | 10.7     |
| + LLaMA (Frozen, Ours) | **0.71** | **1.08** | **10.5** |
| + LLaMA (Random)       | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Frozen, Ours)                   | **50.22** | **82.26** | **90.08** |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Frozen, Ours)      | **48.09** | **89.03** |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

- The second evidence of aligning with the equal parameter counts is presented in Figure 2 (Page 6) of our original submission. We analyze how the performance of image classification changes with different LLaMA or OPT layers. As shown in the figure, the models can have varied accuracy under the same trainable parameter and total parameter counts. Therefore, it indicates the importance of having proper pretrained weights in the inserted LLM blocks, such as our final LLaMA block, to provide benefits across diverse visual tasks.

Reviewer comment (Evaluation):
The total number of parameters, FLOPs, and throughput/latency of ViTs with the LLM layer inserted should be reported to investigate whether the proposed paradigm is more competitive than directly scaling up vanilla ViTs.

Author response:
First, we agree that sharing detailed configurations is beneficial from an application perspective, and per the reviewer’s request, the table below presents these comparisons. Using a frozen LLM transformer **adds negligible trainable parameters** and **slightly decreases the speed** in image classification, while mainly resulting in an increase in overall parameters and FLOPs. 

| Model   | Acc Top1      | Param (Trainable) | Param (All) | FLOPs  | FPS (Batch-Size=1)       | FPS (Batch-Size=256)    |
| ------- | -------- | ----------------- | ----------- | ------ | ------------------------ | ----------------------- |
| ViT-S   | 80.1     | 22.05M            | 22.05M      | 8.5G   | 75.26 batches-per-second | 2.54 batches-per-second |
| + LLaMA | **80.7** | 25.19M            | 227.58M     | 90.11G | 70.18 batches-per-second | 2.36 batches-per-second |

(The FLOPs numbers are evaluated with [`flops_profiler`](https://github.com/cli99/flops-profiler) using a standard image of shape $224\times 224\times 3$. FPS comes from running the inference on 50k ImageNet-val images using batch size of 1 and 256 and a single A6000 GPU).

Second, and more importantly, we would like to clarify the fundamental differences between our work and directly scaling up ViTs. As mentioned earlier, our approach is not tailored for optimizing ViTs, and studies focusing on image classification can develop larger models that surpass our paradigm with proper design and tuning. In contrast, we would like to reiterate that **our goal is to reveal the novel and intriguing discovery** that *despite being trained solely on text data, a frozen LLM transformer can surprisingly function as a visual encoder layer for diverse purely visual tasks, in the absence of language*. Therefore, **our approach is not intended to optimally scale up and enhance individual tasks**, but to reveal that **a frozen set of parameters can be shared across various tasks and backbones**. Even from the perspective of improving models, we would like to clarify:

- Scaling up ViT can lead to improvement in image classification, but **the scaled-up layers may not necessarily extend to diverse tasks as we do**, such as point clouds, videos, and vision-language tasks. 
- To the best of our knowledge, **simply scaling up ViTs does not exhibit the emergent behavior of concentrating on relevant regions as in our information filtering hypothesis**. Furthermore, our approach empirically shows such patterns not limited to images. **Our discovery of this behavior covers videos, point clouds, 2D vision-language, and 3D vision-language tasks**. We summarize three representative results (Figures 3, 5, C from our original submission) in [[this anonymous image link]](https://imgur.com/a/8vEjjqv), for the convenience of checking.

Reviewer comment (Experiments):
Some ablation studies are missing. For example, append the frozen LLM layer behind prior layers instead of the last layer in ViT.

Author response:
We appreciate the reviewer’s excellent suggestions. We would like to again emphasize that our work is **the first** to discover that “*despite being trained solely on text data, a frozen LLM transformer can be used as a visual encoder layer for purely visual tasks, in the absence of language*”. So, **we prioritize validating the general applicability of this discovery across a wide range of visual tasks, in the most straightforward, simplest, and unified manner** – achieved by inserting a single LLM layer at the end of visual backbones. We do not claim that this simple strategy is the optimal approach for achieving the best performance. While we acknowledge that there are various variants to utilize specific LLM layers and insert them into visual backbone architectures which may further improve performance, we deliberately avoid optimizing model selection with more complicated or task-specific architectures. 

Following the reviewer’s suggestions, the table below presents an ablation study on different architecture variants. While some variants improve the performance while others decrease the performance, these ablation results contribute to strengthening our understanding and are orthogonal to our main discovery. We will include the results in the revision. And we leave a more comprehensive investigation on these and additional variants as interesting future work.

|      | Model             | Configuration                                               | Acc Top1     |
| ---- | ----------------- | -------------------------------------------------- | -------- |
| 1    | ViT-S             | Baseline ViT-S                                     | 75.3     |
| 2    | + LLaMA           | Single LLaMA block at the end of ViT | 75.8     |
| 4    | + LLaMA at Middle | Single LLaMA block at the middle of ViT            | 75.5     |
| 5    | + LLaMA at Head   | Single LLaMA block at the head of ViT              | 72.6     |
| 6    | + 2 LLaMA Blocks  | 2 LLaMA blocks at the end of ViT                   | **77.1** |

Experimental setting: We conduct the ablation experiments under our ablation setting of the original submission: ViT-S with 100 epochs on ImageNet. This is shorter than a full training of 300 epochs, due to our resource constraints. However, the results in the table are all compared under the identical and fair setup.

- *Q: Inserting more LLM blocks*. Please refer to Row 6 in the table above "+ 2 LLaMA Blocks." We use the final 2 LLaMA blocks, instead of the final single block, to evaluate the impact of using more LLaMA blocks. We find that using more LLM blocks seems beneficial for the overall performance. We think this is an interesting direction to further broaden our discovery and will include it in the revised paper.

- *Q: Other places to insert*. Please refer to Rows 4 and 5 in the table above "+ LLaMA at Middle" and "+ LLaMA at Head." They perform worse than inserting the LLM transformer block at the end of ViT (Row 2), supporting our design choice. Additionally, inserting at the end of encoders is a simple and unified strategy that can be employed across a wide range of tasks and diverse backbones; in contrast, achieving insertion at the middle may require scrutinizing backbones for each task individually.

Reviewer comment (Experiments):
How does the number of the appended frozen LLM layers affect the model performance?

Author response:
We appreciate the reviewer’s excellent suggestions. We would like to again emphasize that our work is **the first** to discover that “*despite being trained solely on text data, a frozen LLM transformer can be used as a visual encoder layer for purely visual tasks, in the absence of language*”. So, **we prioritize validating the general applicability of this discovery across a wide range of visual tasks, in the most straightforward, simplest, and unified manner** – achieved by inserting a single LLM layer at the end of visual backbones. We do not claim that this simple strategy is the optimal approach for achieving the best performance. While we acknowledge that there are various variants to utilize specific LLM layers and insert them into visual backbone architectures which may further improve performance, we deliberately avoid optimizing model selection with more complicated or task-specific architectures. 

Following the reviewer’s suggestions, the table below presents an ablation study on different architecture variants. While some variants improve the performance while others decrease the performance, these ablation results contribute to strengthening our understanding and are orthogonal to our main discovery. We will include the results in the revision. And we leave a more comprehensive investigation on these and additional variants as interesting future work.

|      | Model             | Configuration                                               | Acc Top1     |
| ---- | ----------------- | -------------------------------------------------- | -------- |
| 1    | ViT-S             | Baseline ViT-S                                     | 75.3     |
| 2    | + LLaMA           | Single LLaMA block at the end of ViT | 75.8     |
| 4    | + LLaMA at Middle | Single LLaMA block at the middle of ViT            | 75.5     |
| 5    | + LLaMA at Head   | Single LLaMA block at the head of ViT              | 72.6     |
| 6    | + 2 LLaMA Blocks  | 2 LLaMA blocks at the end of ViT                   | **77.1** |

Experimental setting: We conduct the ablation experiments under our ablation setting of the original submission: ViT-S with 100 epochs on ImageNet. This is shorter than a full training of 300 epochs, due to our resource constraints. However, the results in the table are all compared under the identical and fair setup.

- *Q: Inserting more LLM blocks*. Please refer to Row 6 in the table above "+ 2 LLaMA Blocks." We use the final 2 LLaMA blocks, instead of the final single block, to evaluate the impact of using more LLaMA blocks. We find that using more LLM blocks seems beneficial for the overall performance. We think this is an interesting direction to further broaden our discovery and will include it in the revised paper.

- *Q: Other places to insert*. Please refer to Rows 4 and 5 in the table above "+ LLaMA at Middle" and "+ LLaMA at Head." They perform worse than inserting the LLM transformer block at the end of ViT (Row 2), supporting our design choice. Additionally, inserting at the end of encoders is a simple and unified strategy that can be employed across a wide range of tasks and diverse backbones; in contrast, achieving insertion at the middle may require scrutinizing backbones for each task individually.

Reviewer comment (Experiments):
The proposed approach adds an additional (frozen) LLM block and (learnable) projection layers to the baseline network, making it unclear whether the performance improvements in Tables 1, 2, 3, 4 and 5 are simply due to increased network capacity.

Author response:
We appreciate the reviewer for the insightful feedback. We are glad to hear that the reviewer finds our discovery of frozen LLM transformers being visual encoders intriguing, and recognizes our efforts in covering a wide range of tasks and developing the analysis of the information filtering hypothesis. Meanwhile, we understand the reviewer’s concerns about our baselines. Below we address all the points.

Following the reviewer’s suggestions, we conducted additional experiments with two sets of baselines: (1) **using additional MLPs**, aligning the number of trainable parameters; (2) **using randomly initialized LLM transformer blocks**, aligning the total number of parameters, to compare with our approach. The tables below summarize the results for all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. 

We will include these results in the revision. The results in the tables show several important findings which validate that our performance gains stem from our method rather than increased network capacity:

- **Our method of adding a frozen LLaMA transformer block consistently outperforms “-MLP” models**.
- More importantly, different from the observation in the image classification task (ablation in Table 6 of our submission), **simply adding MLPs is not a consistently beneficial strategy for all visual tasks and can, in fact, be detrimental**, resulting in inferior performance on some tasks even compared to the pain baseline. This is because naively adding large MLPs without the guidance from the pre-trained LLM transformer block may lead to challenges in optimization, especially for modalities and tasks characterized by weaker semantics, such as point clouds and motion forecasting.
- Similarly, **our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**.
- Similarly, **naively adding a large transformer block can be even detrimental for many tasks**, resulting in inferior performance even compared to the baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image Classification  | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 80.1     | 95.0     |
| + LLaMA (Ours)        | **80.7** | **95.4** |
| + MLP                 | 80.4     | 95.0     |
| + LLaMA (Random Init) | 76.9     | 91.9     |

| Point Cloud Recognition | BG       | OBJ      | T50 (Hardest) |
| ----------------------- | -------- | -------- | ------------- |
| PointBERT               | 87.4     | 88.1     | 83.1          |
| + LLaMA (Ours)          | **88.0** | **88.5** | **83.8**      |
| + MLP                   | 86.5     | 87.3     | 83.4          |
| + LLaMA (Random Init)   | 87.2     | 88.0     | 82.6          |

| Action Recognition    | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 64.7     | 89.2     |
| + LLaMA (Ours)        | **65.9** | **89.9** |
| + MLP                 | 63.8     | 88.9     |
| + LLaMA (Random Init) | 64.1     | 88.8     |

| Motion Forecasting    | ADE↓     | FDE↓     | MR↓      |
| --------------------- | -------- | -------- | -------- |
| MMTransformer         | 0.72     | 1.10     | 10.7     |
| + LLaMA (Ours)        | **0.71** | **1.08** | **10.5** |
| + MLP                 | 0.74     | 1.16     | 11.8     |
| + LLaMA (Random Init) | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Ours)                           | **50.22** | **82.26** | **90.08** |
| + MLP                                    | 49.48     | 81.12     | 89.58     |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Ours)              | **48.09** | **89.03** |
| + MLP                       | 47.14     | 88.08     |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

Reviewer comment (Experiments):
The ablation in Table 6 demonstrates that matching the MLP parameter count already bridges some gap between the proposed method and the baseline, revealing a clear inferiority of the baseline used in the rest of the tables.

Author response:
We appreciate the reviewer for the insightful feedback. We are glad to hear that the reviewer finds our discovery of frozen LLM transformers being visual encoders intriguing, and recognizes our efforts in covering a wide range of tasks and developing the analysis of the information filtering hypothesis. Meanwhile, we understand the reviewer’s concerns about our baselines. Below we address all the points.

Following the reviewer’s suggestions, we conducted additional experiments with two sets of baselines: (1) **using additional MLPs**, aligning the number of trainable parameters; (2) **using randomly initialized LLM transformer blocks**, aligning the total number of parameters, to compare with our approach. The tables below summarize the results for all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. 

We will include these results in the revision. The results in the tables show several important findings which validate that our performance gains stem from our method rather than increased network capacity:

- **Our method of adding a frozen LLaMA transformer block consistently outperforms “-MLP” models**.
- More importantly, different from the observation in the image classification task (ablation in Table 6 of our submission), **simply adding MLPs is not a consistently beneficial strategy for all visual tasks and can, in fact, be detrimental**, resulting in inferior performance on some tasks even compared to the pain baseline. This is because naively adding large MLPs without the guidance from the pre-trained LLM transformer block may lead to challenges in optimization, especially for modalities and tasks characterized by weaker semantics, such as point clouds and motion forecasting.
- Similarly, **our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**.
- Similarly, **naively adding a large transformer block can be even detrimental for many tasks**, resulting in inferior performance even compared to the baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image Classification  | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 80.1     | 95.0     |
| + LLaMA (Ours)        | **80.7** | **95.4** |
| + MLP                 | 80.4     | 95.0     |
| + LLaMA (Random Init) | 76.9     | 91.9     |

| Point Cloud Recognition | BG       | OBJ      | T50 (Hardest) |
| ----------------------- | -------- | -------- | ------------- |
| PointBERT               | 87.4     | 88.1     | 83.1          |
| + LLaMA (Ours)          | **88.0** | **88.5** | **83.8**      |
| + MLP                   | 86.5     | 87.3     | 83.4          |
| + LLaMA (Random Init)   | 87.2     | 88.0     | 82.6          |

| Action Recognition    | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 64.7     | 89.2     |
| + LLaMA (Ours)        | **65.9** | **89.9** |
| + MLP                 | 63.8     | 88.9     |
| + LLaMA (Random Init) | 64.1     | 88.8     |

| Motion Forecasting    | ADE↓     | FDE↓     | MR↓      |
| --------------------- | -------- | -------- | -------- |
| MMTransformer         | 0.72     | 1.10     | 10.7     |
| + LLaMA (Ours)        | **0.71** | **1.08** | **10.5** |
| + MLP                 | 0.74     | 1.16     | 11.8     |
| + LLaMA (Random Init) | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Ours)                           | **50.22** | **82.26** | **90.08** |
| + MLP                                    | 49.48     | 81.12     | 89.58     |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Ours)              | **48.09** | **89.03** |
| + MLP                       | 47.14     | 88.08     |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

Reviewer comment (Experiments):
The main (implicit) claim in this paper is that the (frozen) LLM weights have some transferrable knowledge to vision tasks and can be exploited using the proposed methodology. An ideal baseline to support this claim should be the same model with (learnable) weights that are randomly initialized instead of LLM weights.

Author response:
We appreciate the reviewer for the insightful feedback. We are glad to hear that the reviewer finds our discovery of frozen LLM transformers being visual encoders intriguing, and recognizes our efforts in covering a wide range of tasks and developing the analysis of the information filtering hypothesis. Meanwhile, we understand the reviewer’s concerns about our baselines. Below we address all the points.

Following the reviewer’s suggestions, we conducted additional experiments with two sets of baselines: (1) **using additional MLPs**, aligning the number of trainable parameters; (2) **using randomly initialized LLM transformer blocks**, aligning the total number of parameters, to compare with our approach. The tables below summarize the results for all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. 

We will include these results in the revision. The results in the tables show several important findings which validate that our performance gains stem from our method rather than increased network capacity:

- **Our method of adding a frozen LLaMA transformer block consistently outperforms “-MLP” models**.
- More importantly, different from the observation in the image classification task (ablation in Table 6 of our submission), **simply adding MLPs is not a consistently beneficial strategy for all visual tasks and can, in fact, be detrimental**, resulting in inferior performance on some tasks even compared to the pain baseline. This is because naively adding large MLPs without the guidance from the pre-trained LLM transformer block may lead to challenges in optimization, especially for modalities and tasks characterized by weaker semantics, such as point clouds and motion forecasting.
- Similarly, **our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**.
- Similarly, **naively adding a large transformer block can be even detrimental for many tasks**, resulting in inferior performance even compared to the baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image Classification  | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 80.1     | 95.0     |
| + LLaMA (Ours)        | **80.7** | **95.4** |
| + MLP                 | 80.4     | 95.0     |
| + LLaMA (Random Init) | 76.9     | 91.9     |

| Point Cloud Recognition | BG       | OBJ      | T50 (Hardest) |
| ----------------------- | -------- | -------- | ------------- |
| PointBERT               | 87.4     | 88.1     | 83.1          |
| + LLaMA (Ours)          | **88.0** | **88.5** | **83.8**      |
| + MLP                   | 86.5     | 87.3     | 83.4          |
| + LLaMA (Random Init)   | 87.2     | 88.0     | 82.6          |

| Action Recognition    | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 64.7     | 89.2     |
| + LLaMA (Ours)        | **65.9** | **89.9** |
| + MLP                 | 63.8     | 88.9     |
| + LLaMA (Random Init) | 64.1     | 88.8     |

| Motion Forecasting    | ADE↓     | FDE↓     | MR↓      |
| --------------------- | -------- | -------- | -------- |
| MMTransformer         | 0.72     | 1.10     | 10.7     |
| + LLaMA (Ours)        | **0.71** | **1.08** | **10.5** |
| + MLP                 | 0.74     | 1.16     | 11.8     |
| + LLaMA (Random Init) | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Ours)                           | **50.22** | **82.26** | **90.08** |
| + MLP                                    | 49.48     | 81.12     | 89.58     |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Ours)              | **48.09** | **89.03** |
| + MLP                       | 47.14     | 88.08     |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

Reviewer comment (Experiments):
Without a good baseline, the strong claims across multiple tasks are not well supported.

Author response:
We appreciate the reviewer for the insightful feedback. We are glad to hear that the reviewer finds our discovery of frozen LLM transformers being visual encoders intriguing, and recognizes our efforts in covering a wide range of tasks and developing the analysis of the information filtering hypothesis. Meanwhile, we understand the reviewer’s concerns about our baselines. Below we address all the points.

Following the reviewer’s suggestions, we conducted additional experiments with two sets of baselines: (1) **using additional MLPs**, aligning the number of trainable parameters; (2) **using randomly initialized LLM transformer blocks**, aligning the total number of parameters, to compare with our approach. The tables below summarize the results for all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. 

We will include these results in the revision. The results in the tables show several important findings which validate that our performance gains stem from our method rather than increased network capacity:

- **Our method of adding a frozen LLaMA transformer block consistently outperforms “-MLP” models**.
- More importantly, different from the observation in the image classification task (ablation in Table 6 of our submission), **simply adding MLPs is not a consistently beneficial strategy for all visual tasks and can, in fact, be detrimental**, resulting in inferior performance on some tasks even compared to the pain baseline. This is because naively adding large MLPs without the guidance from the pre-trained LLM transformer block may lead to challenges in optimization, especially for modalities and tasks characterized by weaker semantics, such as point clouds and motion forecasting.
- Similarly, **our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**.
- Similarly, **naively adding a large transformer block can be even detrimental for many tasks**, resulting in inferior performance even compared to the baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image Classification  | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 80.1     | 95.0     |
| + LLaMA (Ours)        | **80.7** | **95.4** |
| + MLP                 | 80.4     | 95.0     |
| + LLaMA (Random Init) | 76.9     | 91.9     |

| Point Cloud Recognition | BG       | OBJ      | T50 (Hardest) |
| ----------------------- | -------- | -------- | ------------- |
| PointBERT               | 87.4     | 88.1     | 83.1          |
| + LLaMA (Ours)          | **88.0** | **88.5** | **83.8**      |
| + MLP                   | 86.5     | 87.3     | 83.4          |
| + LLaMA (Random Init)   | 87.2     | 88.0     | 82.6          |

| Action Recognition    | Acc Top1     | Acc Top5     |
| --------------------- | -------- | -------- |
| ViT-S                 | 64.7     | 89.2     |
| + LLaMA (Ours)        | **65.9** | **89.9** |
| + MLP                 | 63.8     | 88.9     |
| + LLaMA (Random Init) | 64.1     | 88.8     |

| Motion Forecasting    | ADE↓     | FDE↓     | MR↓      |
| --------------------- | -------- | -------- | -------- |
| MMTransformer         | 0.72     | 1.10     | 10.7     |
| + LLaMA (Ours)        | **0.71** | **1.08** | **10.5** |
| + MLP                 | 0.74     | 1.16     | 11.8     |
| + LLaMA (Random Init) | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Ours)                           | **50.22** | **82.26** | **90.08** |
| + MLP                                    | 49.48     | 81.12     | 89.58     |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Ours)              | **48.09** | **89.03** |
| + MLP                       | 47.14     | 88.08     |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

Reviewer comment (Experiments):
What happens to the results in Tables 1, 2, 3, 4 and 5 if we don't use the LLM weights and just use learnable random weights?

Author response:
We thank the reviewer for the prompt response and the follow-up question.

First, yes, the randomly initialized LLaMA block weights **are tuned end to end in the above experiments**.

Second, we would like to note that the results are not surprising, especially given that in our setting, adding a randomly initialized LLaMA block **significantly increases the trainable parameters**, contrary to what one might expect when adding a ViT transformer layer. This observation also aligns with the findings from prior studies – **naively** adding additional transformer blocks and blindly increasing network capacity does not necessarily improve performance. This is because, as mentioned in our earlier response, “this naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.” 

To illustrate further, the table below compares the trainable and overall parameter numbers between ViT-S and ViT-S-LLaMA under both our frozen setting and the new baseline of randomly initialized setting.

|                              | Param (Trainable) | Param (All) |
| ---------------------------- | ----------------- | ----------- |
| ViT-S                        | 22.05M            | 22.05M      |
| + LLaMA (Frozen)             | 25.19M            | 227.58M     |
| + LLaMA (Randomly Initialized) | 227.58M           | 227.58M     |

From the table, we can see that, for the baseline with a randomly initialized LLaMA block, **the number of trainable parameters increases by more than 10 times**, compared with our frozen model and the plain baseline. Notably, these additional 10 times of parameters all **belong to a single huge LLM transformer block with much larger dimensionality and capacity** than the original ViT backbone. These considerably increased trainable parameters lead to significant optimization difficulties. Moreover, our evaluation consists of a wide range of heterogeneous types of visual tasks with different backbones and different training sets (e.g., lack of large training sets for some tasks). It is expected that such high-capacity models with randomly initialized parameters may not provide consistent benefits and can be even detrimental.

In a context relevant but different from ours, prior work has shown that **naively adding capacities to a model may not lead to improvement**, because it also needs associated training techniques and datasets. For instance, the authors of SwinTransformer (Liu *et al.*) attempted to scale ViT-B to ViT-L on ImageNet1K in Table 1 of their paper. However, the accuracy dropped from 77.9% to 76.5%. While this observation is not identical to our experimental setting, we provide it here as an additional source of reference that reflects the challenges associated with training large transformers.

Liu *et al.* Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021.

We hope that our explanation has addressed the reviewer’s follow-up question and concern. We are happy to answer any further questions.

Reviewer comment (Experiments):
The experimental improvements are most of the time rather marginal with a few cases showing practical improvements. One could argue that ViT-S-MLP from Tab. 6 would be a fairer comparison for all experiments as it more closely matches the parameter count. In this case, one would expect the gap to further close.

Author response:
We agree that this is a valid concern and conducted additional experiments with the “-MLP” baseline for all the tasks investigated in our paper, including point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language, in addition to the already reported result for image classification, as shown in the tables below. We will include these results in the revision. The results in the tables show two main findings which validate the advantage of our method: 

- **Our method of adding a frozen LLaMA transformer block consistently outperforms “-MLP” models**.
- More importantly, different from the observation in the image classification task, **simply adding MLPs is not a consistently beneficial strategy for all visual tasks and can, in fact, be detrimental**, resulting in inferior performance on some tasks even compared to the pain baseline. This is because naively adding large MLPs without the guidance from the pre-trained LLM transformer block may lead to challenges in optimization, especially for modalities and tasks characterized by weaker semantics, such as point clouds and motion forecasting.

| Image classification (already presented in Table 5 of paper) | Acc Top1     | Acc Top5     |
| ---------------------------------------------------------- | -------- | -------- |
| ViT-S                                                      | 80.1     | 95.0     |
| + LLaMA (Ours)                                             | **80.7** | **95.4** |
| + MLP                                                      | 80.4     | 95.0     |

| Point cloud recognition | BG       | OBJ      | T50 (Hardest) |
| ----------------------- | -------- | -------- | ------------- |
| PointBERT               | 87.4     | 88.1     | 83.1          |
| + LLaMA (Ours)          | **88.0** | **88.5** | **83.8**      |
| + MLP                   | 86.5     | 87.3     | 83.4          |

| Action recognition | Acc Top1     | Acc Top5     |
| ------------------ | -------- | -------- |
| ViT-S              | 64.7     | 89.2     |
| + LLaMA (Ours)     | **65.9** | **89.9** |
| + MLP              | 63.8     | 88.9     |

| Motion forecasting | ADE ↓    | FDE ↓    | MR ↓     |
| ------------------ | -------- | -------- | -------- |
| MMTransformer      | 0.72     | 1.10     | 10.7     |
| + LLaMA (Ours)     | **0.71** | **1.08** | **10.5** |
| + MLP              | 0.74     | 1.16     | 11.8     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Ours)                           | **50.22** | **82.26** | **90.08** |
| + MLP                                    | 49.48     | 81.12     | 89.58     |

|  3D Vision-language (3D VQA) | EM1       | EM10      |
| -------------- | --------- | --------- |
| SQA3D          | 47.20     | 86.82     |
| + LLaMA (Ours) | **48.09** | **89.03** |
| + MLP          | 47.14     | 88.08     |

Reviewer comment (Experiments):
Some additional ablations might have been of interest.

Author response:
We appreciate the reviewer’s excellent suggestions. We would like to first emphasize that our work is **the first** to discover that “*despite being trained solely on text data, a frozen LLM transformer can be used as a visual encoder layer for purely visual tasks, in the absence of language*”. So, **we prioritize validating the general applicability of this discovery across a wide range of visual tasks, in the most straightforward, simplest, and unified manner** – achieved by inserting a single LLM layer at the end of visual backbones. We do not claim that this simple strategy is the optimal approach for achieving the best performance. While we acknowledge that there are various variants to utilize specific LLM layers and insert them into visual backbone architectures which may further improve performance, we deliberately avoid optimizing model selection with more complicated or task-specific architectures. 

Following the reviewer’s suggestions, the table below presents an ablation study on different architecture variants. While some variants improve the performance while others decrease the performance, these ablation results contribute to strengthening our understanding and are orthogonal to our main discovery. We will include the results in the revision. And we leave a more comprehensive investigation on these and additional variants as interesting future work.

|      | Model             | Configuration                                               | Acc Top1     |
| ---- | ----------------- | -------------------------------------------------- | -------- |
| 1    | ViT-S             | Baseline ViT-S                                     | 75.3     |
| 2    | + LLaMA           | Single LLaMA block at the end of ViT | 75.8     |
| 4    | + LLaMA at Middle | Single LLaMA block at the middle of ViT            | 75.5     |
| 5    | + LLaMA at Head   | Single LLaMA block at the head of ViT              | 72.6     |
| 6    | + 2 LLaMA Blocks  | 2 LLaMA blocks at the end of ViT                   | **77.1** |

Experimental setting: We conduct the ablation experiments under our ablation setting of the original submission: ViT-S with 100 epochs on ImageNet. This is shorter than a full training of 300 epochs, due to our resource constraints. However, the results in the table are all compared under the identical and fair setup.

Reviewer comment (Experiments):
Why only transfer a single LLM layer? Does it also work with more?

Author response:
- *Q: Inserting more LLM blocks*. Please refer to Row 6 in the table above "+ 2 LLaMA Blocks." We use the final 2 LLaMA blocks, instead of the final single block, to evaluate the impact of using more LLaMA blocks. We find that using more LLM blocks seems beneficial for the overall performance. We think this is an interesting direction to further broaden our discovery and will include it in the revised paper.

Reviewer comment (Experiments):
Why is the layer inserted at the end of the ViT? How does the outcome change when the layer is inserted earlier?

Author response:
- *Q: Other ways to insert*. Please refer to Rows 4 and 5 in the table above "+ LLaMA at Middle" and "+ LLaMA at Head." They perform worse than inserting the LLM transformer block at the end of ViT (Row 2), supporting our design choice. Additionally, inserting at the end of encoders is a simple and unified strategy that can be employed across a wide range of tasks and diverse backbones; in contrast, achieving insertion at the middle may require scrutinizing backbones for each task individually.

Reviewer comment (Experiments):
How would the results change if we instead transfer a pre-trained vision layer? (see next point)

Author response:
4. *Q: Inserting layers from a visual model*.

Following the reviewer’s suggestion, we use the last transformer layer of CLIP-B/16 and insert it into ViT-S/16 for image classification. The result in the table below shows that this variant does not improve the performance. We hypothesize that this result may indicate that the pre-encoded knowledge in CLIP does not bring in additional useful knowledge for training visual backbones within our current paradigm.  

| Model   | Acc Top1      |
| ------- | -------- |
| ViT-S   | 75.3     |
| + LLaMA | **75.8** |
| + CLIP  | 74.5     |

Reviewer comment (Experiments):
The transfer from a language model to a vision model is interesting and most surprising that it works. However, to generalize the approach, one could have also tested to transfer layers from other types of foundation models, e.g. vision-language models (CLIP) or self-supervised vision models (DINO). If there is a considerable gap between LLM and vision layers, one could conclude that, while it works, LLM layers might not be optimal for the domain.

Author response:
We appreciate the reviewer’s perspective on interpreting our method as a more general approach to transferring knowledge from any foundation model beyond LLMs. As clarified earlier, our paper focuses on LLM transformers, for the reason exactly pointed out by the reviewer – “the transfer from a language model to a vision model is interesting and most surprising that it works.” Investigating other types of foundation models contributes to strengthening our understanding and is orthogonal to our main discovery on LLM transformers. In addition, the result in the table above suggests that investigating other types of foundation models is not trivial, and we leave this line of exploration as interesting future work.

Reviewer comment (Experiments):
It remains an open question if this approach scales with large-scale models and training. One could expect diminishing returns as data and model allow for learning equally capable layers. This might still make this approach attractive for smaller scales so it would be interesting to know about this potential limitation. While this is difficult to evaluate, are there any indications how it might behave?

Author response:
5. *Q: How will it scale*? 

We thank the reviewer for the insightful question. Based on our current empirical evaluation, we do not have conclusive evidence indicating the performance gain in scale-up scenarios. Moreover, it seems that this may also vary depending on specific tasks. For example, the improvement from LLM blocks decreases from ViT-Tiny to ViT-Small (Table 1, Page 4) in image classification, but increases from ViT-Small to ViT-Base (Table 3, Page 5) in action recognition. As mentioned by the reviewer, “this is difficult to evaluate,” requiring substantially increased computational resources, and we leave this line of exploration as interesting future work.

On the other hand, we would like to further emphasize that our approach is not intended for an optimal direction to scale up models for **individual tasks**. Instead, it is designed to reveal that **a frozen LLM transformer with unchanged parameters** can surprisingly function as an encoding layer for **diverse visual tasks**.

Reviewer comment (Experiments):
Which task and dataset was evaluated for the results in Tab. 6 and Fig. 2?

Author response:
6. *Q: Clarification of Tab. 6 and Fig. 2?* 

Sorry for the confusion and we will revise the paper to better clarify this. Both Table 6 and Figure 2 (Page 6) are for image classification on ImageNet. The only difference is: Table 6 is trained for the standard 300 epochs, while the ablation in Figure 2 is trained for a more efficient 100 epochs, due to our limited resources. Specifically, fully training one model on ImageNet would occupy most of our GPUs for 3-4 days.

Reviewer comment (Experiments):
The information filtering hypothesis allows some valuable insights. At the same time, it seems to be most prominent after $F_{LM}^A$ judging by Fig. 3, but then in Fig. 4, $F_{LM}^A$ obtains a low score for ViT-B-LLaMA. Why is this the case? Which model size is evaluated in Fig. 3? Is this observation not consistent across architectures?

Author response:
7. *Q: Questions on the analysis of the information filtering hypothesis?* 

The visualization in Figure 3 and the quantitative evaluation in Figure 4 are consistent with our information filtering hypothesis. Below we clarify the specific questions one by one. 

- *Q: Which model is evaluated in Fig. 3?* Figure 3 uses ViT-S and ViT-S-LLaMA models for visualization.

- *Q: Why $F_{LM}^A$ has a lower score on ViT-B?* Note that Figure 3 shows the visualization on ViT-S and ViT-S-LLaMA. Now we present additional visualization on ViT-B and ViT-B-LLaMA, shown in [[this anonymous image link]](https://imgur.com/a/XCtD9kz). This image explains why $F_{LM}^A$ in ViT-B-LLaMA has a lower score than ViT-B: **the attention layer $F_{LM}^A$ still clearly distinguishes relevant regions from irrelevant regions**, but the relevant regions are assigned lower activations, which corresponds to a smaller IoU in Figure 4. However, **the illustration in the new ViT-B figure and Figure 4 are still consistent with our information filtering hypothesis**, because the LLM layer finally converts the relevant regions into highly-activated tokens to enlarge their contribution in $F_{LM}^F$ and $F_L^{2}$. 

- *Q: Is the observation consistent across architectures?* **Our information filtering hypothesis has consistent observation across architectures**: the features processed by our LLM blocks ( $F_{LM}^F$ and $F_{L}^2$) consistently show a clearer concentration to relevant regions. Meanwhile, we acknowledge that the specific behaviors and utilities of each layer remain an open-problem, *e.g.*, the observed phenomenon that $F_{LM}^A$ can discern tokens in diverse ways.

Reviewer comment (Experiments):
Although the inserted LLM block is frozen, it is still not clear whether the performance gains truly stem from it. How about adding a randomly initialized LLM block for comparison? In Table 2, regarding the point cloud recognition task, the performance gains are marginal, especially on ModelNet40 (degraded performances for 1k and 4k).

Author response:
2. *Q: Training from a randomly initialized LLM block*. 

In the tables below, we present the ablation studies on *training from a randomly initialized LLM block* for all the tasks investigated in our paper, including image classification, point cloud recognition, action recognition, motion forecasting, 2D vision-language, and 3D vision-language. We will include these results in the revision. The results in the tables show two main findings which validate that the performance gains truly stem from our method:

- **Our method of adding a frozen LLaMA transformer block consistently outperforms using a randomly initialized transformer block**.
- **Naively adding a large transformer block can be detrimental for many tasks**, resulting in inferior performance even compared to the baseline without the addition of a transformer block. This is because the naive strategy increases the challenges for optimization and suffers from a small training dataset compared to LLM pretraining.

| Image classification   | Acc Top1     | Acc Top5     |
| ---------------------- | -------- | -------- |
| ViT-S                  | 80.1     | 95.0     |
| + LLaMA (Frozen, Ours) | **80.7** | **95.4** |
| + LLaMA (Random Init)  | 76.9     | 91.9     |


| Point cloud classification | BG       | OBJ      | T50 (Hardest) |
| -------------------------- | -------- | -------- | ------------- |
| PointBERT                  | 87.4     | 88.1     | 83.1          |
| + LLaMA (Frozen, Ours)     | **88.0** | **88.5** | **83.8**      |
| + LLaMA (Random Init)      | 87.2     | 88.0     | 82.6          |

| Action recognition     | Acc Top1     | Acc Top5     |
| ---------------------- | -------- | -------- |
| ViT-S                  | 64.7     | 89.2     |
| + LLaMA (Frozen, Ours) | **65.9** | **89.9** |
| + LLaMA (Random Init)  | 64.1     | 88.8     |

| Motion Forecasting    | ADE↓     | FDE↓     | MR↓      |
| --------------------- | -------- | -------- | -------- |
| MMTransformer         | 0.72     | 1.10     | 10.7     |
| + LLaMA (Frozen, Ours)  | **0.71** | **1.08** | **10.5** |
| + LLaMA (Random Init) | 0.74     | 1.15     | 11.5     |

| 2D Vision-language (Zero-shot Retrieval) | EM1       | EM5       | EM10      |
| ---------------------------------------- | --------- | --------- | --------- |
| METER                                    | 49.66     | 80.86     | 89.48     |
| + LLaMA (Frozen, Ours)  | **50.22** | **82.26** | **90.08** |
| + LLaMA (Random Init)                    | 49.80     | 81.62     | 89.72     |

| 3D Vision-language (3D VQA) | EM1       | EM10      |
| --------------------------- | --------- | --------- |
| SQA3D                       | 47.20     | 86.82     |
| + LLaMA (Frozen, Ours) | **48.09** | **89.03** |
| + LLaMA (Random Init)       | 47.26     | 88.46     |

Reviewer comment (Experiments):
Why only insert one frozen LLM block for visual encoding? Can adding more blocks bring more performance gains?

Author response:
- *Q: Inserting more LLM blocks*. Please refer to Row 6 in the table above "+ 2 LLaMA Blocks." We use the final 2 LLaMA blocks, instead of the final single block, to evaluate the impact of using more LLaMA blocks. We find that using more LLM blocks seems beneficial for the overall performance. We think this is an interesting direction to further broaden our discovery and will include it in the revised paper.

Reviewer comment (Experiments):
Is there any other way to insert the frozen LLM block in visual encoders? How about adding it at the beginning of visual transformers, or in the middle of visual transformers?

Author response:
- *Q: Other ways to insert*. Please refer to Rows 4 and 5 in the table above "+ LLaMA at Middle" and "+ LLaMA at Head." They perform worse than inserting the LLM transformer block at the end of ViT (Row 2), supporting our design choice. Additionally, inserting at the end of encoders is a simple and unified strategy that can be employed across a wide range of tasks and diverse backbones; in contrast, achieving insertion at the middle may require scrutinizing backbones for each task individually.

Reviewer comment (Experiments):
The authors removed the attention mask and positional embedding for the frozen LLM block. Is there any experimental study for these designs?

Author response:
6. *Q: Attention mask and positional embedding*. 

We would like to clarify that **our approach uses the attention mask strategies and positional embeddings identical to those used in the corresponding visual tasks**. We do not use the auto-regressive attention masks and rotary positional embeddings designed for LLMs, since they are not directly applicable to visual tasks, and thus we do not include experiments for this. Specifically, most visual models do not use auto-regressive attention masks or rotary positional embedding, because their tokens are *2- or 3-dimensional* and should be *considered all at once*, compared to a *1-dimensional temporal sequence* in language. For example, ViT (Dosovitski *et al.*) for classification uses learnable positional embedding and no attention masks; and VectorNet (Gao *et al.*) for motion forecasting encodes 3D coordinates into features and only uses attention masks to indicate padded tokens. Therefore, we directly follow the common practice of each visual task.

Dosovitski *et al.* An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.

Gao *et al.* Vectornet: Encoding hd maps and agent dynamics from vectorized representation. CVPR, 2020.