Paper: From Language to 3D Worlds: Adapting Language Models for Point Cloud Perception (ICLR-2024)

Reviewer comment (Novelty):
This paper shares very similar spirits with many recent papers on Large (Vision) Language Models and Point Cloud Understanding, such as Gao et al. LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model, and Guo et al. Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following.

Author response:
First of all, we express our gratitude for your comprehensive review and the valuable insights you have provided. Your feedback has prompted a closer examination of our work, and we have updated our manuscript to better reflect the novelty and impact of our approach.

* Originality and Novel Contributions:

We thank the reviewer for the constructive suggestion, and we have discussed the difference between these parameter-efficient methods including LLaMa AdapterV2 and Point-Bind & Point-LLM (Sec 2.3). Furthermore, in response to your concerns about the originality of our work, we have elaborated on the distinct aspects of LAMP in Section 3.2, emphasizing its unique contributions beyond attention-based adaptation. Our approach introduces a new alignment strategy for the data distribution of 3D point clouds with language models, which is distinct from the methodologies proposed in the studies you mentioned. This strategy demonstrates a different take on how pretrained language models can be effectively applied to 3D point cloud perception tasks, especially in long-tailed and out-of-distribution problems, which has not been addressed by LLaMA-Adapter V2 or Point-Bind & Point-LLM. Meanwhile, our method achieves outstanding performance jointly learning point cloud and natural language in the 3D visual grounding. These unique contributions distinguish our method from others.

Reviewer comment (Novelty):
Essentially, this paper and other relevant papers are trying to bind point cloud representations to Language Models via adaptation. The attention-based adaptation has also been exploited in LLaMA-Adapter V2, which also supports point cloud inputs.

Author response:
* Methodological Distinction:

We have included a detailed comparison of our methodology with related works in Section 2.3. Here, we describe the unique aspects of our cross-modal self-attention learning and cross-modal cross-attention learning strategies. Unlike the attention mechanisms used in previous studies, our approach employs a novel projection network designed to harmonize point cloud embeddings with the parameters of language models, thus enabling more effective feature extraction and modality bridging.

Reviewer comment (Novelty):
The authors need to discuss the differences between their work and the above-mentioned works, and also compare their method with those methods in the experiments.

Author response:
First of all, we express our gratitude for your comprehensive review and the valuable insights you have provided. Your feedback has prompted a closer examination of our work, and we have updated our manuscript to better reflect the novelty and impact of our approach.

* Originality and Novel Contributions:

We thank the reviewer for the constructive suggestion, and we have discussed the difference between these parameter-efficient methods including LLaMa AdapterV2 and Point-Bind & Point-LLM (Sec 2.3). Furthermore, in response to your concerns about the originality of our work, we have elaborated on the distinct aspects of LAMP in Section 3.2, emphasizing its unique contributions beyond attention-based adaptation. Our approach introduces a new alignment strategy for the data distribution of 3D point clouds with language models, which is distinct from the methodologies proposed in the studies you mentioned. This strategy demonstrates a different take on how pretrained language models can be effectively applied to 3D point cloud perception tasks, especially in long-tailed and out-of-distribution problems, which has not been addressed by LLaMA-Adapter V2 or Point-Bind & Point-LLM. Meanwhile, our method achieves outstanding performance jointly learning point cloud and natural language in the 3D visual grounding. These unique contributions distinguish our method from others. 

* Methodological Distinction:

We have included a detailed comparison of our methodology with related works in Section 2.3. Here, we describe the unique aspects of our cross-modal self-attention learning and cross-modal cross-attention learning strategies. Unlike the attention mechanisms used in previous studies, our approach employs a novel projection network designed to harmonize point cloud embeddings with the parameters of language models, thus enabling more effective feature extraction and modality bridging.

Reviewer comment (Novelty):
Adapting Language Models to 3D point clouds with attention is straightforward and not that novel considering the above-mentioned literature.

Author response:
First of all, we express our gratitude for your comprehensive review and the valuable insights you have provided. Your feedback has prompted a closer examination of our work, and we have updated our manuscript to better reflect the novelty and impact of our approach.

* Originality and Novel Contributions:

We thank the reviewer for the constructive suggestion, and we have discussed the difference between these parameter-efficient methods including LLaMa AdapterV2 and Point-Bind & Point-LLM (Sec 2.3). Furthermore, in response to your concerns about the originality of our work, we have elaborated on the distinct aspects of LAMP in Section 3.2, emphasizing its unique contributions beyond attention-based adaptation. Our approach introduces a new alignment strategy for the data distribution of 3D point clouds with language models, which is distinct from the methodologies proposed in the studies you mentioned. This strategy demonstrates a different take on how pretrained language models can be effectively applied to 3D point cloud perception tasks, especially in long-tailed and out-of-distribution problems, which has not been addressed by LLaMA-Adapter V2 or Point-Bind & Point-LLM. Meanwhile, our method achieves outstanding performance jointly learning point cloud and natural language in the 3D visual grounding. These unique contributions distinguish our method from others. 

* Methodological Distinction:

We have included a detailed comparison of our methodology with related works in Section 2.3. Here, we describe the unique aspects of our cross-modal self-attention learning and cross-modal cross-attention learning strategies. Unlike the attention mechanisms used in previous studies, our approach employs a novel projection network designed to harmonize point cloud embeddings with the parameters of language models, thus enabling more effective feature extraction and modality bridging.

Reviewer comment (Experiments):
In addition to the 3D object datasets, the authors also need to evaluate their method on more realistic indoor 3D datasets such as SUN-RGBD and ScanNet.

Author response:
* Evaluation on Realistic Indoor 3D Datasets:

As suggested, we conducted experiments on the Shapenet-Core55,  PartNet, Objaverse-LVIS, and Scan200 datasets (New Table 7 & Sec.4.5). We hope that these experimental results can address your concerns.

| Model | Shapenet-Core55 (%) | PartNet (%) | Objaverse-LVIS (%)
|-------|---------------------|-------------|-------------|
| LAMP  | 83.0 Acc            | 50.5 mIoU  | 54.2 Acc

| Model    | Head (%) | Common (%) | Tail (%) | All (%) |
|----------|----------|------------|----------|---------|
| LGround [ECCV'22]  | 51.51     | 22.68       | 12.41     | 28.87    |
| LAMP   | 52.65     | 24.13       | 14.25     | 30.47    |


In conclusion, we hope that the revisions and additions to our manuscript, particularly in Sections 4.4 and 4.5, address your concerns regarding the originality and applicability of our work. We believe these enhancements underscore the novelty of LAMP and its contributions to the field of 3D point cloud perception.

We look forward to your further comments and suggestions.

Reviewer comment (Experiments):
The reviewer finds some performance in the paper somewhat unconvincing and also seems to lack a proper baseline to compare, particularly when referring to the results in Table 2. For instance, when comparing Point-MLP elite with LAMP, the performance appears quite similar, or even worse (considering Point-MLP elite has 90.9 as mAcc). While the trainable parameters of the Point-MLP elite are also minimal at 0.68M, its inference speed is anticipated to be notably faster. This is because, during inference, Point-MLP elite maintains its 0.68M parameters. Conversely, LAMP, despite reducing only its trainable parameters, is expected to have a longer inference time, given it leverages a significantly larger frozen language model. Thus, from a practical standpoint, the advantage of LAMP having a small number of trainable parameters but including an expansive language model seems to weaken its asserted advantages.

Author response:
We acknowledge your concern regarding the performance comparison with Point-MLP elite, especially as mentioned in Table 2. To address this, we have expanded our discussion in Section 4.1 and included additional analyses in Table 3 and Table 5, which demonstrate that LAMP outperforms advanced models like Point-MLP in terms of mean IoU scores in the dense prediction tasks of semantic segmentation and part segmentation. Meanwhile, LAMP also outperforms Point-MLP, **particularly in long-tailed classes and out-of-domain recognition tasks​​** (Table 6 and Table 7). We add a detailed discussion between LAMP and Point-MLP in Section 4.

Reviewer comment (Experiments):
Given that ModelNet40 is somewhat of a saturated benchmark, it would enhance the paper's credibility if LAMP were evaluated on more challenging datasets, for example, the ScanObjectNN classification benchmark. This would provide a clearer perspective on its efficacy and potential real-world applications.

Author response:
We agree that evaluating LAMP on more challenging datasets would strengthen our paper. In response, we have included the results of 
ShapeNetCore, PartNet, and challenging Objaverse-LVIS, Scan200 datasets are as follows (New Table 7 in the Manuscript):

| Model | Shapenet-Core55 (%) | PartNet (%) | Objaverse-LVIS (%)
|-------|---------------------|-------------|-------------|
| LAMP  | 83.0 Acc            | 50.5 mIoU  | 54.2 Acc

| Model    | Head (%) | Common (%) | Tail (%) | All (%) |
|----------|----------|------------|----------|---------|
| LGround [ECCV'22]  | 51.51     | 22.68       | 12.41     | 28.87    |
| LAMP   | 52.65     | 24.13       | 14.25     | 30.47    |

We hope that the experimental results on the newly added 4 benchmarks will address your concerns.

Reviewer comment (Experiments):
One of the advantages of the proposed approach is the reduced number of trainable parameters, and the reviewer is curious about what will happen if increasing the trainable parameters for projecting the point cloud features to language models.

Author response:
Your question about the impact of increasing trainable parameters is insightful. In response, we have conducted additional experiments to explore this aspect. For multimodal tasks, the trainable language model backbone can significantly improve performance while increasing trainable parameters (as detailed in Sec 4.6).

| Method            | Venue    | Input | Unique (%)    |       | Multiple (%)  |       | Overall (%)   |       |
|-------------------|----------|-------|---------------|-------|---------------|-------|---------------|-------|
|                   |          |       | Acc@0.25      | Acc@0.5 | Acc@0.25      | Acc@0.5 | Acc@0.25      | Acc@0.5 |
| EDA (Wu et al., 2023b) | CVPR'23 | 3D    | 85.76         | 68.57 | 48.11         | 36.82 | 53.83         | 41.70 |
| 3D-VisTA (Zhu et al., 2023) | ICCV'23 | 3D    | 77.00         | 67.90 | 37.90         | 30.40 | 45.20         | 37.30 |
| LAMP (Base scale + Frozen)               | Ours     | 3D    | 85.62 | 69.34 | 49.83 | 38.33 | 55.17 | 42.96|
| LAMP (Base scale + Finetuning)              | Ours     | 3D    | 86.74 | 69.87 | 50.06 | 39.77 | 55.66 | 43.54|
| LAMP (Large scale + Finetuning)              | Ours     | 3D    | **88.32** | **70.75** | **51.79** | **40.23** | **57.36** | **44.60**|

Reviewer comment (Writing):
If the reviewer understands it correctly, there might be a typo in section 3.3. Instead of W^{T}_q, it makes more sense to be W^{L}_q.

Author response:
Thank you for pointing out the potential typo in Section 3.3. Upon review, we have corrected the notation from $W^{T}_q$ to $W^{L}_q$, aligning it with our intended representation.

Reviewer comment (Evaluation):
In Table 8, the 'Infer.(s)' column seems unclear. Based on the reviewer's interpretation, it represents test times in seconds. Would it be possible to provide a metric showing the average time taken for each sample, possibly in milliseconds, for a more intuitive comparison?

Author response:
We appreciate your feedback for clarity on the 'Infer.(s)' column in Table 8. We have revised this table to present the inference time in milliseconds per sample 'Infer.(ms)', providing a more intuitive and direct comparison of the models' efficiency.

Reviewer comment (Experiments):
Although using LLM (Language Learning Models) for 3D point analysis is a good starting point, I notice that the main experiments in the article still focus on pure point cloud experiments, such as 3D Object Classification and Part Segmentation, etc. Tasks using Language Models typically focus on multimodal tasks (point cloud-text), like the 3D Visual Grounding mentioned in the paper. However, it seems that most of the experiments in the article still conventionally utilize task-specific heads for 3D point cloud analysis.

Author response:
We thank you for your constructive comment on the focus of our experiments. We have expanded our experimental section to include more multimodal tasks, particularly those involving text-3D tasks and 3D visual grounding. As detailed in Section 2, LAMP demonstrates its effectiveness and versatility on both unimodal and cross-modal tasks, validating the approach’s superiority over traditional methods​​. Besides outstanding experimental results on point cloud analysis, **LAMP also delivers a new state-of-the-art performance on 3D visual grounding task**.

Reviewer comment (Theory):
The use of the LLM+adapter pipeline doesn't seem very suitable for traditional 3D point cloud analysis. The reason why LLaVA and minigpt4 can effectively use the LLM+image adapter pipeline is that the final output space is still in the language space, so there's no issue with keeping the LLM fixed without further training. In this article, the output space is a traditional perceptual space, such as 3D classification or segmentation. To address these issues, one could either formulate traditional point analysis tasks as vision-language tasks with an output in the language space, or replace the LLM with a 2D image encoder to initialize parameters. The approach in the article seems somewhat unreasonable and odd. The author might want to reconsider it.

Author response:
We acknowledge your concerns about the use of LLM+adapter in traditional 3D point cloud analysis. We have included a comprehensive discussion in the manuscript (Table 1) to address this issue. We discuss the impact of the language model to understand the point cloud. It also reveals that a plain language model can read point clouds. Furthermore, it is more about the multimodal nature between nature language learning and point cloud analysis. Therefore, we propose a new approach for this task. Specifically, in Section 3, we describe how LAMP leverages language-pretrained models to transfer knowledge to 3D point clouds, exploring an under-utilized modality - language - for 3D tasks​​.

Reviewer comment (Experiments):
The experimental results don't seem to show a significant improvement. The ModelNet dataset is too small. Using a model the size of BERT might lead to overfitting? It seems that even simpler models already achieve good results, such as the PointNet++ from six years ago. Perhaps it's more appropriate to test on a larger dataset and then redefine all tasks as vision-language tasks.

Author response:
We have conducted additional experiments with larger datasets to address the concern of overfitting. Our results, which are now detailed in Section 4, indicate that LAMP achieves outstanding performance without overfitting, outperforming other competitive methods on the large-scale S3DIS dataset​​. Moreover, we demonstrate the robustness of LAMP in relieving the long-tail problem and out-of-distribution problems with substantial improvements over existing models as shown in Table 3 and Table 4​​.

Moreover, we are willing to conduct more experimental results on additional datasets including ShapeNetCore, PartNet, and challenging Scan200 datasets as follows:

| Model | Shapenet-Core55 (%) | PartNet (%) |
|-------|---------------------|-------------|
| LAMP  | 83.0 Acc            | 50.5% mIoU  |

| Model    | Head (%) | Common (%) | Tail (%) | All (%) |
|----------|----------|------------|----------|---------|
| LGround [ECCV'22]  | 51.51     | 22.68       | 12.41     | 28.87    |
| LAMP    | **52.65**     | **24.13**       | **14.25**     | **30.47**    |

Reviewer comment (Writing):
The paper writing needs improvement; it looks a bit rushed.

Author response:
We have made significant revisions to the paper to improve clarity and presentation. The revised sections clearly articulate the methodology and experimental results, ensuring that the paper conveys our contributions effectively.

Reviewer comment (Theory):
My primary concern is that the paradigm of LLM+adapter tends to align all inputs to the language space and then carry out language tasks or redefine the tasks as vision-language tasks. While some successors only use LLM as an inference model (e.g., NextGPT), I believe this might be problematic. Using LLM as an inference model is probably just because large models have only been trained on language. Otherwise, it doesn't make sense to use a language model as an inference model for image generation.

Author response:
We have elaborated on the paradigm of LLM+adapter in our revised manuscript. In Section 5, we discuss how our architecture extends LAMP to multimodal scenarios, such as 3D visual grounding, demonstrating the framework's ability to unify encoding across different input modalities​​. This is further evidenced by the new state-of-the-art performance on the ScanRefer dataset​​. We believe that LAMP's approach to modality divergence in vision-language tasks opens a promising direction for future research.