Paper: CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR-2024)

Reviewer comment (Reproducibility):
In Alg.1, FIND function maps a localized object, a relation and a set of candidate objects to an output localized object. What is the detailed process of FIND? Is the relations between objects known?

Author response:
The relations between objects are not known. Thus, we have to extract them.  
First, we extract the entire mentioned objects and their relations from the utterance using the scene graph parser [[1]](https://aclanthology.org/W15-2812.pdf), [[2]](https://openaccess.thecvf.com/content/CVPR2023/papers/Wu_EDA_Explicit_Text-Decoupling_and_Dense_Alignment_for_3D_Visual_Grounding_CVPR_2023_paper.pdf).  

Our method involves iterating over all anchors extracted from the utterance and searching for candidate objects in P that share the same class.  
When the anchor class is represented singularly in the scene, we return the matched object.  
However, we run the FIND function in scenarios where disambiguation is required due to multiple objects belonging to the anchor class.  
The FIND function leverages the parsed spatial relations and the localized objects, so far, within the scene to identify the remaining anchors accurately.  
However, it is not guaranteed that the FIND function will be able to localize the remaining unlocalized anchors accurately.  
Thus, in this case, as shown in the last step in Algorithm 1, we randomly sample an object of the same class.

Reviewer comment (Experiments):
Alg.1 receive the input of object proposals P. Is it the output from the 3D object detection model which requires extra training? If so, the anchor localization may be bottlenecked by the object proposals.

Author response:
The prominent role of the localization module is to assign an object proposal for each anchor mentioned in the input description without requiring human effort.  
A pertinent example can be drawn from the labeling procedure employed in PhraseRefer, which demanded a cumulative workforce commitment of 3664 hours.  

Thus, we use the GT object proposals P from the ScanNet dataset, following the related works, e.g., [ReferIt3D](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460409.pdf).  

Hence, given a new dataset, our algorithm requires two pieces of information: bounding boxes (GT or predicted) and at least one localized object.  
Then, our algorithm works iteratively, starting from prior knowledge, i.e., at least one localized anchor/target, to localize the rest of the mentioned anchors in the utterance given the previously localized ones without the need for any human efforts.  

To show the influence of this module, we report the results for two variants for each baseline, one using our pseudo-noisy labels (Algorithm 1), which is human-free, and the second one using the anchors GT annotations from “PhraseRefer.”  
Significance performance gain can be shown for the MVT case in Table 4, where using our pseudo labels, we outperform “PhraseRefer” and “ScanEnts,” which utilize GT anchors annotations.  
Furthermore, using their manual annotations instead of our pseudo labels, we achieve SOTA results and outperform them significantly.  
This emphasizes the importance of our proposed approach.

Reviewer comment (Experiments):
Considering the elaborate process of psuedo label generation, the propose frameworks seems not "data efficient" in data preparation, though it is data efficient in 3D grounding training.

Author response:
We believe our answers to the previous two questions should answer your concerns and alleviate any misunderstanding about how our pseudo-label generator works.  

If any more clarifications are needed, please let us know; we would love to elaborate more.

Reviewer comment (Theory):
The interpretability is limited. From the input end, GPT-3.5 is used for extract the logical order of objects given an input utterance, thus adding interpretability. But the logical relations of the objects is not explicit modeled. Instead, it is implicit learned in the Seq-2-Seq network.

Author response:
We argue that modeling the relations implicitly is enough, as to predict the logical order of objects correctly is a much more complex problem.  
To solve it, the model has to extract the relations correctly first.  
To probe our claim, we conducted an experiment where we added a new training objective for the language head.  
Instead of only predicting the target of the anchors from the input utterance, we add another loss on predicting the relationship between the objects as pairs and use [bipartite loss](https://arxiv.org/abs/2005.12872).  
However, as shown in the following Table, there is no gain from learning the relations explicitly.  

| **Method** | **Grounding Accuracy** |  
|----------|------------------------|  
| **MVT**                    | 55.1 |  
| **MVT+CoT**                | **64.4** |  
| **MVT+CoT+Realtions Objective**                | **64.2** |

Reviewer comment (Novelty):
The proposed idea which pays attention to not just the target object, has been proposed before in “PhraseRefer”. They use human annotators to obtain fine grained annotation, which shares the same underlying idea with this paper. To me one of the major novelties of this paper is that they explicitly model the logical order while the “PhaseRefer” uses an implicit manner. I think these two works are highly related, but the author didn’t give enough introduction of this “PhraseRefer”, which makes it a bit difficult to understand the performance comparison e.g. Table 4. SAT with PhraseRefer/ScanEnts and yours. Besides, I believe some numbers are missing in this Table 4, e.g. MVT with PhaseRefer. The authors should give a more clear comparison with this line of works that use fine grain annotation.

Author response:
> 1: The authors should give more clear comparison with this line of works that use fine grain annotation, i.e., “PhraseRefer” and “ScanEnts”.

Indeed, contrasting our work with the two recent works, “PhraseRefer” and “ScanEnts,” is essential, and we incorporated this response into the discussion section to clearly position our contributions compared to these works.

We acknowledge the great work proposed by “PhraseRefer” and “ScanEnts,” which paved the way for the importance of paying attention not just to the target object but also to the anchors.
Thus, underlying approaches have some similarities in terms of demonstrating the importance of including anchors in the pipeline.
However, there are a lot of significant differences:
1) Design a CoT framework for 3D visual grounding that explicitly models causal reasoning while interpreting the instruction inspired by the human perception system.
2) Show that we can design an efficient framework that achieves state-of-the-art results on three challenging benchmarks, i.e., Nr3D, Sr3D, and ScanRefer, without requiring human labels. Proper design of such an approach is pivotal in attaining a good performance  while circumventing the need for intensive human annotations.
A pertinent example can be drawn from the labeling procedure employed in PhraseRefer, which demanded a cumulative workforce commitment of 3664 hours, roughly equivalent to an extensive five-month timespan.

Thus, as shown in Table 4, we integrated our CoT architecture into four different baselines, i.e., LAR, SAT, MVT, and ViL.
Our framework shows a consistent gain across different baselines.
Most importantly, we report the results for two framework variants for each baseline, one using our pseudo-noisy labels (human-free) and the second using the anchors GT annotations from “PhraseRefer”.
Significance performance gain can be shown for the MVT case in the Table below, where using our pseudo labels, we outperform “PhraseRefer” and “ScanEnts,” which utilize GT anchors annotations.
Furthermore, when we use their manual annotations instead of our pseudo labels, we achieve SOTA results and outperform them significantly.
This emphasizes the importance of our proposed CoT framework.

| **Method** | **Grounding Accuracy** |
|----------|------------------------|
| **MVT**                    | 55.1 |
| **MVT+ScanEnts**           | 59.3 |
| **MVT+PhraseRefer**           | 59.0 |
| **MVT+CoT(Pseudo labels)**                | 60.4 |
| **MVT+CoT(GT labels)**                | **64.4** |

We hope this comparison clarifies the differences and poses = work clearly within the literature, specifically “PhraseRefer” and “ScanEnts”.

> 2: I believe some missing number in this TableTable.4 e.g. MVT with PhaseRefer.

Thank you so much for pointing out this point.
We missed this number as it was added in the second revised version of the PhaseRefer paper.
We incorporated this in the revised version in Table 4 and Table 6.

Reviewer comment (Experiments):
Do all numbers in Table 4 use a GPT-generated path? If not, answer my below question: If you have access to the GT anchor label either from “PhaseRefer” or “ScanEnts”, but you still require GPT to give you extra information “the Path”, how is the performance going in this case?

Author response:
> 3: Do all numbers in Table 4 use a GPT-generated path? If not, then, If you have access to the GT anchor label either from “PhaseRefer” or “ScanEnts”, but you still require GPT to give you extra information “the Path”, how is the performance going in this case?

We do not use the GPT-generated path during inference or test time.

The GPT-generated path is only used as a teacher label for training a simple MLP that predicts the path during the inference time.
Therefore, GPT is not involved in the system during inference, as we train a simple MLP layer, i.e., the Pathway-Head in Figure 3, to predict the logical order.
For instance, when the system is deployed on a robot, it will be efficient as our simple MLP Pathway-Head is utilized to generate a reasonable order based on the input description.

Accordingly, GPT could be seen as a design choice to obviate the need for human annotations during training.

In the Table below, the Pseudo and GT labels are for the anchors' localization, not the path.
For the path, we always use our MLP-predicted path.
However, the accuracy will not be affected if we replace it with the GPT-generated path.
As shown in the Table below, we achieve 64.4% and 64.5% using our predicted and GPT-generated paths, respectively.
This is an intuitive result, as when we assess the performance of the Pathway Head separately, we get 98% accuracy w.r.t the GPT-generated path.
Therefore, replacing the Pathway Head generated path by the GPT ones is expected not to impact the final performance.

| **Method** | **Grounding Accuracy** |
|----------|------------------------|
| **MVT**                    | 55.1 |
| **MVT+ScanEnts**           | 59.3 |
| **MVT+PhraseRefer**           | 59.0 |
| **MVT+CoT(Pseudo labels)**                | 60.4 |
| **MVT+CoT(GT labels)**                | 64.4 |
| **MVT+CoT(GT labels+GPT Path)**                | 64.5 |

This experiment is added in the revised version in the Appendix section.

Reviewer comment (Experiments):
In Sec.3.3 Anchors parser “Then, we match the objects to their closest matching class from the ScanNet labels”, would you explain this in more detail? Correct me if I am wrong: I assume if you assign a more fine grain label set, like ScanNet200, you are actually providing richer semantic knowledge to the model. My question is how do you assign the label, do you use the same level of semantic information with the others?

Author response:
> 4: In Sec.3.3 Anchors parser “Then, we match the objects to their closest matching class from the ScanNet labels”, Would you explain this in more detail? 

Due to the free-from nature of Nr3D, the anchors mentioned in the GT descriptions sometimes do not precisely match the ScanNet class labels.
For instance, the GT description is “The plant at the far right-hand side of the bookcase tucked in the furthest corner of the desk.” However, there is no “bookcase” class in ScanNet.

Therefore, we need to match it to the nearest ScanNet class labels, which in this case will be “bookshelf.”
Thus, we first check whether the extracted anchor from the text matches one of the ScanNet class labels. 
If not, then we encode the anchor words and the complete list of the ScanNet class labels.
Then, we measure the cosine similarity on the feature space to get the closest one.
Considering that the anchor's name may contain several words, e.g., office chair or file cabinet, we can not use BERT encodings.
Instead, we leverage its extended version, capable of encoding sentence level instead of word level, termed [SBERT](https://arxiv.org/abs/1908.10084).

> 5: My question is how do you assign the label, do you use the same level of semantic information with the others?

Yes, we use the same semantic information level as the other work for a fair comparison.
However, it is straightforward to use ScanNet200 instead for richer semantics, but then we have to retrain all the existing work on ScanNet200 for fair comparison.
That's why, to avoid this and for a fair comparison, we adhere to the well-established setup by other related work.

Reviewer comment (Experiments):
From the numbers of MVT in Table 4 and Table 5, it seems that when the MVT baseline has access to more data, the performance can improve quite a lot. So is it possible that your proposed method introduces explicit knowledge to the model while MVT can have a chance to learn that out from the data if the dataset is large enough?

Author response:
> 6: From the numbers of MVT in Table.4 and Table.5, it seems that when MVT baseline has access to more data, the performance can improve quite a lot.
So it is possible that your proposed method introduce explicit knowledge to the model while MVT can has a chain to learn that out from the data if the dataset is large enough?

To study the effect of adding more data on the performance of our method compared to MVT, we will explore three case studies:

1) **Nr3D+Sr3D results**
From Table 4 and Table 5, when we use more data by training on Sr3D and Nr3D jointly instead of Nr3D alone, the MVT performance increased by 3.4% (from 55.1 to 58.5). 
On the other hand, our CoT approach performance increases by 2% (from 60.4 to 62.5).
When more data is used, this reduction in the performance gain could be considered a minor impact because the gap is only reduced by 1% despite increasing the training data 3 times.
Wherease the Nr3D dataset consists of 41.5K samples, while the Sr3D dataset contains 83.5K samples.

2) **ScanRefer results**
As shown in Table 6, our method introduces a consistent gain. 
For instance, using 70% of the data, we outperform MVT by 8%, where the grounding accuracies are 62.5% and 54.5% for our method and MVT, respectively.
Moreover, when more data is used, where 100% of the data is used instead of 70%, we still outperform MVT by 7%.

3) **Nr3D and Sr3D results**
As shown in Figure 2, our CoT framework introduces a consistent gain across different amounts of training data (10% - 100%) on four different baselines.

In conclusion, the aforementioned case studies show that adding more data enhances both MVT and our method equally, which implies that our CoT approach introduces a significant gain that can not be replaced by just scaling the data.

Reviewer comment (Evaluation):
**Detailed Failure Case Analysis.** While the paper mentions the identification of failure cases as a benefit of the interpretability, it does not provide a detailed analysis or examples of these cases. Including such an analysis could provide valuable insights into the limitations of the model and areas for future improvement.

Author response:
> 1: **Detailed Failure Case Analysis**. Can you provide a detailed analysis or examples of the identification of failure cases as a benefit of the interpretability?

To demonstrate our interpretability abilities, we visualize attention maps as shown in Figure 5 in the Appendix.
In Figure 5, the input description is “the chair to the far left hand side of the taller desk with the computer monitor on it.”
Accordingly, there are two anchors mentioned in the description, i.e., desk and monitor, and the target is the chair. The correct chair should be number two, however, the model predicts number four. 
By visualizing the attention maps, on the left part of Figure 5, we can identify the main cause of the wrong prediction, whereas, the first anchor localize a wrong desk (desk \#3) instead of desk \#2.
Therefore, the rest of the chain, i.e., the monitor and the chair are localized wrongly.
This example also shows the ambiguity in Nr3D, where it is hard to say which desk is the taller, desk \#2 or desk \#3.

Reviewer comment (Evaluation):
Can you provide a detailed analysis or examples of the identification of failure cases as a benefit of the interpretability?

Author response:
> 1: **Detailed Failure Case Analysis**. Can you provide a detailed analysis or examples of the identification of failure cases as a benefit of the interpretability?

To demonstrate our interpretability abilities, we visualize attention maps as shown in Figure 5 in the Appendix.
In Figure 5, the input description is “the chair to the far left hand side of the taller desk with the computer monitor on it.”
Accordingly, there are two anchors mentioned in the description, i.e., desk and monitor, and the target is the chair. The correct chair should be number two, however, the model predicts number four. 
By visualizing the attention maps, on the left part of Figure 5, we can identify the main cause of the wrong prediction, whereas, the first anchor localize a wrong desk (desk \#3) instead of desk \#2.
Therefore, the rest of the chain, i.e., the monitor and the chair are localized wrongly.
This example also shows the ambiguity in Nr3D, where it is hard to say which desk is the taller, desk \#2 or desk \#3.