Paper: V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection (ICLR-2024)

Reviewer comment (Presentation):
I suggest improving the order of presentation in Sec. 3.2 and Sec. 3.2. For example, I suggest moving the paragraph of "3DV-RPE" before talking about "canonical object space" and other details. When I read this part, I was quite confused by Sec. 3.2, not knowing how $R$ is generated, what is $P_i$, etc.

Author response:
Great suggestion! Thanks for pointing out this issue. **We have revised the organization of Sec. 3.1 and Sec. 3.2 to make it easier to follow. Please check the blue text in the revised PDF.**

Reviewer comment (Experiments):
As position encoding is the focus of this paper, I expect the authors to analyze or conduct ablation studies on more position encoding algorithms. Details are in the "questions" section below.

Author response:
Great suggestions! First, we would like to invite you to refer to Table 13 and the related discussion in our submission. We guess the comparison results with the latest position encoding algorithms (in the first two rows) might be the mentioned ablation studies on more position encoding algorithms. Besides, we also answer your other detailed concerns in the following response.

For your convenience, we also attached Table 13 in the following:

| method                                   | epoch | AP25 | AP50 |
| ---------------------------------------- | ----- | ---- | ---- |
| Baseline (w/o RPE)                       | 540x   | 71.4 | 47.6 |
| Baseline + CRPE (Stratified Transformer[1]) | 540x   | 74.7 | 58.1 |
| Baseline + CRPE (EQNet[2])                  | 540x   | 73.1 | 54.4 |
| Baseline + 3DV-RPE                       | 540x   | 77.8 | 66.0 |

Accordingly, our 3DV-RPE significantly outperforms these two advanced RPE schemes, especially on AP50 metrics (with gains of +7.9 and +11.6 respectively). We observe that both the CRPE (Stratified Transformer) and CRPE (EQNet) can achieve performance gains similar to a special case of our 3DV-RPE that only considers the 3D bounding box center (AP25=73.4/AP50=54.8). Therefore, it's clear that the key to effectiveness is to explicitly consider the relative position information between each point and the 8 vertex points of a given 3D bounding box. We will add the above information in the final revision.

[1] Stratified Transformer for 3D Point Cloud Segmentation, CVPR 2022  

[2] A Unified Query-based Paradigm for Point Cloud Understanding, CVPR 2022

Reviewer comment (Experiments):
I also haven't found the performance of the baseline without relative position encoding. In case I missed it, I suggest the authors put it into Sec. 4.3 or Sec. 4.4 for a clear ablation study.

Author response:
Good point! In fact, we have reported the performance of the baseline without relative position encoding in Table 6 (1st row and 5th row). We would like to reorganize Sec. 4.4 to place Table 6 and the related discussion at the forefront to make it clearer.

We also provide the detailed comparison results on ScanNet for reference:

|                        | \#epoch | AP25 | AP50 |
| ---------------------- | ------- | ---- | ---- |
| Our Baseline           | 540x    | 71.4 | 47.6 |
| Our Baseline + 3D-VPRE | 540x    | 77.8 | 66.0 |

Reviewer comment (Experiments):
**Baseline performance.** As mentioned in the weakness section, could you remind me where you have put the baseline performance? It is critical to recognize the improvement of 3DV-RPE. Technically, I wish to see that under the same normalization and other tricks, 3DV-RPE is indeed helpful.

Author response:
As answered in Q3, we have reported the baseline performance in Table 6 (1st row and 5th row) and attached the results for reference. Accordingly, we observe that the 3DV-RPE can boost the AP25 from 71.4 to 77.8 and AP50 from 47.6 to 66.0. We will explicitly add the above discussion in the final revision.

Reviewer comment (Experiments):
**Additional analysis.** With position encoding being the center of this paper, I think it necessary to conduct ablation studies on other common formats of position encoding, such as: Absolute position encoding, in both the formats you proposed like Eqn. 3 and Eqn. 4, or common sin-cos position encoding.

Author response:
Great point! First, as answered in Q2, we have reported the comparisons with two advanced RPE schemes specially designed for the 3D object detection task, including the CRPE (Stratified Transformer) and CRPE (EQNet). Second, we have also conducted the ablation experiments on the mentioned absolute position encoding in both the formats you proposed, like Eqn. 3 (in the updated PDF Eqn. 5) and Eqn. 4 (in the updated PDF Eqn. 2).

| method                                                        | epoch | AP25 | AP50 |
| ------------------------------------------------------------- | ----- | ---- | ---- |
| Baseline (w/o RPE)                                            | 540x   | 71.4 | 47.6 |
| Baseline + APE w/ Sin-Cos                                     | 540x   | 71.8 | 47.9 |
| Baseline + APE w/ MLP + NonLinear（Eqn. 4, update to Eqn. 2）                    | 540x   | 72.1 | 48.7 |
| Baseline + APE w/ MLP + NonLinear + Predefined Table（Eqn. 3, update to Eqn. 5 ） | 540x   | 72.0 | 48.5 |
| Baseline + 3DV-RPE                                             | 540x   | 77.8 | 66.0 |

According to the above results, we can observe that

- Our 3DV-RPE significantly outperforms all the APE variants by a large margin, especially on AP50 which requires more accurate localization capability.
- By comparing the 3rd row to the 2nd row, we observe that the MLP and non-linear transformation can slightly improve the performance of APE.

Reviewer comment (Experiments):
**Additional analysis (hyper-parameters).** More justifications of hyper-parameters. For example, where does 10 come from in $T$'s shape? May I use another number to replace 10?

Author response:
Good point! We chose $10$ considering the trade-off between performance and memory cost. We conducted ablation experiments on the size of $\mathbf{T}$ and reported the detailed comparison results in Table 12. We also included the results below for reference:

| 3DV-RPE table shape    | \#epoch | AP25 | AP50 |
| ---------------------- | ------- | ---- | ---- |
| $5\times5\times5$    | 360x    | 76.7 | 64.7 |
| $10\times10\times10$ | 360x    | 76.7 | 65.0 |
| $25\times25\times25$ | 360x    | 76.7 | 64.2 |

Accordingly, we have observed that our 3DV-RPE remains robust when faced with different choices for the shape of $\mathbf{T}$.

Reviewer comment (Experiments):
The author combines many technical improvements including normalizing box according to object size, one-to-many assignment as auxiliary loss, a modified resnet34-fpn backbone, and even TTA. They're all irrelevant to the claimed core contribution 3DV-RPE. So to prove the proposed module is effective, the most convincing way could be adding 3DV-RPE directly onto the baseline GroupFree3D. Considering that the 3DV-RPE module can work in a plug-and-play manner in theory, I would expect more results based on other methods such as GroupFree3D, 3DETR or CAGroupFree3D.

Author response:
Please refer to the general response for more details. In summary, our 3DV-RPE can significantly enhance the performance of both 3DETR and GroupFree3D. We will include these detailed comparison results in the final revision.

Reviewer comment (Experiments):
In Table 6, the author reports the one with 3DV-RPE + TTA (77.8 / 66.0). Does all other ablation attention results are also reported with TTA?

Author response:
Yes, we report all the ablation attention experiment results with TTA in Table 6. Furthermore, we presume you might also be interested in **the comparison results without TTA**, for which we provide the details as follows:

| method | # Epochs | AP25 | AP50 |
| ---------------------------- | -------- | ---- | ---- |
| None                         | 360x      | 67.9 | 43.5 |
| 3D Box Mask                  | 360x      | 72.9 | 58.3 |
| 3DV-RPE                      | 360x      | 76.2 | 64.2 |
| 3D Box Mask + 3DV-RPE        | 360x      | 75.3 | 61.5 |
| None                         | 540x      | 70.6 | 46.7 |
| 3D Box Mask                  | 540x      | 74.2 | 59.6 |
| 3DV-RPE                      | 540x      | 77.4 | 65.0 |
| 3D Box Mask + 3DV-RPE        | 540x      | 76.5 | 62.4 |

In conclusion, even without TTA, our 3DV-RPE continues to significantly enhance performance, especially the AP50 metrics, which necessitates precise 3D localization capabilities.

Reviewer comment (Experiments):
The author reported the best results for the proposed method, how about the ablations? How many times have you run for each ablation choice in all tables? Is the fairness of the comparison guaranteed?

Author response:
Yes, we have reported the best results from the ablation experiments.

Following the previous GroupFree3D methodology, we trained each setting $5\times$ times and tested each training trial $5\times$ times. We have reported the maximum performance from these $25\times$ trials to ensure fairness. Additionally, we are prepared to provide the average performance, if necessary.

Reviewer comment (Experiments):
Given the best set of parameters for the authors' final model, changing the choices in loss functions can affect the Hungarian matching cost matrix, thus it's hard to say whether the improvement / performance drop comes from the module or improper cost weights.

Author response:
Good point! We attempt to address your concerns about the impact of the loss function and Hungarian matching cost choices by replacing both with the original versions used in 3DETR, and we've reported **the comparison results without using TTA**:

| method | loss weights                                                                                                      | cost weights                                                                                                      | # Epochs | AP25 | AP50 |
| ------ | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------- | ---- | ---- |
| Ours   | $\lambda_{1}=2$,$\lambda_{2}=1$,$\lambda_{3}=0.5$,$\lambda_{4}=3$,$\lambda_{5}=0.1$,$\lambda_{6}=0.5$ | $\lambda_{1}=2$,$\lambda_{2}=1$,$\lambda_{3}=0.5$,$\lambda_{4}=3$,$\lambda_{5}=0.1$,$\lambda_{6}=0.5$ | 540x     | 77.4 | 65.0 |
| 3DETR  | $\lambda_{1}=1$,$\lambda_{2}=5$,$\lambda_{3}=1$,$\lambda_{4}=2$,$\lambda_{5}=0.1$,$\lambda_{6}=0.5$   | $\lambda_{1}=2$,$\lambda_{2}=0$,$\lambda_{3}=0$,$\lambda_{4}=1$,$\lambda_{5}=0$,$\lambda_{6}=0$       | 540x     | 77.0 | 64.7 |

According to the above comparison experiments, we can see that **the significant differences in the choices of loss functions and matching cost functions only bring slight performance gains**. We will include these results in the final revision.

Reviewer comment (Experiments):
Why are some ablations done with ScanNet while others use SUN-RGBD (e.g., Table 4)? Does it mean the coordinates normalization works similarly on ScanNetv2?

Author response:
The primary reason is that the ground-truth rotation angle annotations for all 3D bounding boxes in ScanNet are set to zero. Hence, the normalization of coordinates does not change the relative coordinates. We will incorporate this information into the final revision.

Reviewer comment (Novelty):
Actually I think the RPE and normalized coords are designed in similar ways: Point-RCNN has adopted converting boxes to canonical coords and doing normalizations on orientation. Moreover, in anchor-based detectors, they already use the anchor boxes' W and H to normalize the regression targets. Here the author uses dynamic bounding boxes from predictions, which has also been explored in methods like MetaAnchor, etc.

Author response:
Please refer to the general response. We will add references and discussions on connections to acknowledge related efforts in the final revision. We sincerely hope that our work will not be rejected solely because some aspects of our V-DETR share similar insights with previous studies.

Reviewer comment (Experiments):
There are many unrelated tricks applied to improve detection performance. I don't like the approach of doing whatever it takes to improve the results. Rather, the author should focus on the main contribution. After the core module is sufficiently discussed, one can further improve its results with more tricks. Here the author puts all stuff together, which makes me doubt where the improvement comes from.

Author response:
Good point! We would like to improve the organization of the experiments for clarity. Additionally, we have applied our 3DV-RPE to several other clean baselines, including 3DETR and GroupFree3D. Please refer to the detailed results in the general response

Reviewer comment (Evaluation):
How much fraction do the queries outside predicted bounding boxes account for with respect to the total number of queries? 10%? 20%? The author should provide an investigation into this problem.

Author response:
We guess you're referring to the fraction of queries localized outside the ground-truth bounding boxes, rather than within the predicted bounding box. Each query will predict a center and dimensions, ensuring that it is localized at the center position of the predicted bounding box.

We compile statistics on the fraction of queries that fall outside the predicted bounding boxes in relation to all queries as follows:

|                                                           | \#8-th decoder layer | \#6-th decoder layer | \#4-th decoder layer | \#2-th decoder layer | first-stage |
| --------------------------------------------------------- | -------------------- | -------------------- | -------------------- | -------------------- | ----------- |
| \# of the queries outside ground-truth box/\# all queries | 24.77%               | 25.62%               | 27.53%               | 32.79%               | 38.21%      |

In addition, we also compile statistics on the fraction of queries localized outside the ground-truth bounding boxes, considering only the matched queries selected by Hungarian matching as follows:

|                                                                       | \#8-th decoder layer | \#6-th decoder layer | \#4-th decoder layer | \#2-th decoder layer | first-stage |
| --------------------------------------------------------------------- | -------------------- | -------------------- | -------------------- | -------------------- | ----------- |
| \# of the matched queries outside ground-truth box/\# matched queries | 1.14%                | 1.77%                | 2.64%                | 4.26%                | 5.92%       |

Reviewer comment (Theory):
Why is the PE added in the way in Eq. 1? I think the form of matmul(Q, K) + R does not match the intended aim of the paper. Instead, I think matmul(Q+R, K) should be more proper, or add a relative PE to Q and a global PE to K.

Author response:
We address your concerns on the PE scheme as follows:

- First, we need to clarify that relative position encoding should explicitly encode the relative spatial relationships between each query and each key. Therefore, by default, the shape of the RPE should be $nQ\times nK$. Our implementation that directly adds RPE to matmul(Q, K) is the most intuitive one.
- Second,  the motivation for adding the RPE to matmul(Q, K) is to explicitly modulate the pairwise cross-attention map values between the 3D bounding box query and each of the 3D point clouds. In other words, the key (and value) point clouds localized in the query 3D boxes will receive larger values and smaller values otherwise.

Therefore, simply adding the PE to a query or the key essentially results in absolute PE instead of relative PE. We also present the results **(w/o TTA)** with absolute PE, which adds PE to both query and key, in the following table. As you can see, the absolute PE does not perform well.

| method                  | epoch | AP25 | AP50 |
| ----------------------- | ----- | ---- | ---- |
| Baseline + abosolute PE | 540x   | 70.7 | 47.1 |
| Baseline + 3DV-RPE      | 540x   | 77.4 | 65.0 |

In summary, the proposed RPE scheme provides a natural and effective implementation. We will include the above information in the final revision if necessary.

We would also like to invite the reviewer to take another look at an insightful study[1] on the relative position encoding scheme for Vision Transformers. Their experiments indicate superior performance of both the bias and contextual modes. **Our 3DV-RPE can be considered a effective designed version of the bias mode, specifically tailored for 3D object detection tasks.**

[1] Rethinking and Improving Relative Position Encoding for Vision Transformer, ICCV 2021

Reviewer comment (Writing):
The precomputed lookup table was the hardest to follow (Eq 3) since the connection to Eq 4 was not immediately obvious. One more sentence there to explicitly connect the two would be helpful. I.e. T represents a discretized set of possible \Delta P that we interpolate into.

Author response:
Thanks for pointing out this issue and providing great suggestions! **We have followed your suggestions to add the mentioned sentence between Eq. 3 and Eq. 4 (in the updated PDF: between Eq. 2 and Eq. 5) to make it clearer in the revised PDF.**

Additionally, following the comments from Reviewer fSn3, we've noted that the explanation could be further clarified by discussing "3DV-RPE" prior to introducing the concept of the "canonical object space".

Reviewer comment (Presentation):
Page 5 has a broken figure reference.

Author response:
Thanks for pointing out this issue! We will fix the typo "fig:rotatedRPE" (refers to Figure 4) in the final revision.

Reviewer comment (Reproducibility):
I don't understand how T in Eq(3) is initialized/set? What range do the T values take? -5 to 5 as indicated by the signed-log function?

Author response:
👉 The $\mathbf{T}$ is initialized/set with a 3D grid of size $10\times10\times10$. Each value is a 3-d vector and the values at position $(i,j,k)$ is computed by $\mathbf{T}[i, j, k] = (\frac{\mathrm{max value}(2i-(\mathrm{table size}-1))}{\mathrm{tablesize}-1},\frac{\mathrm{maxvalue}(2j-(\mathrm{tablesize}-1))}{\mathrm{tablesize}-1},\frac{\mathrm{maxvalue}(2k-(\mathrm{tablesize}-1))}{\mathrm{tablesize}-1})$.  (Eq(3) has been reordered to Eq(5) in the updated PDF)

👉 Yes, the $\mathbf{T}$ values are within the range of -5 to 5. We also provide a concise python implementation of the $\mathbf{T}$ initialization as follows:

```python
MAX_VALUE = 5
TABLE_SIZE = 10

# the initialization of T
relative_coords_table = torch.stack(torch.meshgrid(
    torch.linspace(-MAX_VALUE, MAX_VALUE, TABLE_SIZE, dtype=torch.float32),
    torch.linspace(-MAX_VALUE, MAX_VALUE, TABLE_SIZE, dtype=torch.float32),
    torch.linspace(-MAX_VALUE, MAX_VALUE, TABLE_SIZE, dtype=torch.float32),
), dim=-1).unsqueeze(0)

self.register_buffer("relative_coords_table", relative_coords_table)          
```

We create and initialize $\mathbf{T}$ with the meshgrid function and linspace function in pytorch. **We would like to add the above source code of the efficient implementation of 3DV-RPE in the final revision to make it clearer. We also ensure to release the source code of our approach after the paper is accepted.**

Reviewer comment (Novelty):
the proposed fix is also specific to DETR-based backbones.

Author response:
Good point! We want to point out that DETR-based methods have dominated the SOTA in 2D detection and 3D outdoor detection. Therefore, the proposed fix shows that the 3DV-RPE is key to building the SOTA 3D indoor detection system with the DETR-based method. We would also like to explore how to apply the proposed method to non-DETR-based methods in the future, based on your further valuable comments.

Reviewer comment (Theory):
the proposed fix may be vulnerable if the first stage predicting the coarse bounding boxes fail. For example, if the bounding boxes are very off, then preventing later layers to attend to out-of-the-box regions may make it impossible to recover.

Author response:
Good point! However, we have empirically found that this concern might not be valid, as we explain in detail in the following response.

First, we have compiled statistics on the ratio of (matched) first-stage predictions that fall outside the ground-truth bounding boxes. We found that **only 4.35% of the matched proposal boxes have their center points outside the target box, and a mere 0.06% have zero intersection over union (IoU) with the target box at all**. This is a relatively rare occurrence. **The reason is that DETR-based methods apply the Hungarian matching technique to select the top-ranking high-quality predictions based on cost functions that measure the accuracy of bounding box localization.** Therefore, the (matched) first-stage predictions are generally accurate enough to be used as input for the second stage.

Second, we have also compiled statistics on the ratio of (matched) predictions after refinement through subsequent transformer decoder layers. The proportion of refined predictions with their center points outside the target box decreases from 4.35% to 1.14%, and those with zero IoU at all drop to 0.02%. This indicates that our 3DV-RPE is capable of recovering low-quality first-stage predictions. The key reason is that **our 3DV-RPE can also capture useful long-range contextual information outside the box.** We have conducted a detailed analysis in the Section "Comparison with 3D box mask" (page 7) and in Table 6 (page 9).

We will conduct further studies if you can provide additional valuable comments.

Reviewer comment (Experiments):
there are also claims that are unsupported in the paper. For example, the sentence in the introduction section "We attribute the discrepancy to the limited scale of training data available for 3D object detection" is not well supported. Can you use less data to train 2D detectors to show it's really the data scale issue?

Author response:
Thanks for pointing out this issue! We agree this claim is not well supported and we would like to clarify it as follows.
 
👉 Our primary point is that **the scale of 3D object detection data is significantly smaller compared to that of 2D object detection tasks.** For instance, the number of different scenes (representing valid training samples) in ScanNetv2 is only 1,513, while the number of different images in COCO exceeds 110,000, rendering it approximately 70 times larger. We hypothesize that the limited amount of training data may complicate the learning of inductive biases, including locality.
 
👉 Furthermore, we conduct experiments on the 2D detector DETR, using approximately 1% of the training data (1,200 images), following your suggestion. We train DETR[1] for the same number of iterations (2,217,881 iterations) as the original DETR, which was trained on the full dataset for 300 epochs, while maintaining a batch size of 16. First, we observe that the mAP of the validation set drops from 44.9% to 10.4%, which closely aligns with the performance reported in Table 5 of [2]. Second, in accordance with Figure 6 of the DETR paper, **we visualize the cross-attention maps for the predicted object in Figure 11 of the revised PDF**. **We note that these attention maps fail to focus on local object regions, especially the object extremities. Therefore, it is evident that the scale of data has a substantial impact on the model's ability to effectively learn the locality inductive biases.**

[1] End-to-End Object Detection with Transformers, ECCV 2020

[2] DETReg: Unsupervised Pretraining with Region Priors for Object Detection, CVPR2022