Paper: Organ-DETR: 3D Organ Detection Transfomer with Multiscale Attention and Dense Query Matching (ICLR-2024)

Reviewer comment (Novelty):
Top-down processing is a typical design for detection. The technical novelty should be clearly pointed out in this paper. Another issue is that ICLR might not be the best place for this paper and more technical contributions can be emphasized.

Author response:
We express our appreciation to the reviewer for providing the comments and suggestions. According to the comments, we plan to incorporate a paragraph to the Introduction that highlights the key contribution of this study. We also plan to edit the experimental section to highlight the improvements significant by Organ-DETR. 

We share the belief that this study is well-suited for ICLR, given its emphasis on novel methods in deep learning. Throughout the manuscript, we treat organ detection as a 3D object detection problem, emphasizing efficient representation for detection tasks and a reliable matching strategy.

Once again, we appreciate all the comments, suggestions, and encouragement provided by the reviewer.  We hope we have addressed every comment satisfactorily. Your thoughtful input is greatly appreciated and we kindly invite the reviewer to raise any additional questions or points during the forthcoming reviewer-author discussion period, set to take place from Nov 10 to Nov 22.

Reviewer comment (Novelty):
Limited Novelty: The adoption of multi-scale attention in this paper is not novel, as it has been extensively explored in both medical imaging [1] and general visual recognition [2] contexts. [1] Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation [2] MAFormer: A Transformer Network with Multi-scale Attention Fusion for Visual Recognition

Author response:
The phrase "multi-scale attention" is broadly descriptive and caused confusion by suggesting that Organ-DETR is similar to earlier techniques such as those in references [1] and [2]. It is crucial to emphasize that Organ-DETR is quite distinct from these previous methods in terms of its methodology, conceptual framework, and practical implementation.

In general, [1] and [2] use transformer-based encoders, while Organ-DETR uses a CNN-based encoder. MSA, as a module, refines the features for DETR. Both [1] and [2] lack cross-stage attention, while our MSA incorporates inner-stage self-attention and inter-stage cross-attention. 

In detail, MERIT [1] introduces a cascaded backbone with two hierarchical transformers, generating multiscale features that might appear similar to our method. However, the objectives for utilizing these multiscale features differ significantly. In MERIT, the multiscale features contribute to generating multi-stage prediction maps, employed in a multi-stage feature mixing loss aggregation to enhance segmentation. In contrast, our MSA module focuses on cross-attention between multiscale features to capture long-range feature patterns.
In MAFormer[2], MAF blocks are employed within a single stage, lacking cross-stage operations. In contrast, our MSA module incorporates inner-stage self-attention and inter-stage cross-attention, effectively capturing long/short-range information. Additionally, the means to capture global features differ. MAFormer uses fully connected layers in the proposed Global Learning with Down-sampling (GLD) module, while our MSA module leverages cross-attention between multiscale features.

Reviewer comment (Theory):
One-to-Many Label Assignment: In the organ detection scenario, the use of a one-to-many label assignment strategy is not reasonable, considering that there is typically only one instance of each organ (e.g., liver, pancreas, spleen) in the human body.

Author response:
We have addressed this concern meticulously earlier in Sec. 2 and subsection “Dense Query Matching” in Sec 3. Below we briefly summarize DQM from the manuscript: 
The proposed DQM, as a one-to-many matcher, improves the TRUE POSITIVE MATCHING RATE by increasing the ground truth instances to enhance RECALL. Increasing the number of true positive instances in DQM simultaneously decreases the number of negative queries, leading to lower false positives (and thus, higher precision) and boosting the training’s learning pace. In INFERENCE, DQM selects the BEST-PREDICTED query per class/organ whose prediction score is the HIGHEST among the others. Figure 6 provides an ablation study on DQM parameters.
Based on the comment, there may have been confusion by the reviewer about the exact strategy of DQM and, therefore, also the confusion about its mechanism. To fix that confusion, we plan to mention different steps of DQM in Sec. 3 clearly to make the DQM’s methodology easy to follow.