Paper: Task-Guided Biased Diffusion Models for Point Localization (ICLR-2024)

Reviewer comment (Presentation):
The structure is not ideal. The details of diffusion models should be given in Section 2 under a subsection titled 'background' and not in Section 3. In addition, the position of tables/figures and the places when they are referenced should be improved. Figure 3 is referenced in the introduction, but is located in page 5. While Figure 1 is located in page 3, but is referenced for the first time at the end of page 4. Similarly, Table 1, is placed in page 6, but is referenced for the first time in page 8, while table 2 is introduced and referenced in page 7. This makes reading the paper quite confusing.

Author response:
We have reorganized the paper for better understanding.  Please refer to the revised paper.

Reviewer comment (Evaluation):
Why does the AUC drop in Figure 3b at step 10?

Author response:
The AUC drops at step 10 as only 10 steps can’t produce high-quality images in diffusion models.

Reviewer comment (Experiments):
Why was a discrete setting was implemented instead of an SDE diffusion model [3]? Furthermore, more modern conditioning strategies can be used [4].

Author response:
This work mainly focuses on how to apply diffusion models to point localization tasks. Therefore, we only evaluate DDPM, other improved diffusion models will be investigated in future works.

Reviewer comment (Novelty):
The necessity of applying diffusion models to point localization tasks is not illustrated. In other words, the positive properties of diffusion models seem not incorporated into this approach. Whether the diffusion model here can be replaced with other simple network for the same purpose while the posed two problems are correspondingly absent.

Author response:
**[Reply]**: We employ diffusion models for their ability to predict sparse outputs, beneficial for localization tasks. A comparative illustration between our method and a basic network is added in Figure 6.

Reviewer comment (Experiments):
The proposed method is not correlated to the task of point localization, as the targeted problems are general for diffusion models. So why the point localization tasks are chosen.

Author response:
**[Reply]**: Please refer to Q1.

Reviewer comment (Experiments):
Introducing the task-biased noise in the forward diffusion process have been proposed in literatures [1][2]. The comparison baselines should include these methods instead of all non-diffusion methods and the discussion is recommended.

Author response:
**[Reply]**: Thanks for the suggestion. We have added more discussion in the revised version.

Reviewer comment (Theory):
The motivation and hypothesis of the paper may be questionable: At the beginning of the abstract, the author “hypothesize that diffusion models can be used to enhance the performance of deep learning methods for predictive tasks involving sparse outputs, such as point-localization tasks.” Where is this hypothesis or intuition from? What kind of data distribution is learned by the diffusion model in this formulation? Why should the incorporation of a diffusion model be helpful for the sparse outputs? Although Sec. 3.2.1 describes some motivations, it is not very convincing and clear why the diffusion model is inherently helpful in this case.

Author response:
**[Reply]**: Diffusion models excel in modeling data distribution. In our study, we trained these models with sparse dot maps, enabling them to predict sparse outputs effectively. This is validated through the visualization of the predicted maps in Figure 6.

Reviewer comment (Novelty):
About the novelty of task-guided bias: In order to reduce sampling steps, the method uses the baseline prediction as initial prediction. This looks very similar to a bunch of previous works on conditional diffusion models, such as [1], while the paper seems not to mention any previous works in this line and no comparison as well. From Sec. 3.2.2, it looks like the forward and backward diffusion process all needs a biased noise distribution, does it mean that the training process of the diffusion model also requires the paired data (initial prediction and ground truth)? This may indicate that for each different dataset/task/baseline to obtain the initial prediction, it is required to train a new diffusion model? What kind of data distribution priors are learned in the process?

Author response:
**[Reply]**: Thanks for pointing out the work [1], additional discussion about it has been added in Section 2. Indeed, the diffusion model needs to be trained for a new dataset or task which is a common practice.

Reviewer comment (Theory):
About the purpose of task-guided loss: From the paper, this loss is proposed to reduce the variance of prediction from different noise so as to improve accuracy. First, it is not very clear to me why minimizing the stochasticity of the reconstructed samples can help with accuracy. Second, with such constraint, it seems the prior Gaussian noise distribution is collapsed or converted to a delta data distribution? What does this mean in the predictive task? If the stochasticity of the reconstructed samples is not preferred, why not just use a fixed input or deterministic sampling process?

Author response:
**[Reply]**: We didn't claim that accuracy is improved because we minimized the stochasticity. Since we apply diffusion models to discriminative tasks, the task-guided loss explicitly supervises the model to predict the correct location. At the same time, it reduces the stochasticity of the prediction.

Reviewer comment (Evaluation):
No discussion about the method limitation in the paper.

Author response:
**[Reply]**: The primary limitation of our method is its increased computational demand compared to baseline models. This issue is discussed in Section 5.

Reviewer comment (Experiments):
The "biased" DDPM needs careful design: As I said before, the proposed biased diffusion model is just a diffusion model with prior. The computer vision community has multiple choices to inject priors to the diffusion models like conditions, texture inversions, etc. The author could provide more choices in the ablation step.

Author response:
**[Reply]**: In the proposed method, as illustrated in Figure 1, the condition is employed to inject prior information. It's important to note that the input of this condition is essential, as it provides information about the localization of persons.

Reviewer comment (Experiments):
The experiments part needs to be revised. From my side, the author could have a brief recap of diffusion models in 3.1, then give more space to provide ablations, and more detailed methods (like the proposed framework on different methods in the same task), FPS, training times, etc.

Author response:
**[Reply]**: According to the suggestion, we have reorganized the paper by reducing the content in Section 3.1 and adding more analysis in Section 4.3.