Paper: Diffusion Model for Dense Matching (ICLR-2024)

Reviewer comment (Novelty):
**Novelty** : The novelty is somewhat limited to the specific design of the conditioning to the diffusion model. Computation of cost volume using pretrained network have been used in many flow computation networks (as in Glu-Net). In particular, the main novelty is the smart choice of inputs and outputs for the diffusion model. Elaborating a bit more on the challenges of the design will help highlight the novelty of this approach.

Author response:
Thank you for your valuable feedback. Firstly, we would like to emphasize that a key novelty of our work is the novel formulation of dense correspondence as a diffusion generative process, a point also positively acknowledged by reviewers **8Vji**, **kgjX**, and **nL9F**. Our approach differs from previous methods [1,2,3] that relied on hand-crafted design prior terms or recent studies [4,5,6] that focus on the data term, assuming that the network architecture learns optimal matching priors with large-scale training datasets. Our unique contribution lies in **learning both the “data” and “prior” terms by leveraging a powerful generative model, the diffusion model**, demonstrating its effectiveness in the challenging dense correspondence task.

Regarding the novelty of our architecture design, we have conducted additional ablation studies compared with previous works [7,8,9,10] that applied diffusion models for specific dense predictions, such as semantic segmentation [7,8] and monocular depth estimation [7,9,10]. These studies typically use a single RGB image or feature descriptor as a condition for predictions aligned with the input RGB image. A concurrent study [11] has also applied a diffusion model to predict optical flow, using concatenated feature descriptors from both source and target images. However, this model is limited to scenarios with small displacements, typically in the optical flow task, which differs from the main focus of our study. DiffMatch, in contrast, aims to predict dense correspondence between two RGB images, $I_\mathrm{src}$ and $I_\mathrm{tgt}$, in scenarios involving textureless regions, repetitive patterns, large displacements, or noise. To achieve this, **we introduced a novel conditioning method using a local cost volume $C^{l}$ and initial flow field $F_\mathrm{init}$**, which captures pixel-wise interactions and initial guesses of dense correspondence between the images.

To validate the effectiveness of our architecture, in response to your insightful suggestion, we trained our model using only feature descriptors from the source and target, $D_\mathrm{src}$ and $D_\mathrm{tgt}$, as conditions. Please note that this approach is similar in architectural design to [7] and [11]. The quantitative results, which compare different conditioning methods, show that our conditioning method significantly outperforms those using only two feature descriptors. We attribute these results to our architectural design, which is specifically tailored for dense correspondence. Table 4 in the “Section 5.3 ABLATION STUDY” of our main paper further demonstrates the effectiveness of each component in our architecture. We will ensure to include an extended discussion and additional ablation studies on this aspect in the final version of our paper. Thank you.

| Conditioning Scheme | ETH3D AEPE↓ |
| --- | --- |
| Feature Concat. | 106.83 |
| DiffMatch | **3.12** |

Reviewer comment (Experiments):
**Generalization**: The framework shows impressive performance on dense matching for the given datasets, but providing a sense of how generalizable this is to in-the-wild captures, is potentially helpful.

Author response:
Thank you for your constructive suggestion. First, we would like to clarify that our method is trained by DPED-COCO [1], which consists of augmented synthetic image pairs DPED with randomly and independently moving objects in MS-COCO. This allows our model to alleviate overfitting to synthetic homography and enables our model to be generalized in non-rigid transform. 

As you kindly suggest, to address your concerns regarding generalizability, we have extended our evaluation to include the MegaDepth dataset, known for its large-scale collection of image pairs with extreme viewpoint and appearance variations. Following the procedure of PDC-Net+ [2], we tested on 1600 images. The quantitative comparisons below show that our approach outperforms PDC-Net+ on MegaDepth, highlighting the potential of our method for generalizability. 

| Methods | MegaDepth AEPE↓ |
| --- | --- |
| PDCNet+ | 63.97 |
| DiffMatch | **59.72** |

Furthermore, we have recently focused on integrating DiffMatch with other dense matching methods to improve its generalizability. This integration is achieved by replacing $F_\mathrm{init}$, derived from the feature backbone (VGG-16 in our case), with the flow field obtained from other models. **We evaluated our method on the semantic matching datasets PF-PASCAL [3] and PF-WILLOW [4]**. We used the state-of-the-art semantic matching model, SD-DINO [5], to determine $F_\mathrm{init}$. Please note that, for a comprehensive experiment, our model ideally should have been trained with SD-DINO. However, due to time constraints during the rebuttal period, we only used the flow field from SD-DINO as $F_\mathrm{init}$ for the pre-trained Conditional Denoising Diffusion Module in the sampling phase. **Interestingly, we found that DiffMatch significantly improves semantic matching performance compared to the original SD-DINO on both PF-PASCAL and PF-WILLOW.** **This indicates the potential of our Conditional Denoising Diffusion Module to be integrated on top of other dense correspondence methods to enhance their performance.** We plan evaluate DiffMatch on additional datasets and explore integrating our model as a plug-in for other models to make it more lightweight. We intend to include these findings in our paper as soon as the experiments are complete. Thank you again for your suggestion.

| Datasets |  | PF-PASCAL |  |  | PF-WILLOW |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | PCK@0.05 | PCK@0.1 | PCK@0.15 | PCK@0.05 | PCK@0.1 | PCK@0.15 |
| PDC-Net+ | 34.34 | 56.84 | 70.13 | 30.81 | 54.71 | 68.56 |
| SD-DINO | **71.67** | 86.04 | 91.92 | 67.26 | 88.61 | 94.32 |
| DiffMatch combined with SD-DINO | 70.67 | **88.69** | **95.15** | **67.37** | **89.39** | **95.29** |

Reviewer comment (Writing):
In this setting is $F_{init}$ extracted from VGG-16 considered the prior term ?

Author response:
Thank you for your question. Firstly, we would like to clarify that $F_\mathrm{init}$ is a condition for noise restoration in the Conditional Denoising Diffusion Module, closely related to the “data term” derived from the data, $I_\mathrm{src}$ and $I_\mathrm{tgt}$.

More specifically, in a probabilistic interpretation, the objective for dense correspondence includes a “data” term, which measures matching evidence between source and target features, and a “prior” term, encoding prior knowledge about correspondence. Specifically, the “prior” term in our paper discusses the formation of the matching field manifold and the distribution of the ground-truth flow field.

Traditional methods [1,2,3,4] typically reduce this matching prior to a hand-crafted “smoothness” constraint, suggesting that the flow at one point should be similar to those of its surrounding points. In contrast, current learning-based methods [5,6,7,8] emphasize the “data” term by training neural networks under the assumption that the network architecture itself can learn the optimal matching prior with a large-scale training dataset. However, our paper is the first to propose learning both the “data” and “prior” term simultaneously for dense correspondence, utilizing a diffusion-based generative model. We welcome any further questions you may have. Thank you.

Reviewer comment (Experiments):
**Possible generalization concerns and limited experimental evaluation**: modeling a prior on what a good matching field looks like using a diffusion model exposes the proposed solution to generalization problems since the prior will only model the type of matching flows seen during training. For example in the extreme case where the method is trained only with match fields coming from homographies it will probably not generalize well to other types of non-rigid transformations between frames. The competitors have this type of limitation in a less pronounced way since they focus on improving feature extraction and matching rather than modeling a global prior on what a “good matching field” should look like. Whether this problem arises in practice is hard to estimate from the current paper since the experimental validation is rather limited compared to the main competitors. The proposed method is evaluated only on two datasets for dense correspondence matching and on two corrupted versions of the same datasets. Competitors like GOCor, PDCNet and PDCNet+ are evaluated on other datasets (e.g., MegaDepth and RobotCar) and additional correspondence tasks (e.g., Optical Flow on KITTI, Pose Estimation on YFCC100M and ScanNet).

Author response:
Thank you for your valuable suggestion. First, we would like to clarify that our method is trained using DPED-COCO [1], which augments synthetic image pairs from DPED with randomly sampled independently moving objects from MS-COCO. This allows our model to alleviate overfitting to synthetic homography and enables it to generalize to non-rigid transformations.

As you thankfully suggest, we explore other datasets to evaluate the generalizability of our model. We wish to clarify that our method is specifically tailored for challenging dense correspondence tasks, including textureless regions, repetitive patterns, large displacements, or noise. In contrast, the KITTI and RobotCar datasets, which contain road sequences captured by stereo cameras, typically exhibit relatively small displacements and do not align with our primary objective. Similarly, ScanNet and YFCC100M are used for evaluating sparse matching tasks in outdoor and indoor pose estimation, respectively, while our focus is on dense matching. Recent works [2,3] have been evaluated on these datasets, post-processing the results with RANSAC [4]. This was achieved as the goal of these works is to estimate uncertainty of the predicted matches, so that they selectively choose confident matches to find homography by RANSAC.

To address your concerns regarding generalizability, **we have extended our evaluation to include the MegaDepth dataset**, known for its large-scale collection of image pairs with extreme viewpoint and appearance variations. Following the procedure of PDC-Net+ [2], we tested on 1600 images. The quantitative comparisons below show that **our approach outperforms PDC-Net+ on MegaDepth, highlighting the potential of our method for generalizability.** 

| Methods | MegaDepth AEPE↓ |
| --- | --- |
| PDC-Net+ | 63.97 |
| DiffMatch | **59.72** |

We plan to evaluate DiffMatch on additional datasets and explore the further generalizability of our method. We intend to include this finding in our paper as soon as the experiments are complete. Additionally, we have recently focused on integrating DiffMatch with other dense matching methods to improve their performance and our generazability. This will be further discussed in our response titled "Response to reviewer nL9F (Part 3/4)." Thank you again for your suggestion.

**Citations:**

[1] Prune Truong, et al. Gocor: Bringing globally optimized correspondence volumes into your neural network. NeurIPS 2020.

[2] Prune Truong, et al. Pdc-net+: Enhanced probabilistic dense correspondence network. IEEE PAMI 2023.

[3] Prune Truong, et al. Learning accurate dense correspondences and when to trust them. CVPR 2021.

[4] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381–395, 1981.

[5] Ham, Bumsub, et al. Proposal flow: Semantic correspondences from object proposals. IEEE PAMI 2017.

[6] Ham, Bumsub, et al. Proposal flow. CVPR 2016.

[7] Zhang, Junyi, et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. arXiv preprint arXiv:2305.15347 (2023).

Reviewer comment (Reproducibility):
**Inference time concerns**: Tab. 5 of the paper compares the inference time of PDCNet(+) vs the proposed method with 5 sampling steps and shows that the two proposals are comparable. However in Sec. 4.6 the authors mention that in practice they sample multiple times and average the diffused fields to get the final performance. Depending on how many samples are drawn it will have an impact on the runtime making it grow significantly. From the current paper it is unclear if this multiple sampling strategy is used in the experimental validation or only in Appendix C.2 and whether the inference time of Tab. 5 are taking this into account or not. If not (as it seems like from the text) the inference cost will be significantly higher than competitors.

Author response:
Thank you for pointing this out. We apologize for omitting the number of samples for multiple hypotheses in Table 5 in “Section 5.3 ABLATION STUDY”. We would like to inform you that in this analysis, we randomly sample 3 Gaussian noises as $F_T$ from an i.i.d. normal distribution for multiple hypotheses. Following your valuable suggestion, we have conducted a more thorough investigation into the trade-off between the number of hypotheses and time consumption. Please note that our framework includes batching the inputs, which significantly mitigates the impact of multiple hypotheses on inference time. Additionally, after addressing several coding issues, we now clearly present quantitative results on the relationship between the number of hypotheses and time efficiency. We observe that varying the number of hypotheses has a minimal effect on inference time, largely thanks to the efficiency gained from input batching. We will make sure to include this corrected table and discussion in the finalized version of our paper. Thanks again for pointing this out.

| Method | C-ETH3D AEPE↓ | Time [ms] |
| --- | --- | --- |
| PDC-Net | 29.50 | **112** |
| PDC-Net+ | 27.11 | **112** |
| DiffMatch (1 sample, 5 steps) | 27.52 | **112** |
| DiffMatch (2 sample, 5 steps) | 27.41 | 123 |
| DiffMatch (3 sample, 5 steps) | **26.45** | 140 |

Reviewer comment (Experiments):
**More ablation studies and unclear dependency on the initialization**: the core of the work is the use of a diffusion model to refine an initial estimation of a matching field ($F_{init}$). From the paper it is unclear how much the prior is able to recover in case of a bad initialization or not and whether, if possible, the model will need more diffusion steps to recover from a bad conditioning. I would have liked these aspects to be discussed as part of the ablation study. Another interesting ablation that would have nicely complemented the work would have been using the dense cost volume as conditioning to the diffusion process. If the concern is around hardware limitations a test should still be possible at lower resolutions.

Author response:
Thank you for your insightful question. In the Appendix, Table 6 and Figure 8 in “Section C.3 THE EFFECTIVENESS OF GENERATIVE MATCHING PRIOR” present a comparison between the initial flow field $F_\mathrm{init}$, derived from raw correlation, and the refined flow field resulting from DiffMatch on the harshly corrupted HPatches and ETH3D. While the raw correlation results in poor performance, with Average Endpoint Errors (AEPEs) of 165.0 and 96.30 respectively, DiffMatch effectively refines this poor initialization using the diffusion generative prior, achieving significantly better performance with AEPEs of 33.15 and 26.45, respectively.

Furthermore, as you insightfully suggested, considering the relationship between the initial flow and overall performance, there seems to be substantial potential for performance enhancement by using results from other models as the initial flow field. We appreciate your excellent suggestion. One issue under consideration is the need to extract the results of existing models for training. Due to time constraints, we were unable to conduct this experiment. However, we chose a simpler approach which replaces $F_\mathrm{init}$ in our VGG-16 pre-trained model with the flow field obtained from a pre-trained PDC-Net+. Unfortunately, as shown in the below table, we did not find any improvement over the performance of our paper. We think this is because the flow field from PDC-Net+ differs significantly from the VGG-16 $F_\mathrm{init}$ used in the training phase. The flow field from PDC-Net+ tends to produce much smoother results compared to the flow field from raw correlation, as PDC-Net+ learns smoothness in their network with a large-scale dataset.

|  | ETH3D AEPE↓ | C-ETH3D AEPE↓ |
| --- | --- | --- |
| DiffMatch  | **3.12** | **26.45** |
| PDC-Net+  | 3.14 | 27.11 |
| DiffMatch combined with PDC-Net+  | 3.55 | 26.49 |

Instead, we evaluated our method on the semantic matching datasets PF-PASCAL [1] and PF-WILLOW [2]. We used the state-of-the-art semantic matching model, SD-DINO [3], to determine $F_\mathrm{init}$. Please note that SD-DINO uses intermediate diffusion features in a pre-trained diffusion model, so there is no need for training or fine-tuning. We believe that not training the neural network and directly using the feature descriptors from the diffusion model seems to provide more adequate initial flow field for our network. **Interestingly, we found that DiffMatch notably improves semantic matching performance compared to the original SD-DINO on both PF-PASCAL and PF-WILLOW.** **This indicates the potential of our Conditional Denoising Diffusion Module to be integrated on top of other dense correspondence methods to enhance their performance.**

| Datasets |  | PF-PASCAL |  |  | PF-WILLOW |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | PCK@0.05 | PCK@0.1 | PCK@0.15 | PCK@0.05 | PCK@0.1 | PCK@0.15 |
| PDC-Net+ | 34.34 | 56.84 | 70.13 | 30.81 | 54.71 | 68.56 |
| SD-DINO | **71.67** | 86.04 | 91.92 | 67.26 | 88.61 | 94.32 |
| DiffMatch combined with SD-DINO | 70.67 | **88.69** | **95.15** | **67.37** | **89.39** | **95.29** |

We find your suggestion very promising and plan to explore integrating our model as a plug-in for other models to make it more lightweight. We intend to include these findings in our paper as soon as the experiments are complete. Thank you again for your suggestion.

**Citations:**

[1] Ham, Bumsub, et al. Proposal flow: Semantic correspondences from object proposals. IEEE PAMI 2017.

[2] Ham, Bumsub, et al. Proposal flow. CVPR 2016.

[3] Zhang, Junyi, et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. arXiv preprint arXiv:2305.15347 (2023).

---

Thank you for your insightful suggestion. As you mentioned, leveraging a global cost volume might indeed provide the model with pixel-wise interaction between given images, potentially enhancing performance. However, as you rightly pointed out, we have empirically found that the increased memory consumption required to build the cost volume, along with the increased channel dimension, pose significant challenges. To address this, as you kindly proposed, utilizing a global cost volume at a lower resolution could be an effective strategy to establish a coarse flow field. We are planning to conduct experiments that leverage a coarsely estimated flow field, derived from a global cost volume, as the initial flow field instead of those derived from raw cost volume. This approach, we anticipate, could further improve the overall performance of our model by providing a better initialization. We will include these findings in our paper as soon as the experiments are complete.

Reviewer comment (Theory):
The statement "These approaches assume that the matching prior can be learned within the model architecture by leveraging the high capacity of deep networks" needs clarification. Exactly how is the prior "learned within the model architecture"? Can we say something more precise about how the prior is captured, and how much of it, in these earlier methods?

Author response:
Thank you for your constructive question. We would like to clarify that the concept of “the matching prior within the model architecture” has been already discussed in previous works [1,2,3]. These studies argue that the matching prior inherently exists within the model architecture or is learned from large-scale training datasets. However, as you insightfully asked, “how the prior is captured in the model architecture and how much it is” might be challenging to precisely visualize, because we regard the “prior term” as encoding the general prior knowledge of correspondence and matching field manifold.

As earlier methods [4,5,6] design a hand-crafted prior term as a smoothness constraint, we can assume that the "smoothness" of the flow field is included in this "prior" knowledge of the matching field. Based on this understanding, we reinterpret Table 6 and Figure 8 in "Section C.3 THE EFFECTIVENESS OF GENERATIVE MATCHING PRIOR", showing the comparison between the results of raw correlation and learning-based methods [7,8,9]. Previous learning-based approaches predict the matching field with raw correlation between an image pair as a condition. We observe that despite the absence of an explicit prior term in these methods, the qualitative results from them exhibit notably smoother results compared to raw correlation. This difference serves as indicative evidence that the neural network architecture may implicitly learn the matching prior with a large-scale dataset. 

However, it is important to note that the concept of "prior" extends beyond mere "smoothness”. This broader understanding underlines the importance of explicitly learning both the data and prior terms simultaneously, as proposed in our paper. We will make sure to include this extended discussion in the final version of our paper. Thank you for pointing this out.

**Citations:**

[1] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. CVPR 2018.

[2] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. CVPR 2016.

[3] Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense correspondence. ICCV 2021.

[4] Javier Sánchez Pérez, Enric Meinhardt-Llopis, and Gabriele Facciolo. Tv-l1 optical ﬂow estimation. Image Processing On Line, 2013:137–150, 2013.

[5] Marius Drulea and Sergiu Nedevschi. Total variation regularization of local-global optical ﬂow. ITSC 2011.

[6] Manuel Werlberger, Thomas Pock, and Horst Bischof. Motion estimation with non-local total variation regularization. CVPR 2010. 

[7] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn. Fcss: Fully convolutional self-similarity for dense semantic correspondence. CVPR 2017.

[8] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical ﬂow using pyramid, warping, and cost volume CVPR 2018.

[9] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. CVPR 2017.

Reviewer comment (Writing):
Please don't write $1e^{-4}$ when (I assume) you mean $10^{-4}$. $e$ is the base of natural logarithms. If you must use scientific number formats (please do it only in code, not papers!), do note that it's written $1e-4$, not $1e^{-4}$.

Author response:
Thank you for pointing this out. We will fix our notations according to your suggestion in the revised version of our paper.

Reviewer comment (Novelty):
One notable improvement will be adding more discussions to diffusion based dense prediction networks, especially methods like DDP [1].

Author response:
Thank you for your question. First, we wish to clarify that the previous methods [1,2,3,4] applying a diffusion model for dense prediction, such as semantic segmentation [1,2], or monocular depth estimation [1,3,4], use a single RGB image or its feature descriptor as a condition to predict specific dense predictions, such as segmentation or depth map, aligned with the input RGB image. A concurrent study [5] has applied a diffusion model to predict optical flow, concatenating feature descriptors from both source and target images as input conditions. However, it is notable that this model is limited to scenarios involving small displacements, typical in optical flow tasks, which differ from the main focus of our study. In contrast, our objective is to predict dense correspondence between two RGB images, source $I_\mathrm{src}$ and target $I_\mathrm{tgt}$, in more challenging scenarios such as image pairs containing textureless regions, repetitive patterns, large displacements, or noise. To achieve this, we introduce a novel conditioning method which leverages a local cost volume $C^{l}$ and initial flow field $F_\mathrm{init}$ between two images as conditions, containing the pixel-wise interaction between the given images and the initial guess of dense correspondence, respectively.

To validate the effectiveness of our architecture design, as you have thankfully suggested, we train our model using only feature descriptors from source and target, $D_\mathrm{src}$ and $D_\mathrm{tgt}$, as conditions. Please note that this method could be a similar architecture design to DDP [1] and DDVM [5], which only condition the feature descriptors from input RGB images. We present quantitative results to compare different conditioning methods and observe that the results with our conditioning method significantly outperform those using two feature descriptors. We believe that the observed results are attributed to the considerable architectural design choice, specifically tailored for dense correspondence. Table 4 in "Section 5.3 ABLATION STUDY" of the main paper also shows the effectiveness of each component in our architecture. We will make sure to include this extended discussion and ablation studies on this aspect in the final version of our paper. Thank you for pointing this out.

| Conditioning Scheme | ETH3D AEPE↓ |
| --- | --- |
| Feature Concat | 106.83 |
| DiffMatch | **3.12** |

**Citations:**

[1] Yuanfeng Ji, et al. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303.17559, 2023.

[2] Zhangxuan Gu, et al. Diffusioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.02773, 2022.

[3] Saurabh Saxena, et al. Moncular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816, 2023.

[4] Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303.05021, 2023.

[5] Saurabh Saxena, et al. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. arXiv preprint arXiv:2306.01923, 2023a.

Reviewer comment (Theory):
It is questionable why DDP is not directly applicable to the task of dense matching.

Author response:
Thank you for your question. First, we wish to clarify that the previous methods [1,2,3,4] applying a diffusion model for dense prediction, such as semantic segmentation [1,2], or monocular depth estimation [1,3,4], use a single RGB image or its feature descriptor as a condition to predict specific dense predictions, such as segmentation or depth map, aligned with the input RGB image. A concurrent study [5] has applied a diffusion model to predict optical flow, concatenating feature descriptors from both source and target images as input conditions. However, it is notable that this model is limited to scenarios involving small displacements, typical in optical flow tasks, which differ from the main focus of our study. In contrast, our objective is to predict dense correspondence between two RGB images, source $I_\mathrm{src}$ and target $I_\mathrm{tgt}$, in more challenging scenarios such as image pairs containing textureless regions, repetitive patterns, large displacements, or noise. To achieve this, we introduce a novel conditioning method which leverages a local cost volume $C^{l}$ and initial flow field $F_\mathrm{init}$ between two images as conditions, containing the pixel-wise interaction between the given images and the initial guess of dense correspondence, respectively.

To validate the effectiveness of our architecture design, as you have thankfully suggested, we train our model using only feature descriptors from source and target, $D_\mathrm{src}$ and $D_\mathrm{tgt}$, as conditions. Please note that this method could be a similar architecture design to DDP [1] and DDVM [5], which only condition the feature descriptors from input RGB images. We present quantitative results to compare different conditioning methods and observe that the results with our conditioning method significantly outperform those using two feature descriptors. We believe that the observed results are attributed to the considerable architectural design choice, specifically tailored for dense correspondence. Table 4 in "Section 5.3 ABLATION STUDY" of the main paper also shows the effectiveness of each component in our architecture. We will make sure to include this extended discussion and ablation studies on this aspect in the final version of our paper. Thank you for pointing this out.

| Conditioning Scheme | ETH3D AEPE↓ |
| --- | --- |
| Feature Concat | 106.83 |
| DiffMatch | **3.12** |

**Citations:**

[1] Yuanfeng Ji, et al. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303.17559, 2023.

[2] Zhangxuan Gu, et al. Diffusioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.02773, 2022.

[3] Saurabh Saxena, et al. Moncular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816, 2023.

[4] Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303.05021, 2023.

[5] Saurabh Saxena, et al. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. arXiv preprint arXiv:2306.01923, 2023a.

Reviewer comment (Experiments):
Another possible improvement is to add diffusion-based dense prediction models as baselines to the method (e.g., a DDP model trained on dense flow supervision).

Author response:
Thank you for your question. First, we wish to clarify that the previous methods [1,2,3,4] applying a diffusion model for dense prediction, such as semantic segmentation [1,2], or monocular depth estimation [1,3,4], use a single RGB image or its feature descriptor as a condition to predict specific dense predictions, such as segmentation or depth map, aligned with the input RGB image. A concurrent study [5] has applied a diffusion model to predict optical flow, concatenating feature descriptors from both source and target images as input conditions. However, it is notable that this model is limited to scenarios involving small displacements, typical in optical flow tasks, which differ from the main focus of our study. In contrast, our objective is to predict dense correspondence between two RGB images, source $I_\mathrm{src}$ and target $I_\mathrm{tgt}$, in more challenging scenarios such as image pairs containing textureless regions, repetitive patterns, large displacements, or noise. To achieve this, we introduce a novel conditioning method which leverages a local cost volume $C^{l}$ and initial flow field $F_\mathrm{init}$ between two images as conditions, containing the pixel-wise interaction between the given images and the initial guess of dense correspondence, respectively.

To validate the effectiveness of our architecture design, as you have thankfully suggested, we train our model using only feature descriptors from source and target, $D_\mathrm{src}$ and $D_\mathrm{tgt}$, as conditions. Please note that this method could be a similar architecture design to DDP [1] and DDVM [5], which only condition the feature descriptors from input RGB images. We present quantitative results to compare different conditioning methods and observe that the results with our conditioning method significantly outperform those using two feature descriptors. We believe that the observed results are attributed to the considerable architectural design choice, specifically tailored for dense correspondence. Table 4 in "Section 5.3 ABLATION STUDY" of the main paper also shows the effectiveness of each component in our architecture. We will make sure to include this extended discussion and ablation studies on this aspect in the final version of our paper. Thank you for pointing this out.

| Conditioning Scheme | ETH3D AEPE↓ |
| --- | --- |
| Feature Concat | 106.83 |
| DiffMatch | **3.12** |

**Citations:**

[1] Yuanfeng Ji, et al. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303.17559, 2023.

[2] Zhangxuan Gu, et al. Diffusioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.02773, 2022.

[3] Saurabh Saxena, et al. Moncular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816, 2023.

[4] Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303.05021, 2023.

[5] Saurabh Saxena, et al. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. arXiv preprint arXiv:2306.01923, 2023a.

Reviewer comment (Experiments):
It is common for dense matching models to also test on various tasks such as optical flow estimation (KITTI) and two-view geometry estimation (ScanNet / YFCC100M). Are there specific reasons your model cannot achieve these similar tasks?

Author response:
Thank you for your valuable suggestion. We wish to clarify that our method is specifically tailored for challenging dense correspondence tasks, including textureless regions, repetitive patterns, large displacements, or noise. In contrast, the KITTI dataset, which contains road sequences captured by stereo cameras, typically exhibits relatively small displacements and does not align with our primary objective. Similarly, ScanNet and YFCC100M are used for evaluating sparse matching tasks in outdoor and indoor pose estimation, respectively, while our focus is on dense matching. Recent works [1,2] evaluated on these datasets, by post-processing the results with RANSAC [3]. This was achieved as the goal of these works is to estimate uncertainty of the predicted matches, so that they selectively choose confident matches to find homography by RANSAC.

To address your concerns regarding generalizability, **we have extended our evaluation to include the MegaDepth dataset, known for its large-scale collection of image pairs with extreme viewpoint and appearance variations.** Following the procedure of PDC-Net+ [2], we tested on 1600 images. The quantitative comparisons below show that **our approach outperforms PDC-Net+ on MegaDepth, highlighting the potential of our method for generalizability.** 

| Methods | MegaDepth AEPE↓ |
| --- | --- |
| PDC-Net+ | 63.97 |
| DiffMatch | **59.73** |

Furthermore, we have recently focused on integrating DiffMatch with other dense matching methods to improve its generalizability. This integration is achieved by replacing $F_\mathrm{init}$, derived from the feature backbone (VGG-16 in our case), with the flow field obtained from other models. **We evaluated our method on the semantic matching datasets PF-PASCAL [4] and PF-WILLOW [5]**. We used the state-of-the-art semantic matching model, SD-DINO [6], to determine $F_\mathrm{init}$. Please note that, for a comprehensive experiment, our model ideally should have been trained with SD-DINO. However, due to time constraints during the rebuttal period, we only used the flow field from SD-DINO as $F_\mathrm{init}$ for the pre-trained Conditional Denoising Diffusion Module in the sampling phase. **Interestingly, we found that DiffMatch significantly improves semantic matching performance compared to the original SD-DINO on both PF-PASCAL and PF-WILLOW.** **This indicates the potential of our Conditional Denoising Diffusion Module to be integrated on top of other dense correspondence methods to enhance their performance.**

We plan to evaluate DiffMatch on additional datasets and explore the possibility of integrating our model as a plug-in for other models to make it more lightweight. We intend to include these findings in our paper as soon as the experiments are complete. Thank you again for your suggestion.

| Datasets |  | PF-PASCAL |  |  | PF-WILLOW |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | PCK@0.05 | PCK@0.1 | PCK@0.15 | PCK@0.05 | PCK@0.1 | PCK@0.15 |
| PDC-Net+ | 34.34 | 56.84 | 70.13 | 30.81 | 54.71 | 68.56 |
| SD-DINO | **71.67** | 86.04 | 91.92 | 67.26 | 88.61 | 94.32 |
| DiffMatch combined with SD-DINO | 70.67 | **88.69** | **95.15** | **67.37** | **89.39** | **95.29** |


**Citations:**

[1] Prune Truong, et al. Learning accurate dense correspondences and when to trust them. CVPR 2021.

[2] Prune Truong, et al. Pdc-net+: Enhanced probabilistic dense correspondence network. IEEE PAMI, 2023.

[3] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381–395, 1981.

[4] Ham, Bumsub, et al. Proposal flow: Semantic correspondences from object proposals. IEEE PAMI 2017.

[5] Ham, Bumsub, et al. Proposal flow. CVPR 2016.

[6] Zhang, Junyi, et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. arXiv preprint arXiv:2305.15347 (2023).