Paper: Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence (ICLR-2024)

Reviewer comment (Writing):
The usage of "aggregation" is a little confusing. In my opinion, "aggregation" means combining multiple features into a single one, which should usually be used to describe the process of attention aggregation of ($QK^T$ and $V$). However, I am not sure whether "aggregation" is suitable to be used to indicate the whole learning process of cost volume learning. Because many cost volume learning is not related to attention learning.

Author response:
> Unsure whether the term "aggregation" is suitable for operations performed over cost volumes.

In the stereo matching literature, "cost aggregation" plays a pivotal role. This process involves combining primary matching costs, typically through a filtering process, to generate a comprehensive disparity map, as noted in source [A]. The scope of "aggregation" in this context is broad and versatile. It can encompass various techniques ranging from the application of simple Gaussian kernels to more complex methods like 3D convolutions, as highlighted in source [B]. Additionally, the term also covers techniques such as semi-global or local attentions, as well as other forms of attention mechanisms, indicated in source [C].

These diverse aggregation techniques serve to illustrate the multifaceted nature of cost volume learning in stereo matching. The term "aggregation" is not merely confined to the domain of attention-based feature combination but extends to a variety of methods, each contributing to the accurate and efficient processing of stereo imagery. This wide-ranging application of the term confirms its appropriateness and relevance in the field, demonstrating that cost volume learning indeed benefits from and incorporates a spectrum of aggregation strategies. However, we can change the term to "refinement" if it better fits to describe our approach.
We highly appreciate the reviewer's comment in this regard.

[A] Cross-Scale Cost Aggregation for Stereo Matching
[B] Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation
[C] GA-Net: Guided Aggregation Net for End-to-end Stereo Matching

Reviewer comment (Experiments):
Although the authors analyze the merits of feature/cost aggregation, some claims have not been clarified. For example, feature matching is "challenged by repetitive patterns and background clutters", while cost volume learning enjoys "robustness to repetitive patterns and background clutter". No evidence is shown in this paper to support this claim.

Author response:
> Evidences to claim that feature matching is challenged by repetitive patterns and background clutters and cost aggregation enjoys robustness to repetitive patterns and background clutter. 


There is a well-established understanding in computer vision that feature matching can be particularly challenging in scenes with repetitive patterns and cluttered backgrounds. This difficulty arises from the reliance of feature matching techniques on identifying and aligning distinct keypoints or features between images. In environments where textures are repetitive or backgrounds are complex, these features often become ambiguous, leading to mismatches and inaccuracies. This limitation is widely recognized in the literature and has led to the development of enhanced feature detection algorithms, such as SIFT, and more recent methods like LOFTR, which aim to refine local features to extract high-quality matches in indistinctive or low-texture regions.

To support our claim, we will reference these matching methods in our paper. They exemplify the ongoing efforts in the field to address the challenges posed by such complex environments. However, we acknowledge that the assertion regarding the robustness of cost aggregation to these challenges might be contentious. Nevertheless, while our paper does not provide specific empirical evidence to support this claim (although we provide visualizations that can explain such robustness in Fig. 4 to some extent), it is nonetheless informed by a substantial body of research in the semantic matching literature. Numerous studies in this area have demonstrated the effectiveness of cost aggregation in various contexts.

Nevertheless, in light of the comment, we agree that it is prudent to moderate our claim regarding the robustness of cost aggregation to environments with repetitive patterns and cluttered backgrounds. We recognize that this aspect could be seen as a hypothesis rather than a conclusively proven fact. We appreciate the constructive comment and will accordingly adjust the tone of our claim to reflect a more cautious stance on this matter.

Reviewer comment (Experiments):
The experiments are not solid enough. The proposed method needs to be compared with more recent methods. In the geometric matching results from Tab2, most competitors are from 2020 and 2021, which are far from "state-of-the-art". Only one flow estimation method is considered (GMFlow). However, as discussed in the supplementary, the comparison is not fair, because GMFlow is trained on Sintel rather than DPED-CityScape-ADE+MegaDepth fine-tuning. Besides, all these competitors are trained on DPED-CityScape-ADE **or** MegaDepth (the proposed method is trained with DPED-CityScape-ADE **and** MegaDepth finetuning) as said in supplementary B.1. The authors should clarify this.

Author response:
> Include more recent works in Table 2, and GMFlow is trained on different dataset. Verify effectiveness of the proposed method in optical flow.  

We apologize for any oversight in our initial presentation of recent methods in our comparative analysis. While Table 1 comprehensively includes recent methods, we acknowledge that Table 2 appeared somewhat outdated. To address this, we have now included an evaluation of DKM, a method that has been tested on HPatches, with its results available in its public GitHub repository. We have compared these results with those of our method:

| Methods           |   I  |   II |   III | IV    | V     | Avg   |
|-------------------|:----:|-----:|------:|-------|-------|-------|
| PDC-Net (CVPR'21) | 1.15 | 7.43 | 11.64 | 25.00 | 30.49 | 15.14 |
| DKM (CVPR'23)     | 2.35 | 7.24 | 16.48 | 29.86 | 39.35 | 19.06 |
| Ours              | 1.91 | 6.13 | 5.62  | 6.36  | 19.44 | 7.88  |

As illustrated, our method demonstrates highly competitive performance, even when juxtaposed with newer methods like DKM.

Regarding the comparison with GMFlow, we recognize and apologize for the previously unfair comparison. Unlike other methods in our comparison that were trained on MegaDepth, GMFlow was not, which might have skewed the results.  To address this and to further evaluate the generalizability of our method as suggested by the reviewer, we conducted additional evaluations in the context of optical flow. For these evaluations, we utilized the KITTI and SINTEL datasets. Aligning with the practices of GLU-Net, PDC-Net, and GoCOR, our method was not trained on the Chair and Things datasets but rather on MegaDepth. The results of these additional evaluations are presented below:

| Methods  | KITTI-2015 | SINTEL (Final) |
|----------|:----------:|-------:|
| RAFT     |    5.32    |   2.69 |
| PDC-Net  | 5.40       | 4.54   |
| PDC-Net+ |    4.53    |      - |
| GMFlow   | 7.77       | 3.44   |
| Ours     | 4.39       | 4.16   |

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness.

Reviewer comment (Presentation):
The performance of the proposed method on the optical flow (KITTI, Sintel) task is blurry for me.

Author response:
To address this and to further evaluate the generalizability of our method as suggested by the reviewer, we conducted additional evaluations in the context of optical flow. For these evaluations, we utilized the KITTI and SINTEL datasets. Aligning with the practices of GLU-Net, PDC-Net, and GoCOR, our method was not trained on the Chair and Things datasets but rather on MegaDepth. The results of these additional evaluations are presented below:

| Methods  | KITTI-2015 | SINTEL (Final) |
|----------|:----------:|-------:|
| RAFT     |    5.32    |   2.69 |
| PDC-Net  | 5.40       | 4.54   |
| PDC-Net+ |    4.53    |      - |
| GMFlow   | 7.77       | 3.44   |
| Ours     | 4.39       | 4.16   |

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness. 

We hope this additional information and the results of our extended evaluations address the concerns raised and provide a more complete understanding of the capabilities of our method.

> Optical flow transferability.

We provide quantitative results above.

Reviewer comment (Evaluation):
It's good to see the improved matching performance on the HPatches dataset. However, I want to see whether the improved matching would lead to better Rotation and translation estimations.

Author response:
Pose estimation evaluation

Following reviewer's suggestion, we perform additional evaluation on pose estimation. Specifically, we use YFCC100M and ScanNet. The results are shown below:

| Methods            | 5$\degree$ | 10$\degree$ | 15$\degree$ |
|--------------------|:----------:|------------:|------------:|
| SP+SG (CVPR'19)    |    38.7    |   59.13 | 75.8        |
| LOFTR (CVPR'21)    |    42.4    |        62.5 | 77.3        |
| PDCNet+ (TPAMI'23) |    35.51   |       58.08 | 74.50       |
| Ours               | 37.62      | 59.83       | 73.22       |

| Methods            | 5$\degree$ | 10$\degree$ | 15$\degree$ |
|--------------------|:----------:|------------:|------------:|
| SP+SG (CVPR'19)    |    16.16   |       33.81 | 51.84       |
| LOFTR (CVPR'21)    |    16.88   |   62.533.62 | 77.350.62   |
| 3DG-STFM (ECCV'22) | 23.6       | 43.6        | 61.2        |
| PDCNet+ (TPAMI'23) |    20.25   |       39.37 | 57.13       |
| Ours               | 20.23      | 39.21       | 57.77       |

The upper table is YFCC and the below is ScanNet.As shown, although it surpasses PDCNet and others, it is apparent that our method is not  state-of-the-art. Nevertheless, we wish to highlight that our formulation, which  aims at  dense matching, should be separated from sparse matching (feature matching) works. A pervasive challenge in dense matching methods is their requirement to extract all matches between views. This challenge often results in many dense matching networks facing difficulties in sparse or semi-sparse methods for geometry estimation,  which is shared by many other works. Our framework is also not immune to this limitation. A specific area where our current architecture falls short is in its approach to handling confidence scores. While our cost volumes do encode a confidence score for each tentative match, a recent study like Lightglue has shown the benefits of explicitly estimating these confidence scores directly from the network, particularly for sparse matching tasks. This explicit estimation technique enhances performance in challenging scenarios, such as those involving occlusions or uncertain regions. Through empirical experiments, we've found that our model struggles more in these situations, largely attributed to the absence of a dedicated confidence estimation module.   To provide a complete understanding of our framework's capabilities and limitations, we will introduce a new section in our paper dedicated to discussing this limitation. 

Moreover, we wish to stress that the true essence and value of our work lie in the comprehensive and thorough investigations we have conducted into the relationships and synergies between feature aggregation and cost aggregation, as appreciated by the reviewer HY22, especially in the context of dense matching tasks. Our research goes beyond merely presenting a new technical design for state-of-the-art performance. Instead, it offers a deeper understanding of the underlying principles and interactions within these aggregation methodologies. This in-depth exploration, we believe, provides a more substantial and lasting contribution to the field.

Reviewer comment (Evaluation):
Please show some failure cases.

Author response:
Failure cases

We will visualize failure cases for both geometric and semantic matching and include the limitation discussed above as well. We appreciate the reviewer's comment.

Reviewer comment (Evaluation):
In visualization results Figure 2, features with integrative aggregation methodology preserve the semantic information. However, it seems the proposed method damages the local discriminative ability of features.

Author response:
One of the goals of our proposed aggregation is also to synthesize and incorporate broader contextual information. This often involves balancing local feature details with more global, semantic information. While the process enhances the overall semantic understanding of the image, it can sometimes appear to diminish the acuity of local features. This is not necessarily a flaw but rather a trade-off, aiming for a more holistic understanding of the image data. As demonstrated in Fig. 4 of the supplementary material, top row, where image pairs have extreme geometric deformation and repetitive patterns, our framework can warp all pixels to the corresponding locations unlike others. This shows our framework does not necessarily experience such damage.

Reviewer comment (Evaluation):
The local discriminative ability of features is also important for dense matching tasks. I would like to see an analysis of whether the proposed method causes damage in this perspective or whether this issue can be avoided in some design of the method.

Author response:
One of the goals of our proposed aggregation is also to synthesize and incorporate broader contextual information. This often involves balancing local feature details with more global, semantic information. While the process enhances the overall semantic understanding of the image, it can sometimes appear to diminish the acuity of local features. This is not necessarily a flaw but rather a trade-off, aiming for a more holistic understanding of the image data. As demonstrated in Fig. 4 of the supplementary material, top row, where image pairs have extreme geometric deformation and repetitive patterns, our framework can warp all pixels to the corresponding locations unlike others. This shows our framework does not necessarily experience such damage.

Reviewer comment (Novelty):
Similar ideas on feature and cost volume aggregation have been demonstrated in other literature such as stereo matching [a,b] and optical flow estimation [c]. The related work section (Sec. 2) summarizes those relevant papers very well. Compared to the existing solutions, what would be the new technical design (in self-/cross-attention) of the proposed module, except for applying it to semantic & geometric matching problems? Can the newer technical design from the proposed module also benefit other tasks that use cost volume, e.g., stereo matching, optical flow, scene flow, etc.?  

[a] Attention-Aware Feature Aggregation for Real-time Stereo Matching on Edge Devices, ACCV 2020  
[b] Attention Concatenation Volume for Accurate and Efficient Stereo Matching, CVPR 2022  
[c] GMFlow: Learning Optical Flow via Global Matching, CVPR 2022

Author response:
> What would be the key component to benefit the tasks? And would it also benefit optical flow?

The key technical design to benefit the task is the capturing of synergy between features, cost volumes, and their aggregations. While our approach to achieve this is indeed by incorporating self-attention and cross-attention mechanisms for effective aggregation, it's important to note that these are part of a suite of carefully considered technical designs. These designs are meticulously crafted to meet three key objectives: minimizing computational resources (achieved through Conv4D operations and maintaining a fixed resolution of the cost volume), maximizing performance (enhanced by residual connections among aggregated costs for training stability), and fostering synergy between feature aggregation and cost aggregation (via our innovative attention designs).

However, we contend that these technical aspects, while significant, constitute a smaller part of our overall contribution. The true essence and value of our work lie in the comprehensive and thorough investigations we have conducted into the relationships and synergies between feature aggregation and cost aggregation, especially in the context of dense matching tasks. Our research goes beyond merely presenting a new technical design for state-of-the-art performance. Instead, it offers a deeper understanding of the underlying principles and interactions within these aggregation methodologies, as appreciated by the reviewer HY22. This in-depth exploration, we believe, provides a more substantial and lasting contribution to the field.

This general approach, as the reviewer kindly suggested to show, is not constrained to geometric and semantic and it can also be evaluated  on other tasks, *e.g.,* optical flow, to further demonstrate the generalizability of our framework.  The results are shown below:

| Methods  | KITTI-2015 | SINTEL (Final) |
|----------|:----------:|-------:|
| RAFT     |    5.32    |   2.69 |
| PDC-Net  | 5.40       | 4.54   |
| PDC-Net+ |    4.53    |      - |
| GMFlow   | 7.77       | 3.44   |
| Ours     | 4.39       | 4.16   |

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness. 


We highly appreciate the reviewer's constructive comment, and we will integrate the results into the final manuscript.

Reviewer comment (Presentation):
Discussion on the limitation is missing. What would be the limitation of the method or unsolved problems?

Author response:
> Limitations

A pervasive challenge in dense matching methods is their requirement to extract all matches between views. This challenge often results in many dense matching networks facing difficulties in sparse or semi-sparse methods for geometry estimation. Our framework is not immune to this limitation. A specific area where our current architecture falls short is in its approach to handling confidence scores. While our cost volumes do encode a confidence score for each tentative match, a recent study like Lightglue has shown the benefits of explicitly estimating these confidence scores directly from the network, particularly for sparse matching tasks. This explicit estimation technique enhances performance in challenging scenarios, such as those involving occlusions or uncertain regions. Through empirical experiments, we've found that our model can struggle in these situations, largely due to the absence of a dedicated confidence estimation module.  To provide a complete understanding of our framework's capabilities and limitations, we will introduce a new section into our paper dedicated to discussing this limitation.

Reviewer comment (Evaluation):
Table 4 shows the accuracy improvement by adding more components. I am wondering if the paper can also include some qualitative examples and discuss where the gain mainly originates, such as resolving some particular matching ambiguity. It would be great if the paper can provide more insights related to its improvement.

Author response:
> Qualitative comparisons of Table 4.


In response to the suggestion of including qualitative comparisons, we recognize the importance of such visualizations in complementing our quantitative results, as presented in Table 4. However, the qualitative results as the performance improves would primarily manifest as minor adjustments in the accuracy of warped keypoints or pixels. These changes, while significant in a quantitative sense, may not be visibly distinguishable. However, as an alternative, we present PCA visualization as in Fig.9 and Fig.10 in the supplementary material, where we provide more examples of how our method works. These visualizations are designed to provide an intuitive understanding of how our model achieves its performance gains. They offer a clear illustration of the internal workings and improvements brought about by our framework. For example, if we look at (I-IV), they semantically match with the change from raw features or cost volume to those that are aggregated, which are visualized in Fig.9 and Fig.10 in the supplementary material, while the rest (V-VI) do not provide much intuition but rather are similar to the already presented visualizations with being higher resolution (meaning more fine details). 

Nevertheless, to provide additional information for better understanding, we revised the supplementary material and included Fig.11, where we show PCA visualization comparisons of the each variant introduced in the component ablation table. The caption is colored red to draw attention to the change. As the comparison shows, while there exist apparent differences between (I) and (IV), we acknowledge that the differences are not visually apparent as we add the components from (II) to (IV). This is because the resolutions of features are defined in the coarsest level, which makes it little bit hard for us to interpret the visualization. Nevertheless, we believe that the differences present in the visualizations contributed to the improvements and that we provide more intuitive visualizations in Fig. 9 and Fig. 10.

Reviewer comment (Reproducibility):
How many number of learnable parameters account for the new module (i.e., integrative feature and cost aggregation module)? How significant are they compared to the number of parameters of the entire network (15.5M)? Probably it's also good to include an extra column in Table 3 and 4 for the number of network parameters.

Author response:
> It would be good to include the number of learnable parameters for Table 3 and Table 4.

We appreciate the reviewer's suggestion. A small issue is that we intentionally adjusted all variants' number of learnable parameters to be similar for the experiments in Table 3 and 4 as mentioned in section 5.5, but we agree that it would be more informative to include the number of learnable parameters for each module, i.e., integrative self-attention and cross-attention, in our architecture.   The table below shows the comparison. 

|                                                    | Number of params |
|----------------------------------------------------|:----------------:|
| l = 1, integrative self-attention                  |      1612896     |
| l = 1, cross-attention with matching distribution  | 308576           |
| l = 2, integrative self-attention                  |      2511968     |
| l = 2, cross-attention with matching distribution  | 683104           |
| l = 3, integrative self-attention                  | 6513248          |
| l = 3, cross-attention with matching distribution  | 2692960          |
| l=1,2,3 cross-attention with matching distribution | 3684640          |
| l=1,2,3 integrative self-attention                 | 10638112         |
| Total attention params                             | 14.3 M           |
| Total params                                       | 14.5 M           |

Note that we also include similar ablation study in Table 2 of the supplementary material, which shows the comparison by varying depth of the transformer layers. We will include this in the supplementary material, as a separate table.

Reviewer comment (Reproducibility):
What's the resolution of the cost volume (saying the input image resolution is HxW)? At each level the features are upsampled ($D^{l}_s$), but how can the resolution of the cost volume ($C^{l}$) remain the same over different pyramid levels? Is there any reason to fix the resolution of the cost volume?

Author response:
> Resolution of cost volumes at each level

We apologize for any confusion caused by our initial explanations and appreciate the opportunity to clarify. In our architecture, as depicted in the upper branch of Figure 5, the cost volumes at each level are indeed fixed to the coarsest resolution. This specification applies specifically to the region labeled as "Cost Volume," where Conv4D operations are performed over the output cost volumes resulting from both self- and cross-attention mechanisms.

Additionally, within the aggregation modules illustrated in Figure 4, the dimensions of the cost volumes are maintained to be consistent with those of the feature maps. This design choice ensures that the cost volumes and feature maps are compatible for effective integration within these modules.

The rationale behind fixing the cost volume dimensions to the coarsest resolution in certain parts of the network, particularly for the residual connections, is driven by memory efficiency considerations. Since cost volumes inherently require more memory to store compared to feature maps, reducing their dimensions in specific regions of the network helps in managing the overall memory usage more effectively. This approach strikes a balance between maintaining the integrity and effectiveness of the cost volume information while optimizing the memory demands of the network.

We hope this clarification addresses the concerns and provides a clearer understanding of our architectural choices and their underlying motivations. If there are further questions or aspects that require additional explanation, we are more than willing to provide the necessary information.