Paper: ROBUST SPARSE AND DENSE MATCHING (ICLR-2024)

Reviewer comment (Novelty):
The model architecture is not well motivated. It seems similar to previous optical flow and dense feature matching works. It is not clear why the authors do not start from an established baseline like, e.g., PDCNet or GMFlow.

Author response:
Thanks for pointing out this problem. We provide a detailed explanation for this problem in our first official comment. The primary reasons are twofold. First of all, the iterative optimization mechanism provides an explicit and efficient approach for exploiting geometric similarity, which can be employed in geometry estimation methods for further improvement. Moreover, conducting matching at a higher resolution can empirically introduce additional improvement for downstream pose estimation as in DKM, PATS, etc. However, popular optical flow estimation works like RAFT, FlowFormer, and GMFlow end up with the refinement at a relatively low resolution(1/4 at most). To mitigate this problem, we elaborately construct the model architecture to perform refinement at 1/2 resolution. In future work, we will conduct more comprehensive experiments and utilize your mentioned models(GMFlow, PDCNet+) for generalization evaluations with our training data.

Reviewer comment (Evaluation):
The performance on pose estimation is significantly below previous work. The MegaDepth benchmark, which is commonly used, is only shown briefly in the ablation, but no state of the art comparison is provided. The performance is about 5% lower than DKM. On ScanNet the performance is about 3% lower. Also on optical flow the relation to state of the art methods is not documented.

Author response:
Thanks for your instructive suggestions on our paper. In this work, we aim to improve the generalization performance of the matching problem by collecting large-scale datasets covering optical flow, and geometry estimation datasets. We also propose a generalist model, capable of both optical flow estimation and geometry estimation. Our method achieves the best zero-shot evaluations across all datasets for matching and pose estimation tasks, strongly validating the effectiveness of our proposed method. To mitigate the degeneration of specialized datasets caused by training on data of diverse distributions, we propose a decoupled training strategy by learning matching and uncertainty estimation in a two-stage manner. Overall, we have taken into full consideration both the specificity and generality. The best zero-shot generalization performance compared with prior generalist and specialized models as well as the competitive performance compared with specialized methods can strongly demonstrate the contributions of our work. We also report the performance of optical flow following the standard setting as specialized models. The comparison is illustrated in Tab.6. We provide the updated results by adopting the tile mechanism proposed in FlowFormer. The results are shown below:

| Methods       | Sintel-C &darr; | Sintel-F &darr;  | KITTI EPE &darr; | KITTI F1 &darr; |
| ------------- | --------------- | ---------- | -------- | ---------- |
|PWCNet| 2.6 | 3.9 | 10.4 | 33.7 |
|DKM| 2.6 | 4.4 | 13.1 | 37.7 |
|MS-RAFT| 1.4 | 2.7 | - | - |
|FlowFormer| **1.0** | **2.4** | 4.1 | 14.7 |
|GMFlow| 1.1 | 2.5 | 7.8 | 23.4 |
|RAFT| 1.4 | 2.7 | 5.0 | 17.4 |
|Ours| **1.0** | **2.4** | **3.9** | **12.6** |

This comparison strongly validates that if training on specific datasets under the same setting as specialized models, our method can also obtain superior performance.

Reviewer comment (Experiments):
The ablation on the data is inconclusive. Adding optical flow datasets seem to lower results on pose estimation (Table 1). In the data ablation (Table 2) those results are no longer shown, why? Since those results are not shown, it must be assumed that adding more optical flow datasets further degrade performance.

Author response:
Currently, widely used datasets for training optical flow, such as FlyingThings3D, FlyingChairs, Sintel, and TartanAir, are synthesized datasets with relatively small displacements. In contrast, real-world datasets for geometry estimation often exhibit substantial changes in viewpoint. The integration of a diverse set of datasets with varying distributions, as opposed to training on specific datasets tailored for specialized tasks, may introduce inevitable interference, which accounts for the degeneration. The ablation studies in Table 1 are primarily conducted to reveal this phenomenon and demonstrate the effectiveness of our proposed decoupled training strategy in mitigating this decline. A more straightforward table for downstream pose estimation is presented below:

|Methods|Datasets|Megadepth AUC@5|YFCC AUC@5|
|---|----|----|----|
|Joint|M|55.3|48.2|
|Joint|M+TS|51.9|47.8|
|Decoupled|M|55.7|48.5|
|Decoupled|M+TS|54.0|48.2|

Leveraging our decoupled training strategy efficiently enhances the performance when the optical flow dataset is used. In Tab.2, we utilize our decoupled training strategy in default to explore the effectiveness of scaling up the diversity of training for generalization performance. The uncertainty estimation module is not involved at this matching stage so only generalization evaluations on matching tasks are reported.

Reviewer comment (Experiments):
The proposed model use high-weight parameters, swin-transformer, RAFT. It doesn't present the comparison with other methods.

Author response:
Thanks for pointing out the weakness of our work and we apologize for the unclear writing in this component. We will provide more comprehensive experiments in our revised supplementary materials.

## Comparison with other methods:
In our work, we utilize the shifted-window strategy to perform self-attention and cross-attention for feature enhancement. The compared methods like GMFlow and FlowFormer also adopt the high-weight parameters. It's worth mentioning that FlowFlower utilizes the Twins-L model as its backbone for feature extraction.

Reviewer comment (Writing):
The "Warping Error Map" is not detailed in paper, but it's important.

Author response:
## Warping error computation
As briefly introduced in Sec.3.2, we warp the target frame to the reference frame according to the estimated optical flow with bilinear sampling. Given the input reference and target images $I_r, I_t,$ as well as the estimated optical flow $f$ at the grid $x$, the warped reference image can be computed as:
$$
I_{warped}(x) = I_r(x+f(x)),
$$
and the warped RGB error can be computed as:
$$
E_{RGB} = |I_r - I_{warped}|
$$
We also compute the feature error $E_{feat}$ in the same way with reference and target features $F_r, F_t$:
$$
E_{feat}(x) = |F_r(x) - F_t(x+f(x))|
$$

The final warping error is defined as the concatenation of both warped RGB error and warped feature error. We will add detailed introduction about the implementation of warping error in our revised paper.

Reviewer comment (Reproducibility):
How to use "Uncertainty Map" in ransac filter, it should be given in detail.

Author response:
## Uncertainty map for ransac filter
The uncertainty map serves as a manifestation of the effectiveness of sparse correspondence matching. A higher degree of effectiveness in the sparse matching outcomes for a specific point is associated with an increased magnitude of its corresponding uncertainty. The cumulative uncertainties of all points within an image collectively contribute to the formation of its corresponding uncertainty map. We conduct a point selection procedure involving the uncertainty map and its corresponding matching outcomes, subsequently directing the selected points into a RANSAC filter. Our point selection process comprises two sequential steps. The initial step involves the selection of points using uncertainty as a screening criterion. The second step, based on the distribution among these selected points, constitutes a secondary selection aimed at ensuring diversity in the chosen points. In the specific context of the secondary point screening step, it entails the initial computation of Euclidean distances among the points selected during the initial screening. Euclidean distances are subsequently employed as a reference to establish weights for the secondary point selection, where greater distances correspond to augmented weights. As a preventive measure against the introduction of noise data, weights for points with distances exceeding a predefined threshold are reset to the minimum value of 1e-7.

Reviewer comment (Evaluation):
There is no model size and runtime cost comparison with other methods.

Author response:
## Computation cost and inference speed
Thanks for pointing out the absence of this experiment. We select state-of-the-art methods and make comparisons with them in terms of computation costs and inference speed. The result is shown below:

|Methods|Computation Costs(GFLOPs)|Inference Speed(s)|
|---|----|----|
|FlowFormer|695.0|0.23|
|GMFlow|202.8|0.06|
|RAFT|505.8|0.15|
|Ours|940.7|0.19|

We will explore to construct a more efficient model in our future work.

Reviewer comment (Experiments):
The effectiveness of multi-scale design: The method used the FPN and generated similarity matrices in three scales. However, in the following three GRU Refinement Blocks only one matrix seems to be used. How about the matrices in the other two scales? Besides, further ablations on the multi-scale design should be provided.

Author response:
We are sorry for the confusion caused by the absence of this experiment. We provide the following ablation studies by inferring at previous scales for a quick validation of our multiscale design. The results are shown below and scale 3 corresponds to the refinement of 1/2 resolution utilized in our final model while scale 1 indicates the refinement of 1/8 resolution:

| Output Scale       | HPatches PCK-1 &uarr; | KITTI AEPE&darr;  | KITTI F1 &darr; | ETH3D AEPE &darr; | ETH3D PCK-1 &uarr; |
| ------------- | --------------- | ---------- | -------- | ---------- | ----------- |
| 1   | 40.4 | 4.9 | 15.1 | 11.3 | 40.4 |
| 2 | 46.5 | 4.5 | 11.1 | 10.1 | 46.5 |
| 3 | **47.3** | **4.4** | **10.9** | **9.9** | **47.3** |

Furthermore, we also utilize the matching at the previous scale for pose estimation, and the performance is shown below:
|Output Scale| ScanNet AUC@5 | YFCC@5 | TUM@5 |
|-------|-------|-------|-------|
|1| 22.9 |39.4|15.0|
|2| 25.3 |47.0|16.0|
|3| **26.0** |**48.3**|**16.3**|

The above-listed results across matching and pose estimation tasks validate the effectiveness of the multi-scale design and the necessity of applying refinement at a higher resolution. We will place this comparison in the supplementary materials and conduct more comprehensive ablation experiments in our revised paper.

Reviewer comment (Novelty):
The design of dataset: The proposed dataset seems like a subset of TartanAir dataset with an improved optical flow rendering method. What is the main problem solved by building this dataset? What are the advantages over previous datasets besides better optical flow supervision? More experimental results based on this dataset need to be given.

Author response:
Thanks for pointing out this question. The re-organized of the TartanAir dataset is an important exploration in our work. Notably, the magnitude of displacement varies substantially across different datasets and scenarios. We list the displacement distribution of different datasets as well as our organized data below which can also be found in Tab.5 in our paper. It's worth noticing that our generated optical flow dataset obtains a different distribution compared to the original TartanAir dataset.

|Datasets|Task|s0-10|s10-40|s40+|
|----|----|-----|-----|--------|
|KITTI|Optical Flow|30.3|36.5|33.2|
|HPatches|Geometry Estimation|0.9|4.5|94.6|
|FlyingThings3D|Optical Flow|25.2|45.7|29.1|
|TartanAir|Optical Flow|46.2|47.2|6.6|
|Sintel|Optical Flow|69.0|21.3|9.7|
|ScanNet|Geometry Estimation|0.0|0.6|99.4|
|Megadepth|Geometry Estimation|1.1|10.9|88.0|
|Ours|Optical Flow|0.9|9.2|89.9|

We re-organize the dataset by increasing the sampling interval to obtain a closer distribution of other real-world datasets. The computation of optical flow ground truth is illustrated in our appendix A.1. The effectiveness of our proposed dataset is obvious as shown in Tab.1 and Tab.2. We simplify Tab.2 as below to show the effectiveness of our organized TartanAir sampled(TS) dataset.

| With TS dataset       | HPatches PCK-1 &uarr; | KITTI AEPE&darr;  | KITTI F1 &darr; | ETH3D AEPE &darr; | ETH3D PCK-1 &uarr; |
| ------------- | --------------- | ---------- | -------- | ---------- | ----------- |
| &#10060;   | 44.3            | 4.1        | 10.8     | **2.0**    | 55.9        |
| &#10004; | **46.3**        | **3.5**    | **9.6**  | **2.0**    | **56.3**     |

The adoption of our organized TartanAir dataset efficiently improves the generalization performance across multiple datasets, demonstrating the advantages of our organized dataset.

Reviewer comment (Experiments):
Several results in ablation study are not clear: The data in the experimental table cannot clearly reflect the effectiveness of each module. For example, in Table 1, what is the setting of RSDM? Is it the last row?

Author response:
We apologize for the ambiguity caused by our experiment settings and paper writing. In Tab.1, we illustrate the effectiveness of the decoupled training strategy by learning matching and uncertainty in a two-stage manner. In our final experiment, we adopt this decoupled training strategy. At the matching stage, we first train the dense matching model on the mixture of datasets as shown in Tab.2 to improve the generalization performance, and the effectiveness is validated. In the second stage, we freeze the parameters of the matching model and train the uncertainty estimation model based on the warping error according to the matching model. At this stage, only Megadepth and ScanNet are utilized. The detailed training settings are mentioned in Sec.4.1.