Paper: Certified Robustness on Visual Graph Matching via Searching Optimal Smoothing Range (ICLR-2024)

Reviewer comment (Experiments):
It is advisable to conduct a comparison between the proposed method and other existing techniques for robust GM, such as ASAR[1] and COMMON [2]. Specifically, COMMON addresses robust graph matching by considering noisy correspondence during training, while ASAR takes adversarial attacks into account during training. Evaluating these methods alongside CR-OSRS would provide more comprehensive experimental insights. Furthermore, reporting the certified accuracy and average certified radius for these models is encouraged.

Author response:
We appreciate your valuable feedback. We have performed additional experiments based on your suggestions and further elaborated our proposed certified robustness method. 
* The ASAR and COMMON methods you referred to are the recently proposed visual GM methods with relatively good results. We calculated CR (in link: https://anonymous.4open.science/r/Certified-Robustness-on-Visual-Graph-Matching-via-Searching-Optimal-Smoothing-Range-E5F0/Rebuttal/Figure/ASAR/ and https://anonymous.4open.science/r/Certified-Robustness-on-Visual-Graph-Matching-via-Searching-Optimal-Smoothing-Range-E5F0/Rebuttal/Figure/COMMON) and ACR (in table below) achieved by CR-OSRS and RS-GM on Pascal VOC under keypoint position perturbations for these two models. ***The results indicate that the certified robustness obtained by CR-OSRS is also superior on these two basic models.*** This result has been updated in the new version of the paper, please refer to Tab.1, Fig.6, Fig.7 and Appendix.E.1 in the paper.

Table: ACR achieved by CR-OSRS and RS-GM for six GM solvers on Pascal VOC under keypoint position perturbations. 

|     | CR-OSRS+AUG+REG      | CR-OSRS+AUG+REG      | CR-OSRS+AUG+REG       | RS-GM+AUG+REG        | RS-GM+AUG+REG        | RS-GM+AUG+REG         | 
| --- | ----------------------- | ----------------------- | ------------------------ | ----------------------- | ----------------------- | ------------------------ |
|     | $\|\|\delta_{lower}\|\|$ | $\|\|\delta_{upper}\|\|$ | $\|\|\delta_{volume}\|\|$ | $\|\|\delta_{lower}\|\|$ | $\|\|\delta_{upper}\|\|$ | $\|\|\delta_{volume}\|\|$ |     |
COMMON| 1.550|1.751|1.900|0.952|0.952|1.069|
ASAR|1.541|1.648|1.968|0.683|0.683|0.841|
NGMv2| 1.425|1.586|1.934|0.778| 0.778|1.010|
CIE-H|0.987|1.167|1.354|0.572 |0.572 | 0.731|
PCA-GM|0.954|1.158|1.340|0.546|0.546|0.686|
GMN|0.899|1.076|1.253|0.514| 0.514|0.617|

* ***We offer more explanation on the difference between certified robustness and empirical robustness as well as the significance of CR-OSRS.*** The difference is that the results obtained by certified robustness are rigorously verifiable, that is, when the perturbation is within a certified range, the output is always invariant. However, empirical robustness does not have this theoretical guarantee, which means that it may have well defense capabilities against current attack methods, but it is highly susceptible to future unseen attacks. Therefore, the two types of robustness cannot be directly compared. This paper uses CR-OSRS to transform basic solvers into smoothed models with certified robustness for comparison and analysis. We also observe, in practice, that the better the matching performance of the basic model, the better the certified robustness and matching performance of the smoothed model will be. For instance, COMMON method enhances the model performance by leveraging noisy correspondence, which will make the smoothed models obtained by CR-OSRS more certifiably robust.

Reviewer comment (Presentation):
Since the author outlines four challenges in the Introduction, it would be beneficial to emphasize these points within the Method section, using C1 to C4.

Author response:
Thanks for your suggestion, which will improve the clarity of our paper. We have implemented the corresponding changes in the new version of the paper.

Reviewer comment (Experiments):
In Eq. 10, the authors mentioned that a constraint on b is imposed in the optimization. However, how this constraint works is not well explained. The effectiveness of this constraint is not evaluated in the experiments.

Author response:
Thank you for your questions and suggestions. We will clarify the role of the constraint and its ablation study. We have also revised the paper accordingly, please refer to Appendix.C and Appendix.E.1 in the updated version for more details.
* ***We first explain the rationale behind the design of the optimization algorithm.*** As described in Sec.4.2, we employ a smoothing distribution $\mathbf{B}$ with correlation to transform the basic GM model into a one with certified robustness. However, simply setting the values of $\mathbf{B}_1$ and $\mathbf{B}_2$ in advance may not achieve satisfactory results. Therefore, we propose to optimize $\mathbf{B}_1$ and $\mathbf{B}_2$ around the initial values to enhance the model’s certified robustness. It is worthwhile to mention that we optimize $\sigma$ and $b$ on the dataset, regardless of the architecture and the training procedure of the basic model.
* As mentioned in Sec.4, the proxy radius is a measure of the certified robustness of the algorithm, so our optimization algorithm aims to maximize the proxy radius on the dataset as shown in Eq.10. ***Moreover, we enforce a constraint on the correlation parameter $b$, which is motivated by two main reasons.*** One is that $b$ has to be positive, since a negative $b$ would invalidate the smoothing distribution; the other is to balance the trade-off between certified robustness and model performance. A large $b$ may improve the matching performance but reduce the certified space.
* We have performed ablation experiments to examine the effect of parameter $\kappa$ on the certification results as in figures (https://anonymous.4open.science/r/Certified-Robustness-on-Visual-Graph-Matching-via-Searching-Optimal-Smoothing-Range-E5F0/Rebuttal/Figure/Ablation%20study%20of%20kappa/). The figures shows that $\kappa$ had little overall influence on the outcomes, but a larger $\kappa$ results in a larger $||\delta_{volume}||$ and $||\delta_{upper}||$ as well as a smaller $||\delta_{lower}||$. However, we still stress that the constraint plays a vital role in ensuring that the value of $b$ remains within a reasonable optimization range. This result has been updated in the new version of the paper, please refer to it in Appendix.E.1 and Fig.9.

Reviewer comment (Experiments):
The authors introduced a regularization in Eq. 11, however, the ablation study of the variant without this regularization is missing.

Author response:
Thank you for your question. We actually have presented the ablation study you referred to in Fig.2. We evaluate the certification results of RS-GM and CR-OSRS on the basic model, the model with data augmentation (AUG), and the model with data augmentation and the regularization term (AUG+REG). We also explain in Sec.5.2 that the regularization term is always related to data augmentation as shown in Eq.11. In the absence of data augmentation, the outputs corresponding to all copy data are identical and thus the regularization term is always zero. Fig.2 demonstrates that the proposed data augmentation and the regularization term are effective.

Reviewer comment (Experiments):
Please provide more details of the constraint in Eq. 10 as well as ablation studies.

Author response:
Thank you for your questions and suggestions. We will clarify the role of the constraint and its ablation study. We have also revised the paper accordingly, please refer to Appendix.C and Appendix.E.1 in the updated version for more details.
* ***We first explain the rationale behind the design of the optimization algorithm.*** As described in Sec.4.2, we employ a smoothing distribution $\mathbf{B}$ with correlation to transform the basic GM model into a one with certified robustness. However, simply setting the values of $\mathbf{B}_1$ and $\mathbf{B}_2$ in advance may not achieve satisfactory results. Therefore, we propose to optimize $\mathbf{B}_1$ and $\mathbf{B}_2$ around the initial values to enhance the model’s certified robustness. It is worthwhile to mention that we optimize $\sigma$ and $b$ on the dataset, regardless of the architecture and the training procedure of the basic model.
* As mentioned in Sec.4, the proxy radius is a measure of the certified robustness of the algorithm, so our optimization algorithm aims to maximize the proxy radius on the dataset as shown in Eq.10. ***Moreover, we enforce a constraint on the correlation parameter $b$, which is motivated by two main reasons.*** One is that $b$ has to be positive, since a negative $b$ would invalidate the smoothing distribution; the other is to balance the trade-off between certified robustness and model performance. A large $b$ may improve the matching performance but reduce the certified space.
* We have performed ablation experiments to examine the effect of parameter $\kappa$ on the certification results as in figures (https://anonymous.4open.science/r/Certified-Robustness-on-Visual-Graph-Matching-via-Searching-Optimal-Smoothing-Range-E5F0/Rebuttal/Figure/Ablation%20study%20of%20kappa/). The figures shows that $\kappa$ had little overall influence on the outcomes, but a larger $\kappa$ results in a larger $||\delta_{volume}||$ and $||\delta_{upper}||$ as well as a smaller $||\delta_{lower}||$. However, we still stress that the constraint plays a vital role in ensuring that the value of $b$ remains within a reasonable optimization range. This result has been updated in the new version of the paper, please refer to it in Appendix.E.1 and Fig.9.

Reviewer comment (Experiments):
Please provide the ablation study of the regularization in Eq. 11.

Author response:
Thank you for your question. We actually have presented the ablation study you referred to in Fig.2. We evaluate the certification results of RS-GM and CR-OSRS on the basic model, the model with data augmentation (AUG), and the model with data augmentation and the regularization term (AUG+REG). We also explain in Sec.5.2 that the regularization term is always related to data augmentation as shown in Eq.11. In the absence of data augmentation, the outputs corresponding to all copy data are identical and thus the regularization term is always zero. Fig.2 demonstrates that the proposed data augmentation and the regularization term are effective.

Reviewer comment (Presentation):
The presentation can be improved for better clarity, as it involves multiple areas ranging from graph matching (combinatorial optimization), robustness certification, visual recognition, etc.

Author response:
We appreciate your insightful suggestions. Our research draws on knowledge from multiple fields, and we aim to present and compare them more rigorously and clarify their relevance to this study according to your suggestions. We show the comparison of visual GM methods and certified robustness methods in Tab.1 and Tab.2 below. We have also revised the paper accordingly, please refer to Appendix.B in the updated version for more details.

* **First**, we provide a brief introduction to visual Graph Matching(GM). GM is a fundamental problem in combinatorial optimization that involves identifying node correspondences between graphs. As introduced in Sec.3, deep visual GM deals with images with keypoints as inputs and solves in an end-to-end manner. Visual GM consists of three components: keypoint feature extractor, affinity learning, and final correspondence solver. ***We introduce and compare the main visual GM methods used in this work in Tab.1 below, which we have added to the paper, please refer to Appendix.B for more details.***

Table 1: Summary of main existing literature in learning GM. 

| Method | Introduction|
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|GMN[1]|The seminal work that employs a convolutional neural network to extract node features and constructs an end-to-end model with spectral matching.|
|PCA-GM[2]| Leveraging intra-graph and cross-graph structural information using graph convolutional networks.|
|CIE-H[3]|Enhancing end-to-end training by edge embedding and Hungarian-based attention mechanism.|
|NGMv2[4]|Developing a matching-aware graph convolution scheme with Sinkhorn iteration.|
|ASAR[5]|An appearance-aware regularizer is employed to explicitly increase the dissimilarities between similar keypoints and improve model robustness through adversarial attacks.|
|COMMON[6]|Integrating the momentum distillation strategy to balance the quadratic contrastive loss and reduce the impact of bi-level noisy correspondence.|
* **Second**, we introduce the concept and methods of certified robustness, which aims to design solvers that can verify the consistency of their prediction for any input within a certified perturbation range. As discussed in Sec.2, various methods have been proposed to achieve certified robustness, but only the Randomized Smoothing (RS) class of methods can scale up to large models and ImageNet-level datasets. ***We present some of the representative RS-based methods in Tab.2, along with their applicable scenarios and features.*** We have added the comparison to the paper, please refer to Appendix.B in the updated version for more details. As analyzed in Sec.1, none of these methods can be directly applied to visual GM. Therefore, we propose the CR-OSRS method, which can transform any basic visual GM model (described in Tab.1) into a model with certified robustness.

Table 2: Summary of main existing literature in RS-type methods for robustness certification. 

| Method | Introduction|
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|RS[7]|A pioneering work on certified robustness for classification tasks, demonstrating that Gaussian smoothing distributions can provide a provable $\ell_2$ perturbation bound.|
|DSSN[8]|Providing a novel non-additive smoothing robustness certificate for the $\ell_1$ threat model.|
|Median Smoothing[9]|Developing a new variant of smoothing specifically for detection based on the medians of the smoothed predictions.|
|RS for Segmentation[10]|Presenting a scalable certification method for image and point cloud segmentation based on randomized smoothing.|
|RS for Community Detection[11]|Building a new smoothed community detection method via randomly perturbing the graph structure.|
* **Third**, this work concentrates on the performance of the visual GM model, which only requires visual feature extraction, and not visual recognition. Visual recognition is beyond the scope of this work, so the visual recognition model will not be evaluated and discussed.

Reviewer comment (Evaluation):
The paper lacks some discussion for enlarging its potential impact to other combinatorial tasks or any limitation and difficulty to extend its adaption to other tasks.

Author response:
We appreciate your valuable suggestions. Below we will discuss the challenges of extending certified robustness research to general combinatorial optimization (CO) problems. Then we will introduce the particularities of the visual graph matching (GM) problem that this work focuses on and its advancement for the general issue.
* First, we emphasize that robustness certification aims to provide a mathematical guarantee that the model output is bounded by a certain region when the input is slightly modified. We share your interest in developing a provably robust method that can be applied to a broader range of CO problems. ***However, we encounter several challenges in this direction. First***, different CO problems may have different input and output formats. Therefore, it is necessary to devise a way to unify the diverse input and output formats. For instance, many CO problems can be represented as graph input forms, but there is no standard and unified format for the outputs, to the best of our knowledge. One possible solution is to use a general structured data format to unify the outputs. After unifying the input and output formats, we can design a general robustness certification method. ***Second***, there is currently a lack of sufficient research on the robustness certification of complex models that take graphs as inputs and structured data as outputs. Most of the existing work can only handle simple classification problems. ***Third***, we need to define the notion of certified robustness for CO problems clearly. The paper [12] proposes a heuristic robustness criterion for CO problems, but it does not guarantee the certified robustness of the model.
* Second, visual GM, as a special case of CO, has a distinctive input and output format, as described in Sec.1. The input consists of pairs of images and their corresponding keypoints, and the output is a matching matrix that indicates the correspondence between the keypoints. To address this task, we propose a novel method that designs a randomized smoothing distribution with correlation constraints and a subspace partitioning. This not only provides a theoretical guarantee for the robustness of visual GM, but also preserves the model performance and the rationality of the verification. In addition, we define the certified robustness of visual GM as the condition that when the perturbation is within the certified input space, the smoothed model always predicts the output within the output subspace. ***These address the above-mentioned challenges of CO robustness certification in the visual GM problem.***
***Our work has implications for the robustness certification of general CO problems***. Particularly, our method provides an effective solution for processing matrix output, which is also a special kind of structured data; our proposal of joint smoothing distribution may also be beneficial in inspiring the processing of constrained input formats in many CO problems; the data augmentation and the regularization term may suggest a similar approach to balancing certified robustness and model performance for general CO problems. ***However, the unification of input and output formats, the certification methods for complex graph tasks, and the reasonable certified robustness definitions are still pending to be solved.***
* In conclusion, we attempt to solve a special case of the robustness certification problem for CO in this paper. We are working on addressing the challenges for general CO certification and will share our findings in the future.

Reviewer comment (Novelty):
Have you explored the possibility of extending your approach to address problems beyond graph matching? Given the ubiquity of combinatorial optimization on graphs, a discussion on a potentially more general framework could be beneficial.

Author response:
We appreciate your valuable suggestions. Below we will discuss the challenges of extending certified robustness research to general combinatorial optimization (CO) problems. Then we will introduce the particularities of the visual graph matching (GM) problem that this work focuses on and its advancement for the general issue.
* First, we emphasize that robustness certification aims to provide a mathematical guarantee that the model output is bounded by a certain region when the input is slightly modified. We share your interest in developing a provably robust method that can be applied to a broader range of CO problems. ***However, we encounter several challenges in this direction. First***, different CO problems may have different input and output formats. Therefore, it is necessary to devise a way to unify the diverse input and output formats. For instance, many CO problems can be represented as graph input forms, but there is no standard and unified format for the outputs, to the best of our knowledge. One possible solution is to use a general structured data format to unify the outputs. After unifying the input and output formats, we can design a general robustness certification method. ***Second***, there is currently a lack of sufficient research on the robustness certification of complex models that take graphs as inputs and structured data as outputs. Most of the existing work can only handle simple classification problems. ***Third***, we need to define the notion of certified robustness for CO problems clearly. The paper [12] proposes a heuristic robustness criterion for CO problems, but it does not guarantee the certified robustness of the model.
* Second, visual GM, as a special case of CO, has a distinctive input and output format, as described in Sec.1. The input consists of pairs of images and their corresponding keypoints, and the output is a matching matrix that indicates the correspondence between the keypoints. To address this task, we propose a novel method that designs a randomized smoothing distribution with correlation constraints and a subspace partitioning. This not only provides a theoretical guarantee for the robustness of visual GM, but also preserves the model performance and the rationality of the verification. In addition, we define the certified robustness of visual GM as the condition that when the perturbation is within the certified input space, the smoothed model always predicts the output within the output subspace. ***These address the above-mentioned challenges of CO robustness certification in the visual GM problem.***
***Our work has implications for the robustness certification of general CO problems***. Particularly, our method provides an effective solution for processing matrix output, which is also a special kind of structured data; our proposal of joint smoothing distribution may also be beneficial in inspiring the processing of constrained input formats in many CO problems; the data augmentation and the regularization term may suggest a similar approach to balancing certified robustness and model performance for general CO problems. ***However, the unification of input and output formats, the certification methods for complex graph tasks, and the reasonable certified robustness definitions are still pending to be solved.***
* In conclusion, we attempt to solve a special case of the robustness certification problem for CO in this paper. We are working on addressing the challenges for general CO certification and will share our findings in the future.

Reviewer comment (Presentation):
Can you add a table or figure to summarize and compare related methods from multiple aspects for better accessibility of readers?

Author response:
We appreciate your insightful suggestions. Our research draws on knowledge from multiple fields, and we aim to present and compare them more rigorously and clarify their relevance to this study according to your suggestions. We show the comparison of visual GM methods and certified robustness methods in Tab.1 and Tab.2 below. We have also revised the paper accordingly, please refer to Appendix.B in the updated version for more details.

* **First**, we provide a brief introduction to visual Graph Matching(GM). GM is a fundamental problem in combinatorial optimization that involves identifying node correspondences between graphs. As introduced in Sec.3, deep visual GM deals with images with keypoints as inputs and solves in an end-to-end manner. Visual GM consists of three components: keypoint feature extractor, affinity learning, and final correspondence solver. ***We introduce and compare the main visual GM methods used in this work in Tab.1 below, which we have added to the paper, please refer to Appendix.B for more details.***

Table 1: Summary of main existing literature in learning GM. 

| Method | Introduction|
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|GMN[1]|The seminal work that employs a convolutional neural network to extract node features and constructs an end-to-end model with spectral matching.|
|PCA-GM[2]| Leveraging intra-graph and cross-graph structural information using graph convolutional networks.|
|CIE-H[3]|Enhancing end-to-end training by edge embedding and Hungarian-based attention mechanism.|
|NGMv2[4]|Developing a matching-aware graph convolution scheme with Sinkhorn iteration.|
|ASAR[5]|An appearance-aware regularizer is employed to explicitly increase the dissimilarities between similar keypoints and improve model robustness through adversarial attacks.|
|COMMON[6]|Integrating the momentum distillation strategy to balance the quadratic contrastive loss and reduce the impact of bi-level noisy correspondence.|
* **Second**, we introduce the concept and methods of certified robustness, which aims to design solvers that can verify the consistency of their prediction for any input within a certified perturbation range. As discussed in Sec.2, various methods have been proposed to achieve certified robustness, but only the Randomized Smoothing (RS) class of methods can scale up to large models and ImageNet-level datasets. ***We present some of the representative RS-based methods in Tab.2, along with their applicable scenarios and features.*** We have added the comparison to the paper, please refer to Appendix.B in the updated version for more details. As analyzed in Sec.1, none of these methods can be directly applied to visual GM. Therefore, we propose the CR-OSRS method, which can transform any basic visual GM model (described in Tab.1) into a model with certified robustness.

Table 2: Summary of main existing literature in RS-type methods for robustness certification. 

| Method | Introduction|
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|RS[7]|A pioneering work on certified robustness for classification tasks, demonstrating that Gaussian smoothing distributions can provide a provable $\ell_2$ perturbation bound.|
|DSSN[8]|Providing a novel non-additive smoothing robustness certificate for the $\ell_1$ threat model.|
|Median Smoothing[9]|Developing a new variant of smoothing specifically for detection based on the medians of the smoothed predictions.|
|RS for Segmentation[10]|Presenting a scalable certification method for image and point cloud segmentation based on randomized smoothing.|
|RS for Community Detection[11]|Building a new smoothed community detection method via randomly perturbing the graph structure.|
* **Third**, this work concentrates on the performance of the visual GM model, which only requires visual feature extraction, and not visual recognition. Visual recognition is beyond the scope of this work, so the visual recognition model will not be evaluated and discussed.