Paper: A ROBUST DIFFERENTIAL NEURAL ODE OPTIMIZER (ICLR-2024)

Reviewer comment (Evaluation):
Although the GTSONO optimizer proposed in this paper can improve the robustness of the model to some extent, it adds too much extra computational overhead, which is unacceptable in the training of larger models.

Author response:
- While we admit that our method introduces a second set of weights, we highlight that most adversarial training methods typically include nested optimization schemes that are computationally demanding. The main advantage of our methodology is that the antagonizing weights  are updated through a *single* backpropagation, which reduces computational complexity while also achieving robustness even with natural training bypassing the computational burden of perturbing the input.  In addition, note that we also enhance computational efficiency by exploiting efficient matrix decompositions.

Reviewer comment (Experiments):
The paper's limited comparison with just one other optimizer that improves robustness diminishes the persuasiveness of the results, as a more comprehensive comparison would have strengthened the findings.

Author response:
- Following the suggestions of the reviewer, we present further comparisons against the state-of-the-art optimizers that enhance robustness such as Lyanet[1], and FI-ODE[2]. Additionally, we also evaluate the all optimizers in additional datasets: CIFAR100 and TinyImageNet. These results in the Table below further indicate the robustness of our optimizer as it outperforms state-of-the-art robust optimizers in multiple datasets.  We also highlight the significantly shorter training time required by GTSONO in contrast to Lyanet and FI-ODE. 
    
    | Optimizer | CIFAR100-PGD_20^0.03 | Tiny-ImageNet PGD_20^0.03 | Training TinyImageNet (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | Adam | $10.7\pm1.1$ | $1.9\pm0.1$ | 5:37 | 1.35 M |
    | SNOpt | $19.7\pm0.6$ | $3.6\pm0.9$ | 5:41 | 1.35 M |
    | SGD | $9.6\pm0.5$ | $0.8\pm0.2$ | 5:20 | 1.35 M |
    | LyaNet^* | $20.1\pm0.5$ | $5.4\pm0.4$ | 313:21 | 19.6 M |
    | FI-ODE^* | $15.4\pm0.4  $ | $4.0\pm0.2$ | 607:56 | 2.4 M |
    | c-GTSONO | $23.5\pm0.1$ | $9.1\pm0.8$ | 6:35 | 1.47M |
    
    | CIFAR10-Optimizer | PGD_20^0.03 | PGD_20^0.05 | Time (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | LyaNet^* | $46.1\pm0.4$ | $32.7\pm0.6$ | 32:44 | 19.6 M |
    | FI-ODE^* | $48.5\pm0.3$ | $33.4\pm0.6$ | 143:17 | 2.4 M |
    | c-GTSONO | $50.6 ± 0.3$| $35.0 \pm 0.2$ | 3:53 | 1.35M |

Reviewer comment (Experiments):
One limitation of this paper is that the experiments are confined to the CIFAR10 and SVHN datasets, with no validation on more extensive datasets such as CIFAR100 and ImageNet. This restricts the applicability of the research to a certain extent.

Author response:
- Following the suggestions of the reviewer, we present further comparisons against the state-of-the-art optimizers that enhance robustness such as Lyanet[1], and FI-ODE[2]. Additionally, we also evaluate the all optimizers in additional datasets: CIFAR100 and TinyImageNet. These results in the Table below further indicate the robustness of our optimizer as it outperforms state-of-the-art robust optimizers in multiple datasets.  We also highlight the significantly shorter training time required by GTSONO in contrast to Lyanet and FI-ODE. 
    
    | Optimizer | CIFAR100-PGD_20^0.03 | Tiny-ImageNet PGD_20^0.03 | Training TinyImageNet (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | Adam | $10.7\pm1.1$ | $1.9\pm0.1$ | 5:37 | 1.35 M |
    | SNOpt | $19.7\pm0.6$ | $3.6\pm0.9$ | 5:41 | 1.35 M |
    | SGD | $9.6\pm0.5$ | $0.8\pm0.2$ | 5:20 | 1.35 M |
    | LyaNet^* | $20.1\pm0.5$ | $5.4\pm0.4$ | 313:21 | 19.6 M |
    | FI-ODE^* | $15.4\pm0.4  $ | $4.0\pm0.2$ | 607:56 | 2.4 M |
    | c-GTSONO | $23.5\pm0.1$ | $9.1\pm0.8$ | 6:35 | 1.47M |
    
    | CIFAR10-Optimizer | PGD_20^0.03 | PGD_20^0.05 | Time (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | LyaNet^* | $46.1\pm0.4$ | $32.7\pm0.6$ | 32:44 | 19.6 M |
    | FI-ODE^* | $48.5\pm0.3$ | $33.4\pm0.6$ | 143:17 | 2.4 M |
    | c-GTSONO | $50.6 ± 0.3$| $35.0 \pm 0.2$ | 3:53 | 1.35M |

Reviewer comment (Experiments):
Could the authors provide insights into the feasibility of applying GTSONO to larger and more complex datasets such as CIFAR100 and ImageNet?

Author response:
- Following the suggestions of the reviewer, we present further comparisons against the state-of-the-art optimizers that enhance robustness such as Lyanet[1], and FI-ODE[2]. Additionally, we also evaluate the all optimizers in additional datasets: CIFAR100 and TinyImageNet. These results in the Table below further indicate the robustness of our optimizer as it outperforms state-of-the-art robust optimizers in multiple datasets.  We also highlight the significantly shorter training time required by GTSONO in contrast to Lyanet and FI-ODE. 
    
    | Optimizer | CIFAR100-PGD_20^0.03 | Tiny-ImageNet PGD_20^0.03 | Training TinyImageNet (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | Adam | $10.7\pm1.1$ | $1.9\pm0.1$ | 5:37 | 1.35 M |
    | SNOpt | $19.7\pm0.6$ | $3.6\pm0.9$ | 5:41 | 1.35 M |
    | SGD | $9.6\pm0.5$ | $0.8\pm0.2$ | 5:20 | 1.35 M |
    | LyaNet^* | $20.1\pm0.5$ | $5.4\pm0.4$ | 313:21 | 19.6 M |
    | FI-ODE^* | $15.4\pm0.4  $ | $4.0\pm0.2$ | 607:56 | 2.4 M |
    | c-GTSONO | $23.5\pm0.1$ | $9.1\pm0.8$ | 6:35 | 1.47M |
    
    | CIFAR10-Optimizer | PGD_20^0.03 | PGD_20^0.05 | Time (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | LyaNet^* | $46.1\pm0.4$ | $32.7\pm0.6$ | 32:44 | 19.6 M |
    | FI-ODE^* | $48.5\pm0.3$ | $33.4\pm0.6$ | 143:17 | 2.4 M |
    | c-GTSONO | $50.6 ± 0.3$| $35.0 \pm 0.2$ | 3:53 | 1.35M |

Reviewer comment (Experiments):
Given the focus on robustness improvement, could the authors consider including a more extensive comparison with various state-of-the-art optimizers that enhance robustness?

Author response:
- Following the suggestions of the reviewer, we present further comparisons against the state-of-the-art optimizers that enhance robustness such as Lyanet[1], and FI-ODE[2]. Additionally, we also evaluate the all optimizers in additional datasets: CIFAR100 and TinyImageNet. These results in the Table below further indicate the robustness of our optimizer as it outperforms state-of-the-art robust optimizers in multiple datasets.  We also highlight the significantly shorter training time required by GTSONO in contrast to Lyanet and FI-ODE. 
    
    | Optimizer | CIFAR100-PGD_20^0.03 | Tiny-ImageNet PGD_20^0.03 | Training TinyImageNet (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | Adam | $10.7\pm1.1$ | $1.9\pm0.1$ | 5:37 | 1.35 M |
    | SNOpt | $19.7\pm0.6$ | $3.6\pm0.9$ | 5:41 | 1.35 M |
    | SGD | $9.6\pm0.5$ | $0.8\pm0.2$ | 5:20 | 1.35 M |
    | LyaNet^* | $20.1\pm0.5$ | $5.4\pm0.4$ | 313:21 | 19.6 M |
    | FI-ODE^* | $15.4\pm0.4  $ | $4.0\pm0.2$ | 607:56 | 2.4 M |
    | c-GTSONO | $23.5\pm0.1$ | $9.1\pm0.8$ | 6:35 | 1.47M |
    
    | CIFAR10-Optimizer | PGD_20^0.03 | PGD_20^0.05 | Time (min:sec) | # of Parameters |
    | --- | --- | --- | --- | --- |
    | LyaNet^* | $46.1\pm0.4$ | $32.7\pm0.6$ | 32:44 | 19.6 M |
    | FI-ODE^* | $48.5\pm0.3$ | $33.4\pm0.6$ | 143:17 | 2.4 M |
    | c-GTSONO | $50.6 ± 0.3$| $35.0 \pm 0.2$ | 3:53 | 1.35M |

Reviewer comment (Experiments):
Are there any further insights into the stability and convergence properties of GTSONO, especially under varying hyperparameters and different neural network architectures?

Author response:
- The convergence guarantees are agnostic of the architecture and hyperparameter configuration. The only necessary assumption for the convergence guarantees to hold should be that the function of our accumulated loss $Q$ is locally convex-concave - which can be imposed through appropriate selection of the regularization parameters $R_u, R_v$

Reviewer comment (Theory):
In general, deep learning optimizers have used mini-batch size to obtain a partial batch of datasets, which means that random noise will be introduced, so the optimizer's parameter update rule should correspond to a stochastic differential equation instead of an ordinary differential equation. Therefore, why do the authors use an ODE rather than an SDE for their theoretical analysis?

Author response:
- We would like to kindly emphasize that the stochasticity comes from the input, however this does not affect the dynamics which remain deterministic. Therefore, the appropriate representation is through an ODE instead of an SDE. This allows us to employ deterministic optimal control which naturally leads to a deterministic update law.

Reviewer comment (Experiments):
Despite the effort of reducing computational cost, the proposed method is still very expensive. It would be good to include a complexity comparison between other neural ODE optimizers: first-order adjoint and SNOpt.

Author response:
### **2. Complexity comparison between GTSONO and other Neural ODE optimizers: first-order adjoint and SNOpt.**  

- We updated the paper, in order to present better the complexity comparison between GTSONO and other Neural ODE optimizers, adding a pertinent table in the main part of the paper. For an extensive comparison with regards to the computational requirement of the optimizers, we refer the reviewer to Tables 8-12 in Appendix D, which show training time and memory consumption.

Reviewer comment (Experiments):
My main question is as in weakness 1: the proposed optimizer seems to be beneficial to attacks on neural network weights rather than on the inputs. I hope the authors can clarify why they choose to demonstrate the effectiveness of their optimizer on input-robustness, and will the method be useful for attacks on system weights?

Author response:
### **3. Clarification on the selection to demonstrate the effectiveness of our optimizer on input-robustness**
    
- While we mostly refer the reviewer in weakness 1, regarding our motivation to recast the optimization problem with input disturbance to a game theoretic formulation. We chose to demonstrate the effectiveness on input-robustness, given that this is the most widely used form of attack. We also appreciate the suggestions of the reviewer to include studies on systems weights which we intend to explore in our future work.

Reviewer comment (Theory):
In the experiments, it seems that only having adversarial control on convolution layers is much better than having them on all of the layers. From the theory parts it is not clear why this is the case. The authors should provide more analysis on this.

Author response:
### **4. In the experiments, it seems that only having adversarial control on convolution layers is much better than having them on all of the layers. From the theory parts it is not clear why this is the case. The authors should provide more analysis on this.**
    
- We provide some intuition for this phenomenon. To quantify the similarity between the feature vector of the models from attacked and clean images, we calculate the cosine similarity. We find that the model with adversary weights in the convolution layers evaluated with attacked images is significantly closer to the feature vector from the clean images (cosine_similarity=0.92), compared to the feature vector from attacked images in the model without adversary control in the convolution layers (cosine_similarity=0.21). Therefore, feeding a robust feature vector to the classifier is similar to feeding the feature vector from the clean images, which eliminates the need to include antagonizing weights in the last linear layers, as it is well known that they drop the natural accuracy of model. Therefore, since the disturbance is in the input, this implies that it suffices to add adversary control in the convolution layers in order to capture robust features for robustifying the model. We intend to to investigate more thoroughly the underlying theory in future work.