Paper: Robustness via learned Bregman divergence (ICLR-2024)

Reviewer comment (Theory):
The claim that $(\nabla\phi)^{-1}=\nabla\bar{\phi}$ is based on the assumption that ICNN is of the Legendre type. However, this assumption has not been verified. The proposed numerical method for approximating the inverse $(\nabla\phi)^{-1}$ appears unrelated to Fenchel's (1949) result. According to Equation (7), the minimization problem $\min_f \Vert f(\nabla\phi(x)) - x\Vert_2$ should yield $f=(\nabla\phi)^{-1}$, but it is not demonstrated that the L2 distance is the optimal choice.

Author response:
> "The claim that $(\nabla\phi)^{-1}=\nabla\overline\phi$ is based on the assumption that ICNN is of the Legendre type. However, this assumption has not been verified"

We did not claim that ICNNs are of Legendre type (it would require proving that the limit along the boundary of the interior of the domain of definition of $\nabla\phi$ is $\infty$ which is challenging for an arbitrarily complex $\phi$). We did not claim that this equality holds for ICNNs. We only used this fact to motivate the approximation which we wrote as $\overline\Psi(\boldsymbol{x}) \approx \nabla\overline\phi(\boldsymbol{x})$.

> "The proposed numerical method for approximating the inverse appears unrelated to Fenchel's (1949) result"

The result of Fenchel (1949) is important for our work: it provides the insight that the inverse of $\Psi$ is the gradient of a another convex function (in particular the conjugate). We have relied on this to model $\overline\Psi$ as a gradient of an ICNN, which is convex by our construction. This inspiration is discussed right above Equation 7.

> " ... but it is not demonstrated that the L2 distance is the optimal choice."

Training for this objective (the mean square error) is a design choice that we made and it is a common choice, e.g., also widely used when training autoencoders. The choice gave good results as detailed in Sec.5. We did not claim its optimality and it can be replaced by alternatives if suggested.

Reviewer comment (Evaluation):
The evaluation in this paper has limitations. For instance, it remains unclear how the Bregman divergence training converges. Additionally, it is uncertain whether the method can scale to larger images, such as ImageNet. How does the choice of sampling numbers impact the approximation?

Author response:
> "...it remains unclear how the Bregman divergence training converges."

The training **converges** nicely without over-fitting. The evolution of the training and the validation loss across optimization steps is added in Appendix C.

Reviewer comment (Theory):
The Bregman-based attack finds the projection point $x'$ in the intersection of $D$, which is the intersection of $B_{\phi}$ and $B$. According to the algorithm, the projection is performed by projecting onto $B$ first, and then onto $B_{\phi}$. However, it seems unlikely that the resulting projected point will lie in the intersection.

Author response:
> "... the projection is performed by projecting onto $B$ first, and then onto $B_\phi$. However, it seems unlikely that the resulting projected point will lie in the intersection."

$\def\epsm{{\epsilon_\text{max}}}$

$\def\Bf{{\mathbb{B}_\phi(\vx, \epsilon)}}$

$\def\B{{\mathbb{B}(\vx, \epsilon_\text{max})}}$


The result of this projection is indeed always in the intersection: 

Let $\vy \not \in \Bf \cap \B$. First, we define $\vz$ as the projection of $\vy$ onto $\B$, so $|| \vz- \vx ||  \leq \epsm$. Next, we perform a line search on the segment of endpoints $\vx$ and $\vz$ until we find $\vu$ satisfying $D_\phi(\vu || \vx) \leq \epsilon$. In other words, $\vu = \alpha \vx + (1-\alpha)\vz$ for some $\alpha \in [0,1]$. We have $\vu \in \Bf$ since $D_\phi(\vu || \vx) \leq \epsilon$. We also have $||\vx-\vu|| = ||\vx - \alpha\vx -(1-\alpha) \vz|| = (1-\alpha)||\vx-\vz|| \leq \epsm$ (because $\alpha \in [0,1]$ and $||\vz-\vx|| \leq \epsm$), which means $\vu \in \B$. This concludes the proof that $\vu$, the result of this procedure is always in $\Bf \cap \B$.

---

We should have provided a more intuitive explanation for the projection question. Here is a hopefully improved attempt.

Indeed, as you say, for two arbitrary sets it would not work. In our setting it does because of the special nature of the sets. Concretely, one set is a Bregman ball, the other is an $L^2$ ball and they have the **same center point.**

The projection is done by first a projecting into the $L^2$ ball $\B$, which is a division by a scalar. Then, from this point on we do a line search towards the (joint) center point. This entire line segment is thus in $\B$! We pick the first point found that is also in $\Bf$. At least, the (joint) center point will yield a valid result. 

> "Your proof only shows $\vu\in\Bf \cap \B$, and does not show it is the projection of $\vz$."

Yes, that is all what we need to show, since $\vu$ is the result of our projection procedure used in lines 7-11 of Algorithm 1. This $\vu$ is the result of a fast approximation procedure that always belongs to in intersection $\Bf \cap \B$ and not the result of an exact projection into $\Bf$. This is stated in the paragraph below Algorithm. 1: The projection into the intersection is done as a projection into $\B$ followed by a projection into $\Bf$. Since the latter has no closed-from expression, we approximate it by a binary search over the segment....


> "How to ensure the exisitence of $\vu$ satisfying $D_\phi(\vu || \vx) \leq \epsilon$?"

We search for $\vu$ in the line segment having $\vx$ and $\vz$ as endpoints:

* Step 0: we start the search from $\vz$. If $D_\phi(\vz || \vx) \leq \epsilon$, then we stop and return $\vu \leftarrow \vz$.
* Step 1: otherwise ($D_\phi(\vz || \vx) > \epsilon$), we move to the middle of the line segment: $\vu \leftarrow \frac{1}{2} \vx + \frac{1}{2} \vz$. If $D_\phi(\vu || \vx) \leq \epsilon$, then we stop and return this $\vu$.
* Step 2: otherwise ($D_\phi(\vu || \vx) > \epsilon$), we move $\vu$ to the middle of the line segment having the old $\vu$ and $\vx$ as endpoints : $\vu \leftarrow \frac{3}{4} \vx + \frac{1}{4} \vz$. If $D_\phi(\vu || \vx) \leq \epsilon$, then we stop and return this $\vu$.
* Step 3: otherwise ($D_\phi(\vu || \vx) > \epsilon$), we move $\vu$ to the middle of the line segment having the old $\vu$ and $\vx$ as endpoints : $\vu \leftarrow \frac{7}{8} \vx + \frac{1}{8} \vz$. If $D_\phi(\vu || \vx) \leq \epsilon$, then we stop and return this $\vu$.

.....

So $\vu$ is moving towards $\vx$ starting from $\vz$. In the worst case, $\vu$ ends up very close to $\vx$: $\vu \approx \vx$. Since $\vx$ is in the interior of $\Bf$ (because $D_\phi(\vx || \vx) = 0 < \epsilon$), its neighborhood is also in $\Bf$. In particular, the point $\vu$: $D_\phi(\vu || \vx) \leq \epsilon$.


More formally, the value of $\vu$ at a step $i$ follows a series $\vu_i = \alpha_i \vx + (1-\alpha_i) \vz$, with $\alpha_i=1 - (\frac{1}{2})^i$. This series $\{\vu_i\}$ converges to $\vx$ since $\{\alpha_i\}$ converges to 1. We know that $D_\phi(\vx || \vx) = 0 < \epsilon$ so $\vx$ is in the interior of the ball $\Bf \subset \mathbb{R}^n$. Thus, there exists a value of $i=N$, for which $D_\phi(\vu_N || \vx) \leq \epsilon$. The $\alpha$ in the proof above is set to $\alpha=\alpha_N$, and $\vu=\vu_N$. In summary, the existence of $\vu$ satisfying $D_\phi(\vu || \vx) \leq \epsilon$ is ensured by the convergence of $\{\vu_i\}$ to $\vx$ and by the fact that $\vx$ is in the interior of $\Bf$.

Reviewer comment (Theory):
Although Lemma 1 states that $d$ lies in the range $(0, 1)$, it is unclear why Figure 1 is generated when $d=1$.

Author response:
> "... it is unclear why Figure 1 is generated when $d=1$"

Figure 1 is only a 2D cartoon meant to give visual intuition of our method so we chose $d=1$ to be easily readable. Choosing $d=0.99999$ for example would have resulted in the same visual.

Reviewer comment (Theory):
In the experiment, the fraction of $\Gamma$ functions is defined as $1/\sqrt{n}$. The purpose of introducing $\mu$ as the fraction of Gamma functions in Lemma 1 is not clear.

Author response:
> "The purpose of introducing $\mu$ as the fraction of Gamma functions in Lemma 1 is not clear."

We use $\mu$ to avoid writing the bulky fraction in Equation. 21 multiple times. The fraction ensures that equality (9) holds. But, as suggested by another reviewer, the lemma can be moved to the appendix.

Reviewer comment (Evaluation):
Based on my understanding, each iteration of mirror descent involves one forward pass and one backward pass for the input $x^t$. It is unclear how the computation cost of Bregman divergence training is determined.

Author response:
> "Based on my understanding, each iteration of mirror descent involves one forward pass and one backward pass for the input. It is unclear how the computation cost of Bregman divergence training is determined."

It is not exactly like that: there are two separate phases. First, we train the divergence itself, minimizing the objective in Equation. 10, not involving mirror descent. Then we use the divergence in the inner-loop of the adversarial training (with mirror descent in Algorithm 1). 

The overall computational cost is composed of the cost of computing $\phi$, $\Psi$ and $\overline\Psi$. The computation of $\phi(\boldsymbol{x})$ is a forward pass on the convolutional neural network $\phi$. The computation of $\Psi(\boldsymbol{x})$ and $\overline\Psi(\boldsymbol{x})$ are back-propagation passes on the neural networks representing $\phi$ and $\overline\phi$, respectively. These computations are implemented using the highly optimized automatic differentiation engine of PyTorch (mentioned in Sec. 3.1 and Sec 5.).

Reviewer comment (Writing):
The notations in the equations lack consistency, such as the font used for variables like $\tilde{x}$ and $u$.

Author response:
We argue that our notation is consistent; indeed, we followed the conference recommendations provided in the ICLR template. For example, the font in `\bm{x}` $\vx$ indicates a vector where the font in `\textnormal{x}` $\textnormal{x}$ is for random variables. We also made these distinctions clear throughout the text.

Reviewer comment (Theory):
The idea of learning similarity measures and generating suitable adversarial examples is interesting. However, since the work aims to increase the robustness of classifiers it needs to be discussed to what extend the corruption model can be practically known.

Author response:
We show that we can train models that are robust to real-world corruptions like Fog. CIFAR-10-C actually contains computational models for creating corrupted images from clean ones. Otherwise corruptions can be obtained through samples collected in the real world and are thus available in this sense.

Reviewer comment (Experiments):
The method is compared to $\ell_2$ adversarial and RLAT method in the experiments, but in-distribution performance and data augmentation baselines seem to be missing.

Author response:
Our method approaches the problem of robustness from an adversarial perspective (as RLAT). Importantly, our method is agnostic to the architecture of the classifier (RLAT is not) and thus can be combined with any of the approaches based on data augmentation, which are thus orthogonal. The in-distribution performance is given in Table 2 and Table 3 under the column "standard accuracy".

Reviewer comment (Evaluation):
In Table 3 the corruption robustness accuracy is shown for the different models and on different corruptions. Do the authors have any insights on why the model trained on contrast (via learning the similarity measure) performs well on both contrast and fog, whereas the model trained on fog performs worse in both categories?

Author response:
We had already addressed this question (as good as we could) at the end of the paragraph **Adversarial training** in Sec.5: "One hint is that the three corruptions are similar in nature but this behavior requires further investigation."

Reviewer comment (Experiments):
The experiments are only conducted on CIFAR10-C, which is a single, simplistic dataset. I would like to believe that the results would hold on a more complex dataset like ImageNet-C, but this is not demonstrated in the paper. The Conclusions seem to acknowledge that scaling to larger convex architectures for the base function could be challenging.

Author response:
> No experiments on ImageNet.

We would have liked to but with the limited GPU compute available to us it would take about one month. We are looking for a compute grant for the next or final version.

Reviewer comment (Theory):
One of the issues of using Bregman divergences to measure image similarity is that these divergences are not necessarily symmetric. Many image augmentations are symmetric.

Author response:
> "One of the issues of using Bregman divergences to measure image similarity is that these divergences are not necessarily symmetric. Many image augmentations are symmetric."

Indeed, we have considered the symmetrization $S_\phi$ of the Bregman divergence $D_\phi$, which, however, loses the triangle inequality:

$$
S_\phi(\boldsymbol{x'}, \boldsymbol{x}) = D_\phi(\boldsymbol{x'} \parallel \boldsymbol{x}) +  D_\phi(\boldsymbol{x} \parallel \boldsymbol{x'}) 
$$
We have substituted $S_\phi$ for $D_\phi$ throughout the approach and we re-run the evaluation in Sec. 5. The symmetry did not provide any improvement compared to the proposed $D_\phi$.

Reviewer comment (Presentation):
Lemma 1 is not important to include in the main text. The alternative approach in A.2 is a much simpler method to draw samples, which does not require the complicated statement of the Lemma; in fact the experiments use the simpler version in A.2! Moreover, due to typicality, sampling from a normal distribution in high dimension will essentially sample from the surface of a hypersphere.

Author response:
> "Lemma 1 is not important to include in the main text."

You are right. We will change that.

Reviewer comment (Presentation):
I believe that the paper’s contribution on defining a new method for metric learning should be highlighted in the abstract.

Author response:
We would like to thank the reviewer for the feedback ranging from high-level comments to the finer details of the implementation. As you suggest, we will emphasize better that we propose a novel method for metric learning, which is indeed the core contribution of our work with robustness only as one possible application.

Reviewer comment (Writing):
$\tau$ is used before definition in Equation 7 (def isn’t until Section 3.3).

Author response:
> "Do you draw a single pair $\tau$ $d$ for each training image as an augmentation?"

Yes, the pair is (uniformly) sampled for each image in the mini batch during each training epoch.