Paper: Test like you Train in Implicit Deep Learning (ICLR-2024)

Reviewer comment (Theory):
It is hard to interpret the meaning of main theorems. For example, Eq. (9) in Theorem 1 contains orthogonal projections and $E_N$ while their values are unknown.

Author response:
**Explanation of theorem 1**: Theorem 1 states that it is possible to improve the loss by changing the number of iterations only up to a “point”, and gives a quantitative result for this “point” (i.e. the lower bound). We even show in the subsequent Corollary 1, that in the most extreme case it is not possible to improve the loss.
We will add this clarification after theorem 1 in the camera-ready version of our paper.
We recall that all notations are defined within theorem 1 or its assumptions.

Reviewer comment (Theory):
The tightness of the bounds shown in these theorems is not discussed. It will be beneficial to use a figure to compare the actual value and the analytical bound.

Author response:
**Tightness of the bound**: The bound is tight and we will mention it in the camera-ready version of our paper. It comes from the fact that only triangular inequalities were used in the proof. Fig 1. (left) shows the tightness under the assumptions of Corollary 1.

Reviewer comment (Experiments):
The experiments only consider the case where $N$ is fixed while $\Delta N$ varies (e.g., Figure 2 & 3). It is unclear whether the conclusion holds for other choices of $N$.

Author response:
**Varying $N$**: In each experiment, $N$ is different, in other words we don’t use the same number of inner iterations for different settings. The idea here was to reuse pre-trained DEQs, which generally have very fine hyperparameter tuning to maximize the performance on the task and for a specific number of inner iterations. Each $N$ is specified in Table 1. in the Appendix. 
Using different $N$ would mean retraining all of these models ourselves, which is beyond our computational budget.
We also wanted to make sure to uncover this phenomenon for models that were trained by others in order to strengthen our findings.

Reviewer comment (Experiments):
Theorem 1 is validated on a small scale dataset. It would be helpful to validate it on a real dataset.

Author response:
**Validation of Theorem 1**: we tried to verify Corollary 2 on a more real dataset in Figure 4. Figure 1. was indeed meant as an illustration rather than a validation of the theorem, and we believe that our experiments on diverse real-world dataset demonstrate the applicability of our findings.

Reviewer comment (Evaluation):
Figure 4 plots $D(N, \Delta N)$ of the training loss for different levels of “overparameterization” which they measure using the training loss (lower training loss → more expressive model). They conclude from the experiment that “The lower the training loss, the higher the $D(N, \Delta N)$”. This comparison seems a bit unfair and requires normalizing $D(N, \Delta N)$ with respect to the training loss. For a loss lower bounded by 0 (like most losses), if the training loss is already small, the lowest possible improvement with more fixed point iterations $D(N, \Delta N)$ obviously gets smaller.

Author response:
**Fig 4: fairness of comparison across losses**: We think this is a very good point, and have reworked Fig 4. to include this normalization. It should be noted that with the normalization the result is less striking for MAML, but we still obtain a negative correlation score of $-0.4$ with a p-value of $0.03$ (i.e. statistically significant) without rerunning the experiment. For iMAML the result is still very obvious with a negative correlation score of $-0.6$ and a p-value of $7.10^{-4}$.