Paper: Magnitude Invariant Parametrizations Improve Hypernetwork Learning (ICLR-2024)

Reviewer comment (Experiments):
I might miss the related information. Apologies if that is the case. Are there any results of combining additive output with BatchNorm/LayerNorm in Figure 5? It is interesting to see such results because it verifies that MIP has something that normalization layers cannot offer. If proportionality is the main reason, what are the theoretical reasons that MIP can solve the problem but normalization cannot?

Author response:
The reviewer is correct, Figure 5 does not include results combining the
additive output with BatchNorm/LayerNorm. In our experiments, adding
normalization layers to MIP had little effect in per-epoch convergence,
but it increased the per-epoch computational cost because of the
additional overhead of the normalization layers.

More generally, the fundamental reason normalization layers are not well
suited to deal with this issue is that they remove degrees of freedom by
enforcing zero mean and unit variance in the input. Since hypernetworks
often feature low dimensional input spaces, removing degrees of freedom
in the input introduces substantial information loss. Enforcing constant
magnitude inputs has similar shortcomings. This is why our input
encoding maps each dimension to a separate pair of coordinates.

Reviewer comment (Experiments):
If the problem is proportionality, how about we use other activation functions such as ELU or GELU? Does that solve the problem? If not, can we still say the problem is proportionality?

Author response:
We found that other activations like GELU, SiLU/Swish and Tanh, while
not having the same proportionality property, still propagate the input
magnitude enough for it to be detrimental in training. MIP improves
training for all choices of non-linear activation. See section C.3 of
the supplement.

Reviewer comment (Experiments):
While the authors do empirically show that MIP benefits training, it is not clear whether the increased variance could also be controlled with, e.g., appropriately chosen (i.e., lower) learning rates and (i.e., higher) momentum, in the original parametrisation (which could attain similar performance, albeit slower).

Author response:
This is an important point. In our experiments, we controlled for this by testing a range of learning rates and reporting the best converging models. We consistently found that MIP was less sensitive to hyperparameter choices than the default formulation, and that there were settings, like HyperMorph with momentum SGD, where we could not find any suitable hyperparameter setting (learning rate & momentum factor) that would allow the model to meaningfully train.

Reviewer comment (Experiments):
It would be interesting to see whether the residual formulation of the hypernetwork closes meaningfully the gap between fully task specific parameters and the ones predicted by the hyper network, i.e., $\theta^0 + h(E_{L2}(\gamma); w)$. For example, what is the performance if on a new task $t$ one starts from $\theta^0$ and just optimises for a specific number of steps on that task to get $\theta_t^*$? Is $\theta_t^* - \theta_0$ related to $h(E_{L2}(\gamma); w)$? Is the performance of $\theta_t^*$ similar to the performance of $\theta^0 + h(E_{L2}(\gamma); w)$?

Author response:
While not exactly the same as the setup described by the reviewer, we did perform experiments where we individually trained classical networks for each task $\theta_t$ for the HyperMorph and SSHN settings, to understand how classical networks converged for these settings. When comparing the task-specific networks to the performance of the hypernetwork weights for the same task, we found that hypernetwork weights performed no worse than task-specific weights.

Reviewer comment (Theory):
It is not clear why for non-scalar inputs $\gamma$, i.e., $\gamma \in \mathbb{R}^D$ with $D \geq 2$ a simple unit normalisation transformation, i.e., $\hat{\gamma} = \frac{1}{\|\|\gamma\|\|_2}\gamma$ would not work for removing the dependence on the magnitude. It seems to me that in this case each $\hat{\gamma}$ would just correspond to a different point on the hypersphere and the output of the hypernetwork would not be independent of the input.

Author response:
This is a good question that we should have addressed more directly. Dividing by the norm of the input produces information loss in the hypernetwork input space, even for multi-dimensional hypernetwork inputs. As an example, we can consider inputs $\gamma_A = [0.1, 0.2]$ and $\gamma_B = [0.4, 0.8]$, dividing them by their respective norms results in $\gamma_A / ||\gamma_A|| = \gamma_B / ||\gamma_B|| = [0.447,0.894]$. While $\gamma_A$ and $\gamma_B$ correspond to different points in the input space, they are mapped to the same constant norm representation. This is why our input encoding maps each dimension to a separate pair of coordinates.

Since hypernetworks often feature low dimensional input spaces, removing a degree of freedom in the input introduces substantial information loss.