Paper: Stable Anisotropic Regularization (ICLR-2024)

Reviewer comment (Experiments):
The reasons for contradictions with prior research are unclear, weakening the persuasiveness of the main claim. The authors' main claim that "anisotropy is the key to model performance improvement" isn't reconciled with prior research which posits that "isotropy is the key to model performance improvement". While the authors suggest that the discrepancy arises from the evaluation metrics used (as stated “previous studies have made claims using “flawed” measures of isotropy,” on page 7), both this work and prior studies differ not only in metrics but also in tasks (GLUE tasks vs. word similarity tasks). Therefore, it's not an apple-to-apple comparison. If there's an inverse correlation between anisotropy and performance in word similarity tasks, it becomes challenging to coherently explain the overall results. This issue might possibly be resolved if there were experiments or comprehensive discussions specifically for the word similarity tasks.

Author response:
We agree that more work needs to be done to determine the relationship between isotropy and word similarity tasks. However, most word similarity benchmarks simply calculate the cosine similarity of word embeddings and do not involve any model fine-tuning. Since I-STAR is a regularization algorithm that adjusts isotropy during fine-tuning, I-STAR may not be the most appropriate method to analyze the relationship between word similarity and isotropy. However, a future direction of work could continue the pre-training task of masked language modeling with an added I-STAR penalty to adjust for isotropy. 

Instead, our paper focuses on the relationship between isotropy and fine-tuning performance, as I-STAR is a regularization algorithm requiring fine-tuning tasks. However, our paper evaluates the relationship between isotropy and a model’s ability to rate the similarity of two sentences using the Semantic Textual Similarity Benchmark (STS-B). STS-B consists of sentence pairs annotated with a similarity score from 1-5. 

Although a majority of previous works have argued that isotropy is beneficial for LLM performance, there are several recent studies that support the claim that adjusting isotropy in fine-tuning hurts downstream model performance. Namely, Rajaee & Pilehvar 2021 show that adjusting for isotropy by removing principal components hurts model performance. Further, recent work by Rudman et al. 2023 demonstrates that a dimension’s ability to store task-specific knowledge strongly correlates with high variance in a given dimension.

Reviewer comment (Evaluation):
If anisotropy is a natural consequence of SDG, what is the significance of deliberately adding anisotropy as a regularization term? The computational cost increases slightly, and the disadvantage of increased hyperparameter tuning cost needs to be offset by some substantial benefits. If there are such advantages, the appeal of the proposed method would be enhanced.

Author response:
Thank you for a great question! Previous works have argued that anisotropy helps models better generalize to unseen examples and is important in fine-tuning (Zhou et al. 2018). Our work empirically supports this claim by demonstrating a relationship between isotropy and performance. We hypothesize that the significance of adding our anisotropy penalty term helps models learn fine-tuning tasks faster. In order to verify our hypothesis, we are actively running an experiment where we track how quickly models converge when training with no regularization and different \lambda values in I-STAR.

Reviewer comment (Experiments):
What is the significance of adjusting the hyperparameter $\zeta$ for shrinkage estimation when using IsoScore* as a regularization term? Even if the value of IsoScore is estimated to be on the lower side (Figure 2), if the estimated value has a monotonic relationship with the true value, wouldn't there be no issue in regularization? If the goal is to ensure differentiability, couldn't we just fix it at an appropriate value?

Author response:
Figure 7 in Section C of the Appendix addresses the significance of adjusting the shrinkage parameter $\zeta$ in I-STAR. In all cases, the optimal shrinkage parameter is in the interval [0.2,0.8]. Figure 7 shows that when $\zeta=0$ and we use (a differentiable version of) IsoScore as a regularization penalty model, performance decreases. Further, if we set $\zeta=0$, the isotropy penalty is computed directly from the shrinkage matrix. Namely, no mini-batch information is included in the loss function. When $\zeta=1$, performance is slightly lower than optimal $\zeta$, but significantly higher than when $\zeta=0$.  Results from this experiment demonstrate the need for IsoScore* in order to perform isotropic regularization.

Reviewer comment (Evaluation):
There is no significance testing on the results (Table 1) but there are error bars (good!) — these seem to indicate that most differences outlined are hardly significant (e.g. RTE, 72.56+/-1.29 vs 71.34+/-0.91). This makes it difficult to get a clear picture of the resulting effect of decreasing isotropy.

Author response:
We will run significance testing for each of the model/task setups and display the significance testing results in the paper's main body.

Reviewer comment (Presentation):
Similarly Fig. 3 is difficult to interpret — there are clear decreasing trends in some plots, not so much in most of them.

Author response:
In order to clarify the relationship between isotropy and performance, we have performed Pearson correlation analysis on the results of Figure 3. Initial analysis of our Pearson correlation analysis indicates that isotropy is indeed negatively correlated with performance in all cases (except ALBERT on COLA). Note that the small sample size of each entry (num_tuning_params x num_random_seeds = 35) may limit the generalizability of these results and cause some of the p values to be artificially high.  
    Model & QQP & RTE & MRPC & COLA & STS-B  

    BERT & $-0.573 / 0.001$ & $-0.418 / 0.012$ & $-0.115 / 0.511$ & $-0.227 / 0.190$ & $-0.677 / 0.001$  

    ALBERT & $-0.469 / 0.009$ & $-0.424 / 0.011$ & $-0.184 / 0.330$ & $0.195 / 0.261$ & $-0.624 / 0.004$  
 
    DistBERT & $-0.319 / 0.062$ & $-0.840 / 0.003$ & $-0.532 / 0.003$ & $-0.641 / 0.003$ & $-0.713 / 0.005$

Reviewer comment (Writing):
p.4: Presumably all norms in steps 6 and 7 of Algorithm 1 are 2-norms?

Author response:
Steps 6 &  7 of Algorithm 1 are L2 norms. We will add this to the pseudo-code of the algorithm.

Reviewer comment (Writing):
p.4: Is RDA "regularized discriminant analysis"? It is only introduced on p.5.

Author response:
RDA is indeed a Regularized Discriminant Analysis, which we will introduce earlier in the revised version of our paper.

Reviewer comment (Experiments):
p.4: As \Sigma_S is computed from a large number of tokens and re-estimated after each epoch, this is presumably quite costly computationally. Could you comment on that? As you are computing these anyway, why not use that directly to estimate isotropy? This would essentially correspond to \zeta=1, which is not tested here as far as I can tell.

Author response:
Since we only need to collect a sample of 250k tokens, the partial forward pass on the dataset. Computing the isotropy penalty only on \Sigma_S is equivalent to setting $\zeta=1$. The downside to setting $\zeta=1$ is that no isotropy information about the mini-batches of data will be included in the backward pass. Section C of the Appendix shows the results of fine-tuning BERT, ALBERT, and DistilBERT on RTE using different values of \zeta. Figure 7 (Sec C) demonstrates that setting $\zeta=1$ results in a marginal decline in performance compared to incorporating mini-batch isotropy information during gradient descent.

Reviewer comment (Experiments):
p.9: Why are -1 and +1 lambdas excluded from Fig. 5? -1 seems the most popular choice in Fig. 2.

Author response:
We have updated Figure 5 to add \lambda values of +1 and -1.

Reviewer comment (Experiments):
Regularization to a manifold in parameter space is well studied in Machine Learning, and indeed supports the argument that anisotropy may benefit performance. This is also linked to the idea that there is the level of representativity in the model that must match the intrinsic dimension or complexity in the data. Fig. 5 actually seems to show roughly consistent intrinsic dimensions for the three models used here. Is there a way you could put this in perspective with e.g. model size? Showing for exemple how anisotropy favors a roughly stable internal dimensionality in parameter space as model size increases?

Author response:
That is a very interesting question! Based on your comment, there are two potential experiments to design. First, we can fix the task and fine-tune different model sizes for that task. For example, we can fine-tune BERT-base & BERT-large as well as all of the available ALBERT model sizes from ALBERT-base (12m parameters) to ALBERT-xxl (235M parameters) with different tuning parameter values in I-STAR and calculate the intrinsic dimension of the fine-tuned models. However, given that BERT, DistilBERT, and ALBERT all have a different number of parameters, an interesting follow-up experiment would be to run the experiments in Figure 5 for all of the tasks considered in this paper to verify if there is a desirable intrinsic dimension for model representations for each of the different datasets.

Reviewer comment (Evaluation):
If we see a trend in Figure 3 on how higher IsoScore* leads to lower accuracy, some correlation and significance score should be added to support this claim.

Author response:
In order to clarify the relationship between isotropy and performance, we have performed Pearson correlation analysis on the results of Figure 3. Initial analysis of our Pearson correlation analysis indicates that isotropy is indeed negatively correlated with performance in all cases (except ALBERT on COLA). Note that the small sample size of each entry (num_tuning_params x num_random_seeds = 35) may limit the generalizability of these results and cause some of the p-values to be artificially high.

    Model & QQP & RTE & MRPC & COLA & STS-B 

    BERT & $-0.573 / 0.001$ & $-0.418 / 0.012$ & $-0.115 / 0.511$ & $-0.227 / 0.190$ & $-0.677 / 0.001$

    ALBERT & $-0.469 / 0.009$ & $-0.424 / 0.011$ & $-0.184 / 0.330$ & $0.195 / 0.261$ & $-0.624 / 0.004$
 
    DistBERT & $-0.319 / 0.062$ & $-0.840 / 0.003$ & $-0.532 / 0.003$ & $-0.641 / 0.003$ & $-0.713 / 0.005$

Reviewer comment (Experiments):
Could you add in the Appendix the same experiments done in Figure 3 but for CosReg and also the same in Figure 4 but for IsoScore*? This will give a better intuition of how these 2 different methods work.

Author response:
The updated version of our paper will contain the results of Figure 3 for CosReg and Figure 4 for IsoScore*.