Paper: Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs (ICLR-2024)

Reviewer comment (Evaluation):
I think even though the spike of the BLiMP/GLUE score follows closely after the spike of the SAS score, it is not well substantiated to say that SAS is necessary for the capabilities required for BLiMP and GLUE. The intervention experiment in Sec 4.2, 4.3 can not support the causal relationship between them either. If there is a latent factor X that causes the better SAS, BLiMP, GLUE scores, adding the regularization term may suppress that latent factor X in addition to the SAS score. In this case, suppressing SAS also leads to worse BLiMP and GLUE scores. So that SAS is necessary for BLiMP and GLUE is not the only explanation for the observation in Sec 4.3, 4.3.

Author response:
Thank you for highlighting these subtleties of causal reasoning. Our regularization explicitly suppresses only the maximum attention weights associated with syntactic structure – if our suppression somehow also suppresses some other latent factor X, then X is likely associated with an internal representation of syntax. Furthermore, our manipulations of SAS consistently impact the timing of both the structure onset and capabilities onset (Figure 5; Figure 18, Appendix N) – *i.e.*, the BLiMP breakthrough always occurs shortly after the UAS spike (if the UAS spike occurs), regardless of whether we accelerate or delay the UAS spike using our SAS regularization. Taken together, our many experiments provide overwhelming evidence that SAS is causally related with the development of complex linguistic capabilities.
    - It is difficult to completely eliminate the possibility of a latent cause, but our insights about the relationship between the timing of phase transitions and the interactions between different learning strategies hold regardless of whether the underlying cause is exactly SAS or another internal structure that is strongly associated with syntax.

Reviewer comment (Theory):
It seems that this paper suggests that SAS indicates that the model suffers from some simplicity bias issue, which is counterintuitive to me. In general I think people use simplicity bias to explain some robustness issues because some spurious (unreliable/non-causal) features are simpler to learn than causal features, or say the model is doing some shortcut learning. However, it is hard to imagine that the syntactic structure is something the model shouldn’t rely on to solve any NLP problem.

Author response:
Thank you for mentioning this point – we see how it may be unclear. It is true that many studies of simplicity bias employ experiments with heuristics that are not just simplistic, but also spurious. However, generally the definitions of simplicity bias characterize only the complexity of the learner and not its use of spurious features ([Dingle et al.](https://www.nature.com/articles/s41467-018-03101-6), [Pérez et al.](https://arxiv.org/abs/1805.08522), [Nakkiran et al.](https://arxiv.org/abs/1905.11604), [Shah et al.](https://proceedings.neurips.cc/paper/2020/hash/6cfe0e6127fa25df2a0ef2ae1067d915-Abstract.html)). It may so happen that these simplistic heuristics are "shortcuts" and therefore non-causal features, but this is not a requirement for the definition of simplicity bias. Instead, we are merely referring to the observation that *too much* focus on a simplistic strategy (even an effective, non-spurious one) may distract from a more nuanced and complex learning strategy. In some cases, this more complex strategy may simply add more features on top of the simplistic strategy, rather than replacing it.
    - We have added a brief explanation at the end of Section 4.1 (in red) that also clarifies the definition of simplicity bias – thank you for helping us improve our paper. Appendix C.1 also expands upon the relevant prior work.
   - As for evidence that the model's prediction is really biased, our aim is only to show that the model's learning is biased towards learning human interpretable artifacts such as SAS, not that the predictions are biased by SAS. We show that BERT-Base naturally learns SAS both without intervention (Fig. 1) and even after limited amounts of suppression (Section 4.3.1).

Reviewer comment (Novelty):
It’s unclear to me why we should look at the development of these “capabilities”. I would like to know how the findings in this paper can potentially direct future research directions?

Author response:
Understanding phase transitions guides our understanding of how models acquire the skills necessary for completing complex tasks and highlights potential optimization inefficiencies. These insights can often be used to improve model training – for instance, we show that briefly suppressing SAS can accelerate training (Section 4.3), that training instabilities (or critical points) often occur during the alternative strategy onset (Section 5.2, Figure 4), and that an interpretable dependency exists between internal structure and external capabilities (Section 4.2). In other work, [Achille et al.](https://arxiv.org/abs/1711.08856) identify how data corruption during the early phases of training has a disproportional effect on long-term model performance. [Gur-Ari et al.](https://arxiv.org/abs/1812.04754) identify a phase transition early in training wherein the gradient quickly concentrates within the subspace of the top few eigenvalues of the Hessian; this later inspired techniques for low-dimensional representations of DNN parameters ([Lu et al.](https://dl.acm.org/doi/10.1145/3583780.3614873)) and proofs for the emergence of SGD alignment with outlier eigenspaces ([Arous et al.](https://arxiv.org/abs/2310.03010)) Similarly, detailed analyses of the phase transitions in grokking ([Power et al.](https://arxiv.org/abs/2201.02177), [Thilak et al.](https://arxiv.org/abs/2206.04817)) led to discoveries of how to eliminate grokking (and accelerate generalization) on algorithmic datasets ([Liu et al.](https://openreview.net/forum?id=zDiHoIWa0q1)).

Reviewer comment (Writing):
This paper should be more specific about the definition of “capabilities”.

Author response:
Thank you for pointing out this potentially confusing language – we understand that these terms have been widely used in the literature to loosely describe many distinct things. We give a more specific definition of what we mean by "capabilities" in the first bullet point of the contributions part of the introduction: "After the spike, the model starts handling complex linguistic phenomena correctly, as signaled by a break in BLiMP score (which we call the **capabilities onset**)." In other words, we refer to "capabilities" as the ability to handle complex linguistic phenomena or challenging grammatical constructions. 
    - As for the terms "emergence" and "phase transition," we follow the literature ([Wei et al.](https://arxiv.org/abs/2206.07682), [Chowdhery et al.](https://arxiv.org/abs/2204.02311), [Caballero et al.](https://arxiv.org/abs/2210.14891)) and use these terms interchangeably with "break" or "breakthroughs." We precisely define the formula used to identify a breakthrough in Section 2.3 (Equation 2). We have also added a clarification (in red) to that section indicating how we use these terms interchangeably.

Reviewer comment (Theory):
It’s unclear to me how the findings in section 4.1.1 agree with what part of the IB theory.

Author response:
Thank you for the feedback – we have updated Section 4.1 (revisions in red) to clarify this point. In short, we are demonstrating how SAS is a phase transition in both the model's structural syntax representation and functional complexity. IB theory simply provides another lens to analyze the phase transitions with – in addition to the emergence of internal syntactic structure followed by the capabilities onset, we also see a brief memorization phase (marked by the increase in functional complexity, Figs. 2(a) and 14(b)) coinciding with the structure onset, followed by an extended compression phase (marked by the gradual decrease of functional complexity, Figs. 2(a) and 14(b)) which starts exactly at the capabilities onset.

Reviewer comment (Writing):
In Sec 4.2, the term “phase transition” — could you clarify what it refers to? Does it refer to the period between the structure onset and the capabilities onset?

Author response:
We refer to multiple phase transitions throughout the paper – the structure onset and capabilities onset are phase transitions of the UAS and BLiMP, respectively. We mention this point in the first bullet point in the introduction.

Reviewer comment (Writing):
I understand that every paper needs a reasonable scope to work on and I don’t expect that one single paper explains everything. However, I would suggest that the authors scope this paper more explicitly. “Emergence”, “phase transition” and “capability”, for example, I think are some very general terms, and this paper focuses only on some specific aspects of them. Scoping more clearly and explicitly in the introduction section will help readers (at least me) understand this paper more easily, especially when this paper is discussing MLM models while these terms are usually co-occur with autoregressive language models.

Author response:
Thank you for pointing out this potentially confusing language – we understand that these terms have been widely used in the literature to loosely describe many distinct things. We give a more specific definition of what we mean by "capabilities" in the first bullet point of the contributions part of the introduction: "After the spike, the model starts handling complex linguistic phenomena correctly, as signaled by a break in BLiMP score (which we call the **capabilities onset**)." In other words, we refer to "capabilities" as the ability to handle complex linguistic phenomena or challenging grammatical constructions. 
    - As for the terms "emergence" and "phase transition," we follow the literature ([Wei et al.](https://arxiv.org/abs/2206.07682), [Chowdhery et al.](https://arxiv.org/abs/2204.02311), [Caballero et al.](https://arxiv.org/abs/2210.14891)) and use these terms interchangeably with "break" or "breakthroughs." We precisely define the formula used to identify a breakthrough in Section 2.3 (Equation 2). We have also added a clarification (in red) to that section indicating how we use these terms interchangeably.

Reviewer comment (Experiments):
I would like to know what kind of method/technique is used for encoding positional information. Is it the same as the original positional encoding in Vaswani et al. 2017 (https://arxiv.org/abs/1706.03762) or something more recent variants such as Rotary Positional Encoding (https://arxiv.org/abs/2104.09864v4)? I'm asking this because the syntactic dependency relates to positional information in the sentence. I wonder how much it affects (or does not affect) the experimental setup in this paper.

Author response:
"I would like to know what kind of method/technique is used for encoding positional information. Is it the same as the original positional encoding in Vaswani et al. 2017 (https://arxiv.org/abs/1706.03762) or something more recent variants such as Rotary Positional Encoding (https://arxiv.org/abs/2104.09864v4)? I'm asking this because the syntactic dependency relates to positional information in the sentence. I wonder how much it affects (or does not affect) the experimental setup in this paper."  
    - We use the same positional embeddings implementation as in the original BERT paper (Devlin et al., https://arxiv.org/abs/1810.04805) – i.e., fully trained absolute sinusoidal positional embeddings. It would definitely be interesting to explore how various positional embeddings affect the emergence of SAS and the subsequent capabilities onset. Although we were unable to re-run our entire suite of experiments with other positional encodings, there does exist limited evidence that language models learn internal syntax representations even without positional embeddings (such as in the ELMo model, in [Hewitt and Manning](https://aclanthology.org/N19-1419/)) and in decoder-only models with causal attention (such as in the GPT-2 model, in [Vig and Belinkov](https://arxiv.org/abs/1906.04284)). These syntax probes are not exactly the same as the one we use, but are likely related (especially the attention-based probe in [Vig and Belinkov](https://arxiv.org/abs/1906.04284)).

Reviewer comment (Experiments):
This is a more open-ended question but I wonder whether we can observe similar breakthrough (steep drop in loss) in auto-regressive (causal / decoder-only) LMs.

Author response:
"This is a more open-ended question but I wonder whether we can observe similar breakthrough (steep drop in loss) in auto-regressive (causal / decoder-only) LMs."  
    - This is an interesting question – there seems to be some empirical evidence that similar breakthroughs exist in auto-regressive LMs. For example, Fig. 2 in [Kaplan et al.](https://arxiv.org/pdf/2001.08361.pdf) shows initial steep loss drops in large decoder-only Transformers, and [Olsson et al.](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html#phase-change-more-closely) show that the steep loss drop in shallow attention-only models is associated with a break in in-context learning abilities and induction head formation.

Reviewer comment (Presentation):
There are so many experiments and interesting observations that the main paper makes very frequent reference to a plethora of appendices for more detail. This makes it a bit hard in places to figure out _exactly what_ is being reported and what it all means. (E.g. the discussion of the Information Bottleneck was fairly hard to follow, even to someone who knows a bit about that literature.)

Author response:
Thank you for this feedback, we have revised the Information Bottleneck section (Section 4.1) to try to simplify and clarify its connection to the rest of the paper. It's true that there are a lot of results in this paper, and we appreciate your feedback on how to clarify it all.

Reviewer comment (Evaluation):
Fig 1b: why do you think there's so much more variance in the BLiMP results than in the loss curves and UAS scores?

Author response:
The confidence interval we computed for the BLiMP results in Fig. 1(b) is based on variance across different BLiMP challenges (mentioned in the caption). In general, it is common for models to vary in accuracy across the different challenges ([Warstadt et al.](https://arxiv.org/abs/1912.00582)), since each tests a very specific linguistic capability. The loss curve, on the other hand, has a confidence interval computed from the variance across different examples, which we expect to be lower since the model is solving the same task for each example (masked language modeling). The UAS scores do not have a confidence interval in this particular figure (but in Fig. 3 we include confidence intervals across the three model seeds) since the implementation from [Clark et al.](https://arxiv.org/abs/1906.04341) did not admit straightforward calculations of variance. Since each example may have different syntax relations, it would be difficult to interpret the variance of UAS across different examples.

Reviewer comment (Experiments):
I'm curious about whether it matters that silver dependencies were used in regularization. Did you try any other "data-free" regularizers to see if they impact SAS similarly? E.g. since each token has one head, a regularizer that promotes sparsity of attention should implicitly promote SAS as well and vice versa.

Author response:
We did at one point try a *uniform* regularizer that would uniformly regularize any attention weight corresponding to a pair of syntactically related tokens. Although this still required dependency relation labels (and is therefore not "data-free" as you asked about), it would still promote sparsity of attention for $\lambda>0$. However, we found that this uniform regularizer did not suppress SAS as effectively as our proposed regularizer. This is likely because if we try to suppress sparsity with the uniform regularizer, the model can still achieve a low loss with a mostly uniform distribution that spikes only on a single attention weight in a specific head and layer between syntactically related pairs of tokens.

Reviewer comment (Novelty):
Missing references: (i) Liu et al 2021, "Probing Across Time": https://aclanthology.org/2021.findings-emnlp.71/. (ii) p 4: "Causal methods..." I have an idea of what works the authors have in mind, but think they should be explicitly cited here.

Author response:
Thank you, we have added Liu et al. to our related work section in Appendix C.2. We have already cited multiple causal methods in App. C.2.1 but have also added these citations to the first paragraph of Section 4.

Reviewer comment (Reproducibility):
Will code and data be made publicly available?

Author response:
Yes, we will make our code available. Our pretraining uses publicly available datasets ([BookCorpus](https://huggingface.co/datasets/bookcorpus) and [English Wikipedia](https://huggingface.co/datasets/wikipedia)), whereas our evaluation on the WSJ portion of the Penn Treebank requires a membership to LDC, but is available [here](https://catalog.ldc.upenn.edu/LDC99T42).