Paper: Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition (ICLR-2024)

Reviewer comment (Presentation):
The presentation in this paper is extremely ineffective. The classification of critical points is crucial to understanding the paper, yet a complete and accurate description is relegated to the appendix. Similarly, there are many k-gon diagrams in the main text with no direct explanation on how to read them. I believe the paper would be much easier to read with a compact set of definitions clearly stated in the main text.

Author response:
You highlighted the importance of detailing the classification of critical points within the main text rather than relegating it to the appendices. Please note the definition of $k$-gon critical points is already presented in Section 3.2 of the main text. Although Figures A.1 to A.3 in Appendix A enhance the visualization of these $k$-gons, we regret that space constraints prevent their inclusion in the main text.

Reviewer comment (Theory):
While I personally believe that studying simple models can be interesting, I struggle to see any practical interest in studying the model the authors propose, and I am not sure how feasible it can be to relax the many underlying assumptions: having more than 2 hidden units makes the structure of the critical points significantly more complicated, and changing the data model would make the whole theory collapse.

Author response:
It is true that changing the system would invalidate the current classification. However, as remarked above, the classification is *not* the primary contribution of the paper: it is one piece of the puzzle, that allows us to establish the other contributions listed above as the actually important contributions (about phase transitions). So the toy model we study is interesting not because the classification of critical points generalises, but because the Bayesian theory of phase transitions *may* generalise to larger models. This expectation is very reasonable theoretically, and the experiments in this paper are the first step towards verifying this theory empirically.

Reviewer comment (Evaluation):
Even accepting all the limitations above, I don't think the authors provide an adequate quantitative study of the phase transitions of this model beyond some general remarks. I would argue that the analytical results on the loss play no role in the discussion of the dynamical transition picture, as the energy levels could have been identified from the experiments, and knowing the exact values has no importance in what is presented.

Author response:
You state that "I don't think the authors provide an adequate quantitative study of the phase transitions of this model beyond some general remarks". This paper contains, by far, the most detailed study so far in the literature of phases and phase transitions in any neural network and goes far beyond general remarks, providing a classification (both theoretically and empirically backed) of phases and phase transitions.  
Re: the role of the analytical results in the dynamical transition picture. There are ample evidence that properties of the critical points other than their energy level, play an important role. Analytical results about the geometrical properties like the learning coefficients can gives us a more detailed picture. For example, it leads to the curious observation that as loss drops during SGD training, there is a corresponding increase in complexity.

Reviewer comment (Experiments):
One significant weakness is the limited exploration of how the results obtained from this small-scale learning task can be applied to larger-scale and more practical problems.

Author response:
Regarding the link between the small-scale learning task studied in our paper and larger-scale and more practical problems, our response is as follows. The paper makes several main claims, including:

- (A) Bayesian phase transitions are theoretically predicted and actually occur in TMS
- (B) Dynamical transitions occur in TMS, and
- (C) There is some relation between Bayesian and dynamical transitions
- (D) The local learning coefficient is a powerful tool for reasoning about phase transitions

In both (A) and (B) we don't see the scope of our results as really being limited to small-scale systems. Since (A) is a general phenomena of Bayesian learning once the general principles are seen to apply in TMS, it is reasonable to expect such transitions to occur in larger systems, since the larger the system the *more* we expect its behaviour to be governed by general laws. Dynamical transitions have already been observed in larger scale systems, including language models with billions of parameters in the work on Induction Heads and In-Context Learning from Anthropic.

We agree that in (C) the current evidence linking Bayesian and dynamical transitions is restricted to one toy system (TMS). It will require follow-up work to assess whether this can be extended to a general principle for working with larger-scale systems.

However in the case of (D) we believe our work is *already* informing people's plans about how to think about phase transitions in real-world systems and so we feel confident asserting that our approach will have a substantial impact. For example there has been some preliminary work done already looking at modular addition and grokking using the learning coefficient.

Reviewer comment (Theory):
The paper does not clearly articulate how the insights gained from this Toy Model of Superposition (TMS) could be extended to real-world, complex learning scenarios.

Author response:
Regarding the link between the small-scale learning task studied in our paper and larger-scale and more practical problems, our response is as follows. The paper makes several main claims, including:

- (A) Bayesian phase transitions are theoretically predicted and actually occur in TMS
- (B) Dynamical transitions occur in TMS, and
- (C) There is some relation between Bayesian and dynamical transitions
- (D) The local learning coefficient is a powerful tool for reasoning about phase transitions

In both (A) and (B) we don't see the scope of our results as really being limited to small-scale systems. Since (A) is a general phenomena of Bayesian learning once the general principles are seen to apply in TMS, it is reasonable to expect such transitions to occur in larger systems, since the larger the system the *more* we expect its behaviour to be governed by general laws. Dynamical transitions have already been observed in larger scale systems, including language models with billions of parameters in the work on Induction Heads and In-Context Learning from Anthropic.

We agree that in (C) the current evidence linking Bayesian and dynamical transitions is restricted to one toy system (TMS). It will require follow-up work to assess whether this can be extended to a general principle for working with larger-scale systems.

However in the case of (D) we believe our work is *already* informing people's plans about how to think about phase transitions in real-world systems and so we feel confident asserting that our approach will have a substantial impact. For example there has been some preliminary work done already looking at modular addition and grokking using the learning coefficient.

Reviewer comment (Experiments):
Providing a bridge between the toy model and practical applications is essential for enhancing the paper's impact.

Author response:
Regarding the link between the small-scale learning task studied in our paper and larger-scale and more practical problems, our response is as follows. The paper makes several main claims, including:

- (A) Bayesian phase transitions are theoretically predicted and actually occur in TMS
- (B) Dynamical transitions occur in TMS, and
- (C) There is some relation between Bayesian and dynamical transitions
- (D) The local learning coefficient is a powerful tool for reasoning about phase transitions

In both (A) and (B) we don't see the scope of our results as really being limited to small-scale systems. Since (A) is a general phenomena of Bayesian learning once the general principles are seen to apply in TMS, it is reasonable to expect such transitions to occur in larger systems, since the larger the system the *more* we expect its behaviour to be governed by general laws. Dynamical transitions have already been observed in larger scale systems, including language models with billions of parameters in the work on Induction Heads and In-Context Learning from Anthropic.

We agree that in (C) the current evidence linking Bayesian and dynamical transitions is restricted to one toy system (TMS). It will require follow-up work to assess whether this can be extended to a general principle for working with larger-scale systems.

However in the case of (D) we believe our work is *already* informing people's plans about how to think about phase transitions in real-world systems and so we feel confident asserting that our approach will have a substantial impact. For example there has been some preliminary work done already looking at modular addition and grokking using the learning coefficient.