Paper: Big Learning (ICLR-2024)

Reviewer comment (Writing):
The presentation is poor, without the examples the definition of the big learning problem is unclear. The authors should define $x_T$ and $x_S$. Most of the examples only have one pair of $(T, S)$, not a collection.

Author response:
**Q1: ...the definition of the big learning problem is unclear. The authors should define $x_T$ and $x_S$. Most of the examples only have one pair of $(T,S)$, not a collection.**

Thank you for the constructive suggestions. In the revised paper, we have added a new Fig. 1 to the main paper and a new Fig. 6 to the Appendix to highlight
- the definitions of $(x_T, x_S)$ and $(T,S)$ (with a collection of examples), 
- our main motivations, and 
- the big picture of the big learning, 

to improve the clarity.

Reviewer comment (Experiments):
This paper lacks of systematic quantitative comparisons between the proposed approach and existing methods on pretraining foundation models with large-scale data.

Author response:
**Q2: This paper lacks of systematic quantitative comparisons between the proposed approach and existing methods on pretraining foundation models with large-scale data. ... quantitative experiments (table 2 only) is too small and cannot be representative for training FMs.**

We argue that systematic quantitative comparisons with/among existing foundation models (FMs) are not the primary concern when proposing a unified, consistent, and principled big learning framework that contains most FMs and learning paradigms as special cases. 

We address your concerns by first providing detailed discussions on ``systematic quantitative comparisons with/among existing/SOTA foundation models'' (see our general response $1$), followed by highlighting our main contributions (see our general response $2$), which we believe are valuable to the machine learning community.

Reviewer comment (Experiments):
The claims are not sufficiently supported by the experiments: while the authors argue the potential of the learning framework, the quantitative experiments (table 2 only) is too small and cannot be representative for training FMs.

Author response:
**Q2: This paper lacks of systematic quantitative comparisons between the proposed approach and existing methods on pretraining foundation models with large-scale data. ... quantitative experiments (table 2 only) is too small and cannot be representative for training FMs.**

We argue that systematic quantitative comparisons with/among existing foundation models (FMs) are not the primary concern when proposing a unified, consistent, and principled big learning framework that contains most FMs and learning paradigms as special cases. 

We address your concerns by first providing detailed discussions on ``systematic quantitative comparisons with/among existing/SOTA foundation models'' (see our general response $1$), followed by highlighting our main contributions (see our general response $2$), which we believe are valuable to the machine learning community.

Reviewer comment (Evaluation):
It is important to compare the computational cost of the proposed approach with prior method. Can the authors discuss this?

Author response:
**Q3: It is important to compare the computational cost of the proposed approach with prior method. Can the authors discuss this?**


The big learning principle is a general concept that contains most FMs and many learning paradigms as special cases. Accordingly, similar to the discussions on ``systematic quantitative comparisons with/among existing/SOTA FMs,'' the big learning has the same computational cost with its special cases.

Should the reviewer concern about the comparisons among different big-learning instantiations (i.e., different FM algorithms) like masked LM versus BigLearn-GAN, we highlight that the comparisons might not be necessary, because they are often used for different applications. 
Despite that, the BigLearn-GAN is likely more computationally expensive, because of its additional discriminator, regularizer, and communication loss (see the newly introduced Appendix Algorithm 2 of the revised paper). But one also receives the valuable benefits from the corresponding adversarial training and big learning.


Considering a general analysis on the computational cost of the big learning, we have provided a new Algorithm 1 in the Appendix of the revised paper.
In each iteration, we randomly sample **one** $(S,T)$ pair and then perform the selected matching of $p_{\theta}(x_T|x_S) \rightarrow q(x_T|x_S)$, which is similar to one step of conventional update of $p_{\theta}(x) \rightarrow q(x)$, with limited additional overhead to rearrange the data $(x_S,x_T)$ and to embed $(S,T)$.
Therefore, the computational cost for each iteration is roughly the same. 
However, since the big learning masters many/all joint, conditional, and marginal data capabilities simultaneously during training, it needs moderately more iterations to converge. Empirically, we find that $200K$ iterations are sufficient for the big learning.