Paper: Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR-2024)

Reviewer comment (Presentation):
In terms of presentation, the authors may want to add more description on the bert2BERT method, since it is one main competitor to the proposed MSG.

Author response:
We have added introductions to bert2BERT in Appendix (C) in the new version.

Reviewer comment (Theory):
For Eq. (11) on page 5: For d_2 < i < d_2', it should be: d_2 < i <= d_2'.

Author response:
Equation 11, 12, 14 are fixed. We appreciate your careful reading.

Reviewer comment (Theory):
For Eq. (14) on page 5: For d_1 < j < d_1', should be: d_1 < j <= d_1'.

Author response:
Equation 11, 12, 14 are fixed. We appreciate your careful reading.

Reviewer comment (Writing):
Last line on page 7: The stated (Qin et al., 2022b) is not the reference for the bert2BERT method. Do you really mean the bert2BERT or the ELLE here?

Author response:
ELLE reported an evolved version of bert2BERT which is applied to their own tasks. The methods are very similar with a main difference being an interpolation (instead of stacking) strategy in depth growth. We mean the implementation from ELLE here.

Reviewer comment (Experiments):
While the manuscript makes careful comparisons to competing progressive growth methods, some of the baseline performances (without progressive growth) are a little lower than expected.

Author response:
We tried our best with our current computational resources to study the performance gap problem you mentioned. Specifically, we first doubted that the high perplexity of our GPT model was because the model was undertrained. Thus, we controlled the training data to be exactly the same with nanoGPT, which is 9B OpenWebText tokens from huggingface. We performed a more sufficient training with a similar setting to nanoGPT and achieved very close training loss to the one they reported (3.11). Thus, we believe that our implementation is close enough to strong public repositories with this specific dataset, which makes our experiments meaningful. The settings and results are detailed in Appendix (J) in the new version. However, although we reduced the ppl from 41 to 37, there is still a gap. We guess it is because of the data: the public OpenWebText may have similar distribution to the closed-source data used to train GPT-2, but it may be short in number or quality. Another evidence is that none of the related work we cited successfully matches the original GPT-2 perplexities. For the SquaD results, we guess that it's a similar story, and code will be available to ensure reproductivity.

Reviewer comment (Experiments):
Please ensure all baseline approaches are faithful so that readers know MSG is effective in high-performing training runs.

Author response:
We tried our best with our current computational resources to study the performance gap problem you mentioned. Specifically, we first doubted that the high perplexity of our GPT model was because the model was undertrained. Thus, we controlled the training data to be exactly the same with nanoGPT, which is 9B OpenWebText tokens from huggingface. We performed a more sufficient training with a similar setting to nanoGPT and achieved very close training loss to the one they reported (3.11). Thus, we believe that our implementation is close enough to strong public repositories with this specific dataset, which makes our experiments meaningful. The settings and results are detailed in Appendix (J) in the new version. However, although we reduced the ppl from 41 to 37, there is still a gap. We guess it is because of the data: the public OpenWebText may have similar distribution to the closed-source data used to train GPT-2, but it may be short in number or quality. Another evidence is that none of the related work we cited successfully matches the original GPT-2 perplexities. For the SquaD results, we guess that it's a similar story, and code will be available to ensure reproductivity.

Reviewer comment (Experiments):
In Tables 4 and 5, the SQuAD scores are 2-3% lower than what is reported in the BERT paper.

Author response:
We tried our best with our current computational resources to study the performance gap problem you mentioned. Specifically, we first doubted that the high perplexity of our GPT model was because the model was undertrained. Thus, we controlled the training data to be exactly the same with nanoGPT, which is 9B OpenWebText tokens from huggingface. We performed a more sufficient training with a similar setting to nanoGPT and achieved very close training loss to the one they reported (3.11). Thus, we believe that our implementation is close enough to strong public repositories with this specific dataset, which makes our experiments meaningful. The settings and results are detailed in Appendix (J) in the new version. However, although we reduced the ppl from 41 to 37, there is still a gap. We guess it is because of the data: the public OpenWebText may have similar distribution to the closed-source data used to train GPT-2, but it may be short in number or quality. Another evidence is that none of the related work we cited successfully matches the original GPT-2 perplexities. For the SquaD results, we guess that it's a similar story, and code will be available to ensure reproductivity.

Reviewer comment (Writing):
Please summarize the MSG operator conclusions/discussion from the referenced technical report (Li et al., 2023).

Author response:
(Li et al., 2023) is summarized in Appendix (I).

Reviewer comment (Presentation):
Consider moving tables and figures to the page they are discussed on, or to the next page, to help readability.

Author response:
Most Tables, figures, and minors are fixed in the new version, with a new citation to (Bartoldson et al., 2023) in Related Work. For the positions of Tables and figures, we will handle them in the final version.

Reviewer comment (Presentation):
Clean up figures and tables.

Author response:
Most Tables, figures, and minors are fixed in the new version, with a new citation to (Bartoldson et al., 2023) in Related Work. For the positions of Tables and figures, we will handle them in the final version.

Reviewer comment (Presentation):
Table 2 could have uniform line lengths.

Author response:
Most Tables, figures, and minors are fixed in the new version, with a new citation to (Bartoldson et al., 2023) in Related Work. For the positions of Tables and figures, we will handle them in the final version.

Reviewer comment (Presentation):
Figure 2 could have larger axis text, lines, legend text, etc.

Author response:
Most Tables, figures, and minors are fixed in the new version, with a new citation to (Bartoldson et al., 2023) in Related Work. For the positions of Tables and figures, we will handle them in the final version.

Reviewer comment (Novelty):
All of the methods listed in the Related Work section (progressive growth, weight sharing, MoEs, etc.) are discussed in the survey "Compute Efficient Deep Learning" (Bartoldson et al., 2023).

Author response:
Most Tables, figures, and minors are fixed in the new version, with a new citation to (Bartoldson et al., 2023) in Related Work. For the positions of Tables and figures, we will handle them in the final version.

Reviewer comment (Theory):
I am a little bit confused by the introduced method; intuitively, it seems to me that the proposed MSG operator would extend the parameter space to a more rugged space when compared with the average-based methods. Thus I cannot understand why it works well intuitively.

Author response:
Thanks for your insightful comments and suggestions!
- An explanation of the effectiveness of MSG: (1) it introduces masks that make the process smoother for the model to adapt to the more “rugged” space caused by initialization; (2) average-based methods can underperform a naïve random initialization due to problems such as the Symmetry issue (Section 2.3.3). A more general speaking is: minimizing the difference to the original weights does not guarantee improving the training dynamics, since the latter is a research problem itself (also see Section 2.3.3). Another intuition (may be not so good) is that sometimes distillation works better than data-driven pruning for model compression.

Reviewer comment (Evaluation):
The reported results can not demonstrate the real effectiveness of the proposed method in terms of ability to generalize, in fact, it seems that the results are mainly based on training loss while the most important metric should be a set of metric to measure the generalization ability of the model.

Author response:
- Metrics: for the Bert models, we used an average of the GLUE scores. It already includes multiple metrics including Matthews correlation for CoLA, Pearson/Spearman correlation for STS-B and accuracy/f1 for MRPC, QQP, and SQuAD. Since these are a diversity of downstream NLP tasks, they would reflect some generalization ability of the models. The detailed results for each subtask are listed in Table 6 and 7. These may not be loss-based metrics since we observe that they are not strictly aligned with training loss in late phases (see footnote 4 and Figure 5). For GPT models, we follow common practices to report perplexity since they are aimed for text generation.