Original: C URRICULUM L EARNING : A R EGULARIZATION
Revised: C URRICULUM L EARNING : A R EGULARIZATION

Original: METHOD
Revised: M

Original:  FOR E FFICIENT AND S TABLE
Revised: ETHOD FOR E FFICIENT AND S TABLE

Original: B ILLIONS CALE GPT M ODEL P RE -T RAINING
Revised: B ILLION S CALE GPT M ODEL P RE -T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent works have demonstrated great success in training high-capacity autoregressive language models (GPT, GPT-2, GPT-3) on a huge amount of unlabeled text corpus for text generation.
Revised: Recent works have demonstrated great success in training high-capacity autoregressive language models (GPT, GPT-2, GPT-3) on a huge amount of unlabeled text corpus for text generation.

Original: Despite showing great results, autoregressive models are facing a growing training instability issue.
Revised: Despite showing great results, autoregressive models are facing a growing training instability issue.

Original: Our study on GPT-2 models (117M and 1.5B parameters) show that larger model sizes, sequence lengths, batch sizes, and learning rates would lead to lower training stability and increasing divergence risks.
Revised: Our study on GPT-2 models (117M and 1.5B parameters) show that larger model sizes, sequence lengths, batch sizes, and learning rates would lead to lower training stability and increasing divergence risks.

Original: To avoid divergence and achieve better generalization performance, one has to train with smaller batch sizes and learning rates, which leads to worse training efﬁciency and longer training time.
Revised: To avoid divergence and achieve better generalization performance, one has to train with smaller batch sizes and learning rates, which leads to worse training efﬁciency and longer training time.

Original: To overcome this stability-efﬁciency dilemma, we present a study of a curriculum learning-based approach, whichhelps improves the pre-training convergence speed of autoregressive models.
Revised: To overcome this stability-efﬁciency dilemma, we present a study of a curriculum learning-based approach, which improves the pre-training convergence speed of autoregressive models.

Original: More importantly, we ﬁnd that curriculum learning, as a regularization method, exerts a gradient variance reduction effect and enables to train autoregressive models with much larger batch sizes and learning rates without training instability, further improving the training speed.
Revised: More importantly, we ﬁnd that curriculum learning, as a regularization method, exerts a gradient variance reduction effect and enables to train autoregressive models with much larger batch sizes and learning rates without training instability, further improving the training speed.

Original: Our evaluations demonstrate that curriculum learning enables training
Revised: Our evaluations demonstrate that curriculum learning enables training

Original: GPT-2 models with 8x larger batch size and 4x larger learning rate, whereas the baseline approach struggles with training divergence.
Revised: GPT-2 models with 8x larger batch size and 4x larger learning rate, whereas the baseline approach struggles with training divergence.

Original: To achieve the same validation perplexity targets during pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 61% and 49%, respectively.
Revised: To achieve the same validation perplexity targets during pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 61% and 49%, respectively.

Original: To achieve the same or better zero-shot WikiText-103/LAMBADA evaluation results at the end of pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 54% and 70%, respectively.
Revised: To achieve the same or better zero-shot WikiText-103/LAMBADA evaluation results at the end of pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 54% and 70%, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale Transformer-based language models have powered breakthroughs in many natural language processing tasks.
Revised: Large-scale Transformer-based language models have powered breakthroughs in many natural language processing tasks (Vaswani et al., 2017).

Original: These models contain up to billion-scale parameters and are trained on massive open-domain web text corpus with unsupervised learning.
Revised: These models contain up to billion-scale parameters and are trained on massive open-domain web text corpus with unsupervised learning.

Original: Among them, one of the most successful cases is the GPT family: from GPT (110M parameters)
Revised: Among them, one of the most successful cases is the GPT family: from GPT (110M parameters)

Original: (Radford et al., 2018a), to GPT- (1.5B parameters)
Revised: (Radford et al., 2018a), to GPT-2 (1.5B parameters)

Original: (Radford et al., 2018b), and to GPT-3 with 175B parameters (Brown et al., 2020) that hit record high accuracy for many NLP tasks.
Revised: (Radford et al., 2018b), and to GPT-3 with 175B parameters (Brown et al., 2020) that hit record high accuracy for many NLP tasks.

Original: Recent studies (Kaplan et al., 2020) show that these models continue to achieve better accuracy as their sizes increase (together with richer training data), indicating the model size will continue growing in the future.
Revised: Recent studies (Kaplan et al., 2020) show that these models continue to achieve better accuracy as their sizes increase (together with richer training data), indicating the model size will continue growing in the future.

Original: Despite achieving remarkable model accuracy, training these models raise huge challenges on training efﬁciency and instability.
Revised: Despite achieving remarkable model accuracy, training these models raise huge challenges on training efﬁciency and instability.

Original: On one hand, training large models require an extremely huge amount of computation.
Revised: On one hand, training large models require a huge amount of computation.

Original: As an example, it requires approximately  14.8 days on 10000 V100 GPUs to train  GPT-3 (Patterson et al., 2021).
Revised: As an example, it requires approximately 9.2 days on 512 V100 GPUs to train a 8.3B GPT-2 (Shoeybi et al., 2019), and 14.8 days on 10000 V100 GPUs to train a 175B GPT-3 (Patterson et al., 2021).

Original: To reduce the training wall clock time of these large models, one of the most common solutions is to employ distributed training with hundreds or even thousands of GPUs across multiple machines such that the model can process the training data with much higher throughput.
Revised: To reduce the training wall clock time of these large models, one of the most common solutions is to employ distributed training with hundreds or even thousands of GPUs across multiple machines such that the model can process the training data with much higher throughput.

Original: Meanwhile, to avoid the huge overhead incurred by cross-GPU/node communication in distributed training, practitioners increase the batch sizes in the optimization algorithm to increase the computation-communication ratio, to improve training efﬁciency and reduce training wall clock time.
Revised: Meanwhile, to avoid the huge overhead incurred by cross-GPU/node communication in distributed training, practitioners increase the batch sizes in the optimization algorithm to increase the computation-communication ratio, to improve training efﬁciency and reduce training wall clock time.

Original: Despite increased training throughput and efﬁciency, increasing the batch size is not always the panacea, as it often leads to training instability during pre-training GPT models — training would diverge or recover from divergence but overall lead to slow/worse convergence.
Revised: Despite increased training throughput and efﬁciency, increasing the batch size is not always the panacea, as it often leads to training instability during pre-training GPT models — training would diverge or recover from divergence but overall lead to slow/worse convergence.

Original: We conduct a thorough study of the GPT-2 pre-training task (Radford et al., 2018b; Shoeybi et al., 2019) with different batch sizes, models sizes, sequence lengths, and learning rates.
Revised: We conduct a thorough study of the GPT-2 pre-training task (Radford et al., 2018b; Shoeybi et al., 2019) with different batch sizes, models sizes, sequence lengths, and learning rates.

Original: We ﬁnd that larger batch sizes and longer sequence lengths reduce training stability and increase divergence risk, especially when combined with larger model sizes and learning rate.
Revised: We ﬁnd that larger batch sizes and longer sequence lengths reduce training stability and increase divergence risk, especially when combined with larger model sizes and learning rate.

Original: This often leads to slower and worse convergence that hurts ﬁnal generalization performance.
Revised: This often leads to slower and worse convergence that hurts ﬁnal generalization performance.

Original: To achieve stable training and avoid suboptimal generalization performance, common practice often resorts to using smaller batch sizes and learning rates, which would adversely affect training efﬁciency.
Revised: To achieve stable training and avoid suboptimal generalization performance, common practice often resorts to using smaller batch sizes and learning rates, which would adversely affect training efﬁciency.

Original: In this work, we explore a curriculum learning-based approach to address the above training efﬁciencystability dilemma.
Revised: In this work, we explore a curriculum learning-based approach to address the above training efﬁciencystability dilemma.

Original: Curriculum learning (CL) was proposed to improve training convergence speed by presenting easier/simpler examples earlier during training and gradually increasing the sample difﬁculties (Elman, 1993; Sanger, 1994; Bengio et al., 2009).
Revised: Curriculum learning (CL) was proposed to improve training convergence speed by presenting easier/simpler examples earlier during training and gradually increasing the sample difﬁculties (Elman, 1993; Sanger, 1994; Bengio et al., 2009).

Original: CL was explored and veriﬁed for
Revised: CL was explored and veriﬁed for

Original: NLP  ﬁne-tuning tasks (Kocmi & Bojar, 2017; Bojar et al., 2017; Zhang et al., 2018;
Revised: NLP one-stage and ﬁne-tuning tasks (Kocmi & Bojar, 2017; Bojar et al., 2017; Zhang et al., 2018;

Original: Platanios et al., 2019; Zhang et al., 2019; Sachan & Xing, 2016; 2018; Tay et al., 2019; Xu et al., 2020), but its application to pre-training GPT models is not well studied.
Revised: Platanios et al., 2019; Zhang et al., 2019; Sachan & Xing, 2016; 2018; Tay et al., 2019; Xu et al., 2020), but its application to pre-training GPT models is not well studied.

Original: We ﬁnd that curriculum learning, as expected, could improve the convergence speed under the same training hyperparameters.
Revised: We ﬁnd that curriculum learning, as expected, could improve the convergence speed under the same training hyperparameters.

Original: More importantly, we ﬁnd that curriculum learning, as a regularization method, can enable stable and efﬁcient training with much larger batch sizes and learning rates than existing baseline approaches.
Revised: More importantly, we ﬁnd that curriculum learning, as a regularization method, can enable stable and efﬁcient training with much larger batch sizes and learning rates than existing baseline approaches.

Original: Our analysis shows that one reason for curriculum learning’s better training stability lies in its gradient variance reduction effect: Although the gradient variance is generally reduced under larger batch size, the largest variance on certain dimensions (i.e., the outliers) is increased and it reaches extreme values when training divergence happens.
Revised: Our analysis shows that one reason for curriculum learning’s better training stability lies in its gradient variance reduction effect: Although the gradient variance is generally reduced under larger batch size, the largest variance on certain dimensions (i.e., the outliers) is increased and it reaches extreme values when training divergence happens.

Original: Using Adam optimizer’s variance term, we observe that curriculum learning helps reduce both thegradient variance norm  and the maximum variance outliers.
Revised: Using Adam optimizer’s variance term, we observe that curriculum learning helps reduce both the norm of this variance and the maximum variance outliers.

Original: This variance reduction effect helps curriculum learning to achieve stable large-batch training without hurting the token-wise convergence speed and ﬁnal generalization performance.
Revised: This variance reduction effect helps curriculum learning to achieve stable large-batch training without hurting the token-wise convergence speed and ﬁnal generalization performance.

Original: We implement curriculum learning using sequence length as a primary difﬁculty metric and apply it to the GPT-2 pre-training with up to 1.5B parameters.
Revised: We implement curriculum learning using sequence length as a primary difﬁculty metric and apply it to the GPT-2 pre-training with up to 1.5B parameters.

Original: Evaluations show that curriculum learning enables stable and efﬁcient training with 8x larger batch size and 4x larger learning rate, where the baseline approach struggles with training divergence.
Revised: Evaluations show that curriculum learning enables stable and efﬁcient training with 8x larger batch size and 4x larger learning rate, where the baseline approach struggles with training divergence.

Original: To achieve the same validation perplexity targets during pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 61% and 49%, respectively.
Revised: To achieve the same validation perplexity targets during pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 61% and 49%, respectively.

Original: To achieve the same or better zero-shot WikiText-103/LAMBADA evaluation results at the end of pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 54% and 70%, respectively.
Revised: To achieve the same or better zero-shot WikiText-103/LAMBADA evaluation results at the end of pre-training, curriculum learning reduces the required number of tokens and wall clock time by up to 54% and 70%, respectively.

Original: We make the following contributions: i)
Revised: We make the following contributions: i)

Original: We conduct an extensive study of the GPT-2 pre-training task, which provides detailed insights about the training stability-efﬁciency dilemma that motivate our work (Section 3).
Revised: We conduct an extensive study of the GPT-2 pre-training task, which provides detailed insights about the training stability-efﬁciency dilemma that motivate our work (Section 3).

Original: ii)
Revised: ii)

Original: We present an implementation of curriculum learning based on sequence length for GPT-2 model (and autoregressive model in general), which is both efﬁcient and easy to integrate (Section 4).
Revised: We present an implementation of curriculum learning based on sequence length for GPT-2 model (and autoregressive model in general), which is both efﬁcient and easy to integrate (Section 4).

Original: iii)
Revised: iii)

Original: We conduct large-scale experiments to demonstrate the proposed work’s ability to provide superior training stability and efﬁciency at the same time.
Revised: We conduct large-scale experiments to demonstrate the proposed work’s ability to provide superior training stability and efﬁciency at the same time.

Original: To the best of our knowledge, we are the ﬁrst work to demonstrate the beneﬁt of curriculum learning as a regularization method that improves training stability (Section 5).
Revised: To the best of our knowledge, we are the ﬁrst work to demonstrate the beneﬁt of curriculum learning as a regularization method that improves training stability (Section 5).

Original: iv)
Revised: iv)

Original: The curriculum learning implementation as well as the necessary changes to the GPT-2 pre-training framework has been open sourced in a deep learning optimization library (name hidden to maintain anonymity).
Revised: The curriculum learning implementation as well as the necessary changes to the GPT-2 pre-training framework has been open sourced in a deep learning optimization library (name hidden to maintain anonymity).

Original:  B ACKGROUND AND R ELATED W ORKLanguage model pre
Revised:  B ACKGROUND AND R ELATED W ORK

Original: -training The accuracy of transformer-based language models grows substantially with its model size (Radford et al., 2018a;b; Brown et al., 2020).
Revised: Language model pre-training The accuracy of transformer-based language models grows substantially with its model size (Radford et al., 2018a;b; Brown et al., 2020).

Original: Today, a large language model such as GPT-3 (Brown et al., 2020) contains up to 175B parameters, and recent studies show that model accuracy can continue to improve with even larger model sizes (Kaplan et al., 2020).
Revised: Today, a large language model such as GPT-3 (Brown et al., 2020) contains up to 175B parameters, and recent studies show that model accuracy can continue to improve with even larger model sizes (Kaplan et al., 2020).

Original: However, training these large models often incurs excessively long training time and training difﬁculties (Brown et al., 2020).
Revised: However, training these large models often incurs excessively long training time and training difﬁculties (Brown et al., 2020).

Original: Therefore, there are a lot of demands of performing efﬁcient and stable training for large-scale LMs.
Revised: Therefore, there are a lot of demands of performing efﬁcient and stable training for large-scale LMs.

Original: To have the pre-training ﬁnished in a reasonable amount of time, the most common way is to leverage data parallelism to train models on multiple GPUs.
Revised: To have the pre-training ﬁnished in a reasonable amount of time, the most common way is to leverage data parallelism to train models on multiple GPUs.

Original: However, the speedup gains often saturate beyond a few tens of GPUs, because communication becomes the major bottleneck, i.e., the workers will spend more time communicating gradients than computing them, as the number of GPUs increases.
Revised: However, the speedup gains often saturate beyond a few tens of GPUs, because communication becomes the major bottleneck, i.e., the workers will spend more time communicating gradients than computing them, as the number of GPUs increases.

Original: To mitigate this bottleneck, recent works such as 1-bit Adam (Tang et al., 2021) have studied gradient compression and demonstrate their effectiveness against auto-encoding models such as BERT (Devlin et al., 2019).
Revised: To mitigate this bottleneck, recent works such as 1-bit Adam (Tang et al., 2021) have studied gradient compression and demonstrate their effectiveness against auto-encoding models such as BERT (Devlin et al., 2019).

Original: An alternative approach to alleviate these overheads is to use large batch sizes.
Revised: An alternative approach to alleviate these overheads is to use large batch sizes.

Original: For example, LAMB (You et al., 2020) and 1-bit LAMB (Li et al., 2021) enable stable and efﬁcient distributed BERT pre-training with batch size up to 64K/32K (for sequence length 128/512, i.e., 8M/16M tokens per batch) while maintaining the sample-wise convergence speed.
Revised: For example, LAMB (You et al., 2020) and 1-bit LAMB (Li et al., 2021) enable stable and efﬁcient distributed BERT pre-training with batch size up to 64K/32K (for sequence length 128/512, i.e., 8M/16M tokens per batch) while maintaining the sample-wise convergence speed.

Original: For encoder-decoder models such as T5, Raffel et al.
Revised: For encoder-decoder models such as T5, Raffel et al.

Original: (2020) use batch size up to 2K (for sequence length 512, i.e., 1M tokens per batch).
Revised: (2020) use batch size up to 2K (for sequence length 512, i.e., 1M tokens per batch).

Original: For autoregressive models such as the GPT family (Radford et al., 2018a;b; Brown et al., 2020), existing works use batch size up to 1.6K (for sequence length 2K, i.e, 3.2M tokens per batch).
Revised: For autoregressive models such as the GPT family (Radford et al., 2018a;b; Brown et al., 2020), existing works use batch size up to 1.6K (for sequence length 2K, i.e, 3.2M tokens per batch).

Original: Despite the beneﬁt of reduced communication overhead, large-batch training is sensitive to hyperparameters and often leads to issues such as slow convergence, training instabilities, and model divergence.
Revised: Despite the beneﬁt of reduced communication overhead, large-batch training is sensitive to hyperparameters and often leads to issues such as slow convergence, training instabilities, and model divergence (You et al., 2020; Li et al., 2021).

Original: Curriculum learning Inspired by the highly organized curriculum when training human and animal, curriculum learning aims to improve machine learning model training convergence speed by presenting easier (or less complex) examples earlier during training (Elman, 1993; Sanger, 1994; Bengio et al., 2009).
Revised: Curriculum learning Inspired by how humans and animals are trained, curriculum learning aims to improve machine learning model training convergence speed by presenting easier (or less complex) examples earlier during training (Elman, 1993; Sanger, 1994; Bengio et al., 2009).

Original: Previous studies have demonstrated the beneﬁt of faster convergence speed by curriculum learning in many domains such as natural language processing, computer vision, and neural evolutionary computing.
Revised: Previous studies have demonstrated the beneﬁt of faster convergence speed by curriculum learning in many domains such as natural language processing, computer vision, and neural evolutionary computing.

Original: In this section we will discuss recent related works in the natural language processing area.
Revised: In this section we will discuss recent related works in the natural language processing area.

Original: For a more complete literature review, we recommend these recent survey papers (Wang et al., 2020; Soviany et al., 2021).
Revised: For a more complete literature review, we recommend these recent survey papers (Wang et al., 2020; Soviany et al., 2021).

Original: In the natural language processing area, most of the curriculum learning works focus on  downstream ﬁne-tuning tasks, such as neural machine translation (NMT) (Kocmi & Bojar, 2017; Bojar et al., 2017; 
Revised: In the natural language processing area, most of the curriculum learning works focus on small-scale one-stage tasks and downstream ﬁne-tuning tasks, such as neural machine translation (NMT) (Kocmi & Bojar, 2017; Bojar et al., 2017; Zhang et al., 2018;

Original: Zhang et al., 2018; Platanios et al., 2019; Zhang et al., 2019) and natural language understanding (NLU) (Sachan & Xing, 2016; 2018; Tay et al., 2019;
Revised: Platanios et al., 2019; Zhang et al., 2019) and natural language understanding (NLU) (Sachan & Xing, 2016; 2018; Tay et al., 2019;

Original: Xu et al., 2020).
Revised: Xu et al., 2020).

Original: These works show that curriculum learning can improveﬁne-tuning convergence speed, reduce training time, and improve accuracy.
Revised: These works show that curriculum learning can improve convergence speed, reduce training time, and improve accuracy.

Original: In these works, the curriculum difﬁculty metrics for each training sample are usually deﬁned as the sentence length, vocabulary frequency, the inference loss on smaller/cheaper models, or based on self-paced learning (Kumar et al., 2010).
Revised: In these works, the curriculum difﬁculty metrics for each training sample are usually deﬁned as the sentence length, vocabulary frequency, the inference loss on smaller/cheaper models, or based on self-paced learning (Kumar et al., 2010).

Original: For the pacing function (i.e., to decide the curriculum difﬁculty range when sampling next training data batch), these works usually use ﬁxed predeﬁned functions (e.g., gradually increase difﬁculty upper bound by linear, root, and exponential functions), bucketing heuristics (group data with similar difﬁculties, and sample from a subset of buckets every time), or based on self-paced learning.
Revised: For the pacing function (i.e., to decide the curriculum difﬁculty range when sampling next training data batch), these works usually use ﬁxed predeﬁned functions (e.g., gradually increase difﬁculty upper bound by linear, root, and exponential functions), bucketing heuristics (group data with similar difﬁculties, and sample from a subset of buckets every time), or based on self-paced learning.

Original: Only a few works explore curriculum learning for language model pre-training.
Revised: Only a few works explore curriculum learning for language model pre-training.

Original: Press et al.
Revised: Press et al.

Original: (2020) apply curriculum learning to neural language modeling pre-training, speciﬁcally a transformer model with 247M parameters (Baevski & Auli, 2018).
Revised: (2020) apply curriculum learning to neural language modeling pre-training, speciﬁcally a transformer model with 247M parameters (Baevski & Auli, 2018).

Original: They ﬁnd that by adding an additional ﬁrst training stage with a shorter sequence length, it is possible to achieve the same dev.
Revised: They ﬁnd that by adding an additional ﬁrst training stage with a shorter sequence length, it is possible to achieve the same dev.

Original: set perplexity with shorter total training time.
Revised: set perplexity with shorter total training time.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) apply curriculum learning to BERT-base pre-training, a transformer model with 110M parameters.
Revised: (2021) apply curriculum learning to BERT-base pre-training, a transformer model with 110M parameters.

Original: They ﬁnd that by grouping sequences with similar length and with curriculum learning, it is possible to achieve similar downstream task accuracy with shorter pre-training time.
Revised: They ﬁnd that by grouping sequences with similar length and with curriculum learning, it is possible to achieve similar downstream task accuracy with shorter pre-training time.

Original: Campos (2021) apply curriculum learning to ELMo pre-training, a bi-directional
Revised: Campos (2021) apply curriculum learning to ELMo pre-training, a bi-directional

Original: LSTM model with 93.6M parameters (Peters et al., 2018).
Revised: LSTM model with 93.6M parameters (Peters et al., 2018).

Original: They test a variety of curricula on the
Revised: They test a variety of curricula on the

Original: WikiText-2 and WikiText-103 but do not ﬁnd strong evidence that the use of curriculum learning can improve language model pre-training.
Revised: WikiText-2 and WikiText-103 but do not ﬁnd strong evidence that the use of curriculum learning can improve language model pre-training.

Original:  M OTIVATION AND I NSIGHTS
Revised:  M OTIVATION AND I NSIGHTS

Original: In this section we perform an in-depth analysis of the GPT-2 model pre-training baseline (without curriculum learning).
Revised: In this section we perform an in-depth analysis of the GPT-2 model pre-training baseline (without curriculum learning).

Original: We follow the training pipeline from the NVIDIA Megatron-LM work (Shoeybi et al., 2019) 1 .
Revised: We follow the training pipeline from the NVIDIA Megatron-LM work (Shoeybi et al., 2019) 1 .

Original: All of the experiments are performed on 128 NVIDIA V100 GPUs (32GB memory).
Revised: All of the experiments are performed on 128 NVIDIA V100 GPUs (32GB memory).

Original: There are 16 nodes, and each node has 8 GPUs.
Revised: There are 16 nodes, and each node has 8 GPUs.

Original: GPUs inside the same node are connected by NVLink 2.0, and nodes are connected by a 100 Gigabit InﬁniBand EDR inter-node network.
Revised: GPUs inside the same node are connected by NVLink 2.0, and nodes are connected by a 100 Gigabit InﬁniBand EDR inter-node network.

Original: Model and dataset We evaluate two GPT-2 model sizes from the original GPT-2 work (Radford et al., 2018b): one with 117M parameters (12 layers, 768 hidden size, 12 attention heads), and  one with 1.5B parameters (48 layers, 1600 hidden size, 25 attention heads).
Revised: Model and dataset We evaluate two GPT-2 model sizes from the original GPT-2 work (Radford et al., 2018b): one with 117M parameters (12 layers, 768 hidden size, 12 attention heads), and [Figure] one with 1.5B parameters (48 layers, 1600 hidden size, 25 attention heads).

Original: For training data, we collect and use the same dataset blend as the Megatron-LM work: Wikipedia (Devlin et al., 2019),[Figure]
Revised: For training data, we collect and use the same dataset blend as the Megatron-LM work: Wikipedia (Devlin et al., 2019),

Original: CC-Stories (Trinh & Le, 2018), RealNews (Zellers et al., 2019), and OpenWebtext (Radford et al., 2019).
Revised: CC-Stories (Trinh & Le, 2018), RealNews (Zellers et al., 2019), and OpenWebtext (Radford et al., 2019).

Original: Training parameters We evaluate two sets of training parameters.
Revised: Training parameters We evaluate two sets of training parameters.

Original: The ﬁrst set follows the
Revised: The ﬁrst set follows the

Original: Megatron-LM work: batch size 512, 300K total training steps (157B tokens), and learning rate 1 .
Revised: Megatron-LM work: batch size 512, 300K total training steps (157B tokens), and learning rate 1 .

Original: 5 × 10 − 4 with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 297K steps (with minimum learning rate 1 × 10 − 5 ).
Revised: 5 × 10 − 4 with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 297K steps (with minimum learning rate 1 × 10 − 5 ).

Original: The second parameter set tests a more aggressive training strategy: batch size 4K ( 8 × larger), 37.5K total training steps (157B tokens), and learning rate 6 × 10 − 4 ( 4 × larger) with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 34.5K steps (same minimum learning rate).
Revised: The second parameter set tests a more aggressive training strategy: batch size 4K ( 8 × larger), 37.5K total training steps (157B tokens), and learning rate 6 × 10 − 4 ( 4 × larger) with a linear warmup of 3K steps and a single cycle cosine decay over the remaining 34.5K steps (same minimum learning rate).

Original: For sequence length/context size, we mainly use 1K which is the default for GPT-2.
Revised: For sequence length/context size, we mainly use 1K which is the default for GPT-2.

Original: But we also test 2K (on the smaller 117M model with batch size 512 and 157B tokens) which is the default for GPT-3 (Brown et al., 2020).
Revised: But we also test 2K (on the smaller 117M model with batch size 512 and 157B tokens) which is the default for GPT-3 (Brown et al., 2020).

Original: All experiments are performed with mixed precision/
Revised: All experiments are performed with mixed precision/

Original: FP16 training, Adam optimizer ( β 1 = 0 .
Revised: FP16 training, Adam optimizer ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) = 1 × 10 − 8 ) (Kingma & Ba, 2015), weight decay of 0.01, checkpoint activation, same random seed (for Python, NumPy, PyTorch, and CUDA), and gradient clipping.
Revised: 999 , (cid:15) = 1 × 10 − 8 ) (Kingma & Ba, 2015), weight decay of 0.01, checkpoint activation, same random seed (for Python, NumPy, PyTorch, and CUDA), and gradient clipping.

Original: The stability-efﬁciency dilemma Figure 1 presents the training loss and validation perplexity for 5 cases (baseline without curriculum learning) under different model sizes, batch sizes (and LR), and sequence lengths.
Revised: The stability-efﬁciency dilemma Figure 1 presents the training loss and validation perplexity for 5 cases (baseline without curriculum learning) under different model sizes, batch sizes (and LR), and sequence lengths.

Original: First, the 1.5B model training is much more unstable and has many divergences.
Revised: First, the 1.5B model training is much more unstable and has many divergences.

Original: This is even worse when applied with a larger batch size/learning rate, where the ﬁnal validation perplexity becomes worse as shown in Figure 1(c).
Revised: This is even worse when applied with a larger batch size/learning rate, where the ﬁnal validation perplexity becomes worse as shown in Figure 1(c).

Original: Similarly for the 117M model, larger batch size/LR or longer 2K sequence length both lead to worse ﬁnal validation perplexity (and worse zero-shot WikiText-103/LAMBADA evaluation results as we will later see in Table 2 Section 5).
Revised: Similarly for the 117M model, larger batch size/LR or longer 2K sequence length both lead to worse ﬁnal validation perplexity (and worse zero-shot WikiText-103/LAMBADA evaluation results as we will later see in Table 2 Section 5).

Original: From the token-wise convergence, it seems that a smaller batch size/learning rate is preferable because of its better generalization performance.
Revised: From the token-wise convergence, it seems that a smaller batch size/learning rate is preferable because of its better generalization performance.

Original: However, Figure 1(b) and 1(d) show that such cases would lead to much worse training efﬁciency.
Revised: However, Figure 1(b) and 1(d) show that such cases would lead to much worse training efﬁciency.

Original: With a larger batch size/learning rate, it is possible to complete the same 157B-token training with much less wall clock time.
Revised: With a larger batch size/learning rate, it is possible to complete the same 157B-token training with much less wall clock time.

Original: For the 1.5B model, it takes 341 and 151 hours to complete the training under batch size 512 and 4K, respectively.
Revised: For the 1.5B model, it takes 341 and 151 hours to complete the training under batch size 512 and 4K, respectively.

Original: Overall, this demonstrates the stability-efﬁciency dilemma for baseline pre-training: With smaller batch size/learning rates, the training is more stable and can achieve better ﬁnal generalization, but presumably with poorer training efﬁciency; With larger batch size/learning rates, the training is more efﬁcient, but with lower stability and worse generalization.
Revised: Overall, this demonstrates the stability-efﬁciency dilemma for baseline pre-training: With smaller batch size/learning rates, the training is more stable and can achieve better ﬁnal generalization, but presumably with poorer training efﬁciency; With larger batch size/learning rates, the training is more efﬁcient, but with lower stability and worse generalization.

Original: As a solution, in the next section we will describe how we design and implement curriculum learning, which resolves this dilemma for the
Revised: As a solution, in the next section we will describe how we design and implement curriculum learning, which resolves this dilemma for the

Original: GPT-2 model pre-training task.
Revised: GPT-2 model pre-training task.

Original:  D ESIGN AND I MPLEMENTATION
Revised:  D ESIGN AND I MPLEMENTATION

Original: Difﬁculty metric: sequence length After considering various metrics used in related works, we choose the sequence length as the curriculum difﬁculty metric (start from shorter sequence training data, then gradually increase the sequence length) for two reasons: i) Sequence length as the curriculum difﬁculty metric has been proven to be effective in the NLP area; ii)
Revised: Difﬁculty metric: sequence length After considering various metrics used in related works, we choose the sequence length as the curriculum difﬁculty metric (start from shorter sequence training data, then gradually increase the sequence length) for two reasons: i) Sequence length as the curriculum difﬁculty metric has been proven to be effective in the NLP area; ii)

Original: For each Transformer block, the self-attention and intermediate layers have time complexity of O ( B × L 2 × H ) and
Revised: For each Transformer block, the self-attention and intermediate layers have time complexity of O ( B × L 2 × H ) and

Original: O ( B × L × H 2 ) , respectively 2 .
Revised: O ( B × L × H 2 ) , respectively 2 .

Original: By reducing the sequence length, we reduce the time complexity quadratically for the self-attention sub-layer and linearly for the intermediate sub-layer of Transformer blocks.
Revised: By reducing the sequence length, we reduce the time complexity quadratically for the self-attention sub-layer and linearly for the intermediate sub-layer of Transformer blocks.

Original: To support curriculum learning, traditional implementation would require "training data reordering" to sample from data subsets that are below certain difﬁculty levels.
Revised: To support curriculum learning, traditional implementation would require "training data reordering" to sample from data subsets that are below certain difﬁculty levels.

Original: This adds additional cost before training (to compute the difﬁculty of data) and during training (data subset sampling).
Revised: This adds additional cost before training (to compute the difﬁculty of data) and during training (data subset sampling).

Original: For sequence length metric, we develop a truncation-based implementation that incurs less cost: During the baseline GPT-2 pre-training, the raw text inputs are truncated into sequences with the same length to form a mini-batch regardless of the lengths of the actual sentences.
Revised: For sequence length metric, we develop a truncation-based implementation that incurs less cost: During the baseline GPT-2 pre-training, the raw text inputs are truncated into sequences with the same length to form a mini-batch regardless of the lengths of the actual sentences.

Original: To enable curriculum learning, we still let the dataloader truncate the raw text into samples with the same full sequence length.
Revised: To enable curriculum learning, we still let the dataloader truncate the raw text into samples with the same full sequence length.

Original: Then at each step given the sequence length determined by the pacing function, we truncate sequences to form a modiﬁed mini-batch for training.
Revised: Then at each step given the sequence length determined by the pacing function, we truncate sequences to form a modiﬁed mini-batch for training.

Original: Pacing function: step-wise linear We choose a step-wise linear pacing function with the following deﬁnition:
Revised: Pacing function: step-wise linear We choose a step-wise linear pacing function with the following deﬁnition:

Original: Given a starting sequence length seqlen 1 , an ending sequence length seqlen 2 (baseline full sequence length), and a curriculum learning duration T (number of steps), the sequence length used for the training batch at step t is seqlen t = seqlen 1 + ( seqlen 2 − seqlen 1 ) × min ( tT , 1) .
Revised: Given a starting sequence length seqlen 1 , an ending sequence length seqlen 2 (baseline full sequence length), and a curriculum learning duration T (number of steps), the sequence length used for the training batch at step t is seqlen t = seqlen 1 + ( seqlen 2 − seqlen 1 ) × min ( tT , 1) .

Original: We choose this pacing function for two reasons: First, such ﬁxed heuristic pacing functions are easy to implement and proved effective in the NLP area.
Revised: We choose this pacing function for two reasons: First, such ﬁxed heuristic pacing functions are easy to implement and proved effective in the NLP area.

Original: Second, step-wise linear function conveys two different functions that are intuitively convincing: i) step-wise linear means that we will perform optimization and update the model for the same number of times for each distinct sequence length; ii) because different sequence lengths lead to different number of tokens in a batch, at longer sequence lengths (higher difﬁculty) we will learn from more tokens.
Revised: Second, step-wise linear function conveys two different functions that are intuitively convincing: i) step-wise linear means that we will perform optimization and update the model for the same number of times for each distinct sequence length; ii) because different sequence lengths lead to different number of tokens in a batch, at longer sequence lengths (higher difﬁculty) we will learn from more tokens.

Original: Besides step-wise linear, we also explored 3 other pacing functions: i)
Revised: Besides step-wise linear, we also explored 3 other pacing functions: i)

Original: We tried a discrete 2-stage pacing function from (Press et al., 2020), but it provides less stability beneﬁt (Section 5.4).
Revised: We tried a discrete 2-stage pacing function from (Press et al., 2020), but it provides less stability beneﬁt (Section 5.4).

Original: ii)
Revised: ii)

Original: We tried a step-wise root function ( seqlen t = seqlen 1 + ( seqlen 2 − seqlen 1 ) × min (( tT ) r , 1) , where r is the root degree), which performs similar to linear and requires one extra hyperparameter.
Revised: We tried a step-wise root function ( seqlen t = seqlen 1 + ( seqlen 2 − seqlen 1 ) × min (( tT ) r , 1) , where r is the root degree), which performs similar to linear and requires one extra hyperparameter.

Original: iii)
Revised: iii)

Original: We tried an adaptive pacing function based on the progress of training/validation loss, which also performs similar and requires extra tuning.
Revised: We tried an adaptive pacing function based on the progress of training/validation loss, which also performs similar and requires extra tuning.

Original:  E VALUATION
Revised:  E VALUATION

Original: In this section we evaluate curriculum learning for GPT-2 pre-training.
Revised: In this section we evaluate curriculum learning for GPT-2 pre-training.

Original: Our analysis shows that curriculum learning can provide stable training at batch size 4K where baseline struggles with divergence issue.
Revised: Our analysis shows that curriculum learning can provide stable training at batch size 4K where baseline struggles with divergence issue.

Original: In addition, curriculum learning greatly reduces the required number of tokens and training time to reach the same validation perplexity and zero-shot evaluation results.
Revised: In addition, curriculum learning greatly reduces the required number of tokens and training time to reach the same validation perplexity and zero-shot evaluation results.

Original: Methodology For model, dataset, and hardware, we follow the same methodology in Section 3.
Revised: Methodology For model, dataset, and hardware, we follow the same methodology in Section 3.

Original: For the curriculum learning conﬁgurations (deﬁned in Section 4), we use seqlen 1 = 8 / 64 (for 117M/1.5B model) and seqlen 2 = 1 K (i.e., the full sequence length).
Revised: For the curriculum learning conﬁgurations (deﬁned in Section 4), we use seqlen 1 = 8 / 64 (for 117M/1.5B model) and seqlen 2 = 1 K (i.e., the full sequence length).

Original: To fully utilize the Tensor Core acceleration in NVIDIA V100 GPU, we add a seqlen t = seqlen t − ( seqlen t mod 8) postprocessing to make sure the sequence length is always a multiple of 8.
Revised: To fully utilize the Tensor Core acceleration in NVIDIA V100 GPU, we add a seqlen t = seqlen t − ( seqlen t mod 8) postprocessing to make sure the sequence length is always a multiple of 8.

Original: For the curriculum duration T , we tune this parameter and use different numbers for each case.
Revised: For the curriculum duration T , we tune this parameter and use different numbers for each case.

Original: We analyze and ﬁnd a low-cost tuning strategy that only requires running the ﬁrst tens of steps (details in Appendix A.1).
Revised: We analyze and ﬁnd a low-cost tuning strategy that only requires running the ﬁrst tens of steps (details in Appendix A.1).

Original: For the training parameters, for curriculum learning we use the same shared parameters as the baseline in Section 3 except two parameters: 1) Because during curriculum learning the number of tokens in a data batch is smaller, we modify the training termination condition so that all cases stop when reaching the same 157B training tokens.
Revised: For the training parameters, for curriculum learning we use the same shared parameters as the baseline in Section 3 except two parameters: 1) Because during curriculum learning the number of tokens in a data batch is smaller, we modify the training termination condition so that all cases stop when reaching the same 157B training tokens.

Original: Because of 1) curriculum learning cases now have more training steps, which make it necessary to modify the learning rate decay schedule to have a fair comparison with the baseline.
Revised: Because of 1), curriculum learning cases now have more training steps, which make it necessary to modify the learning rate decay schedule to have a fair comparison with the baseline.

Original: For 117M model cases, we increase the number of learning rate decay steps by half of the curriculum learning duration.
Revised: For 117M model cases, we increase the number of learning rate decay steps by half of the curriculum learning duration.

Original: However, we ﬁnd that this is still not ideal.
Revised: However, we ﬁnd that simply increasing decay steps still leads to sub-optimal learning rate schedule.

Original: So for 1.5B model cases, we change the learning rate decay to token-wise (same cosine decay over the 157B tokens) instead of step-wise.
Revised: So for 1.5B model cases, we change the learning rate decay to token-wise (same cosine decay over the 157B tokens) instead of step-wise.

Original: We describe the underlying rationale in Appendix A.2.
Revised: We describe the underlying rationale in Appendix A.2.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5. GPT-2 117M EVALUATIONS
Revised: 5. GPT-2 117M EVALUATIONS

Original: Convergence speed.
Revised: Convergence speed.

Original: Figure 2 demonstrates that curriculum learning provides better token and time-wise convergence compared to the baseline under the same batch size/LR and sequence length. 
Revised: Figure 2 demonstrates that curriculum learning provides better token and time-wise convergence compared to the baseline under the same batch size/LR and sequence length. In addition [Table]

Original: In addition, “Bsz 4K, Seqlen 1K, CL 20K” provides better token and time-wise convergence than “Bsz 512, Seqlen 1K, Baseline”
Revised: “Bsz 4K, Seqlen 1K, CL 20K” provides better token and time-wise convergence than “Bsz 512, Seqlen 1K, Baseline”

Original: [Table]
Revised: [Table]

Original: which demonstrates that curriculum learning provides stable and efﬁcient large-batch training while maintaining the convergence speed and ﬁnal generalization performance.
Revised: which demonstrates that curriculum learning provides stable and efﬁcient large-batch training while maintaining the convergence speed and ﬁnal generalization performance.

Original: As a result, curriculum learning combined with large-batch training provides a solution to the training stability-efﬁciency dilemma.
Revised: As a result, curriculum learning combined with large-batch training provides a solution to the training stability-efﬁciency dilemma.

Original: Tokens/time to reach the same validation perplexity.
Revised: Tokens/time to reach the same validation perplexity.

Original: To quantitatively measure the token and time-wise convergence speedup, Table 1 summarizes that for baseline and curriculum learning to reach the same validation perplexity during pre-training, what is the required number of tokens and training time.
Revised: To quantitatively measure the token and time-wise convergence speedup, Table 1 summarizes the required number of tokens and training time for baseline and curriculum learning to reach the same validation perplexity during pre-training.

Original: In this table, we compare each pair of baseline and curriculum learning cases under the same batch size and sequence length.
Revised: In this table, we compare each pair of baseline and curriculum learning cases under the same batch size and sequence length.

Original: Results show that curriculum learning reduces the required number of tokens and training time by up to 61% and 49%, respectively.
Revised: Results show that curriculum learning reduces the required number of tokens and training time by up to 61% and 49%, respectively.

Original: In addition, curriculum learning achieves validation perplexity that baseline is unable to achieve during the whole training.
Revised: In addition, curriculum learning achieves validation perplexity that baseline is unable to achieve during the whole training.

Original: Zero-shot evaluation.
Revised: Zero-shot evaluation.

Original: Last, we perform zero-shot evaluation of the trained models on the WikiText103 and LAMBADA datasets using the same methodology in the Megatron-LM work (Shoeybi et al., 2019).
Revised: Last, we perform zero-shot evaluation of the trained models on the WikiText103 and LAMBADA datasets using the same methodology in the Megatron-LM work (Shoeybi et al., 2019).

Original: In Table 2 we compare all cases to the baseline case with batch size 512 and sequence length 1K, the original conﬁguration in Megatron-LM work.
Revised: In Table 2 we compare all cases to the baseline case with batch size 512 and sequence length 1K, the original conﬁguration in Megatron-LM work.

Original: For each curriculum learning (CL) case we present two rows: one evaluated at the earliest checkpoint that provides better eval results; the other one evaluated at the end of full training.
Revised: For each curriculum learning (CL) case we present two rows: one evaluated at the earliest checkpoint that provides better eval results; the other one evaluated at the end of full training.

Original: At batch size 512, CL provides up to 40%/32% token/time  reduction while achieving better evaluation results (case 1, 2).
Revised: At batch size 512, CL provides up to 40%/32% token/time [Table] reduction while achieving better evaluation results (case 1, 2).

Original: After full training, CL takes 8% more time because it needs more steps to reach the same 157B tokens, but more importantly CL provides
Revised: After full training, CL takes 8% more time because it needs more steps to reach the same 157B tokens

Original: [Table]
Revised: [Table]

Original:  even better eval results (case 3).
Revised: but more importantly CL provides even better eval results (case 3).

Original: When increasing the batch size or sequence length for baseline [Table] the training time is reduced but the evaluation results get worse due to less stable training and worse convergence (case 4, 7).
Revised: When increasing the batch size or sequence length for baseline, the training time is reduced but the evaluation results get worse due to less stable training and worse convergence (case 4, 7).

Original: On the other hand [Table] CL with a larger batch size/sequence length can provide both great token/time saving (up to 54%/70%) and better eval results (case 5, 6, 8, 9) [Table]
Revised: On the other hand, CL with a larger batch size/sequence length can provide both great token/time saving (up to 54%/70%) and better eval results (case 5, 6, 8, 9).

Original: 5. GPT-2 1.5B EVALUATIONS
Revised: 5. GPT-2 1.5B EVALUATIONS

Original: Convergence speed.
Revised: Convergence speed.

Original: Figure 3 demonstrates that curriculum learning provides better token-wise convergence compared to the baseline under the same batch size.
Revised: Figure 3 demonstrates that curriculum learning provides better token-wise convergence compared to the baseline under the same batch size.

Original: In addition, “Bsz 4K, CL 15K” provides better token and time-wise convergence than “Bsz 512, Baseline”, which demonstrates that curriculum learning provides stable and efﬁcient large-batch training while maintaining the convergence speed.
Revised: In addition, “Bsz 4K, CL 15K” provides better token and time-wise convergence than “Bsz 512, Baseline”, which demonstrates that curriculum learning provides stable and efﬁcient large-batch training while maintaining the convergence speed.

Original: For the batch size 512 case, curriculum learning provides smaller time-wise speedup.
Revised: For the batch size 512 case, curriculum learning provides smaller time-wise speedup.

Original: This is because in this case, the communication overhead is too high so that the time saving provided by curriculum learning’s shorter sequences becomes relatively smaller.
Revised: This is because in this case, the communication overhead is too high so that the time saving provided by curriculum learning’s shorter sequences becomes relatively smaller.

Original: Another reason is that the baseline is more stable at batch size 512.
Revised: Another reason is that the baseline is more stable at batch size 512.

Original: Tokens/time to reach the same validation perplexity.
Revised: Tokens/time to reach the same validation perplexity.

Original: Table 3 summarizes the required number of tokens and training time for baseline and curriculum learning to reach the same validation perplexity.
Revised: Table 3 summarizes the required number of tokens and training time for baseline and curriculum learning to reach the same validation perplexity.

Original: Results show that curriculum learning reduces the required number of tokens and training time by up to 45% and 41%, respectively.
Revised: Results show that curriculum learning reduces the required number of tokens and training time by up to 45% and 41%, respectively.

Original: Compared to baseline with batch size 512, curriculum learning with batch size 4K can achieve better ﬁnal validation perplexity together with much better training efﬁciency, demonstrating the dual stability-efﬁciency capability.
Revised: Compared to baseline with batch size 512, curriculum learning with batch size 4K can achieve better ﬁnal validation perplexity together with much better training efﬁciency, demonstrating the dual stability-efﬁciency capability.

Original: Zero-shot evaluation.
Revised: Zero-shot evaluation.

Original: At batch size 512, curriculum learning can achieve better eval results with 22%/16% less token/time and achieve even better eval results after full training (case 10, 11, 12 in
Revised: At batch size 512, curriculum learning can achieve better eval results with 22%/16% less token/time and achieve even better eval results after full training (case 10, 11, 12 in

Original: Table 2).
Revised: Table 2).

Original: Baseline with a larger batch size 4K has better efﬁciency but worse stability/eval results (case 13), while CL with a larger batch size provides both efﬁciency and stability (case 14, 15).
Revised: Baseline with a larger batch size 4K has better efﬁciency but worse stability/eval results (case 13), while CL with a larger batch size provides both efﬁciency and stability (case 14, 15).

Original: 5. V ARIANCE REDUCTION HELPS STABILIZE TRAINING .
Revised: 5. V ARIANCE REDUCTION HELPS STABILIZE TRAINING .

Original: For stochastic gradient optimization, when the gradient variance is large, the algorithm might spend much time bouncing around, leading to slower convergence and potential divergence (Wang et al.,2013).
Revised: For stochastic gradient optimization, when the gradient variance is large, the algorithm might spend much time bouncing around, leading to slower convergence and potential divergence (Wang et al.,

Original: Previous studies show that variance reduction methods improve training stability in areas such as reinforcement learning (Mao et al., 2018; Cheng et al., 2019; Anschel et al., 2017).

Original: During our

Original: [Figure]
Revised: [Figure]

Original: 2020) and “Bsz
Revised: 2020) and “Bsz

Original: Warmup” (Brown et al., 2020)) described in Section 5.4.
Revised: Warmup” (Brown et al., 2020)) described in Section 5.4.

Original: These two related works are not presented in 3(b) and 3(d) because they are performed on different hardware.
Revised: These two related works are not presented in 3(b) and 3(d) because they are performed on different hardware.

Original: All subﬁgures share the same legend as 3(f) (“CL 45K” means applying curriculum learning to the ﬁrst 45K steps).
Revised: All subﬁgures share the same legend as 3(f) (“CL 45K” means applying curriculum learning to the ﬁrst T =45K steps).

Original: [Table]
Revised: [Table]

Original: study of curriculum learning [Table]
Revised: Previous studies show that variance reduction methods improve training stability in areas such as reinforcement learning (Mao et al., 2018; Cheng et al., 2019; Anschel et al., 2017).

Original:  one surprising ﬁnding is that curriculum learning can, in addition to the well-known beneﬁt of faster convergence speed, help to reduce this gradient variance norm/max element, which is presumably why it leads to much more stable training under large model size and large batch size/LR.
Revised: During our study of curriculum learning, one surprising ﬁnding is that curriculum learning can, in addition to the well-known beneﬁt of faster convergence speed, help to reduce this gradient variance norm/max element, which is presumably why it leads to much more stable training under large model size and large batch size/LR.

Original: Figure 3(e) and 3(f) plot the l 1 norm 3 and max element of Adam’s variance term ( √ v t , where v t
Revised: Figure 3(e) and 3(f) plot the l 1 norm 3 and max element of Adam’s variance term ( √ v t , where v t

Original: =
Revised: =

Original: β 2 v t − 1 + (1 − β 2 )( g t ) 2 ) for the GPT-2 1.5B case.
Revised: β 2 v t − 1 + (1 − β 2 )( g t ) 2 ) for the GPT-2 1.5B case.

Original: We plot them step-wise since the optimization and model update are performed at every step.
Revised: We plot them step-wise since the optimization and model update are performed at every step.

Original: We see that when batch size/LR increases, the variance norm decreases but the max element increases.
Revised: We see that when batch size/LR increases, the variance norm decreases but the max element increases.

Original: When comparing GPT-2 117M and 1.5B cases (not shown in ﬁgure), we see that when model size increases both the variance norm and max element increase.
Revised: When comparing GPT-2 117M and 1.5B cases (not shown in ﬁgure), we see that when model size increases both the variance norm and max element increase.

Original: When sequence length increases for the GPT-2 117M case, the variance norm stays the same but the max element increases.
Revised: When sequence length increases for the GPT-2 117M case, the variance norm stays the same but the max element increases.

Original: These demonstrate that large gradient variance norm/max element is one of the symptoms of the training instability issue, similar to what was observed in the aforementioned studies in reinforcement learning area.
Revised: These demonstrate that large gradient variance norm/max element is one of the symptoms of the training instability issue, similar to what was observed in the aforementioned studies in reinforcement learning area.

Original: We ﬁnd that curriculum learning stabilizes training and reduces both the Adam variance norm and the variance max element.
Revised: We ﬁnd that curriculum learning stabilizes training and reduces both the Adam variance norm and the variance max element.

Original: Importantly, curriculum learning avoids all the spikes of the variance max element, which all happen to be where the baseline training diverges.
Revised: Importantly, curriculum learning avoids all the spikes of the variance max element, which all happen to be where the baseline training diverges.

Original: This is one of the reasons why curriculum learning can provide substantial token/time-wise speedup for large-batch training.
Revised: This is one of the reasons why curriculum learning can provide substantial token/time-wise speedup for large-batch training.

Original: One may wonder why gradient clipping cannot help avoid these extreme gradient variance outliers.
Revised: One may wonder why gradient clipping cannot help avoid these extreme gradient variance outliers.

Original: Although gradient clipping can avoid too large/small gradient at every single step, it cannot avoid the gradient variance getting accumulated at certain dimensions, especially for large batch sizes.
Revised: Although gradient clipping can avoid too large/small gradient at every single step, it cannot avoid the gradient variance getting accumulated at certain dimensions, especially for large batch sizes.

Original: 5. C OMPARING WITH RELATED WORKS
Revised: 5. C OMPARING WITH RELATED WORKS

Original: In the end, we compare the proposed work with two related works on the “1.5B model with batch size 4K” case which is the most challenging one.
Revised: In the end, we compare the proposed work with two related works on the “1.5B model with batch size 4K” case which is the most challenging one.

Original: The ﬁrst work is a “2-stage curriculum learning” where the ﬁrst stage uses a shorter sequence length and the second stage uses the full sequence length (Press et al., 2020).
Revised: The ﬁrst work is a “2-stage curriculum learning” where the ﬁrst stage uses a shorter sequence length and the second stage uses the full sequence length (Press et al., 2020).

Original: Following the tuning strategy in the related work, we set the sequence length as 128 for the ﬁrst stage and set its duration at about half of the baseline duration (20K steps).
Revised: Following the tuning strategy in the related work, we set the sequence length as 128 for the ﬁrst stage and set its duration at about half of the baseline duration (20K steps).

Original: The second work is the “batch size warmup” technique used by GPT-3 pre-training where the training starts with a smaller batch size (sequence length unchanged) and gradually increases to full batch size (Brown et al., 2020).
Revised: The second work is the “batch size warmup” technique used by GPT-3 pre-training where the training starts with a smaller batch size (sequence length unchanged) and gradually increases to full batch size (Brown et al., 2020).

Original: In our experiment, we set the starting batch size at 128 and then gradually increase it to 4K, and set the warmup duration at 45K steps (same as the proposed work).
Revised: In our experiment, we set the starting batch size at 128 and then gradually increase it to 4K, and set the warmup duration at 45K steps (same as the proposed work).

Original: Both related works use the same training parameters as the proposed work.
Revised: Both related works use the same training parameters as the proposed work.

Original: Because we perform the two experiments on a different cluster, we do not report the training time as it is not comparable.
Revised: Because we perform the two experiments on a different cluster, we do not report the training time as it is not comparable.

Original: Figure 3 compares the training loss, validation perplexity, and Adam variance norm/max element.
Revised: Figure 3 compares the training loss, validation perplexity, and Adam variance norm/max element.

Original: Both related works provide convergence speedup but it is less than the proposed work.
Revised: Both related works provide convergence speedup but it is less than the proposed work.

Original: More importantly, both related works still have training instability issues.
Revised: More importantly, both related works still have training instability issues.

Original: The two-stage curriculum learning has an obvious training divergence at step 20K when the sequence length switches fromto 1K (the spike at 20K in Figure 3(f)).
Revised: The two-stage curriculum learning has an obvious training divergence at step 20K when the sequence length switches fromto 1K (the spike at 20K in Figure 3(f)).

Original: This is because when staying at the same shorter sequence length for too long, the model becomes heavily overﬁtted for that length which leads to divergence risk when switching to full length.
Revised: This is because when staying at the same shorter sequence length for too long, the model becomes heavily overﬁtted for that length which leads to divergence risk when switching to full length.

Original: The batch size warmupcase does not provide any training stability beneﬁt compared to the baseline even though it has a similar “token reduction” effect as curriculum learning.
Revised: Although both batch size warmup and curriculum learning reduce the number of tokens per batch in a similar fashion, batch size warmup does not provide any training stability beneﬁt compared to the baseline.

Original: This indicates that providing the same number of shorter (less complex) sequences leads to better training stability than providing fewer number of same length (same complexity) sequences.
Revised: This indicates that providing the same number of shorter (less complex) sequences leads to better training stability than providing fewer number of same length (same complexity) sequences.

Original: In addition, batch size warmup has a limitation that the batch size must be multiple of data-parallel size, which will be large for distributed training.
Revised: In addition, batch size warmup has a limitation that the batch size must be multiple of data-parallel size, which will be large for distributed training.

Original: On the other hand, for curriculum learning the sequence length only needs to be multiple of 8 to enable Tensor Core acceleration.
Revised: On the other hand, for curriculum learning the sequence length only needs to be multiple of 8 to enable Tensor Core acceleration.

Original: Both related works provide worse zero-shot evaluation results (case 16, 17 in Table 2).
Revised: Both related works provide worse zero-shot evaluation results (case 16, 17 in Table 2).

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: It is well known that curriculum learning, by presenting easier or simpler training data ﬁrst, could provide faster convergence speed under the same training parameters.
Revised: It is well known that curriculum learning, by presenting easier or simpler training data ﬁrst, could provide faster convergence speed under the same training parameters.

Original: On the other hand, our study reveals another important beneﬁt of curriculum learning as a regularization method: enable stable training at larger batch size and learning rate by providing a gradient variance reduction effect.
Revised: On the other hand, our study reveals another important beneﬁt of curriculum learning as a regularization method: enable stable training at larger batch size and learning rate by providing a gradient variance reduction effect.

Original: Our evaluations on GPT-2 pre-training with up to 1.5B parameters demonstrate that by enabling stable and efﬁcient training with 8x batch size and 4x learning rate, curriculum learning can reduce up to 54%/70% training token/time while still achieving better zero-shot evaluation results.
Revised: Our evaluations on GPT-2 pre-training with up to 1.5B parameters demonstrate that by enabling stable and efﬁcient training with 8x batch size and 4x learning rate, curriculum learning can reduce up to 54%/70% training token/time while still achieving better zero-shot evaluation results.

Original: We hope that our work could motivate more studies of curriculum learning as a regularization method and its applications in other cases such as autoencoding language models and even computer vision models.
Revised: We hope that our work could motivate more studies of curriculum learning as a regularization method and its applications in other cases such as autoencoding language models and even computer vision models.

Original: We believe with a proper curriculum learning design, it is possible to observe similar training stability/efﬁciency beneﬁts by curriculum learning in other areas.
Revised: We believe with a proper curriculum learning design, it is possible to observe similar training stability/efﬁciency beneﬁts by curriculum learning in other areas.

Original: We also hope that our work could motivate more studies of training data efﬁciency in general: as training data keep accumulating, sooner or later one will not be able to ﬁnish even one epoch for many training tasks.
Revised: We also hope that our work could motivate more studies of training data efﬁciency in general: as training data keep accumulating, sooner or later one will not be able to ﬁnish even one epoch for many training tasks.

Original: In this situation, intelligent data pipeline techniques (such as but not limited to curriculum learning) would become increasingly critical to the training stability/efﬁciency.
Revised: In this situation, intelligent data pipeline techniques (such as but not limited to curriculum learning) would become increasingly critical to the training stability/efﬁciency.

Original: A A PPENDIX A. C URRICULUM LEARNING HYPERPARAMETERS TUNING STRATEGY
Revised: A A PPENDIX A. C URRICULUM LEARNING HYPERPARAMETERS TUNING STRATEGY

Original: To ﬁnd a good tuning strategy for curriculum learning
Revised: To ﬁnd a good tuning strategy for curriculum learning

Original: [Table]
Revised: [Table]

Original: we ﬁrst perform a simple grid search on the smaller 117M model.
Revised: we ﬁrst perform a simple grid search on the smaller 117M model.

Original: Table 4 summarizes the grid search results for the curriculum learning duration ( T in Section 4) where we set the starting sequence length ( seqlen 1 in Section 4) ﬁxed at 8.
Revised: Table 4 summarizes the grid search results for the curriculum learning duration ( T in Section 4) where we set the starting sequence length ( seqlen 1 in Section 4) ﬁxed at 8.

Original: We then choose the “best” duration based on the test data perplexity and zero-shot evaluation results at the end of 157B-token pre-training.
Revised: We then choose the “best” duration based on the test data perplexity and zero-shot evaluation results at the end of 157B-token pre-training.

Original: One thing to note is that all the CL cases have quite comparable zero-shot evaluation results [Table] indicating that CL’s performance is not very sensitive to the CL duration within a reasonable range.
Revised: One thing to note is that all the CL cases have quite comparable zero-shot evaluation results, indicating that CL’s performance is not very sensitive to the CL duration within a reasonable range.

Original: On the other hand
Revised: On the other hand

Original: [Table]
Revised: [Table]

Original: the above grid search also sheds light on a better tuning strategy: for all three batch size-sequence length combinations, we ﬁnd that the “best” curriculum duration is always the longest duration that does not have signiﬁcant validation perplexity ﬂuctuation during the ﬁrst 10K steps (i.e., a few multiples of the LR warmup steps), as shown in Figure 4.
Revised: the above grid search also sheds light on a better tuning strategy: for all three batch size-sequence length combinations, we ﬁnd that the “best” curriculum duration is always the longest duration that does not have signiﬁcant validation perplexity ﬂuctuation during the ﬁrst 10K steps (i.e., a few multiples of the LR warmup steps). In Figure 4(a) the CL 60K is the longest duration we tested that does not have signiﬁcant validation ﬂuctuation during the ﬁrst 10K steps, and in Figure 4(b) and Table 4 CL 60K does provide the best validation perplexity during second half of training, best test perplexity at the end, and best LAMBADA accuracy.

Original: Since this does not require training the model until full convergence, this heuristic reduces the hyperparameter tuning cost of the CL-based approach.
Revised: Since this does not require training the model until full convergence, this heuristic reduces the hyperparameter tuning cost of the CL-based approach.

Original: For the 1.5B model, we used this strategy to select the curriculum learning duration based on the validation perplexity curve at the beginning.
Revised: For the 1.5B model, we used this strategy to select the curriculum learning duration based on the validation perplexity curve at the beginning.

Original: For the starting sequence length seqlen 1 , our grid search results show that it’s generally better to set it as small as possible, to maximize the stability and convergence speedup beneﬁt.
Revised: For the starting sequence length seqlen 1 , our grid search results show that it’s generally better to set it as small as possible, to maximize the stability and convergence speedup beneﬁt.

Original: However, if the validation perplexity has signiﬁcant ﬂuctuation at the very beginning, it is better to increase the starting sequence length.
Revised: However, if the validation perplexity has signiﬁcant ﬂuctuation at the very beginning, it is better to increase the starting sequence length.

Original: This is why we choose 8 and 64 as the starting sequence length for 117M and 1.5B models, respectively.
Revised: This is why we choose 8 and 64 as the starting sequence length for 117M and 1.5B models, respectively.

Original: Note that this starting sequence length should be a multiple of 8 to enable Tensor Core acceleration.
Revised: Note that this starting sequence length should be a multiple of 8 to enable Tensor Core acceleration.

Original: Overall, a general tuning strategy can be summarized as:
Revised: Overall, a general tuning strategy can be summarized as:

Original: 1) Start with seqlen 1 = 8 and T =
Revised: 1) Start with seqlen 1 = 8 and T =

Original: a few multiples of LR warmup steps.
Revised: a few multiples of LR warmup steps.

Original: 2) Increase seqlen 1 until the validation perplexity no longer has signiﬁcant ﬂuctuation at the very beginning.
Revised: 2) Increase seqlen 1 until the validation perplexity no longer has signiﬁcant ﬂuctuation at the very beginning.

Original: 3) Perform a binary search to ﬁnd the largest T that does not have signiﬁcant validation perplexity ﬂuctuation during the ﬁrst few multiples of LR warmup steps.
Revised: 3) Perform a binary search to ﬁnd the largest T that does not have signiﬁcant validation perplexity ﬂuctuation during the ﬁrst few multiples of LR warmup steps.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (“CL 270K” means applying curriculum learning to the ﬁrst 270K steps).

Original: A. L EARNING RATE SCHEDULE FOR CURRICULUM LEARNING
Revised: A. L EARNING RATE SCHEDULE FOR CURRICULUM LEARNING

Original: As described in Section 5, for 117M model CL cases we increase the number of learning rate decay steps by half of the curriculum learning duration since CL cases require more steps to reach 157B tokens.
Revised: As described in Section 5, for 117M model CL cases we increase the number of learning rate decay steps by half of the curriculum learning duration since CL cases require more steps to reach 157B tokens.

Original: However, we ﬁnd that this is still not ideal, and for 1.5B model cases we change the learning rate decay to token-wise (same cosine decay over the 157B tokens) instead of step-wise.
Revised: However, we ﬁnd that simply increasing decay steps still leads to sub-optimal learning rate schedule, and for 1.5B model cases we change the learning rate decay to token-wise (same cosine decay over the 157B tokens) instead of step-wise.

Original: This is because for CL cases there are fewer tokens per step at the beginning.
Revised: This is because for CL cases there are fewer tokens per step at the beginning.

Original: So even if we increase the LR decay steps, it still cannot avoid decaying faster token-wise at the beginning.
Revised: So even if we increase the LR decay steps, it still cannot avoid decaying faster token-wise at the beginning.

Original: As shown in Figure 5, 
Revised: As shown in Figure 5, CL with step-wise LR decay has faster token-wise LR decay compared to baseline, which leads to a worse validation perplexity curve. On the other hand, the same CL case with token-wise LR decay has a comparable LR schedule compared to baseline, which leads to better convergence. A. A DDITIONAL ANALYSIS ABOUT TRAINING STABILITY This section aims to provide additional information and analysis about training stability.

Original: CL with step-wise LR decay has faster token-wise LR decay compared to baseline, which leads to  a worse validation perplexity curve.
Revised: First, because in main paper we need to show the ﬁnal convergence difference between baseline and curriculum learning, we had to truncate a lot on y axis when plotting. This makes it harder to observe baseline’s training instability issue, which mostly happened at the ﬁrst half of training when the loss is still higher.

Original: On the other hand, the same CL case with token-wise LR decay has a comparable LR schedule compared to baseline, which leads to better convergence.
Revised: Figure 6 presents the re-plotting of Figure 2(a) and Figure 3(a) but with less y-axis truncation. At 117M, the baseline has a few training loss spikes at batch size 4K. At 1.5B, the baseline and related works experience signiﬁcant training instability exhibited as many loss spikes when training with either batch size 512 or 4K. In contrast, curriculum learning leads to a much healthier training curve and shows no loss spikes in the entire training process under both model sizes and batch sizes,

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: and this healthier training curve leads to better convergence and zero-shot evaluation results in main paper Section

Revised: [Table]

Revised: To quantitatively study “training instability”

Revised: [Table]

Revised: we deﬁne “loss ratio” as an indicative measurement of training instability, which measures the ratio between the current step training loss and the minimum loss among all previous steps.

Revised: A ratio larger than 1.0 means that current step’s loss is larger than the minimum loss in previous steps, thus larger ratio indicates a larger instability.

Revised: Table 5 summarizes the number of steps with loss ratio larger than 1.2 and 1.

Revised: [Table]

Revised: and the maximum loss ratio during the training for all baseline and curriculum learning cases reported in the main paper.

Revised: Similar to what we observe in Figure 6(a), at 117M model size only the baseline with batch size 4K has 22 steps with high loss ratios up to 1.421, and all other cases demonstrate much smoother training loss trend.

Revised: On the other hand, at 1.5B model size the baseline with both batch size 512 and 4K has many steps with large loss ratios, and with the maximum loss ratio as high as 5.65.

Revised: Baseline with batch size 4K is less stable than baseline with batch size 512, indicating that larger batch sizes could lead to more training instability risks.

Revised: In the case of related works, the 2-stage curriculum learning reduces the number and max of large loss ratios but could not fully resolve the instability issue, and the batch size warmup technique provides very minimal training stability gain.

Revised: Finally, the proposed curriculum learning method has zero steps with large loss ratios under both batch size 512 and 4K, demonstrating its superior training stability.

Revised: A. A DDITIONAL ANALYSIS ABOUT TRAINING HYPERPARAMETERS

Revised: In Appendix A.1 above we demonstrate that curriculum learning’s two hyperparameters can be tuned with very low cost only running the very beginning of the training (the third hyperparameter, ending sequence length, does not require tuning since it will always be the full length).

Revised: To understand more about how curriculum learning affects the choice and tuning of normal training hyperparameters, this

Revised: [Table]

Revised: section provides additional analysis about learning rates and gradient clipping.

Revised: Results demonstrate that (a) Compared to baseline, curriculum learning requires less tuning effort on these hyperparameters to provide a stable training; (b) By enabling stable training on larger learning rates, curriculum learning could provide better training efﬁciency and convergence (as demonstrated in main paper Section 5); (c) Tuning gradient clipping for baseline could not provide the same training stability as curriculum learning.

Revised: A.4. L EARNING RATE

Revised: In Section 5 we demonstrate that curriculum learning can provide stable and more efﬁcient training at larger batch size and learning rate, where baseline suffers from training instability.

Revised: We increased both batch size 4 and learning rate at the same time because (a) Large-batch training is more efﬁcient for large-scale training, so larger batch was necessary in our study (b)

Revised: In order to maintain the same convergence speed, it is necessary to increase the learning rate under larger batch size.

Revised: A well-known rule of thumb is that the learning rate should at least increase by the square root of the batch size’s increase ratio.

Revised: As a controlled experiment, here we perform additional studies about what if we keep the batch size the same and only tune learning rate for baseline and curriculum learning.

Revised: We do not consider the case of “same learning rate, different batch sizes” due to the reason (b) above.

Revised: Table 6 presents the number of steps with training loss ratios (deﬁned in Appendix A.3 as an indicative measurement of training instability) larger than 1.5 during GPT-2 1.5B

Revised: Seqlen 1K pre-training (ﬁrst 3K steps only) with batch size 2K 5 , 5 different seeds, and different learning rates for baseline and curriculum learning.

Revised: And Figure 7 illustrates some of the cases with seed 1236 to show how the loss spikes look like.

Revised: Results show that curriculum learning provides stable training during this ﬁrst 3K steps for all ﬁve seeds at learning rates up to 6 × 10 − 4 , while baseline with seed 1237 still has 7 large loss ratios at learning rate as low as 1 . 5 × 10 − 4 .

Revised: At learning rate 12 × 10 − 4 both cases have large loss ratios, but curriculum learning reduces the frequency by 2.1x.

Revised: This demonstrates that (a) Larger learning rates lead to higher training instability risk for both cases.

Revised: (b) With the same amount of tuning effort, curriculum learning has a higher probability of providing a stable training because of the wider range of learning rates it enables; (c) Since curriculum learning enables stable training at larger learning rate, it could provide better and faster training convergence as shown in main paper Section 5.

Revised: A.4. G RADIENT CLIPPING

Revised: In main paper Section 5 we used gradient clipping at 1.0 (global gradient l 2 norm is clipped to 1.0) following the previous work (Shoeybi et al., 2019).

Revised: Here we perform additional studies about what if we apply more gradient clipping to baseline.

Revised: Figure 8(a) presents the training loss during GPT-2 1.5B

Revised: Seqlen 1K pre-training (ﬁrst 5K steps only) with batch size 4K (the same hyperparameters as the second set in Section 3), comparing the baseline and curriculum learning under different gradient steps because for language model pre-training it is common to keep number of training tokens the same for a fair comparison about zero-shot evaluation.

Revised: Similarly, under the same batch size the curriculum learning case would have more total training steps are tokens per step at the beginning.

Revised: Batch size 2K is used here because this analysis was performed at an early stage of this work, and we did not have enough resource to rerun the same analysis with batch size 4K.

Revised: [Figure]

Revised: clipping levels 6 .

Revised: Results show that when applying more gradient clipping to baseline, the training has less and smaller loss spikes.

Revised: And the Adam varaince norm is also reduced as shown in Figure 8(c).

Revised: However, more gradient clipping does not fully resolve the training stability issue.

Revised: Even baseline with the lowest gradient clipping norm cannot avoid all training loss spikes, while curriculum learning with default gradient clipping has no loss spike.

Revised: As described in main paper end of Section 5.3, we believe that this is a limitation of common gradient clipping technique: Although gradient clipping can avoid too large gradient at every single step, it cannot avoid the gradient variance getting accumulated at certain dimensions (as shown in Figure 8(d)), especially for large batch sizes.

Revised: Another concern about applying more gradient clipping is that the momentum norm is also reduced due to more clipping (Figure 8(b)).

Revised: This indicates that when later the training reaches a more stable stage, more gradient clipping could hurt the convergence speed.

Revised: On the other hand, curriculum learning will not affect the convergence speed after the full sequence length is reached.

Revised: Another thing to note is that curriculum learning relies less on gradient clipping: at gradient clipping norm 1.0, baseline has 798 clippings in the ﬁrst 5K steps while curriculum learning has 628 clippings (21% less).

Revised: Overall, this analysis demonstrates that curriculum learning requires less or no tuning on gradient clipping, while baseline still has training stability issue with more gradient clipping.

Revised: It is possible that more complex and adaptive gradient/variance/activation clipping techniques could potentially achieve the same level of training stability as curriculum learning.

Revised: However, inventing and applying such techniques would require an effort no lower than the proposed curriculum learning solution, which is both easy to integrate and low-cost to tune.

Revised: A. L ONGER SEQUENCES DO INCUR HIGHER TRAINING INSTABILITY RISKS

Revised: To learn more about whether sequence length (as used by many previous works in Section 2) is an appropriate curriculum learning metric, here we study whether longer sequences incur higher training instability risks.

Revised: We design an experiment where we artiﬁcially insert full-length sequences during curriculum learning.

Revised: Speciﬁcally, for every 375 steps, we insert 37 steps of full-length sequences (by disabling the truncation) after 338 steps of normal curriculum learning with short sequence lengths 7 .

Revised: Figure 9 presents the step-wise training loss for baseline, normal curriculum learning, and the curriculum learning with the artiﬁcial 10% of full-length sequences.

Revised: There are three observations based on the results: (a) After inserting 10% of full-length sequences, curriculum learning starts to have training loss spikes and those spikes tend to happen at where the full-length sequences are inserted, which indicates that longer sequences do incur higher training instability risks; (b) However, the loss spikes are smaller than the baseline’s, which indicates that curriculum learning could be more resilient to rare and complex data; (c) The loss spikes happen less frequently from step 4K, which indicates that the training with curriculum learning gets more and more stable.

Revised: [Figure]

Revised: [Figure]

Revised: as the second set in Section 3), comparing baseline, curriculum learning, and curriculum learning plus inserting 10% of full-length sequences everysteps.