Insights into Pre-training via Simpler Synthetic Tasks
Abstract
Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective gains.
Notably, recent work shows that even pre-training on synthetic tasks can achieve significant gains in downstream tasks.
In this work, we perform three experiments that iteratively simplify pre-training and show that the simplifications still retain much of its gains.
First, building on prior work, we perform a systematic evaluation of three existing synthetic pre-training methods on six downstream tasks.
We find the best synthetic pre-training method, LIME , attains an average of 67% of the benefits of natural pre-training.
Second, to our surprise, we find that pre-training on a simple and generic synthetic task defined by the Set function achieves 65% of the benefits, almost matching LIME .
Third, we find that 39% of the benefits can be attained by using merely the parameter statistics of synthetic pre-training.
We release the source code at https://github.com/felixzli/synthetic_pretraining .
Introduction
Pre-training on a large amount of data from sources such as text from the web— natural pre-training — is effective for a wide range of downstream tasks (Devlin et al., 2019; Brown et al., 2020; Chen et al., 2021; Lu et al., 2021).
More surprisingly, recent works (Wu et al., 2021; Krishna et al., 2021; Ri &
Tsuruoka, 2022; Anderson & Farrell, 2022) show that pre-training on data that is fully synthetically generated— synthetic pre-training —can also provide substantial gains, partially closing the gap between training from a randomly initialized model and a naturally pre-trained model.
What properties of pre-training are necessary for effective gains?
In this paper, we conduct a careful empirical study to help understand this question.
We perform three experiments that iteratively simplify natural pre-training, while still retaining much of the benefits of natural pre-training.
See
Figure 1 for an overview of the results obtained in our three experiments.
First, we perform a systematic evaluation of three previously proposed synthetic pre-training methods over six downstream tasks. Prior works only evaluated each synthetic pre-training method on a single downstream domain: Wu et al.
(2021) on mathematical reasoning benchmarks, Chiang & Lee (2021) on GLUE, Krishna et al.
(2021) on summarization, and Ri & Tsuruoka (2022) on language modeling and dependency parsing.
We instead show how general these methods are as well as how different synthetic pre-training methods compare against each other.
We find that the best performing synthetic task, LIME (Wu et al., 2021), closes a significant fraction of the gap between random initialization and natural pre-training for summarization ( 85% ), semantic parsing ( 83% ), reading comprehension ( 55% ), code translation ( 43% ), and retrosynthesis ( 49% ).
[Figure]
Second, we discover a surprisingly simple and generic synthetic task that captured nearly the same effectiveness provided by the existing best synthetic task LIME , closing an average of 65% of the gap to natural pre-training (compared to 67% for LIME ).
This simple task, which we call Set , consists of outputting the input sequence without duplicate tokens in its original order, for example:
[Equation]
Third, to simplify the initialization even more, we ask whether only the statistics of the pre-trained weights can explain the effectiveness of synthetic pre-training.
We find that initializing parameters with statistics of a synthetic pre-trained model provides large benefits for some downstream tasks, closing 39% of the gap to natural pre-training.
Simplifying further allows us to see that by initializing all pre-attention layer norm parameters to a lower value (just one number!), we close more than 40% of the gap to natural pre-training performance for summarization and semantic parsing, and an average of 21% over all six tasks.
Experimental Setup
Architecture We trained a T5-Small model, which is a 60 million parameter encoder-decoder transformer model with 6 encoder layers, 6 decoder layers, 8 heads, head dimension 64, and MLP dimension 2048 (Raffel et al., 2020).
Training Details For synthetic pre-training, we use the same hyperparameters that the off-the-shelf language pre-trained T5-small was trained with: AdaFactor optimizer, batch size 128, sequence length 512, and inverse square root learning rate 1 / (cid:112) max( n, 10000) where n is the current training step.
We evaluate token validation accuracy every 5000 training steps.
For all synthetic tasks besides
Dyck , we fine-tuned with the first checkpoint that the model reaches above 99% token validation accuracy.
For Dyck , the model’s validation accuracy plateaued without reaching above 99%, so we chose to fine-tune using the checkpoint with max validation accuracy of 77.7%.
We provide further pre-training details in Appendix A and fine-tuning details in Appendix B.
Downstream Tasks We fine-tuned synthetically pre-trained models on a diverse suite of downstream tasks: 1) Java to C# code translation (10K training examples)
(Lu et al., 2021); 2) two semantic parsing benchmarks, MTOP (17K training examples)
(Li et al., 2021) and WebQSP (2.7K training
[Table]
examples) (Yih et al., 2016), consisting of converting a natural language query to a logical form; 3) USPTO-50K retrosynthesis (40K training examples) (Liu et al., 2017), a task that consists of predicting possible reactants when given a product as input; 4) the reading comprehension benchmark
SQuAD 1.1 (87K training examples) (Rajpurkar et al., 2016); and 5) the summarization benchmark
CNNDM-10K 3 which is 10K training examples from the CNNDM (Krishna et al., 2021).
See full descriptions and input-output examples for every downstream task in Appendix D.
Baselines We compared against randomly initialized ( Random_Init ) and off-the-shelf natural language pre-trained T5-Small ( Pre-trained_T5 ).
Pre-trained_T5 was trained for 524K steps on the C4 dataset with the span-corruption objective (Raffel et al., 2020).
We also compared against T5-Small trained on Wikipedia (Guo et al., 2020) for 10K steps ( Wiki_10K ) with the same pre-training objective as Pre-trained_T5 .
Source code We release the source code to reproduce the experiments at https://github.com/ felixzli/synthetic_pretraining .
Benchmarking Existing Synthetic Pre-training
We benchmark three previously proposed synthetic pre-training tasks across six downstream tasks.
We compute how much of the benefits using natural pre-training are gained by each synthetic pre-training method by taking the average over six downstream tasks of Performance(Initialization) − Performance( Random_Init )
Performance( Pre-trained_T5 ) − Performance( Random_Init ) .
We show these numbers in the right-most column of
Table 1.
We find that LIME achieves the best performance, gaining 67% of the benefits using natural pre-training.
3.1 Synthetic Tasks
For each synthetic task, we generated one million examples.
To generate synthetic task data, we sampled tokens a vocabulary of size 32K, matching the vocabulary size of Pre-trained_T5 .
Details about data generation can be found in the original papers.
We provide an example of each task in
Figure 2.
LIME LIME is a set of three tasks—Deduct, Induct, and Abduct—inspired by Charles Peirce’s three reasoning primitives (Wu et al., 2021).
task consists of three elements: a rule string, a dictionary that represents substitutions, and a result string that is the result of applying those substitutions to the rule string.
The three tasks are then constructed by using two of the three elements as inputs to predict the remaining element.
Dyck Artificial Language We consider a Dyck language with multiple bracket types (taken from
Ri & Tsuruoka (2022)), which we refer as Dyck .
We turn generated text into a sequence-to-sequence task by using the T5 span-corruption pre-training objective (Raffel et al., 2020).
The motivation behind the design of Dyck is how “sentences of natural language often have dependency relations where the existence of a certain word can be predictive of another word” (Ri & Tsuruoka, 2022).
Nonsense Summary Krishna et al.
(2021) developed the nonsense summarization task, which we refer to as Nonsense_Summary , by conducting a qualitative analysis of summarization data and identifying 21 elementary operations used to create summaries such as “CopyFirstSentence” and “CopyQuoted.”
Each synthetic task example consists of an input that is a nonsense document made of randomly sampled tokens and an output that is the corresponding nonsense summary consisting of applying a random subset of the 21 operations to the document.
3.2 Results
Table 1 shows results of evaluating the synthetic tasks across six downstream tasks, and Appendix
Table 4 shows results of ablating the dataset size for retrosynthesis, CNNDM, and SQuAD.
Synthetic pre-training provides large gains over random initialization.
We observed large gains from all three synthetic tasks we evaluated.
Out of these, LIME provided the most benefits across all six downstream tasks.
We computed how much of the natural pre-training benefit is captured by
LIME as
[Table]
LIME [Table] [Table] [Table]
[Table]
[Table]
5% , and 48 .
7% of the gap for CNNDM-10K, MTOP, WebQSP, SQuAD, Code Translation, and
Retrosynthesis respectively.
Task or domain prior is not required for pre-training benefits.
For example, LIME benefits tasks including summarization (CNNDM-10K) and reading comprehension (SQuAD) despite LIME pre-training data involving no natural language like Pre-trained_T5 , representing no structure of natural language like Dyck , and not being explicitly designed to reflect the operations used in summarization like Nonsense_Summary .
Varying degrees of benefits.
Comparing Nonsense_Summary and Dyck , we see that the two synthetic tasks benefit MTOP, WebQSP, and SQuAD similarly, yet their benefits for the three other tasks are very different.
The difference between Nonsense_Summary and Dyck as well as the superiority of LIME are examples of how different synthetic pre-training can have significantly different effectiveness.
Synthetic pre-training is not more efficient.
One of the advantages argued for synthetic pretraining is its computational efficiency (Wu et al., 2021; Krishna et al., 2021).
To explore this potential advantage, we pre-trained our model using the T5 span-corruption pre-training objective on
Wikipedia (Guo et al., 2020) for 10K steps, and the result is shown in Table 1 labelled Wiki_10K .
Wikipedia pre-training for 10K steps provided more significant gains than LIME pre-training for 30K steps and Set pre-training for 10K steps.
Based on this result, we see no evidence that the evaluated synthetic pre-training tasks provides computational efficiency benefits over natural language pre-training, unlike what Wu et al.
(2021) suggested.
Natural language pre-training benefits non-natural language tasks.
Natural language pretraining, including Wiki_10K , outperforms synthetic pre-training on the non-natural language tasks of Code Translation and Retrosynthesis.
Simpler Synthetic Pre-training
Encouraged by the results in the previous section, we want to explore what is necessary for synthetic pre-training benefits.
We evaluate pre-training with simpler and more generic synthetic tasks similar to basic Python API functions.
Surprisingly, a simple and generic task defined by the Set function provides significant benefits ( 65% ), almost matching the previous best synthetic task LIME ( 67% ).
Furthermore, another simple task defined by the Identity function also provided significant benefits ( 46% ).
4.1 Simpler Synthetic Tasks
We constructed 18 simpler synthetic tasks similar to basic Python API functions.
As we did for the three existing synthetic tasks, for each simpler synthetic task we generated one million data points with a vocabulary of 32K tokens, matching the vocabulary size of Pre-trained_T5 .
In
Appendix G.1, we define each task, explain how examples are randomly generated, and provide input-output examples.
We first evaluated all tasks on CNNDM-10K. We then evaluated Set , one of the best performing tasks on CNNDM-10K, and Identity , one of the simplest tasks, on all six downstream tasks.
Set
The Set task consists of outputting the input sequence in its original order except without tokens appearing more than once.
An example input is [ a b b a a a c d c ] , and the corresponding output is [ a b c d ] .
The Set task data generation is explained in Appendix G.
Identity The Identity task consists of copying the input tokens to the output.
An example input is [ a d e a a ] , and the corresponding output is [ a d e a a ] .
To generate one example, we uniformly sample a sequence of tokens between length 10 and length 220 to be the input and output.
Results: simpler synthetic tasks provide similar benefits as existing synthetic tasks
Figure 3 shows results on CNNDM-10K for models pre-trained on each of 18 simpler synthetic tasks.
Table 1 shows results on all six downstream tasks for Set and Identity pre-trained models.
Averaged across the six downstream tasks, Set and Identity gain 65% and 46% of the natural pretraining benefits compared to 67% for LIME , the best previously proposed synthetic task evaluated in
Section 3.
Identity provides at least as much benefit as Nested Language and Nonsense Summary for all tasks besides Code Translation, for which Nested Language benefits more.
The significant benefits of Set and Identity pre-training suggest that the complexities of the three previously proposed synthetic tasks we evaluated—representing three fundamental logical operations, having a structural property that mimics natural language, and reflecting operations used for summarization—may not be the necessary factors for why pre-training on those tasks provides benefits.
Also, due to their simplicity, Set and Identity may be useful for future work towards understanding pre-training and how to design better synthetic tasks.
Even Simpler Synthetic Pre-training with Statistics
Inspired by the previous section, we tried to find even simpler initializations that achieve the the benefits of pre-training.
We explored how benefits of synthetic pre-training can be attained by initialization based only on statistics of pre-trained weights.
We found: (1) Initialization using the mean and standard deviation (SD) of synthetic pre-trained parameters can gain more than 70% of the benefit from synthetic pre-training for CNNDM-10K and MTOP (or an average of 39% of natural pre-training benefits).
(2) Initialization using simpler schemes involving less statistics can still provide benefits.
(3) Specifically the pre-attention layer norm plays an important role for initialization benefits.
5.1 How much of synthetic pre-training benefit can be attributed to better statistics for initialization?
We compare normal fine-tuning, which we call Full_Init , with fine-tuning a model that is initialized with the mean and standard deviation (SD) of each synthetic pre-trained model’s parameter.
We call this initialization scheme Per_Param_Mean_SD and illustrate it in Figure 4.
Specifically, for each parameter p (e.g. an attention query matrix) in the synthetic pre-trained model, we compute the the mean µ p and the standard deviation σ p of all entries in p .
We then randomly initialize the model with the same µ p and σ p for each parameter p .
For example, if a pre-trained model had a 2 × 2 parameter matrix p =
[[ − 1 , 2] , [2 , 5]] , then µ p = 2 and σ p = 2 . 4 , and we would initialize p by sampling the each entry of this 2 × 2 matrix from a normal distribution N (2 , 2 . 4 2 ) .
We list all all the parameters we initialize for T5-Small in this manner in Appendix K.
As shown in Table 2, using the Per_Param_Mean_SD initialization scheme achieves a large proportion of the benefit from normally fine-tuning a synthetic pre-trained model.
For example, for LIME
Identity and Set , the Per_Param_Mean_SD initialization recovered 78% , 85% , and 97% 2 of
MTOP Full_Init performance (or 65%, 66%, 69% of the benefits by natural pre-training).
The remaining gap between Per_Param_Mean_SD and Full_Init supports the possibility that other factors besides better initialization statistics may cause the benefits of synthetic pre-training.
We note that it does not preclude the possibility that better initialization statistics is the sole cause of benefits: there may exist other initialization schemes such that the statistics of the pre-trained model are a subset of the statistics from the initialization scheme.
5.2 Can simpler initialization schemes still produce benefits?
Next we explore if simpler initialization schemes that use a smaller number of statistics than Per_Param_Mean_SD can still provide benefits.
We present three initialization schemes:
Per_Param_Scale , Whole_Model_Scale , and Subset_Per_Param_Scale , with results shown in Table 2.
We also tried two other initialization schemes, Across_Layers_Scale and
Per_Layer_Scale , discussed in Appendix H.
[Table] [Table] Per Param Scale We use the term “scale statistic” to denote the mean of layer norm parameters or the SD of non layer norm parameters.
Unlike the Per_Param_Mean_SD init scheme in the previous subsection, where we initialize each parameter with two statistics (mean and SD), Per_Param_Scale initializes each parameter with only the scale statistic: initialize layer norm parameters by setting their value to the mean of the corresponding pre-trained parameter weights, and initialize non layer norm parameters by sampling from a Gaussian with mean 0 and the SD of the corresponding pre-trained parameter weights.
Whole Model Scale We initialize the whole model with only two statistics: the SD over all non layer norm weights σ m and the mean over all layer norm weights µ m .
Initialize each non layer norm weight by sampling from N (0 , σ 2 m ) and Initialize each layer norm weight as the value µ m .
Per Param Scale:
Subset The motivation for this init scheme is to find a smaller set of parameters whose initialization can provide significant benefit.
We initialize a subset of the parameters (e.g., attention parameters) with the Per_Param_Scale initialization and initialize the rest of the parameters with the default Random_Init .
With this initialization scheme, we tried four different subsets: attention parameters, MLP parameters, pre-attention layer norms, and pre-MLP layer norms.
Results Per_Param_Scale provided almost identical benefits as the Per_Param_Mean_SD initialization, suggesting that just the scale statistics, rather than both the mean and SD, are sufficient to provide benefits of initialization.
Whole_Model_Scale initialization was much worse than Per_Param_Mean_SD , yielding almost no improvements.
A finer scale initialization rather than simply increasing the scale of weights across the whole model as Whole_Model_Scale does may be necessary for benefits.
For the Subset_Per_Param_Scale init scheme, pre-attention layer norms was the only subset we tried that provided significant initialization benefits, covering 46% and 39% of the gap between
From Scratch and Pre-trained_T5 for CNNDM-10K and MTOP.
See results in Table 2 (labeled
Pre-attn_LN_Per_Param_Scale ) and results for all four subsets tried in Appendix Table 5.
5.3 A Better Pre-attention Layer Norm Initialization
Synthetic pre-training results in lower pre-attention layer norm means.
Motivated by benefits from the Subset_Per_Param_Scale initialization scheme—initializing pre-attention layer norms using their synthetic pre-trained means and all other parameters with the default initialization—we wanted to see if the pre-attention layer norm mean values in synthetic pre-trained models have a [Table] pattern.
In Appendix Figure 6, we plotted pre-attention layer norm parameter means of each layer for
LIME , Set [Table] and Identity pre-trained models.
We noticed that the pre-attention layer norm means from the synthetic pre-trained models are much smaller values (below 0 .
4 for most layers) compared to the default initialization of 1 . 0 .
Lower value pre-attention layer norm initialization provides benefits.
Inspired by the above observation, we tried an initialization scheme where all the pre-attention layer norm parameters are initialized to a specific value, and all other parameters are initialized with the default initialization.
We tried five different values ranging from 0 .
05 to 0 .
Results in Table 3 show that initializing the pre-attention layer norm parameters to a lower value than the default initialization of 1 . 0 provided significant benefits: initializing them to 0 .
2 improved CNNDM-10K from 18 .
9 to 28 .
2 ROUGE1,
MTOP from 38 .
6% to 56 .
4% EM, and SQuAD from 16.0 to 22.8 EM.
Related Work
Synthetic Pre-training Past work has shown that synthetic pre-training can benefit a wide range of downstream tasks.
For example, it has been shown that pre-training on artificial languages that intuitively mimic properties of natural language can benefit language modeling, dependency parsing, and GLUE (Ri & Tsuruoka, 2022; Papadimitriou & Jurafsky, 2020; Chiang & Lee, 2021).
Pretraining on LIME, a mixture of three synthetic tasks motivated from reasoning primitives, benefits mathematical reasoning benchmarks (Wu et al., 2021).
For summarization, Krishna et al.
(2021) designed a suite of synthetic summarization tasks on a corpus of random tokens, and they show such pre-training can almost match up with natural pre-training.
Synthetic pre-training also has been applied outside of text-based domains: Anderson & Farrell (2022) shows pre-training on synthetically generated fractal images nearly matches the benefits of pre-training with ImageNet.
Cross-domain Transfer Learning Synthetic pre-training benefiting downstream tasks across a wide range of domains is an instance of cross-domain transfer learning, which past work has demonstrated the efficacy of.
Pre-training language models with many languages benefits downstream tasks in a new unseen language (Liu et al., 2020).
More surprisingly, pre-training on music, code, and amino acid sequences benefits natural language downstream tasks (Papadimitriou & Jurafsky, 2020; Chiang & Lee, 2020).
It was also shown recently that pre-training on natural language benefits offline RL (Reid et al., 2022).
Understanding Pre-training Past work has explored what properties of the pre-training data are important for benefits.
Ri & Tsuruoka (2022) and Papadimitriou & Jurafsky (2020) use synthetic pretraining to identify general properties of pre-training data that correlate with improved downstream language task performance by leveraging how synthetic data is easy to modify in a controlled manner.
Papadimitriou & Jurafsky (2020) found that changing the distribution of tokens in an artificial language can affect downstream performance, and Ri & Tsuruoka (2022) found that for an artificial brackets language using a nesting structure as well as different tokens for bracket pairs results in better performance than using a flat structure and the same token for bracket pairs.
K et al.
Conneau et al.
(2020), and Dufter & Schütze (2020) explore data properties related to cross-language transfer learning.
Sinha et al.
(2021) shows that word order within sentences is not essential for natural langauge pre-training benefits.
Other past work has used probing methods to explore what potentially transferable knowledge is gained from pre-training.
Ri & Tsuruoka (2022) explored how much knowledge of "position-aware context dependence" was gained from artificial language and natural language pre-training.
Chi et al.
and Papadimitriou et al.
(2021) explored potentially transferable knowledge gained from multi-lingual language pre-training.
Initialization Our work tried to understand pre-training through the lens of initialization statistics that benefit optimization.
There has been past work on improving transformer initialization to achieve more stable training and benefits with deeper models (Zhang et al., 2019; Huang et al., 2020; Zhang et al., 2020; Wang et al., 2022).
These papers do not explore benefits from initializing with pre-trained model statistics.
Discussion
Limitations of current synthetic tasks In this work, we show that synthetic pre-training can provide significant benefits across many different tasks.
However, synthetic pre-training still lags behind natural pre-training by an average of 33% for the six downstream tasks we fine-tuned on.
This is expected for natural language downstream tasks, but it is important to note that even on non-natural language tasks, natural language pre-training still outperforms synthetic pre-training.
This result suggests that natural language pre-training produces effects that are missing from the evaluated synthetic pre-training.
Exploring the differences between different pre-training may be an interesting direction for future work.
We emphasize that this paper does not support any claims regarding the practical efficacy of synthetic pre-training.
The metric of "percentage of natural pre-training gap closed" measures relative performance along a meaningless linear scale, and for downstream tasks retrosynthesis and code translation, the magnitude of the improvement over from scratch for natural pre-training is low to begin with.
This paper’s goal is not to communicate that synthetic pre-training is practically effective, but rather to communicate that synthetic pre-training produces nontrivial differences in model behavior compared to a randomly initialized model, and also that there is nontrivial variation in model behavior from differing synthetic pre-training and initialization schemes involving pre-trained weight statistics.
Synthetic pre-training requires deeper understanding Our results show that initialization using the statistics from synthetic pre-trained models can recover more than 70% of the benefit from synthetic pre-training.
The remaining gap leaves open the possibility that the benefits of synthetic pretraining are due to other factors besides better statistics for initialization.
Due to their simplicity, Set and Identity may be useful for future work towards deeper understanding of synthetic pre-training.
In the long term, as we learn more about synthetic pre-training and pre-training in general, we believe that for some downstream tasks it may be possible to develop synthetic pre-training that outperforms natural pre-training: the complexity of existing natural data is fixed, while in some sense the complexity of fully synthetically generated data is infinite.
Privacy and Ethics This work focuses on the understanding of synthetic pre-training.
But synthetic pre-training can also have practical implications for privacy and ethics.
Standard pre-training data, even when its public, can contain private user information.
Large pre-trained models are capable of memorizing training examples, which makes them vulnerable to privacy attacks (Carlini et al., 2020).
Synthetic data on the other hand is divorced from the real world, which can help mitigate privacy issues.
Large pre-trained corpora can contain harmful text (e.g., hate speech).
With synthetic pre-training, since we have complete control over what goes into the data, we can potentially mitigate some of the harmful behavior of existing models.
Unfortunately, synthetic pre-training is currently not as performant as natural pre-training, so real-world data would still have to be used to close the performance gap.
Acknowledgements
We thank Google TPU Research Cloud for the experimental support.
We thank Christian Szegedy,
Eric Zelikman, Tri Dao, Dan Fu, Sidd Karamcheti and Rishi Bommasani, for their useful feedbacks on the draft.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Sections 2,3.1,4.1,G,D (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix Section I 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We release the source code at https://github.com/felixzli/synthetic_ pretraining .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] A Synthetic Pre-training Details We do synthetic pre-training with the same hyperparameters that the off-the-shelf pre-trained T5-small was trained with: AdaFactor optimizer, batch size 128, sequence length 512, and inverse square root learning rate 1 / (cid:112) max ( n, 10000) where n is the current step.
We evaluate token validation accuracy and save a checkpoint every 5000 steps.
For synthetic tasks besides the artificial language, we fine-tuned with the first checkpoint that the model reaches above 99% token validation accuracy.
For LIME, Nonsense Summary, Set, and Identity this was 30K steps, 5K steps, 10K steps, and 5K steps respectively.
For the artificial language pre-training, after training for 300K steps the model’s token validation accuracy had plateaued without converging to above 99% token validation accuracy, so we chose to fine-tune with the max accuracy checkpoint, which was at step 165K with accuracy of 77.7%.
For reference, off-the-shelf pre-trained T5-small was trained for 524,288 steps (Raffel et al., 2020).
B Fine-tuning Details
We fine-tuned with the same hyperparameters (Raffel et al., 2020) fine-tuned with, with the exception of learning rate, for which we do a sweep over 1e-2, 3e-3, 1e-3.
We use the AdaFactor optimizer and batch size 128 for all except WebQSP and Code Translation, for which we use batch sizedue to their long sequence length.
We list the learning rate and sequence length for each task in
Appendix D.3.
We use the default T5 tokenization (Raffel et al., 2020).
We fine-tuned for a max of 150K steps for code translation, 400K steps for retrosynthesis, 250K steps for SQuAD, 50K steps for
WebQSP, 100K steps for MTOP, and 100K steps for CNNDM-10K.
We evaluate and save checkpoints every 10K steps for all tasks besides WebQSP, for which we evaluate and save checkpoints every 5K steps.
For all tasks besides CNNDM-10K and SQuAD, we report the test set metric achieved with the checkpoint corresponding to the max token validation accuracy checkpoint.
For CNNDM-10K, we reported the test set metric achieved at step 50K, because we observed the test set metric achieved with the max token validation accuracy checkpoint was often significantly worse than later step checkpoints, and the test set metric always plateaued before and up to about 50K steps.
For SQuAD, we report the best validation set metric, as was done in (Raffel et al., 2020), because evaluating on the test set requires running inference on a benchmark server.
C Off-the-shelf Pre-trained T5v1.1-Small for CNNDM-10K
The differences between Pre-trained T5v1.1 and Pre-trained T5 are:
T5v1.1 Uses the GEGLU activation in feed-forward hidden layers rather than ReLU.
Dropout was turned off in pre-training for T5v1.1.
No parameter sharing between the embedding and classifier layer for T5v1.1.
T5v1.1 was pre-trained on C4 only, without mixing in the downstream tasks.
The last difference is why we fine-tune CNNDM-10K with Pre-trained T5v1.1 instead of Pre-trained
T5.
Using Pre-trained T5 would be an unfair comparison because it has already trained been on all 290K CNNDM task training data.
D Downstream Tasks D.1 Task Descriptions
CNNDM-10K is 10K training examples from the CNNDM benchmark, which consists of news articles from CNN and Daily Mail and summaries of each article (See et al., 2017).
We evaluated on
CNNDM-10K instead of the full CNNDM because this was the dataset that (Krishna et al., 2021) evaluated their nonsense summarization synthetic task on, which was one of the synthetic tasks we evaluate in Section 3.
MTOP (Li et al., 2021) and WebQSP (Yih et al., 2016) are two benchmarks for semantic parsing, the task of converting a natural language query to a logical form.
For these tasks, we use the data from (Xie et al., 2022) that is already processed to be in a Seq2Seq format compatible with T5.
MTOP has 17K training examples and WebQSP has 2.7K training examples.
SQuAD 1.1 is a reading comprehension dataset consisting of “questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text or span from the corresponding reading passage” (Rajpurkar et al., 2016).
There are 87K training examples.
Java to C# code translation is one task within Microsoft’s CodeXGLUE: General Language
Understanding Evaluation benchmark for Code (Lu et al., 2021).
The task consists of 10K training examples that were collected from the code of several open-source projects that were originally developed in Java and then ported to C#.
USPTO-50K is a dataset of 50K chemical reactions that is a commonly used to benchmark deep learning methods for the chemistry task of single-step retrosynthesis, which is predicting possible reactants when given a product as input.
(Liu et al., 2017).
For this task, we use the data from (Lu &
Zhang, 2022) that is already processed to be in a Seq2Seq format compatible with T5.
There are 40K training examples.
D.2 Task Examples
[Table]
to celebrate his goal against qpr with friend januzaj .
the disco effect in the background adds to the theory, but januzaj doesn’t seem to mind as they later pose on the dance floor with other friends.
united haven’t had too many reasons to have a song and dance this season so it seems they may be hitting the discotheques as another form of release .
however, victory against newcastle on wednesday would leave manager louis van gaal at least tapping his toes as they continue to fight for a champions league spot this season.
januzaj and robin van persie join fellaini in celebrating in front of the manchester united fans at west brom .
januzaj receives some words of wisdom from manchester united’s dutch manager louis van gaal .
januzaj and fellaini are joined by some friends as they take to the dance floor ahead of the newcastle game .
[Table]
the
[Table]
[Table]
in the premier
[Table]
on wednesday .
red devils will be looking for just their second
[Table]
louis
[Table]
currently
[Table]
[Table]
[Table]
D.3 Downstream Task Specific Hyperparameters
More general fine-tuning hyperparameters and details are in Appendix B. For each task, we chose the learning rate by trying 1e-2, 3e-3, 1e-3 for Random Init finetuning, and picking the best performing value. .
CNNDM-10K
We use the same input length, taget length, and max decoding length parameters as the Nonsense Summary synthetic task paper (Krishna et al., 2021).
• learning rate: 0.
• input length:• target length:• max decode length:
MTOP
• learning rate: 0.
• input length:• target length: WebQSP • learning rate: 0. • input length:• target length:
SQuAD
• learning rate: 0. • input length:• target length:
Code Translation
• learning rate: 0. • input length:• target length:[-
-]
Retrosynthesis
• learning rate: 0. • input length:• target length:[-
-]
E Three Previously Proposed Synthetic Tasks from SectionE.1 LIME
The input token length ranges from 68 to 214 and the output token length ranges from 7 to 74.
For the following LIME examples, the “rule string” is A ∗
A + B = C , the “case dictionary” that represents substitutions is { A : a, B : b, C : d + e } , and the “result string” is a ∗
a + b = d + e .
In real generated data, the "Rule Symbols" and "Math Symbols" are randomly sampled from a vocabulary of 32K tokens for each data point. [Table] [Table] 
 {A : a, B : b,
[Table]
: d + e} 
 a * a
+ b = d + e
[Table]
=
[Table]
[Table]
E.2 Dyck Artificial Language
The input token length ranges from 3 to 512 and the output token length ranges from 3 to 114.
For clarity, the following example will involve only 3 head and tail tokens represented by the symbols {, }, [, ], (, ).
In real generated data, there will be 16K possible head-tail token pairs.
An example sentence:
[Equation]
This sentence converted to a Seq2Seq task with the T5 pre-training objective:
[Equation]
E.3 Nonsense Summary
The input token length ranges from 175 to 709 and the output token length ranges from 12 to 825.
We provide an example of a nonsense summary data point in Appendix Figure 5, which is directly taken from the task’s original paper (Krishna et al., 2021).
F Dataset Size Ablation Full Results
The table showing the full training dataset size ablation results is in Appendix Table 4.
G Simpler Synthetic Tasks from Section
All data is generated by sampling over 32K tokens.
But for clarity, all examples are written with individual alphabetic characters representing tokens.
All simpler synthetic tasks have input between length 10 and length 220.
G.1 Set
Definition Input is a token sequence.
Output is the input token sequence with no duplicates (in original order). [Table] [Figure] [Table]
[Figure]
[Table]
Input Generation To generate one data point, we do not just uniformly sample tokens.
Because over a vocabulary of 32K tokens, that simple sampling would result in few sequences with duplicate tokens where the task input and output are different.
Instead, we uniformly sample the length l of the input from min length to max length, then sample t tokens that will be in the input where t < l , then sample the amount of each token by randomly partitioning l to t parts { p 1 , p 2 ...
p t } , then finally generate a random input sequence based on the t sampled tokens and corresponding amounts.
G.2 Delete
Definition Input is token sequence 1 and token sequence 2.
Output is sequence 1 with the first occurrence of sequence 2 deleted.
[Figure]
Input Generation With 0.75 probability, generate sequence 1 and sequence 2 such that sequenceappears in sequence 1 at least once via the following process: sample sequence 2, sample sequencelength, sample location that sequence 2 appears in sequence 1, sample remaining tokens in sequence 1.
With 0.25 probability, generate sequence 1 and sequence 2 such that sequence 2 does not appear in sequence 1 via the following process: sample sequence 1, sample sequence 2 and keep re-sampling sequence 2 until it does not appear in sequence 1.
G.3 Sort
Definition Input is token sequence 1 and token sequence 2.
Output is input sequence 1 sorted in according to order specified by token sequence 2.
[Figure]
Input Generation To generate token sequence 1, uniformly sample the length l of the input from min length to max length, then sample t tokens that will be in the input where t < l , then sample the amount of each token by randomly partitioning l to t parts { p 1 , p 2 ...
p t } , then finally generate a random input sequence based on the t sampled tokens and corresponding amounts.
To generate sequence 2, shuffle the set of tokens in sequence 1.
G.4 Identity
Definition Input is a token sequence.
Output is the same as the
[Table]
G.5 Union
Definition Input is token sequence 1 and token sequence 2.
Output is token sequence that is the union of the set of tokens in sequence 1 and set of tokens in sequence 2.
[Table]
Input Generation Sample max length l , then sample partition of l to l 1 and l 2 , which will be the lengths of sequence 1 and sequence 2.
Sample l/ 2 unique tokens.
To get sequence 1, sample l 1 of the unique tokens.
To get sequence 2, sample l 2 of the unique tokens.
[-G.6 Set
-][Table]
[-Minus Set
-]
Definition Input is token sequence 1 and token sequence 2.
Output is a token sequence that is the subtraction of the set of tokens in sequence 2 from the set of tokens in sequence 1.
[Table]
Input Generation Sample max length l , then sample partition of l to l 1 and l 2 , which will be the lengths of sequence 1 and sequence 2.
Sample l/ 2 unique tokens.
To get sequence 1, sample l 1 of the unique tokens.
To get sequence 2, sample l 2 of the unique tokens. [Table] [Table]
Replace
Definition Input is token sequence 1 and token sequence 2.
Token sequence 2 always has two tokens, which we will denote as t 1 and t 2 .
Output is token sequence 1 with t 1 replaced by t 2 .
[Table]
Input Generation To get sequence 1, sample input sequence length l uniformly between min and max possible length, then sample l tokens.
To get sequence 2, sample one token from the set of tokens in sequence 1, and sample a token from the entire vocab.
G.8 Duplicate
Definition Input is a token sequence.
Output is the input token sequence with every token duplicated.
[Table]
Input Generation Sample input sequence length l uniformly between min and max possible length, then sample l tokens.
[Table]
Definition Input is token sequence 1 and token sequence 2.
Output is token sequence that is the intersection of the set of tokens in sequence 1 and set of tokens in sequence 2.
[Table]
Input Generation Sample max length l , then sample partition of l to l 1 and l 2 , which will be the lengths of sequence 1 and sequence 2.
Sample l/ 2 unique tokens.
To get sequence 1, sample l 1 of the unique tokens.
To get sequence 2, sample l 2 of the unique tokens.
G.
[Table]
Definition Input is a token sequence.
Output is the input token sequence in reverse order.
[Table]
Input Generation Sample input sequence length l uniformly between min and max possible length, then sample l tokens.
G.11 Set2 Minus [-Set
-]Set
Definition Input is token sequence 1 and token sequence 2.
Output is a token sequence that is the subtraction of the set of tokens in sequence 1 from the set of tokens in sequence 2.
[Table]
Input Generation Sample max length l , then sample partition of l to l 1 and l 2 , which will be the lengths of sequence 1 and sequence 2.
Sample l/ 2 unique tokens.
To get sequence 1, sample l 1 of the unique tokens.
To get sequence 2, sample l 2 of the unique tokens.
G.12 Deduplicate
Definition Input is a token sequence.
Output is the input token sequence with no adjacent duplicate tokens.
[Table]
Input Generation Uniformly sample the length l of the input from min length to max length, then sample t tokens that will be in the input where t < l , then sample the amount of each token by randomly partitioning l to t parts { p 1 , p 2 ...
p t } , then generate the input sequence by iterating through the t tokens and repeating each one p i times.
G.13 Last Token
Definition Input is a token sequence.
Output is the last token in the input sequence.
[Figure]
Input Generation Sample input sequence length l uniformly between min and max possible length, then sample l tokens.
G.14 First Token
Definition Input is a token sequence.
Output is the first token in the input sequence.
[Figure]
Input Generation Sample input sequence length l uniformly between min and max possible length, then sample l tokens.
G.15 Longest Word
Definition Input is a token sequence.
The ‘input token sequence has a special format, with special separator tokens distributed throughout.
Output is the largest number of contiguous tokens.
[Figure]
Input Generation Sample the number of “words” w defined as a contiguous block of normal tokens.
Sample the length of the input token sequence l .
Partition l − w + 1 to w parts p 1 , p 2 , ...
p w .
Build the input sequence by sampling the “words” according to the word lengths p 1 , p 2 , ...p w , and then joining the words with the special separator token.
G.16 Search
Definition Input is token sequence 1 and token sequence 2.
Output is “yes” if token sequence 2 is contained in token sequence 1, and “no” otherwise.
[Figure]
Input Generation With 0.75 probability, generate sequence 1 and sequence 2 such that sequenceappears in sequence 1 at least once by using the following procedure: sample sequence 2, sample sequence 1 length, sample location that sequence 2 appears in sequence 1, sample remaining tokens in sequence 1.
With 0.25 probability, generate sequence 1 and sequence 2 such that sequence 2 does not appear in sequence 1 by using the following procedure: sample sequence 1, sample sequenceand keep re-sampling sequence 2 until it does not appear in sequence 1.
Definition Input is a token sequence.
Output is the number of tokens in the input token sequence.
Source:aabc w
Target:
Input Generation
G.18 Count
Definition Input is token sequence 1 and token sequence 2.
Token sequence 2 is only 1 token which we will denote t .
Output is the number of times t occurs in token sequence 1.
Example
Source:
[Equation]
[-Target:
-]Target:
Input Generation Start by sampling the single token t in sequence 2.
Then sample l , which will be the length of sequence 1, and sample the number of times c that t will appear in token 1.
Sample l − c remaining tokens.
H More Initialization Schemes
We define two other initialization schemes we tried, and show results in Appendix Table[-
-]
Across Layers Scale We tried the Across_Layers_Scale initialization scheme, where we initialized each type of parameter with the scale statistic of it computed across all layers.
Some examples of types of parameters include query matrices, pre-MLP layer norms, and pre-attention layer norms.
Using this initialization, all query matrices across the whole model would be initialized with a single SD, which is the SD over all the pre-trained Q matrix weights, across all layers.
We find that
Across_Layers_Scale provides significant benefits for LIME , Identity , and Set , preserving at least 87% of the Per Param Mean/SD initialization scheme benefit for CNNDM-10K and MTOP.
This result suggests that parameter statistics varying from layer to layer might not be an important factor for synthetic pre-training initialization benefits.
Per Layer Scale We tried the Per_Layer_Scale initialization scheme, where every layer is initialized with just two statistics: the SD of all the non-layer norm parameter weights in that layer and the mean of all the layer norm parameter weights in that layer.
Results are mixed.
There is no benefit for Identity CNNDM-10K and minimal benefit for LIME MTOP.
For the other four pre-training and downstream task settings, there was significant benefit, preserving at least 80% of the Per Param Mean/SD benefit for CNNDM-10K and MTOP.
I Compute Used
Each experiment was ran on a Google Cloud v3-8 TPU VM.
We ran most experiments using a total of 5 such VMs.
J Additional Experiments
Multiple Seeds for Section 3 and 4 Experiments In Appendix Table 6, we show the mean and standard deviation for five seeds from evaluating Pre-trained T5, Random Init, LIME, and Set on the six downstream tasks.
For LIME and Set, for each different seed, we generated and pre-trained on a new synthetic dataset.
More Tasks for Section 5.1 Per Param Mean/SD Initialization In Appendix Table 7, we show results for the Section 5.1 Per Param Mean/SD init scheme for all six tasks, not just CNNDM-10K and MTOP which were the only two tasks shown in Table 2.
Multiple Seeds for Section 5.1 Per Param Mean/SD Initialization
On CNNDM-10K, MTOP, and SQuAD, we run the Per Param Mean/SD ininitialization experiments for five seeds.
Results are shown in Appendix Table 7.
We initialized using a model pre-trained on a different generated synthetic dataset for each seed.
More Tasks for Section 5.
Pre-attention Layer Norm Initialization Sweep In Appendix Table 8, we show results for the Section 5.3 pre-attention layer norm initialization sweep for all six tasks, not just CNNDM-10K and MTOP which were the only two tasks shown in Table 3.
Larger Model Size In Appendix Table
, we run experiments with T5 Base, which is a larger model size than T5 Small that was used for experiments in the main paper.
T5 Base has 220 million parameters with 12 encoder/decoder layers compared to 60 million parameters with 6 encoder/decoder layers for T5 Small.
LIME , Set , LIME Per_Param_Mean_SD , and Set Per_Param_Mean_SD close a large proportion of the gap to natural pre-training, similar to results for the T5 Small experiments.
[Table]
[Table] [Table]
[Table]
Set 32.8 (0.5) 71.7/93.7 (0.9/0.2) 73.6/88.6 (1.0/0.3) 48.2/60.0 (0.5/0.7) 58.8 (0.8) 40.5 (0.9) Identity 30.
[Table] [Table]
[Table] [Table]
[Table] [Table]