Transfer of Structural Knowledge from Synthetic Languages (2025)

Abstract: This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark — a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

## introduction large language models (llms) are becoming increasingly powerful and useful. however, the role of data properties in model training and what exactly models learn from the training data remains to a large extent out of the scope of most llm papers. yet surprisingly pre-training a model on a simple algorithmic task can lead to improvements in natural language modelling @xcite . such insights can be used to improve the construction of data sets for language models. therefore, exploring the mechanisms of knowledge transfer is an important open question. scaling language models is a popular way to improve their performance foot_0 . however, as the detailed analysis in @xcite shows, the amount of data, especially high-quality text data, is limited and will become the main bottleneck in the coming years. such circumstances motivate research into more data-efficient learning algorithms and a better understanding of the mechanisms of generalization and transfer learning @xcite . after all, humans are exposed to orders of magnitude less data than modern frontier models, yet demonstrate strong performance across many domains and outperform machines in some areas, even considering recent algorithmic advances. inspired by this, @xcite demonstrate that training roberta @xcite on language acquisition data, together with some tweaks to model architecture and training, leads to 6000× gains in data efficiency. similarly, @xcite achieve significant model compression while retaining the ability to produce fluent and coherent english by using a generated dataset of stories for children, i.e. with small vocabulary and simple plots. and @xcite find that filtering for or generating data with higher educational value is also very helpful. thus, there is a growing body of evidence that the choice of data matters a lot and simply scraping the data from the web is suboptimal. however, there is a limited understanding of what properties of the data are important in different training stages. @xcite show that pretraining the lstm @xcite on structured but not linguistic data, such as midi music, java code, or even nested parentheses, reduces its perplexity when testing on spanish. @xcite find that removing all word order information from the pre-training phase does not significantly affect the final performance, given a fine-tuning phase with the correct word order. @xcite sample the training data from a completely artificial language consisting of random n-grams and observe that pre-training objectives that require processing this information somehow, such as copying sentences in the right order, still improve the performance of the model on summarization tasks compared to a randomly initialized version. however, research in this area currently tends to focus on reporting surprising observations rather than explaining them. papadimitriou and jurafsky (2020) illustrate such observations using figure 1 . this work attempts to build up on those observations and make a small further step studying the mechanisms of transfer learning. in this paper, we address the following research questions: our contributions are summarized as follows: first, as can be seen in the diagram above, different pre-training datasets, even if they all are unrelated to the target task, lead to different final performance. this suggests that some datasets are inherently more complex or more similar to the target language. we introduce a new synthetic "language" by combining ideas from the previous work and use it, as well as two existing synthetic datasets, to pre-train the models. we then fine-tune them on english using three different fine-tuning pipelines. we also provide an algorithm to assess the impact of the pretraining data on the resulting model parameters. second, since one of the settings for transfer learning involves fine-tuning only the embeddings, they are the natural target for investigation. we investigate the structure of the learned embeddings, namely the spectrum of their singular values to understand the effective dimensionality of the data, and the kmeans clustering of to check how uniformly the embeddings are distributed. to check what information is contained in the embeddings, we train linear probes to predict certain features of the words given their embeddings. linear probes are a popular interpretability technique, but to our knowledge they have not been used to study the embeddings of models pre-trained on different datasets and fine-tuned to the same task. finally, we evaluate the performance of these models in natural language understanding. since existing nlu datasets such as glue @xcite and mmlu @xcite are designed for more powerful models, we use gpt-4 (openai, 2023) to generate a similar benchmark consisting of 12 different subtasks foot_1 . ## related work one way of understanding the pre-training of language models is that we transfer some linguistic knowledge from a task with lots of available data to a downstream task @xcite . the recent findings suggest that this is not the only relevant effect, and sometimes not even the most important one. papadimitriou and jurafsky (2020), mentioned above, pre-trained an lstm on structured but not linguistic data and found that adapting such a model to spanish by fine-tuning only its input and output embeddings gave better perplexity than starting with a randomly initialised model. their results and experimental setup established a framework that has been used in subsequent work, including this one. @xcite improved these results replacing lstm with a transformer and changing the synthetic pre-training languages. papadimitriou and jurafsky (2023) used gpt-2 (radford et al., 2019). chiang and lee (2022) introduce a family of shuffle languages. @xcite used a similar technique to combine a task-specific corpus in english with a corpus in the target language unrelated to the task. such transfer also works in the opposite direction, from natural language to other domains. @xcite get performance comparable to training from scratch on different modalities by finetuning only the input and output embeddings of the pre-trained gpt-2. they also try different finetuning approaches, tuning the layer norm parameters and the last transformer block in addition to the input and output embeddings. @xcite show that pre-training moves the model parameters into a flat basin of the loss landscape and suggest it as a reason why pretrained models are less prone to catastrophic forgetting during fine-tuning. neyshabur et al. (2020) also observe this and also show that models finetuned from the same checkpoint stay in the same basin. however, past data alone is almost never enough to predict unseen data, unless one makes some assumptions, i.e. "inductive bias". a useful inductive bias can be injected into the model by pre-training on data that has it. @xcite use pre-training on natural languages with certain properties by model-agnostic meta-learning @xcite to find which biases are needed to quickly acquire these languages. @xcite design synthetic datasets requiring deduction, induction, and abduction and pre-train on them to extract inductive bias for general mathematical reasoning. @xcite pre-train models to simulate finite state transducers given their de-scription and achieve better generalization in nlp tasks with similar structure. ## synthetic languages in this paper we report a series of experiments with several synthetic languages. following hyperparameter choices from @xcite , for each of the languages described below, we use a sequence length of 512, a vocabulary size of 500, and generate 2 • 10 6 sequences so the total size of the dataset is approximately 10 9 tokens in each case. we focus on three synthetic languages: nested, the k-dyck nested bracket language; flat, the shuffle dyck language with no nesting; and flat_shuffle, a block-wise shuffled variant of the flat language. ## experiments for all experiments, we used the tinystories-8m model @xcite . ## conclusion we introduced a new synthetic language flat_shuffle, and the model pre-trained on it was shown to outperform the models based on the languages from previous work. investigation of the structure of the embeddings leads to a hypothesis that the reason behind the superior performance of some synthetic languages is that they require more structured embeddings, which causes the intermediate layers to be adapted to work with such embeddings, and in turn allows effectively using a higher dimension subspace of the embedding space during fine-tuning, which gives more flexibility. also, we haven't observed direct transfer of structure from synthetic languages to english, i.e. english tokens weren't splitted by the model into "opening" and "closing" ones. so it seems that models are working in a reservoir computing style where the computations for an unrelated task are adapted to the task at hand in arbitrary ways. at the same time, it means that the models are not strictly limited by the complexity or structure of the original task in transfer learning, and as long as they have enough complexity of computations, they can use it to adapt to the new task.