Evaluating distillation methods for data-efficient syntax learning (2025)

Abstract: Data-efficient training requires strong inductive biases. To the extent that transformer attention matrices encode syntactic relationships, we would predict that knowledge distillation (KD) targeting attention should selectively accelerate syntax acquisition relative to conventional logit-based KD. To test this hypothesis, we train GPT-2 student models on datasets ranging from 10K to 5M sentences using both distillation methods, evaluating them on both syntactic benchmarks and perplexity. Surprisingly, while logit-based KD dramatically improves data-efficiency, attention-based KD provides minimal benefit even for syntactic tasks. This suggests that output distributions provide sufficient supervisory signal for syntax acquisition, indicating that syntactic knowledge may be distributed throughout the network rather than localized in attention patterns.

## introduction modern language models successfully capture many aspects of human linguistic abilities, from the fundamentals of grammar @xcite @xcite to more sophisticated uses of world knowledge @xcite . however, they achieve these capabilities only after training on vastly more data than human children receive during language acquisition (frank, 2023), motivating research into inductive biases @xcite , predispositions that guide learning toward particular solutions with less data. these biases include architectural modifications @xcite , curriculum learning strategies @xcite , training objectives @xcite , and specialized weight initialization techniques @xcite . figure 1 : we compare knowledge distillation through logits and attention matrices as a form of inductive bias. we hypothesize that syntactic structures reflected in attention could provide an inductive bias that selectively enhances the student's performance in syntactic phenomena. in this paper, we use knowledge distillation to study which aspects of a model's learned representations are most critical for scaffolding particular linguistic capabilities. we focus specifically on learning syntax -an ability classically theorized to require strong (innate) biases @xcite . previous research has shown that syntactic information is encoded in the attention mechanism of transformer models @xcite , and that constraining these attention matrices can serve as an effective inductive bias for syntax @xcite @xcite . if syntax is primarily localized in attention matrices, then kd targeting these representations should be sufficient to transfer syntactic abilities, potentially even outperforming conventional logit-based distillation. moreover, if attention serves as the primary locus of syntactic information, we would predict that attention-based kd should show selective advantages for syntactic tasks relative to general language modeling performance, providing a targeted inductive bias for syntax acquisition under data-limited conditions. in other words, we use distillation as an analytical tool to probe whether syntactic knowledge is indeed localized in specific model components or distributed more broadly throughout the network (figure 1 ). to investigate these two possibilities, we conducted controlled experiments using a pretrained gpt-2 model @xcite as teacher, training architecturally-identical students on datasets ranging from 10k to 5m sentences. our contributions are twofold. first, we demonstrate that conventional logit-based distillation drastically reduces the amount of data required for learning syntax, reaching teacher-level performance with only 500k sentences of training data. second, surprisingly, attention-based kd provides limited benefit for syntactic tasks despite prior evidence that these matrices encode syntactic structure. our work illustrates how knowledge distillation can serve as a powerful analytical tool for understanding which aspects of a model's representations are effective for achieving data-efficiency with respect to specific linguistic capabilities. ## approach we ask whether distillation through attention provides a stronger inductive bias for syntax acquisition compared to conventional distillation through logits. to investigate this question, we conducted controlled experiments using an identical gpt-2 small architecture @xcite for both the teacher and student models. we focused on gpt-2 small as it is the smallest available pretrained language model with adequate performance, enabling feasible experiments. the teacher model was a fully pre-trained gpt-2, while the student models were trained from scratch on different subsets of the babylm dataset @xcite , ranging from 10k to 5m sentences. by varying the dataset size, we assessed how different distillation methods affect data efficiency. all results reported are averages across three random seeds. complete training details are provided in appendix a. ## results before testing the effects of kd on syntactic performance, we check how well each kd approach achieves its objective. as shown in figure s2 , logitbased kd successfully enables the student model to achieve a much lower kl divergence from the teacher model, and attention-based kd enables the student model to achieve a much more similar attention pattern to the teacher model. next, we turn to our main question: how does each kd method affect the linguistic abilities of the student models? ## discussion our results reveal a striking contrast in the ability to improve data-efficiency among different kd methods. while kd via logits enabled student models to achieve teacher-level syntactic performance with just 500k sentences, kd via attention matricesdespite their capacity to encode syntactic structures -offered only marginal benefits. our results contrast with previous studies showing the benefits of attention distillation @xcite . we identify two potential sources of this divergence: the evaluation metric, and the training objective. first, we specifically focused on syntactic task performance, whereas prior work used more general benchmarks such as squad and glue. this raises the possibility that attention patterns may actually be more important for semantics than purely syntac-tic processing. second, previous studies focused on model compression (reduction in parameter counts), which is different from our objective (reduction in training set size, using identical model architectures for the teacher and student). for the compression objective, explicit attention alignment may help compensate for reduced capacity, but may be unnecessary (or hurt performance) when the size of the student model is equal to the teacher model. one key advantage of kd is that it requires minimal assumptions about the specific form of inductive biases. in fact, our results demonstrate that strong syntactic performance can be achieved without relying on explicit grammatical rules. on the other hand, kd-based approaches present certain challenges. kd can be computationally intensive, requiring forward passes through the teacher model for the entire training dataset (see appendix b for the rough estimates), and the inductive biases transferred via kd are less interpretable than those from explicit grammar-based approaches @xcite . taken together, our findings highlight how feature-based kd can serve as a powerful analytical tool to investigate which features are necessary or sufficient for specific capabilities. effective distillation through a particular feature suggests that it contains enough information to serve as an inductive bias for the target capability. our results suggest that the information contained in attention matrices was not a strong enough inductive bias for syntax acquisition, but future work must systematically compare different feature-based kd methods to better understand how different linguistic competencies are encoded within transformer representations.