Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations (2024)

Abstract: Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

## introduction inductive biases play a critical role in nlp, particularly in learning from limited data and in systematic generalization beyond the training distribution. while standard seq2seq models excel on in-distribution data, they often lack structural inductive biases and hence perform poorly on structural generalization, i.e. generalization to unseen combinations of known phrases @xcite , extrapolation to longer inputs @xcite and deeper recursion @xcite . while pre-training on large amounts of text improves structural generalization to a certain extent @xcite , it remains challenging @xcite . this seems to conflict with observations that pre-training equips models with knowledge about syntax @xcite @xcite , which should enable structural generalizations. in this paper, we start from the hypothesis that the lack of structural inductive bias is partly due to limited knowledge of how to use syntactic information for structural tasks. traditionally, nlp has heavily relied on syntactic theories and has phrased many tasks as transformations of syntax trees, ranging from conversion of a sentence from active to passive voice @xcite to constructing a semantic representation for a sentence @xcite . transformations of syntax trees can address a task in a very generalizable way by using the right abstractions. for example, when constructing the semantic representation of an np, by the principle of compositionality, the same transformations can be used for nps whether they serve as direct objects or as indirect objects. inspired by this perspective, we propose a new method of strengthening the structural inductive bias of a pre-trained model with an additional intermediate pre-training step to perform syntactic transformations (see fig. 1 ). we create a dataset of automatically generated syntactic transformations of english dependency trees. given a description of the transformation as a prefix and an input sentence, the model is pre-trained to predict the output of the transformation without access to the underlying dependency tree. this pre-training procedure encourages the model to strengthen its representations of syntax and acquire reusable dynamics of syntactic transformations that can be leveraged for downstream tasks. during fine-tuning, goldstandard descriptions of transformations are not available, and we use a prefix of embeddings that are fine-tuned with the rest of the model instead. contributions we demonstrate that our intermediate pre-training strengthens the structural inductive bias of the model, resulting in a better few-shot performance for syntax-dependent seq2seq tasks, such as conversion from active to passive or chunking. our method also improves structural generalization in the context of semantic parsing. analysis of the pre-trained model shows that it uses attention heads to track what transformation needs to be applied to which input token, and that these heads tend to follow syntactic patterns. in addition, we find that fine-tuning re-uses these attention heads, suggesting that the model can leverage the transformations acquired during pre-training. ## related work pre-training with synthetic data training on synthetic data to shape the inductive bias of transformers has been explored in several recent works. @xcite pre-train on a synthetic language to investigate the impact on language modelling of english. mccoy and griffiths (2023) pre-train on a distribution of tasks using meta-learning @xcite and show improvements for low-resource language modelling of child-directed language. our work builds conceptually on sip @xcite , in which a transformer is pretrained to simulate the behaviour of finite state transducers (fsts) to introduce a structural induc-tive bias for fst-like behaviour. that is, given a representation of an automatically generated fst and an input string, a transformer is pre-trained to predict what the output of the fst is on the given input. during fine-tuning, sip uses a prefix of tunable embeddings in place of an fst description. while sip and this work share similar methodology, i.e. pre-training a model with a description of a transformation and fine-tuning the model with a prefix of tunable embeddings, they address different problems: sip focuses on the sequential inductive bias of fsts, whereas the present work strengthens the inductive bias for transformations of syntax trees. another major difference is that sip's pre-training task is fully deterministic and unambiguous as there is only a single output for any fst and input string. in contrast, here, performing the transformation requires knowledge about the underlying syntax tree, which is not provided to the model. this forces the model to learn the syntax or reuse its existing syntactic knowledge. ## strengthening structural inductive bias standard pre-training objectives, e.g. with denoising objectives @xcite , encourage models to acquire syntactic knowledge but provide little information about syntactic transformations, which are central to many syntactic and semantic seq2seq tasks. our research hypothesis is that intermediate pre-training to perform transformations of syntax trees encourages the model to (i) strengthen its representations of the syntactic categories to which transformations can be applied (e.g. subjects, objects) and (ii) acquire reusable dynamics of transformations that are useful for downstream applications. by providing an explicit description of the transformation as a prefix, different transformations that the model has learned during pretraining can be 'activated' by the right choice of prefix. for this reason, we also fine-tune the model with a prefix of tunable embeddings to make it easy to leverage these transformations on downstream tasks similar to sip @xcite . in addition to learning about transformations of trees, we also want the model to incorporate knowledge about the syntax of the underlying language (i.e. english, in this case). hence, we do not provide syntax trees to the model during training, which also enables us to perform inference and fine-tuning without a parser. ## evaluation we evaluate on syntactic and semantic tasks for which a structural inductive bias should be helpful. specifically, we consider learning from a small amount of task-specific data (few-shot learning) and structural generalization outside of the training distribution to unseen combinations of known phrases, novel syntactic phenomena and deeper recursion than seen during training. ## analysis our research hypothesis is that our intermediate pre-training encourages the model to acquire reusable dynamics of syntactic transformations that can be leveraged during fine-tuning. in this section, we analyze the representations used by our model after its intermediate pre-training, and to what degree they are reused during fine-tuning. ## conclusion we propose a new method of strengthening the structural inductive bias of a transformer by pretraining the model to perform syntactic transformations based on dependency trees. we show that this results in a better few-shot performance for syntax-dependent seq2seq tasks, and also improves structural generalization for semantic parsing. analysis of the pre-trained model shows that it uses attention heads to track what transformation needs to be applied to which input token, and that these heads tend to follow syntactic patterns. in addition, we find that fine-tuning re-uses these attention heads, suggesting that the model can leverage the transformations acquired during pre-training.