Improving BERT Pretraining with Syntactic Supervision (2023)

Abstract: Bidirectional masked Transformers have become the core theme in the current NLP landscape. Despite their impressive benchmarks, a recurring theme in recent research has been to question such models’ capacity for syntactic generalization. In this work, we seek to address this question by adding a supervised, token-level supertagging objective to standard unsupervised pretraining, enabling the explicit incorporation of syntactic biases into the network’s training dynamics. Our approach is straightforward to implement, induces a marginal computational overhead and is general enough to adapt to a variety of settings. We apply our methodology on Lassy Large, an automatically annotated corpus of written Dutch. Our experiments suggest that our syntax-aware model performs on par with established baselines, despite Lassy Large being one order of magnitude smaller than commonly used corpora.

## introduction in recent years, the advent of transformers @xcite has paved the way for high-performing neural language models, with bert @xcite and its many variants being the main exemplar @xcite @xcite . bert-like models achieve state-of-the-art scores in most major nlp benchmarks via a two-step process. first, they are trained on massive-scale, minimally processed raw text corpora by employing the so-called masked language modeling (mlm) objective. task-specific refinements are then obtained by fine-tuning the pretrained model on labeled corpora, usually orders of magnitude smaller in size. this pipeline, despite its attested performance, suffers from two key limitations. on the one hand, training a bert-like model from scratch requires an often prohibitive amount of data and computational resources, barring entry to research projects that lack access to either. on the other hand, a naturally emerging question is whether such models develop an internal notion of syntax. discovery of structural biases is hindered by their distributed, opaque representations, requiring manually designed probing tasks to extract evidence of syntactic awareness @xcite @xcite @xcite . alternatively, when syntactic evaluation becomes the focal point, it is usually deferred to downstream tasks @xcite , owing both to the lack of sufficiently large labeled corpora as well as the computational bottleneck imposed by hard-to-parallelize operations. in this work, we seek to alleviate both points by considering them in tandem. contrary to prior work, we consider the case of introducing explicit syntactic supervision during the pretraining process and investigate whether it can allow for a reduction in the data needs of a bert-like language model. to facilitate this, we couple the standard unsupervised mlm task with a supervised task, mapping each distinct word to a supertag, an abstract syntactic descriptor of its functional role within the context of its surrounding phrase. in essence, this amounts to simple token-level classification, akin to traditional supertagging @xcite , except for parts of the input now being masked. in employing both objectives, we ensure that our model is syntax-aware by construction, while incurring only a negligible computational overhead. we evaluate the trained model's performance in a variety of downstream tasks and find that it performs on par with established models, despite being trained on a significantly smaller corpus. our preliminary experiments suggest an improvement to pretraining robustness and offer a promising direction for cheaper and faster training of structure-enhanced language models. reflecting on the added objective, we call our model tagbert. ## background embedding structural biases in neural language models has been a key theme in recent research. most syntax-oriented models rely on computationally intensive, hard-to-parallelize operations that constrain their integrability with the state of the art in unsupervised language modeling @xcite @xcite . this can be ameliorated by either asynchronous pretraining, relying on accurate but slow oracles @xcite , or multi-task training, where the system is exposed to a syntactic task for only part of its training routine @xcite . in the bert setting, there have been attempts at modifying the architecture by either overlaying syntactic structure directly on the attention layers of the network @xcite or imposing shallow syntactic cues and/or semantic information in a multi-task setting @xcite . while such a setup allows for efficient parallel pretraining, the rudimentary nature of the utilized annotations typically forfeits fine aspects of sentential structure, such as function-argument relations. in this paper, we adopt lexicalism in the categorial grammar tradition @xcite @xcite @xcite , according to which (most of) the grammatical structure of a language is encoded in its lexicon via an algebra of types that governs the process of phrasal composition. under such a regime, the parse tree underlying a sentence can be partially (or even fully, in the case of an adequately "strict" grammar) recovered from its constituent words and their respective types alone. in applied terms, the lexical nature of categorial grammars provides us with the opportunity of capturing syntax in a fully-parallel fashion that is straightforward to incorporate with the masked language modeling objective of bert-like architectures, a fact so far generally overlooked by machine learning practitioners. this perspective is in line with recent insights arguing for the necessity of explicit supervision for syntactic acquisition @xcite . the only prerequisite for our methodology is an adequately sized, categorially annotated corpus. even though gold standard corpora exist for a variety of languages and grammars @xcite @xcite @xcite , their size is generally insufficient for training a parameterrich neural language model. this limiting factor can be counteracted by either lexicalizing existing silver-standard corpora of a larger size, or by using an off-the-shelf, high-performance supertagger to annotate the source data prior to pretraining. in both cases the trained system is likely to inherit common errors of the data-generating teacher; the question is whether the added structural biases facilitate faster training of more general language models, despite potential tagging inaccuracies. ## evaluation to evaluate the trained model, we measure its performance on the below selection of downstream tasks, after fine-tuning. we keep our fine-tuning setup as barebones as possible, using adam (kingma and ba, 2014) with a batch size of 32 and a learning rate of 3 × 10 -5 . we apply model selection based on the validation-set performance and report testset results (averaged over three runs) against the available baselines of each task in lassy small is a gold-standard syntactically annotated corpus for written @xcite . we fine-tune a pos tagger on the subset of the corpus that has been converted to universal dependency format @xcite . sonar-1 is a curated subset of lassy small that includes several layers of manually added annotations @xcite . we employ the named entity recognition (ner), part-of-speech (pos), semantic role labeling (srl) and spatio-temporal relation tags (str) that come packed with the corpus and treat their classification as downstream tasks. ner contains approximately 60 000 samples and 6 class labels encoded in the iob scheme. pos tagging contains about 16 000 samples and comes in two varieties: coarse (12 classes) and fine-grained (241 classes, out of which only 223 appear in the training data, many just once). srl comes also in two varieties: a) predicate-argument structures, encoded with iob scheme, and b) modifiers, enhanced with modified phrase labels, for a total of ## discussion our model performs on par across all tasks considered, indicating pretraining robustness comparable to the heavy weight baselines of bert- @xcite and roberta-based @xcite models. 3 considering the non-ideal nature of the silver-standard tags, the significantly smaller size of our corpus compared to competing models, and the ca. 30% reduced parameter count (79m against 119m for bertje and 117m for robbert), our results can be seen as strong evidence in favor of explicitly encoding structural biases in the pretraining process of neural language models. opting for a lexicalized representation of structure allows for a seamless and cost-efficient integration with bert's core architecture, essentially removing the computational bottleneck of alternating between tensor optimization and structure manipulation. ## conclusion we introduced tagbert, a variation of bert that is biased towards syntax through coupling the standard mlm loss with a supertagging objective. we trained tagbert on a modestly sized, silverstandard corpus of written dutch -after first lexicalizing its annotations -and evaluated the trained model on a number of downstream nlp tasks after fine-tuning. despite a reduced parameter count and the corpus' modest size, our method is achieving performance comparable to established stateof-the art models. this result is contrary to the ongoing trend of utilizing increasingly more data and augmenting model capacity, instead suggesting potential benefits from incorporating richer annotations in convenient representation formats. our work aims towards a syntactically-transparent, costefficient language model that combines both the rigor of formal linguistic theories and the representational power of large-scale unsupervised learning. retroactive placement the current work is situated in the historical landscape where probing for syntactic awareness and the possibility of injecting syntactic structures in the network were still novel enterprises. alongside and following our original endeavour, many more studies have investigated the role of syntax and ways to incorporate it within large language models, with the end-goal of either jointly acquiring the two, or of using explicit syntactic guidelines to constrain language generation @xcite @xcite @xcite @xcite . other than technical differences on the neural front, our work diverges in opting for a linearized representation of syntax through categorial grammars. this choice stands out for its elegance and formal coherence, setting it apart from more widely used alternatives like constituency trees and dependency arcs. indeed, categorial grammars (regardless of the particular flavor adopted) offer the means for an expressive, yet fully lexicalized, modeling of syntax and compositional meaning. their integration into the modern nlp toolkit is facilitated by this inherent flexibility, offering the potential for intricate interplay between structure and form -a potential that still remains, for the most part, untapped. future work given the embedding of this paper within the landscape concerning syntactic awareness and large language models, future work would be based on recent developments in neurosymbolic approaches to lexicalized grammar formalisms. for instance, recent developments in neural supertagging could be exploited, for instance by updating the supertagging from the discriminative setting to a constructive one @xcite . besides developments relevant to model architecture, several novel evaluation tasks for dutch have been developed that may shed light on the distinction between vanilla transformer-based models and syntactically informed ones. for example, the two-sentence classification task of natural language inference (nli) is a typical task that tests for lexical, syntactic, and sentence-level understanding, for which two dutch benchmarks exist @xcite . a comparison between a vanilla model, the syntactically informed tagbert, and the neurosymbolic approach of @xcite is in place to put the relationship of syntax and nli in perspective. further on the dutch front, we would be keen to test the model's ability to understand discontinuous verb-subject dependencies as in @xcite , or to disambiguate relative clauses as in @xcite . finally, we invite and look forward to different research directions, such as experimentation with different languages and grammar formalisms, integration with existing pre-trained models in an intermediate-training fashion @xcite and exploring architectural adjustments that would allow a two-way dependence or a stronger interfacing between the lexical and syntactic modalities.