Using Roark-Hollingshead Distance to Probe BERT ’s Syntactic Competence (2022)

Abstract: Probing BERT’s general ability to reason about syntax is no simple endeavour, primarily because of the uncertainty surrounding how large language models represent syntactic structure. Many prior accounts of BERT’s agility as a syntactic tool (Clark et al., 2013; Lau et al., 2014; Marvin and Linzen, 2018; Chowdhury and Zamparelli, 2018; Warstadt et al., 2019, 2020; Hu et al., 2020) have therefore confined themselves to studying very specific linguistic phenomena, and there has still been no definitive answer as to whether BERT “knows” syntax. The advent of perturbed masking (Wu et al., 2020) would then seem to be significant, because this is a parameter-free probing method that directly samples syntactic trees from BERT’s embeddings. These sampled trees outperform a right-branching baseline, thus providing preliminary evidence that BERT’s syntactic competence bests a simple baseline. This baseline is underwhelming, however, and our reappraisal below suggests that this result, too, is inconclusive. We propose RH Probe, an encoder-decoder probing architecture that operates on two probing tasks. We find strong empirical evidence confirming the existence of important syntactic information in BERT, but this information alone appears not to be enough to reproduce syntax in its entirety. Our probe makes crucial use of a conjecture made by Roark and Holling-shead (2008) that a particular lexical annotation that we shall call RH distance is a sufficient encoding of unlabelled binary syntactic trees, and we prove this conjecture.

## introduction bert @xcite is a structurally rich system both because of its neural architecture and because of the knowledge it captures through pretraining. many have studied the similarities between these architectures and linguistically moti-vated structures or processing pipelines through probing @xcite @xcite @xcite to argue for an enigmatic "bert revolution," into which large parts of previous research in computational linguistics are now subsumed. in this paper, we present rh probe foot_0 , a novel encoder-decoder-based probing architecture that utilises @xcite syntactic distance (rh distance) to examine the overall capacity of bert's comprehension of syntax. we introduce this probing architecture because there still has been no definitive answer as to whether the knowledge that bert acquires during its pre-training process can in fact serve as a replacement for a phrase-structure-based notion of syntax. limited by their probing methodology, prior performance-based probing attempts only investigated individual, fine-grained linguistic phenomena, until @xcite . @xcite proposed perturbed masking, a parameter-free probing method that directly looks for traces of well-established linguistic structures such as constituency trees latent in bert representations. the baseline used in @xcite , a right-branching tree, is overly simplistic. upon conducting a thorough reappraisal of their results, we found that their parameter-free constituencytree parser only marginally outperforms a naïve, right-branching baseline, and their impact matrix which formed the basis of their analysis only has weak to no correlation with constituency tree information. we believe @xcite has overly fixated on parameter-free probing. they want to address the criticism that, because traditional supervised probes introduce supervised data, we cannot directly attribute evidence of "knowledge" to the pre-trained base language model itself -perhaps everything is learned post hoc by the classifier through the probe dataset @xcite . but this risk can be mitigated. in particular, the ablation study is still a valid experimental design for probing @xcite . in our view, the only satisfactory evidence of "knowledge" that there has ever been in artificial intelligence has been the ability to use the model in question to perform inference. the present case should be no different. what we need then is parsing, or some other derivative task, which is easy to perform given a sufficient grasp of syntactic structure and much more difficult to perform without it, but in an experimental setting in which several sources of information are provided as input. bert's output is one such potential source of information. ablation studies then take the form of removing one or more of those sources during training. we also need alternative encodings of that knowledge for the purposes of comparison. @xcite annotates word tokens in sequence but has been conjectured to encode unlabelled syntactic constituency trees. it is used by prpn @xcite as an important component of its internal model of tree structure that the network develops in the course of learning to parse unsupervised. rh distances are an important bridge between token sequences (language model) and arboreal structure (traditional syntax). therefore, we propose rh probe, an encoderdecoder-based probing architecture. with rh probe, we conduct two experiments: (1) an ablation probe that observes whether the removal of a feature source during training (language model output, rh distance, or part-of-speech (pos) information for reference) can decrease the probe's performance; and (2) an "attack" probe that observes whether randomization of certain features during testing can cause the performance to drop. the results of these experiments suggest that word embeddings contain important syntactic information, but that this information alone is not enough to reproduce traditional syntactic representations such as phrase structure in their entirety. in our experiments, we have also not been able to find any correlation between the quality of the language model and the amount of syntactic information. after surveying previous probing methods, including probability probes, performance-based probes, and @xcite parameter-free probe, we introduce our rh probe architecture and prob-ing task design, including the definition of rh distance. we prove @xcite conjecture that rh distance encodes unlabelled binary syntactic trees, and then present two experimental trials that use rh probe to study the syntactic information carried by various sources. our analysis of these results broadly confirms the existence of important syntactic information within bert, but militates against the conclusion that this information alone can reproduce syntax in its entirety. ## experimental setup constituency trees were obtained from the penn treebank (ptb) @xcite . the corpus was processed and split into training/validation/test sets using @xcite scripts, available online 3 . trees were binarized, as usual, but unary branches except for the pre-terminal connections were removed, in accordance with the limitations on the expressive capacity of sequences of rh distances. a hyperparameter search was conducted using optuna @xcite for 150 trials with bert-base-cased embeddings and no rh distances nor pos tags were provided as hints. the search result concluded with a 50 ptb-tag embedding size, a 0.0005 adam @xcite learning rate, and a 5-layer gru encoder-decoder with a 350 hidden-unit size. finally, a beam search algorithm was implemented in the decoder with a width of 5. this set of hyperparameters was used for every probe conducted here. this hyperparameter selection process can be generalised to other large-scale pretrained language models. ## conclusion does bert know syntax? our rh probe results demonstrate that bert contains important syntactic information, although this information alone cannot reproduce syntax in its entirety. our empirical evidence is not subject to the criticism that the observed syntactic knowledge is not obtained by the language model through pretraining, but rather emerges from the probe classifier itself. the evidence is drawn from two carefully designed tasks, in which the substantial performance changes can be observed. nevertheless, we strongly agree with the insight @xcite ) that the internal structure of bert, although it may not embody a complete syntactic characterization of its input, can provide useful information for downstream applications. recent probing attempts @xcite @xcite have proved that the utility of knowledge acquired through pretraining is not limited to lexical semantics -word embedding also encodes syntax-adjacent information. our probing results reinforce this argument through a more general examination of the relation between bert and syntax.