Discontinuous Constituency and BERT : A Case Study of D utch (2022)

Abstract: In this paper, we set out to quantify the syntactic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite based on a mildly context-sensitive formalism, from which we derive grammars that capture the linguistic phenomena of control verb nesting and verb raising. The grammars, paired with a small lexicon, provide us with a large collection of naturalistic utterances, annotated with verb-subject pairings, that serve as the evaluation test bed for an attention-based span selection probe. Our results, backed by extensive analysis, suggest that the models investigated fail in the implicit acquisition of the dependencies examined.

## introduction assessing the ability of large-scale language models to automatically acquire aspects of linguistic theory has become a prominent theme in the literature ever since the inception of bert @xcite and its many variants, largely due to their unanticipated performance. standard practice involves attaching bert to a shallow neural model of low parametric complexity, and training the latter at detecting various linguistic patterns of interest, revealing in the process the amount to which they are encoded within bert's representations. the consensus points to bert-like models having some capacity for syntactic understanding @xcite . their contextualized representations encode structural hierarchies @xcite that can be projected into parse structures, using linear (hewitt and manning, 2019) or hyperbolic transformations @xcite , from which one can even obtain an accurate reconstruction of the underlying constituency tree @xcite . despite their broadening scope, a latent bias persists in the insights provided by the probing literature, due to its focus being, by default, on en-glish. english, albeit boasting a rich collection of evaluation resources, is characterized by a simple grammar with relatively few complications over the syntactic and morphological axes. specifically when it comes to syntax, english lies in close proximity to a context-free language, a class characterized by its low rank in terms of formal complexity and expressive power @xcite . perhaps more importantly, several commonly used evaluation test beds, including the penn treebank @xcite , are in themselves contextfree, muddying the territory between probing for acquired syntactic generalization and arbitrating pattern extraction. as such, claims about the syntactic skills of language models should not be assumed to freely transfer between languages (and, in some cases, even datasets). in this paper, we seek to evaluate bert in the face of patterns that go beyond context-freeness. we employ a mildly context-sensitive grammar formalism to generate complex patterns that do not naturally occur in english. we choose instead to experiment on dutch, a language long-argued to be non-context free, due it its capacity for exhibiting an arbitrary number of cross-serial dependencies. in dutch, cross-serial dependencies arise in sentences where verbs form clusters, causing their respective dependencies with their arguments to intersect when drawn on a plane: figure 1 portrays an adaptation of the example of @xcite . to that end, we first identify two well-studied constructions in dutch that commonly involve cross-serial dependencies: control verb nesting and verb raising. we produce an artificial but naturalistic dataset of annotated samples for each construction; each sample contains span annotations for the verband noun-phrases occurring within, as well as a mapping that associates each verb to its corresponding subject. we then implement a probing model intended to select a verb's subject from a number of candidate phrases, train it on a gold-standard resource of dutch, and employ it on our data. our experimental results convey a rapidly declining performance in the presence of discontinuous syntax, suggesting that the dutch models investigated do not automatically learn to resolve the complex dependencies occurring in the language. to facilitate further research on the topic, our code is publicly available online. 1 2 background ## conclusion we implemented a test suite based on multiple context-free grammars to generate a large collection of sentences containing complicated syntactic phenomena specific to dutch. we trained a probing model on extracting verb-to-subject pairings from the contextualized representations of state-of-theart pretrained dutch language models using an external resource of generic text accompanied by gold standard annotations. we then tested the probe on our generated data, and found it to perform substantially below its own validation benchmarks. after conducting extensive analysis aimed at identifying the source of this discrepancy, we showed that the probe's predictions are inconsistent and its accuracy quickly diminishes as the complexity of the syntactic patterns increases. based on the above, we conclude that neither of the bert models investigated has learned to internalize syntactic and semantic subjecthood in nested constructions involving cross-serial dependencies. our findings serve as empirical evidence hinting at unsupervised language models having difficulty in the automatic acquisition of discontinuous syntactic patterns. we leave several directions open for future work. to begin with, one could mirror the patterns analyzed to other languages and compare model performance cross-linguistically, juxtaposed by the corresponding grammar complexity. alternatively, one could render more elaborate grammars intended to capture other syntactic or semantic phenomena of interest. finally, it is worth investigating the extent to which the "real-world" validation samples incorrectly classified are exemplars of the types of discontinuity captured by our grammars.