Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of D utch Clause Relativization (2023)

Abstract: This paper addresses structural ambiguity in Dutch relative clauses. By investigating the task of disambiguation by grounding, we study how the presence of a prior sentence can resolve relative clause ambiguities. We apply this method to two parsing architectures in an attempt to demystify the parsing and language model components of two present-day neural parsers. Results show that a neurosymbolic parser, based on proof nets, is more open to data bias correction than an approach based on universal dependencies, although both set-ups suffer from a comparable initial data bias.

## introduction ambiguity pervades natural language and as such forms one of the central challenges for natural language understanding (nlu) systems. given the fact that most such systems rely on large-scale deep learning architectures, the presence of structural biases in the training data used may affect a system's capacity for disambiguation. specifically in the case of parsing, typical architectures rely on the assumption of just a single correct parse, although many may exist. this assumption then may force a bias into the training, both on the lexical and on the syntactic level. in this paper, we study syntactic ambiguities in dutch, where a structural ambiguity affects the interpretation of relative clauses. the preferred reading as subject or object relativisation will typically be determined by lexical choice. a running example in dutch, with its two possible interpretations in english, is given below: in this example, the verb 'cure' displays a strong selectional preference for a doctor as its subject and a patient as its object, so one would expect that the subject-relative interpretation (b) is the preferred reading. however, grounding the phrase in a prior sentence disambiguates the ambiguous relative clause: by prepending the phrase "the patient cured the doctor", one is led to infer that the object-relative reading of (c) is in fact preferred. in extreme cases, one finds examples of lexical choice that block one of the readings, typically due to the semantic class of the subject and object being different. an example is "de man die de boterham eet" (the man who eats the sandwich), where it is semantically implausible for the verb's arguments to be reversed. the situation is summarized in arguing that a parser typically exploits statistical properties of its training corpus, but should additionally rely on both lexical and syntactic cues inside of that corpus, we carry out an experiment test a parser's capacity for disambiguation in context. specifically, we extract a set of selectional preferences for dutch transitive verbs, that we classify according to their reversibility, i.e. whether subject and object can be interchanged, additionally indicating whether there is a strong preference for a noun as subject or object. this leads to three classes of (s, v, o) triples that we then use to generate a test set of dutch relative clauses together with prior sentences, to test a parser's capability of disambiguation in context. by evaluating two different parsing regimes that both are built on top of a language model, we investigate the encoding of structural bias in the parser training data, the possibility of mitigating structural bias, and attempt to pinpoint to what extent lexical knowledge or syntactic information is employed by the parsers in question. our contributions in this paper are therefore threefold: we (1) provide a dataset of selectional preferences for dutch (s, v, o) triples with an addi- ## background probing and syntactic sensitivity previous work has used probing, where a small neural network is attached to a large language model to extract taskspecific information, to argue that large scale language models like bert have internalized some linguistic knowledge during pretraining @xcite , and there appears to be some consensus of the syntactic awareness of bert models @xcite . specifically, studies have indicated the possibility of extracting parse trees from bert representations succesfully @xcite . another line of research into syntactic sensitivity investigates the probabilities of language models in a masked language modelling environment, where studies typically define surprisal rates to measure the degree to which the language model's predictions coincide with human-like behaviour in the face of syntactic ambiguities @xcite @xcite , typically focusing on garden-path effects. a related approach uses priming to investigate the language models' response to structurally 1 https://github.com/gijswijnholds/relpron_disambiguation similar sentences @xcite . these studies differ from the current paper in that we explicitly target sentences that are not disambiguated without adding extrasentential context, which should for a parser to infer the intended syntactic analysis. hence, our setup relies on a probing-like paradigm where we evaluate a parser on top of a language model. dutch nlp the rising interest in large scale language models in the nlp community has led to a number of investigations for dutch specifically. two dominant dutch language models have been developed, based on the respective bert @xcite ) and the roberta @xcite architecture @xcite ). on the evaluation side there have been several studies using dutch-specific phenomena to evaluate the respective monolingual language models: the work of @xcite introduces a parallel natural language inference (nli) dataset for dutch, showing that dutch nli is more difficult to tackle than its original english version. more recently, a range of probing studies has been performed investigating verb-subject dependencies in the face of syntactic constructions involving discontinuity @xcite and ellipsis @xcite , aside from a more subtle dutch nli challenge @xcite . more closely related to constructions involving relative pronouns is the work of allein et al. (2020), which introduces relative pronoun prediction for dutch die and dat as a binary classification task, where the surrounding context determines the choice of the neuter (dat) or non-neuter (die) pronoun. this can be modelled as a masked language modelling task @xcite , reaching high performance. a more complex variation of this task is defined by @xcite , where the experiment investigates the language model's capacity to predict relative pronoun attachment. against these studies, our approach differs in that we do not directly target the language model probabilities, but rather investigate the attached parsers, thereby indirectly assessing the contextualization power of the underlying language model. in that sense, the current work is more in line with prior, more theoretical work that takes parser ambiguity into account @xcite @xcite . ## data generation in order to generate a suitably large set of relative clause patterns, we proceed with a pipeline for extracting suitable subject-verb-object triples from a large corpus. first we perform a probability-based extraction of base triples. then we use lexical information from a dictionary to filter out relevant nouns and classify them according to their seman-tic class, after which in a final step we perform a manual filtering and classification of the obtained subject-verb-object triples to guarantee correctness. triple extraction for the first step, we need a way to extract subject-verb-object triples from a large corpus. to this end, we iterate over lassy large @xcite -a 700m word corpus with automatically assigned syntactic annotations -and extract all the cases of transitive verbs and their respective subjects and objects. to compensate for parsing errors and infrequent observations, we rely on posterior probability @xcite , the simplest measure that was shown to give best performance on a selectional preference acquisition task @xcite . posterior probability allows us to easily filter out those triples that occurred most frequently, allowing us to consider the most canonical triples only. for a given (s, v, o) triple, its posterior probability is defined as follows: dictionary-based classification after gathering all (s, v, o) triples and removing stopwords, we perform a dictionary-based filtering and classification in two steps. first, we obtain an exhaustive list of dutch nouns with their respective semantic class foot_0 from the algemeen nederlands woordenboek foot_1 , a comprehensive online dictionary of dutch. we use these categories to classify all the subjects and objects in the extracted (s, v, o) triples, after which we organize triples according to the most frequent noun categories, preventing as much as possible infrequent or illicit combinations of a verb with a given pair of nouns. by organizing triples by whether subject and object fall into the same semantic category, we have an initial estimate of whether a triple is irreversible, and in cases where the semantic category of the subject and object coincide, we estimate how strong the preference of the verb for the particular subject and object is based on the frequency of the (s, v, o) triple relative to its inverted (o, v, s) triple. manual filtering this initial estimation gives us a comprehensive list of subject-verb-object triples, that we finally filter manually, making a per case decision on whether a triple is irreversible, reversible with strong preference for the given subject and object, or reversible with a weak such preference, meaning that object and subject could be swapped without leading to an implausible triple. an example of the latter is "de toerist herkent de reiziger" (the tourist recognizes the traveller). ## parsing regimes in our experiments we want to evaluate parsers that were built on top of a large language model, in order to distinguish the effect of the language model from that of the specific parsing strategy employed. for comparison purposes we test two different parsing regimes. ## evaluation & results in our experiments, we test the parsers on three different scenarios: first, we inspect the existing structural bias present in the parsers due to training data statistics, in a setting where the parser only gets fed the relative clause. in such a setting, we would expect the parser to have a strong bias for irreversible cases, but an even distribution in accuracy on weakly reversible cases. next, we observe the effect of parsing the relative clause when the underlying language model is allowed to contextualize against the prior sentence, expecting this to aid the parser in assigning the correct reading. in this setting, we would ideally hope for high accuracy across the board. finally, we examine the effect of additionally finetuning the parser components on a small amount of training data to see if the parser can pick up on the task. ## discussion in the setup of our experiments, we were careful to develop test cases that target the parsers in a few different settings. rather than expecting high performance overall, the aim of the experiment is to both measure the prevalence of structural bias in the parser, as well as measuring to what extent such bias can be mitigated, if present. as such, the experimental results shouldn't be taken as proof that the parsers in themselves are necessarily insufficient. rather, it should be taken as a point to argue that parsers should generally take ambiguity into account, and while lexical ambiguity can be addressed by means of a neural model iterating over a balanced training dataset, the inability to accommodate syntactic ambiguity, both in the training corpora used, as well as in the parser architectures involved, poses a problem that our experiments confirm. selectional preferences readers may be critical about the choice of a ternary definition of reversibility since selectional preferences are considered to be graded, expressed in terms of degree of plausibility of a given combination of verbs and nouns. indeed, the selectional preference dataset of @xcite considers degrees of plausibility. it is important to note that such a modelling is incompatible with our current experimental setup, and exactly for the reason we outline above, that the parsers by default assume a single output parse and the disambiguation experiment intends to pinpoint exactly one such parse. allowing a parser to express a probability distribution over parse trees would allow one to more closely match the way selectional preferences are modelled, but that is (unfortunately) outside of the scope of this work. finetuning aside from the methodological viewpoint above, one may argue that we need not care about the presented experiment as we could simply finetune underlying language model together with the parser on top and achieve high performance on the experiments. while it is true that finetuning the bert model alongside the parser leads to peak performance, this is most likely due to the language model picking up on positional information quickly, and not due to the model becoming a better parser. to back this claim, we measure parser accuracy metrics on the original training corpus, for finetuned models that only adapt the parser or include the underlying language model. the metrics are displayed in ## conclusion by introducing a synthetic test set of naturalistic dutch relative clauses, we carried out an experiment to investigate the sensitivity to structural bias of two parsing architectures that are both based on a bert-style language model. the experiments show that both parsers pick up on a structural preference for a subject-relative reading of the relative clause, following a strong statistical bias coming from the data they were trained on. further experimentation shows that a more complex neurosymbolic parsing regime adapts more easily to a bias correcting finetuning setup than a universal dependencies parser implemented as a sequence labelling model. however, in both cases performance on the task is severely below expectation, and we hope that this work inspires further work on careful data augmentation and parser development.