Tree Transformer’s Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects (2024)

Abstract: This work studies two types of ambiguity in natural language: prepositional phrase (PP) attachment ambiguity, and garden path constructions. Due to the different nature of these ambiguities – one being structural, the other incremental in nature – we pretrain and evaluate the Tree Transformer of Wang et al. (2019), an unsupervised Transformer model that induces tree representations internally. To assess PP attachment ambiguity we inspect the model’s induced parse trees against a newly prepared dataset derived from the PP attachment corpus (Ratnaparkhi et al., 1994). Measuring garden path effects is done by considering surprisal rates of the underlying language model on a number of dedicated test suites, following Futrell et al. (2019). For comparison we evaluate a pretrained supervised BiLSTM-based model trained on constituency parsing as sequence labelling (Gómez-Rodríguez and Vilares, 2018). Results show that the unsupervised Tree Transformer does exhibit garden path effects, but its parsing ability is far inferior to the supervised BiLSTM, and it is not as sensitive to lexical cues as other large LSTM models, suggesting that supervised parsers based on a pre-Transformer architecture may be the better choice in the presence of ambiguity.

## introduction one of the core challenges for natural language understanding (nlu) systems is that they must advertently deal with ambiguity; depending on the type of ambiguity, systems may or may not incorporate mechanisms for handling ambiguous input. a typical example of structural ambiguity, i.e. an ambiguity that is not resolved without external context, is that of prepositional phrase attachment. an often used example, with its two possible interpretations, is given below: here, the reader initially interprets the horse to be the subject of 'raced', only to revise this assumption after reading the final verb 'fell'. in this paper, we investigate to what extent transformer-based models are sensitive to these ambiguities.to that end, we rely on the language modelling and parsing components of a model; the language modelling component allows for measuring surprisal, and thus measuring the presence and sensitivity of garden path effects @xcite . the parsing component is required in order to measure a model's capacity to disambiguate cases of pp attachment. rather than assessing whether models can identify ambiguity, we investigate to what extent models can disambiguate by means of their parsing components. thus, we focus on the tree transformer model @xcite , which adds a constrained attention layer to the base transformer architecture to do unsupervised parsing while training on a language modelling objective. in doing so, the model allows us to evaluate the parser as well as the language model on the two different evaluation tasks. for comparison, we evaluate the parsing as sequence labelling approach of @xcite , where a bilstm-based language model is trained in a supervised fashion. as an extra baseline, we compare against an end to end setup with llm prompting @xcite . in terms of evaluation data, we rely on two sources: for measuring pp attachment ambiguities, we modernize the existing prepositional phrase attachment corpus @xcite , using a standard bert language model, to be more naturalistic and closer to the kind of data our parsers will have observed during training. second, for measuring garden path effects, we employ existing datasets from previous work @xcite . the results from our comparative evaluation not only assess the tree transformer's proficiency in parsing and language modeling but also provide a deeper understanding of its intrinsic mechanisms when handling complex grammar structures and ambiguity. we summarize our main contributions as follows: all our code and data is available online. foot_0 2 background pp attachment ambiguity due to the widespread presence of prepositional phrase attachment ambiguity in natural language, it has garnered significant attention from scholars. in early research, structure-based methods have been explored, such as the right association @xcite and minimal attachment methods @xcite , which despite their popularity due to simplicity, exhibit notable shortcomings and perform suboptimally in practical applications. additionally, statistics-based methods have been employed. @xcite introduced the first corpus-based co-occurrence statistical method, known as "lexical association". to address sparsity issues in these methods, @xcite proposed a back-off model and other work has applied wordnet @xcite classes @xcite . however, these methods are limited due to their specificity, making them cumbersome for practical applications. especially in the context of neural language modelling, the exploration of pp attachment ambiguity is limited. garden path effects on the other hand, garden path effects have been studied in the context of neural language models: van schijndel and linzen (2018a,b) demonstrate garden path effects in lstm models by simulating human reading times. surprisal theory @xcite proposed that observed slowdowns are a result of the unpredictability of each word appearing in a sentence. subsequently, the surprisal introduced by grammarbased language models has been shown to be correlated with reading time by @xcite . other work has shown that the surprisal in rnns is a powerful predictor of human reading times @xcite . van schijndel and linzen (2018a) demonstrated the ability of rnns to make reading time predictions comparable to grammar-based language models. in addition to validating the surprisal theory across different models, @xcite tested multiple lstms and rnng models to determine if they exhibit the garden path effect and observed their levels of surprisal in disambiguating sentences. their results serve as a baseline in our work. on the side of unsupervised parsing, researchers have explored various techniques, hoping that models could learn latent tree structures from unlabeled data without explicit syntactic annotations. among them, @xcite depicted the problem as a reinforcement learning task. in addition, there have been attempts based on recurrent neural networks, such as prpn @xcite , on-lstm @xcite , and tree-lstms @xcite , an variations like urnng @xcite and diora @xcite . more recently, adaptations of the base transformer architecture @xcite have been proposed for unsupervised parsing, like transformer grammars @xcite , and the tree transformer wang et al. (2019) which is our focus. category word verb v provide noun n 1 services preposition p for noun n 2 customers label l v ## results we initially validate the retrained tree transformer model and the pretrained bilstm model on the wsj-test data, giving the f1 scores in ## conclusion in this work, we prepared a novel prepositional phrase attachment ambiguity dataset, suitable for evaluating modern parsers on their capacity to recognize such ambiguities. we evaluated the tree transformer model, which combines an unsupervised parsing objective with masked language modelling. as such, we could evaluate both on prepositional phrase attachment ambiguity by analyzing the induced parse trees, as well as compare with prior work on measuring garden path effects through language model surprisal rate @xcite . in order to align with the datasets for our experiment, we trained a tree transformer from scratch and compared its performance with a supervised bilstm-based parser trained on constituent parsing as sequence labelling @xcite . our study reveals that the tree transformer model performs suboptimal in disambiguating pp attachments at the sentence level, and less effective than the bilstm-based model. primary factors are its poor ability to identify prepositional phrases and a tendency to attach prepositional phrases to verbs. we argue that this is likely to be caused by tree transformer's unsupervised nature. additionally, the tree transformer demonstrates garden path effects across multiple datasets and exhibits varying sensitivity to different subtle lexical cues, generally being more sensitive to the presence of an object than to specific verb form. overall, the evaluation suggests that an unsupervised model like tree transformer may be the lesser choice compared to a supervised model based on a pre-transformer architecture, when it comes to natural language ambiguities.