Resolving Prepositional Phrase Attachment Ambiguities with Contextualized Word Embeddings (2021)

Abstract: This paper applies contextualized word embedding models to a long-standing problem in the natural language parsing community, namely prepositional phrase attachment. Following past formulations of this problem, we use data sets in which the attachment decision is both a binary-valued choice as well as a multi-valued choice. We present a deep learning architecture that fine-tunes the output of a contextualized word embedding model for the purpose of predicting attachment decisions. We present experiments on two commonly used datasets that outperform the previous best results, using only the original training data and the unannotated full sentence context.

## introduction prepositional phrase (pp) attachment is a subproblem of natural language parsing in which the objective is to determine the likely attachment site of the preposition. the attachment site should correspond to the preferred semantic interpretation of the sentence. i bought a computer with a gpu in the above example, the preposition with could attach to bought or computer. the likely interpretation is that gpu is a sub-component of computer, which implies an attachment to computer. ## previous work early work on this task uses relationships between head words of the phrases involved in the attachment decision. @xcite predict attachments using co-occurrence statistics between the preposition and the candidate heads, drawn from an automatically built corpus of partial parses. @xcite ; @xcite ; @xcite use a wider variety of machine learning techniques to learn the attachment decision from annotated tuples of head words. later work uses a variety of external data sources to help with the attachment decision. e.g., @xcite use features from wordnet @xcite while @xcite use features from framenet @xcite and co-occurrence statistics drawn from the world wide web. more recent work uses word embeddings and neural models. @xcite explore a number of neural composition architectures to combine the embeddings from the head words, while @xcite ## data in this work, we report results on the englishlanguage data sets from @xcite and @xcite , henceforth referred to as the rrr and blbg data sets, respectively. both datasets were extracted from the penn treebank @xcite . each example in the rrr data set foot_0 consists of a tuple (h 1 , h 2 , p, n2, l), where h 1 is the candidate verb head, h 2 is the candidate noun head, p is the preposition, n2 is the noun head child of the preposition, and @xmath0, v } is the label that indicates a noun or verb attachment. we map {n, v } to {2, 1} for consistency with the blbg format, described below. each example in the blbg data set foot_1 consists of a tuple (h 1 . . . h n , p, n2, l), in which h 1 . . . h n are a list of candidate noun and verb heads, p is the preposition, n2 is the noun head child of the preposition, and @xmath1. . . n} denotes the index of the correct attachment in the list of heads. compared to the rrr dataset, the blbg dataset is a better approximation of the attachment decision faced in a full parsing task since it allows more than two in order to enable experiments in which the full sentence context is used as input, the original head word tuples were joined with the full sentences from which they were extracted. starting from data generation scripts provided to us by the author foot_2 of the blbg data set, we joined each example in the original blbg data set with its full sentence context from the penn treebank version 3. the sentences were lower cased to match the convention of the blbg data. as we could not obtain the matching version of the penn treebank for the rrr dataset foot_3 , the full sentence experiments were only conducted on the blbg data set. ## our method in recent years, contextualized word embeddings (cwe), particularly those built using the transformer @xcite architecture, have accelerated progress in many nlp tasks. past work on nlp tasks has typically followed a transfer learning strategy, in which a large pre-trained model is fine-tuned on a small annotated training set with task-specific labels. we apply this strategy using pre-trained bert @xcite and roberta @xcite models on data specific to the prepositional phrase attachment task, with the objective of improving task accuracy. figure 1 introduces a deep learning architecture that fine-tunes the output of a cwe module in order to predict attachments. we henceforth refer to it as the fine tuning for headword attachment (ftha) model foot_4 . the full sentence or phrase is first passed through the cwe module, which yields a vector of token embeddings, shown in layer (1). in cases where the original word has been split into multiple tokens by the cwe tokenizer, we follow the convention in @xcite and use the full sentence or phrase bert or roberta h1 h2 h3 n2 p h1, p, n2 h2, p, n2 h3, p, n2 hidden layer 1 hidden layer 1 hidden layer 1 out 1+ h1, p, n2 out 1 out 2+ h2, p, n2 out 2 out 3+ h3, p, n2 out 3 hidden layer 2 hidden layer 2 hidden layer 2 softmax (1) (2) (3) (4) (5) (6) figure 1: the fine tuning for headword attachment (ftha) model. the diagram shows only 3 candidate head word attachment sites for the preposition, but the network can operate on an arbitrary number of candidate head words. each input example contains the information shown in tables 1 and 2 , together with a token sequence, and token indices of the head words. depending on the experiment configuration, the token sequence is either the full sentence from which the example was drawn, or a phrase synthesized by concatenating the words in (h 1 . . . h n , p, n2). ## experiments the ftha model is trained and evaluated on both the rrr and blbg data sets. furthermore, we compare with past results on both data sets, as well as against an existing implementation foot_5 of a multiple choice fine-tuning architecture in the hug-gingface (hf) code library @xcite . the hf multiple choice implementation was designed for the swag @xcite ) task, and is described in @xcite . we configure the implementation to predict up to 8 choices, where each choice is a phrase formed from the head word triple (h i , p, n2), or a dummy label. for context input, we also give it either a full sentence or a phrase synthesized from all of the head words, depending on the experiment configuration. for our experiments, we configure it to use roberta. the hf implementation encodes the text of the choice (i.e., the head word triples) and any additional input, "pools" the embeddings, and then passes them to linear and softmax layers. in contrast, the ftha model retains the head word embeddings that were computed in the context of the full sentence or phrase. this representation allows the ftha model to work with a higher granularity of information from the input, compared to the hf implementation. all experiments are measured using accuracy of the label classification. the allowable labels for each data set and their distributions are shown in the higher granularity representation of the ftha model (with roberta) gives slight accu-paper dev test @xcite n/a 88.1% hf multiple choice 89.1% 88.4% ftha (roberta) 89.3% 88.8% ftha (bert) 88.8% 87.9% all roberta and bert experiments use the roberta-base and bert-base-uncased models, respectively. all models have at most 126m parameters, and training times were at most 1 hour on a unix server with an nvidia titan rtx gpu with 24gb ram. ## discussion in past work, data sparsity has been a major challenge for corpus-based approaches to prepositional phrase attachment. searchers have incorporated features from external resources like wordnet and framenet, with the idea that semantic features on words will have far less than sparsity than words themselves. in our work, the cwe, built from a huge amount of unsupervised data, seem to compensate for the sparsity in the relatively small training sets. our results on both the rrr and blbg sets exceed those reported in past work, but without using wordnet, framenet, or any other external resources. all other results compared in tables 6 and 5 use resources external to the training set. we do not compare our experimental results with those in @xcite and @xcite due to differences in test data. the approach of do and rehbein (2020), conducted on german-language pp attachment data, resembles our work in that it uses cwe in a neural model. it also presents a scoring function for (h i , p, n2) triples, using a biaffine transformation of word representations derived from a bidirectional lstm. it differs from our approach in that it considers all words in the sentence as potential attachment sites. and unlike our work, it incorporates additional information like part-of-speech tags, topological field tags, and auxiliary distributions computed from a large newspaper corpus. the approach of olteanu and moldovan (2005) uses support vector machines and requires rich features that cannot be derived from the rrr dataset. therefore it creates a new and larger data set with complex features from syntax trees, framenet, and co-occurrence statistics derived from an internet search engine. in contrast, our work focuses on only the cwe and excludes external resources. both the ftha and hf experiments show benefit from using the full sentence context. for ftha, the embeddings computed in the context of the whole sentence are likely more accurate than those computed from the head word phrase. for the hf implementation, the pooled embedding of the sentence carries enough information to boost the accuracy over just using the head words. notably, in both cases, the sentence information is not explicitly annotated with any syntactic information, yet gives a sizable boost (+ 2% to 3%) in pp attachment accuracy. our work studies the pp attachment problem in isolation, and does not compare against the attachment decisions of a full parser. other work @xcite shows that pp attachment is still a problematic area for full parsers, and that an independently trained pp attachment model can improve the decisions of a full parser. ## conclusion we present deep learning experiments for the prepositional phrase attachment task that exceed the accuracy of all previously published results on two widely used data sets. the results in our paper are 0.7% and 4.1% higher in absolute percentage points over the best previously published results on the rrr and blbg data sets, respectively. we present a novel fine-tuning architecture that uses a higher granularity of information from the input, compared to a baseline implementation from hug-gingface. all our results were obtained without using external semantic data sources like wordnet or framenet. lastly, we observe a big accuracy gain when the model is given the full sentence context vs. only the head words, despite having no syntactic annotation in the full sentence context.