Parsers Know Best: G erman PP Attachment Revisited (2020)

Abstract: In the paper, we revisit the PP attachment problem which has been identified as one of the major sources for parser errors and discuss shortcomings of recent work. In particular, we show that using gold information for the extraction of attachment candidates as well as a missing comparison of the system’s output to the output of a full syntactic parser leads to an overly optimistic assessment of the results. We address these issues by presenting a realistic evaluation of the potential of different PP attachment systems, using fully predicted information as system input. We compare our results against the output of a strong neural parser and show that the full parsing approach is superior to modeling PP attachment disambiguation as a separate task.

## introduction prepositional phrase (pp) attachment disambiguation, the task of identifying the correct attachment site for each preposition in the syntax tree, has often been described as the canonical case of structural ambiguity in nlp, with crucial impact on semantic interpretation. what makes pp attachment such a challenging task is that morpho-syntactic information is often insufficient to resolve the ambiguity, in particular for (semi-)free word order languages, and additional semantic information or even world knowledge is needed. even though the pp attachment problem has been studied since the nineties @xcite @xcite , it is still one of the hardest problems for syntactic parsing. for constituency parsing of english, @xcite showed that pp attachment errors are the largest error category across all parsers included in their evaluation. this also holds for dependency parsing. in experiments with the biaffine parser of @xcite on german, we found that 21.8% of the unlabeled attachment errors are due to incorrect pp attachments. to identify the correct attachment site for each preposition, the parser either needs to see these head-pp pairs during training or has to be able to deduce the correct attachment based on seen examples. we can improve lexical coverage by adding more training data, however, treebanking is extremely timeconsuming and requires linguistic expertise. in addition, most pps are adjuncts, which means that they can choose their heads more freely (in comparison to arguments) and it is thus harder to achieve sufficient coverage. a different approach tries to improve pp attachment accuracy by modeling the problem as a separate task. this has the advantage that we do not need fully annotated parse trees as input, and that such a setup makes it easy to integrate a wide range of heterogeneous features. many studies have tried to solve this task, however, only few have shown that their system is able to improve the output of a strong syntactic parser (see §2). in the paper, we present a new pp attachment disambiguation system, based on biaffine attention and contextual word embeddings. while obtaining substantial improvements over previous work, we show that modeling all head-dependent pairs jointly (as done in full parsing) allows the system to make more effective use the training data and is thus superior to modeling pp attachment as a separate task. the paper is structured as follows. we first review statistical methods for pp attachment ( §2), focusing on german ( §2.1). after outlining some shortcomings of recent work, we reproduce a state-of-the-art pp attachment disambiguation system and evaluate it in a realistic scenario, comparing its performance to that of a strong neural parser ( §3). in §4 we propose a new approach based on contextualized word embeddings that overcomes limitations of previous work, and we summarize our results in §5. features a diverse set of features has been proposed in the literature. lexical association scores computed on external, large corpora can help to compensate for limited training data in supervised approaches. the unannotated data are parsed to extract tuples of the form (v, n 1 , p, n 2 ), and their counts are used as features for pp attachment disambiguation @xcite @xcite . alternatively, lexical associations can be estimated based on word co-occurrences in a very large corpus like the world wide web (www), queried by a search engine @xcite . semantic information can also help to overcome data sparsity since words that are semantically related are likely to occur in similar contexts, and thus will show similar selectional preferences regarding pp attachment. the semantic word class can be determined based on external resources like wordnet @xcite @xcite , or automatically via mutual information clustering @xcite . alternatively, lexical sparsity can also be tackled by extending the lexicon with semantically similar words from syntactic collocation databases and thesauri @xcite . semantic information of verbs from framenet @xcite or verbnet @xcite have also been used as features in pp attachment disambiguation systems @xcite . syntactic features have been used to recover missing context information in the extracted system input. examples are the distance between the candidate head and the preposition @xcite @xcite , verb subcategorization @xcite or topological fields @xcite . pre-trained word embeddings that capture both syntax and semantics are frequently used as input to pp attachment disambiguation systems with a neural network architecture @xcite @xcite to improve lexical coverage and enrich the lexical information. comparison with syntactic parsing early work focuses on evaluating pp attachment disambiguation as an independent task rather than comparing results to parser output or integrating it with parsing. exceptions are @xcite and @xcite who integrate lexical preferences in rulebased parsers. disambiguation systems that rely on an oracle (the mechanism that extracts the two candidate attachment sites, based on gold standard data) have been criticized by @xcite . the authors argue that using gold information for candidate extraction is highly unrealistic as it will always include the correct solution in the candidate set, while in real applications the correct solution might not be available. therefore, pp attachment disambiguation systems should be used to refine the preliminary syntactic analysis of a sentence provided by a parser, or to reattach pp attachments. their experiments with three pp reattachment systems show that none of the systems was able to obtain a significant improvement over the baseline parser. @xcite later follow their evaluation setup and report a small but significant improvement in parsing accuracy using word sense information. up to now, the pp attachment disambiguation problem has been studied extensively, with the latest models incorporating novel techniques like neural networks and word embeddings. most proposed approaches revolve around improving lexical coverage, either by utilizing large, external corpora or by integrating semantic information. despite the fact that some systems report better results than a baseline parser, results for pp attachment still fall behind those for state-of-the-art syntactic parsing. in recent work on pp attachment, @xcite employ a non-neural network parser with 94.17% accuracy on the penn treebank (using gold pos tags) as a baseline parser while the state-of-the-art (without contextualized embeddings) is 96.09% @xcite . moreover, while most syntactic parsers perform well with predicted information like pos tags, pp attachment systems are usually evaluated in a setup based on gold information. as a final point, restricting pp attachment sites to nouns and verbs only is very limited compared to full parsing where all possible attachment sites for a pp are considered. ## evaluating pp attachment in a realistic setup the goal of our experiments is to overcome the limitations outlined above. in particular, we experiment with pp attachment disambiguation in a more realistic scenario where gold information like pos tags and topological field information is not available. our reference systems are the dependency parser with biaffine attention @xcite which is among the best systems for parsing german, and the pp attachment disambiguation system from de @xcite which has been introduced in §2.1. ## pp attachment without restrictions in the previous section, we have shown that techniques to extract plausible head candidates for german pp attachment disambiguation based on topological fields decreases the upper bound of the system, either by reducing coverage or by error propagation when combining the candidate extraction rules with predicted information at test time. in this section, we will thus consider all words in a sentence as possible heads instead of restricting candidate selection to certain nouns and verbs, and focus on techniques that can cope with data sparseness and deal with noisy input data in real-world scenarios. pp attachment disambiguation with biaffine transformations our first model is pp-biaffine, a pp attachment disambiguation system with biaffine transformations similar to the reference parser of @xcite . the input to the system consists of words, pos tags and the position of the preposition and its object. prepositions are identified based on predicted pos tags, and their objects are extracted based on gold standard dependency trees for training and predicted parse trees at test time. 10 words and tags are converted to embeddings. the score for word h being the head of preposition p with pp object o is computed using a biaffine transformation: we evaluate the potential of pp-biaffine on both tasks, pp attachment disambiguation and pp reattachment ( pp attachment disambiguation with contextualized word embeddings although having similar architectures, the performance of pp-biaffine is still behind that of the full parser. the main reason for this is that the pp attachment disambiguation system has less training data: it is only trained on pp attachments while the parser trains a joint classifier for all attachment types. instead of using more data, we propose to improve pp-biaffine by transfer learning using bert @xcite . bert is a language model based on multi-layer bidirectional transformers @xcite that are trained to be sensitive to positional context information, resulting in embeddings that represent contextualized word information (contextualized word embeddings). @xcite have shown that the bert large model (with 340m parameters) achieves state-of-the-art results on a wide range of nlp tasks. we now replace the pre-trained word embeddings in pp-biaffine with embeddings provided by the bert base multilingual cased model. 11 the rest of the system remains the same. we call this model pp-biaffine+bert. we do not fine-tune bert on our data, as our experiments showed that this decreases results over simply using the pretrained word embeddings. the performance of pp-biaffine+bert is shown in parsing with contextualized word embeddings in a similar fashion, we can replace the pre-trained word embeddings in the reference parser with bert to further improve its performance. using the same bert base multilingual cased model without fine-tuning, parsing accuracy increases for both pp attachment (+∼2%) and parsing in general (+0.78%) (parser+bert in our experiments suggest that parsing systems are in general superior to systems specialized for pp attachment disambiguation. this, however, is only true for high resource languages like english and german where we have enough training data to train a good parser. for low resource languages, on the other hand, acquiring more data for pp attachment disambiguation is much easier than getting more annotated full trees for parser training because pp attachment disambiguation systems only require input in form of triples of (head, preposition, pp object). moreover, systems for pp attachment disambiguation can utilize data from different treebanks, even if they are based on different underlying linguistic theories as long as they agree on the attachment site of the prepositions. we reduce the amount of data used to train the reference parser to see when the specialized system has an advantage over the parser. the german spmrl training set contains 40k sentences and 720k tokens. in the first experiment, we create training datasets that are 25%, 50% and 75% of the original size. the hyperparameters of the parser are kept the same as in previous experiments ( §3.2, §3.3) foot_9 . in the second experiment, we simulate a low resource scenario by creating training datasets of sizes ranging from 1,000 to 5,000 tokens. we heuristically reduce the dimensions and number of layers of the reference parser. likewise, we train the pp-biaffine+bert 13 system in low resource mode with only 2,500 triples. in both experiments, we use the same pos tags as in previous experiments although pos accuracy could be lower in a real low resourced languages scenario. the results are illustrated in figure 2. pp-biaffine+bert trained on the full pp attachment data (73k triples, or 146k dependency relations) only outperforms the parser when reducing the training data for the parser to 25% (180k tokens/dependency relations). in the simulated low resource setting, pp-biaffine+bert trained on 2.5k triples (5k dependency relations) clearly outperforms the parser trained on 5k tokens. the experiments confirm that with the same amount of data annotation, the pp attachment disambiguation system has an advantage over the parser. as the creation of a pp attachment dataset consisting of triples is less expensive than annotating full syntax trees, we believe this could be a way to improve pp attachment accuracy for low resourced languages. ## conclusion we presented an extensive study of the pp attachment disambiguation problem and proposed a new system that combines biaffine attention and pretrained contextualized word embeddings. while our system outperforms recent work on german by a large margin, its performance is still inferior compared to that of a strong neural parser, thus questioning the approach of modeling pp attachment disambiguation as a separate task. we showed that the lower results for the pp attachment system are caused by error propagation due to using predicted syntactic information for candidate extraction. in addition, the parser can make more efficient use of the training data. while the pp attachment disambiguation system is only trained on the pp attachment edges, the parser makes use of all edge types in the tree to train a joint classifier that predicts the head of each word in a sentence. however, we argue that our system might still be useful for lower resourced languages where, due to a lack of training data, no strong parser is available. while our results are for german, the pp-biaffine+bert version of our system is language-agnostic and we expect that our findings will carry over to other languages. we leave this for future work.