Neural Reranking for Dependency Parsing: An Evaluation (2020)

Abstract: Recent work has shown that neural rerankers can improve results for dependency parsing over the top k trees produced by a base parser. However, all neural rerankers so far have been evaluated on English and Chinese only, both languages with a configurational word order and poor morphology. In the paper, we re-assess the potential of successful neural reranking models from the literature on English and on two morphologically rich(er) languages, German and Czech. In addition, we introduce a new variation of a discriminative reranker based on graph convolutional networks (GCNs). We show that the GCN not only outperforms previous models on English but is the only model that is able to improve results over the baselines on German and Czech. We explain the differences in reranking performance based on an analysis of a) the gold tree ratio and b) the variety in the k-best lists.

## introduction neural models for dependency parsing have been a tremendous success, pushing state-of-the-art results for english on the wsj benchmarking dataset to over 94% las @xcite . most state-of-the-art parsers, however, are local and greedy and are thus expected to have problems finding the best global parse tree. this suggests that combining greedy, local parsing models with some mechanism that adds a global view on the data might increase parsing accuracies even further. in this work, we look into incorporating global information for dependency parsing via reranking. different model architectures have been proposed for neural reranking of dependency parse trees @xcite @xcite . despite achieving modest or even substantial improvements over the baseline parser, however, all the systems above only report performance on english and chinese data, both morphologically poor languages with a configurational word order and mostly projective tree structures. in the paper, we thus try to reproduce results for different reranking models from the literature on english data and compare them to results for german and czech, two morphologically rich(er) languages (mrls) with a high percentage of nonprojective structures. in addition, we present a new discriminative reranking model based on graph convolutional networks (gcns). our gcn reranker outperforms the other rerankers on english and is also the only model able to obtain small improvements over the baseline parser on german and czech while the other rerankers fail to beat the baselines. the improvements, however, are not significant and raise the question what makes neural reranking of mrls more difficult than reranking english or chinese. we analyze the differences in performance on the three languages and show that the reasons for this failure are due to the composition and quality of the k-best lists. in particular, we show that the gold tree ratio in the english k-best list is much higher than for german and czech, and that the trees in the english k-best list show a higher variety, thus making it easier for the reranker to distinguish between high-and low-quality trees. the paper is structured as follows. in §2, we review related work on reranking for neural dependency parsing. the different reranking models are described in detail in §3. in §4, we first reproduce reranking results for english and evaluate our new reranker on the english data. then we test the different models on the two morphologically rich(er) languages and present the results of our evaluation and our analysis, before we conclude in §5. ## related work reranking is a popular technique to improve parsing performance of the output of a base parser. first, the top k candidate trees are generated by the base parser, then these trees are reranked using additional features not accessible to the base parser. this adds a more global and complete view of the trees, in contrast to the local and incomplete features used by the parser. discriminative rerankers have been a success story in constituency parsing @xcite . a disadvantage of the traditional feature-rich rerankers is that the large number of potentially sparse features makes them prone to overfitting, and also reduces the efficiency of the systems. neural rerankers offer a solution to that problem by learning dense, low-dimensional feature representations that are better at generalization, and so reduce the risk of overfitting. neural reranking the first neural reranker has been presented by @xcite for constituency parsing, based on a recursive neural network which processes the nodes in the parse tree bottom-up and learns dense feature presentations for the whole tree. this approach was adapted for dependency parsing by @xcite . @xcite improve on previous work by proposing a recursive convolutional neural network (rcnn) architecture for reranking which can capture syntactic and semantic properties of words and phrases in the parse trees (see §3 for a more detailed description of the two models). in our work, we focus on neural reranking of a k-best list of parses generated by a base parsing system as we could not find any available parsers that are both non-projective and produce packed parse forests as output. ## neural reranking models for dependency parsing in this section, we look into reranking for dependency parsing and compare two different types of models: the generative inside-outside recursive neural network (iornn) reranker @xcite and the discriminative reranker based on recurrent convolutional neural networks (rcnns) @xcite . in addition, we propose a new reranking model for dependency parsing that employs graph convolutional networks (gcns) to encode the trees. ## evaluating neural rerankers for dependency parsing we are now providing a systematic evaluation of different neural reranking models used to rank the k-best lists generated by different parsers. in our first experiments, we try to reproduce the results for the available rerankers (iornn, rcnn) on english. after that, we compare the performance of the rerankers on german and czech data. unless stated otherwise, results are compared based on uas and las including punctuation. ## conclusion we have evaluated recent neural techniques for reranking dependency parser output for english, german and czech and presented a novel reranking model, based on graph convolutional networks (gcns). we were able to reproduce results for english, using existing rerankers, and showed that our novel gcn-based reranker even outperformed them. however, none of the rerankers works well on the two morphologically rich(er) languages. our analysis gave some insights into this issue. we showed that the failure of the rerankers to improve results for german and czech over the baseline is due to the lower quality of the k-best lists. here the gold tree ratio in the k-best list plays an important role, as the discriminative rerankers are very well able to distinguish the gold trees from other trees in the list, but their performance drops notably when we remove the gold trees from the list. in addition, we observe a higher diversity in the english k-best list, as compared to german and czech, which helps the rerankers to learn the differences between high-and low-quality trees. we conclude that the prerequisite for improving dependency parsing with neural reranking is a diverse k-best list with a high gold-tree ratio. the latter is much harder to achieve for mrls where the freer word order and high amount of non-projectivity result in a larger number of tree candidates, reflected by a lower gold tree ratio.