A Benchmark of Rule-Based and Neural Coreference Resolution in D utch Novels and News (2020)

Abstract: We evaluate a rule-based (Lee et al., 2013) and neural (Lee et al., 2018) coreference system on Dutch datasets of two domains: literary novels and news/Wikipedia text. The results provide insight into the relative strengths of data-driven and knowledge-driven systems, as well as the influence of domain, document length, and annotation schemes. The neural system performs best on news/Wikipedia text, while the rule-based system performs best on literature. The neural system shows weaknesses with limited training data and long documents, while the rule-based system is affected by annotation differences. The code and models used in this paper are available at https://github.com/andreasvc/crac2020

## introduction in recent years, the best results for coreference resolution of english have been obtained with end-to-end neural models @xcite @xcite @xcite . however for dutch, the existing systems are still using either a rule-based @xcite or a machine learning approach @xcite . the rule-based system dutchcoref (van cranenburgh, 2019) outperformed previous systems on two existing datasets and also presented a corpus and evaluation of literary novels @xcite . in this paper we compare this rule-based system to an end-to-end neural coreference resolution system: e2e-dutch. this system is a variant of @xcite with bert token representations. we evaluate and compare the performance of e2e-dutch to dutchcoref on two different datasets: (1) the sonar-1 corpus @xcite , a genre-balanced corpus of 1 million words, and (2) the riddlecoref corpus of contemporary novels @xcite . this provides insights into (1) the relative strengths of a neural system versus a rule-based system for dutch coreference, and (2) the effect of domain differences (news/wikipedia versus literature). the two datasets we consider vary greatly in terms of overall size and length of the individual documents; the training subset of riddlecoref contains only 23 documents (novel fragments) compared to 581 documents for sonar-1. however, the average number of sentences per document is higher for riddlecoref than for sonar-1 (295.78 vs. 64.28 respectively). we also conduct an error analysis for both of the systems to examine the types of errors that the systems make. ## related work the main differences between traditional and neural approaches can be summarized as follows: system conll rule-based @xcite 58.3 perceptron @xcite 58.7 hybrid: rules + ml @xcite 63.2 embeddings @xcite 63.4 + rl @xcite 65.3 + entity embeddings @xcite 65.7 system conll end-to-end @xcite 68.8 higher-order + ctf + elmo @xcite 73.0 finetuning bert base @xcite 73.9 finetuning bert large @xcite 76.9 pretraining spanbert @xcite 79.6 spanbert + qa @xcite 83. 1 the rest of this section discusses the current best systems for dutch and english. ## coreference corpora in this paper we consider entity coreference and focus on the relations of identity and predication. the rest of this section describes the two dutch corpora we use. ## coreference systems we now describe the two coreference systems, dutchcoref and e2e-dutch, which we evaluate on the coreference corpora described in the previous section. ## evaluation before presenting our main benchmark results, we discuss the issue of coreference evaluation metrics. ## analysis the previous section showed some surprising results. we now take a closer look at the differences between the two coreference systems, datasets, and the annotations. ## conclusion we found large gaps in performance for the two systems across the two domains, but this result is not conclusive due to several reasons, which are as follows. the neural system shows a weakness with the long documents in the novel corpus, but also needs more training data to reach its full potential. the rule-based system should be better adapted to the sonar-1 annotation scheme, but the neural system's capacity to adapt to arbitrary annotation conventions does not necessarily imply better linguistic performance. to maximize the comparability and usefulness of the corpora, their annotations should be harmonized, which involves manual mention annotation. in future work we want to improve the neural system by using genre metadata and finetuning bert, and the rule-based system should be extended to a hybrid system by adding supervised classifiers.