A Hybrid Rule-Based and Neural Coreference Resolution System with an Evaluation on D utch Literature (2021)

Abstract: We introduce a modular, hybrid coreference resolution system that extends a rule-based baseline with three neural classifiers for the subtasks mention detection, mention attributes (gender, animacy, number), and pronoun resolution. The classifiers substantially increase coreference performance in our experiments with Dutch literature across all metrics on the development set: mention detection, LEA, CoNLL, and especially pronoun accuracy. However, on the test set, the best results are obtained with rule-based pronoun resolution. This inconsistent result highlights that the rule-based system is still a strong baseline, and more work is needed to improve pronoun resolution robustly for this dataset. While end-to-end neural systems require no feature engineering and achieve excellent performance in standard benchmarks with large training sets, our simple hybrid system scales well to long document coreference (>10k words) and attains superior results in our experiments on literature.

## introduction this paper reports on a hybrid rule-based and neural coreference resolution system foot_0 evaluated on dutch literary texts. we use neural classifiers for the following three subtasks: these subtasks have been selected based on the expected return on investment given the particular weaknesses of the rule-based model @xcite and specific challenges of literary coreference @xcite . to keep the approach as simple as possible, we implement these classifiers as independent modules operating in a pipeline. the classifiers can be trained on a laptop without gpu in ten minutes, and are therefore substantially less resource-intensive than state-of-the-art neural models; e.g., spanbert @xcite requires pre-training a bert model with span representations on specialized hardware (tpu); moreover, end-to-end neural coreference resolution systems are generally memory and cpu intensive, especially when longer contexts are taken into account @xcite . the output of coreference resolution is a set of mention spans, partitioned into clusters (example based on @xcite : (1) [de chirurg] 1 kon [de patiënt] 2 niet behandelen. [hij] 2 was [[haar] 1 zoon] 2 ! [the surgeon] 1 couldn't treat [the patient] 2 . [he] 2 was [[her] 1 son] 2 ! ## related work rule-based coreference resolution provides a reasonable baseline @xcite , and its advantages are that it can exploit global features of entities based on the whole document. in contrast to end-to-end systems, information from parse trees and named-entity recognition can be used, as well as other components from the natural language processing (nlp) pipeline. feature-based models also use the nlp pipeline, but use machine learning classifiers that make local decisions (mention-pair and mention-ranking architectures), or attempt to take global context into account, but this runs into computational challenges with long documents. end-to-end neural systems do not need the nlp pipeline and are able to optimize all steps of coreference resolution jointly, which has enabled large advances in standard benchmarks @xcite @xcite . however, there are several challenges with end-to-end neural models: long documents with long coreference chains @xcite , domain and annotation differences across datasets @xcite , and needing a large number of training examples @xcite . moreover, gender bias is a general challenge in coreference resolution systems @xcite . each of these areas is potentially easier to address with a well engineered rule-based or feature-based approach to coreference resolution, and we therefore choose to explore this direction. hybrid coreference resolution systems have been presented before; @xcite present a system in which most steps of the rule-based system are implemented with random forest classifiers. they obtain improvements in accuracy and efficiency, but neural systems have since eclipsed these results. their classifiers include mention detection and pronoun resolution, which we also pursue in this work. in addition, previous work shows that neural representations and surface features have complementary strengths @xcite . this is another sense in which our system is hybrid: we use both manually selected features as well as contextualized word embeddings. parts of the neural architecture and features are inspired by @xcite , but we use bert @xcite for embedding features instead of static word embeddings, since bert representations have shown to bring about significant improvements in natural language tasks that rely on the context. there has been some work on improving detection of mention attributes (animacy, gender, number) using external datasets and machine learning. @xcite extract attributes from a large corpus with dependency parses using heuristic patterns. @xcite focus on animacy and use wordnet and semcor combined with machine learning. these methods aim to learn general patterns for detecting attributes of noun phrases. in contrast, we will annotate attributes of the entities in our coreference corpus in context and train the classifier on those annotations. we hope to handle more difficult, ambiguous cases which require context with this approach. although most coreference resolution systems are trained and evaluated on domains contained in benchmark datasets, such as news texts and phone conversations in the case of ontonotes, we train and evaluate our hybrid system on dutch literature. the reason we are interested in the literary domain is that, while literary texts are increasingly subject to computational analysis in the field of digital humanities, there is still a lot of work required to adapt nlp models to the literary domain, of which coreference resolution is a particularly challenging instance. importantly, the literary domain contains unique characteristics, such as long coreference chains and dialogue, which do not appear in typical benchmark data for coreference resolution @xcite . ## data we use riddlecoref (van cranenburgh, 2019), with the same train/dev/test splits as used in @xcite . the corpus consists of 162k tokens of contemporary @xcite @xcite @xcite bestselling novels in dutch (translated and original), with a total of 33 documents (fragments of novels), and an average of 4897.4 tokens per document. the entity coreference annotations follow the dutchcoref annotation guidelines. the 38,466 mentions in the corpus have been manually corrected and exclude non-referring expressions. we did an additional round of corrections on the whole corpus, mostly to fix mention boundaries to exclude relative clauses and remove nonreferring expressions (idioms, verbal expressions, negated mentions). we also made small improvements to the mention detection of the rule-based system: bare nouns in conjunctions are extracted as mentions, and subordinate clauses are removed from mention spans, in accordance with the annotation guidelines. three mention attributes, namely animacy, gender, and number, have been manually annotated for each of the 11,684 entities in the training and development sets. the gender attribute has four possible values: f (female), m (male), fm (unknown or mixed gender), and n (neuter, non-human). any gender except neuter implies a human (person) entity; the animacy attribute is therefore implied. note that dutch has noun classes with grammatically gendered and neuter words; however, our annotations concern the gender with which individuals are identified. for example, the noun phrase het meisje (the girl) is grammatically neuter, but annotated as female, since it would be referred to by female pronouns. the number attribute has two possible values: sg (singular) and pl (plural; an entity consisting of multiple individuals/objects). we annotate the semantic number (e.g., "the group" is plural since it is a collective noun that could be referred to by "they"), regardless of the syntactic number. ## method the base system is a rule-based coreference resolution system (van cranenburgh, 2019) which takes parse trees as input. we extend this system with neural classifiers for three subtasks; see figure 1 for an overview of our hybrid system. ## results we first report the results for each module on the development set, and then report the results for the system in various configurations on the development and test sets. the results for the mention span classifier are shown in the results for the mention attributes classifier are shown in for coreference evaluation, we use the averaged conll score @xcite and the lea coreference metric @xcite . in addition we report mention scores and pronoun accuracy. pronoun accuracy includes demonstrative and indefinite pronouns in addition to third person personal and possessive pronouns. see in order to isolate the effect of mention detection (which is known to introduce pipeline errors), we also perform an evaluation on the test set with gold mentions, see all our evaluations include singletons; evaluating without singletons does not change the ranking of the systems on each metric. we conclude that the mention attribute classifier robustly improves the performance, but that the pronoun resolution classifier yields inconsistent results. finally, while the result is puzzling, a similar result was reported by @xcite , where the rule-based system performed better on the test set than on the development set, while the end-to-end neural system showed the opposite effect (better on development set than on test set). ## analysis 6.1 analysis of differences it could be that the development and test set differ in difficulty. we consider several basic statistics to compare the two sets. we first consider differences in the out-of-vocabulary (oov) rate and word frequencies with the jensen-shannon distance. we find that the development set actually has a lower oov rate than the test set, with respect to the training set (16.3% and 13.3%, respectively). the jensen-shannon distance shows the same pattern (0.307 vs 0.290, respectively). moreover, the average sentence length is similar between the development and test sets (18.39 and 18.26, respectively), but higher than the train set (15.51). the development set does have a lower number of mentions (6548 vs 6869) and entities (2643 vs 3008). finally, the development set has a higher percentage of names (14.9 vs 9.1). genre is another potential explanation for the difference. there are four different genres in the riddlecoref dataset: (literary) fiction, suspense, romance, and other. the development set contains 4 fiction and 1 other novel, while the test set contains 3 fiction, 1 romance, and 1 suspense novel. we now take a closer look at the difficulty of these genres using an out-of-domain training set. we evaluate the pronoun resolution module on each genre in riddlecoref; the results are in ## discussion and conclusion we have presented a hybrid coreference resolution system that extends a rule-based baseline with three simple neural classifiers. the classifiers substantially increase the coreference performance in our experiments on dutch literature, except for pronoun resolution on the test set. the strongest improvements is on pronoun accuracy, which is especially important in longform narrative text. there are several areas in which the system can be improved. in our approach we erred on the side of simplicity, but in the case of pronoun resolution the approach was too simple, leading to an improvement on the development set but not on the test set. the simplicity can be relaxed in several ways. the modules are trained with gold standard input, but using predictions of previous modules may give better results. if possible, the modules should be trained jointly. adding more and more varied training data, such as from sonar-1 can be expected to yield better results. bert performs better when finetuned and when encoding segments of 128 to-kens, as reported by @xcite . finally, other modules could be added. anaphoricity classifiers are used in most state-of-the-art systems. in literature, dialogue is particularly important; annotating and predicting speakers of direct speech will help in resolving first and second person pronouns. future work should investigate in more detail the trade-offs between rule-based systems using an nlp pipeline and modern end-to-end neural models, especially in the challenging case of longdocument coreference in narrative text.