Revealing the Myth of Higher-Order Inference in Coreference Resolution (2020)

Abstract: This paper analyzes the impact of higher-order inference (HOI) on the task of coreference resolution. HOI has been adapted by almost all recent coreference resolution models without taking much investigation on its true effectiveness over representation learning. To make a comprehensive analysis, we implement an end-to-end coreference system as well as four HOI approaches, attended antecedent, entity equalization, span clustering, and cluster merging, where the latter two are our original methods. We find that given a high-performing encoder such as SpanBERT, the impact of HOI is negative to marginal, providing a new perspective of HOI to this task. Our best model using cluster merging shows the Avg-F1 of 80.2 on the CoNLL 2012 shared task dataset in English.

## introduction coreference resolution has always been considered one of the unsolved nlp tasks due to its challenging aspect of document-level understanding @xcite @xcite @xcite . nonetheless, it has made a tremendous progress in recent years by adapting contextualized embedding encoders such as elmo @xcite and bert @xcite @xcite . the latest state-of-the-art model shows the improvement of 12.4% over the model introduced 2.5 years ago, where the major portion of the improvement is derived by representation learning (figure 1 ). most of these previous models have also adapted higher-order inference (hoi) for the global optimization of coreference links, although hoi clearly has not been the focus of those works, for the fact that gains from hoi have been reported marginal. this has inspired us to analyze the impact of hoi on modern coreference resolution models in order to envision the future direction of this research. to make thorough ablation studies among different approaches, we implement an end-to-end coreference system in pytorch (sec 3.1), and two hoi approaches proposed by previous work, attended antecedent and entity equalization (sec 3.2), along with two of our original approaches, span clustering and cluster merging (sec 3.3). these approaches are experimented with two transformer encoders, bert and spanbert, to assess how effective hoi is even when coupled with those high-performing encoders (sec 4). to the best of our knowledge, this is the first work to make a comprehensive analysis on multiple hoi approaches side-by-side for the task of coreference resolution. ## related work most neural network-based coreference resolution models have adapted antecedent-ranking @xcite @xcite @xcite , which relies on the local decisions between each mention and its antecedents. to achieve deeper global optimization, @xcite ; @xcite ; @xcite built entity representations in the ranking process, whereas @xcite ; @xcite ## experiments for our experiments, the conll 2012 english shared task dataset is used @xcite . given the end-to-end coreference system in section 3.1, six models are developed as follows: foot_1 ## conclusion we implement the end-to-end coreference resolution model and investigate four higher-order inference methods, including two of our own methods. our best model shows the new result of 80.2 on the conll 2012 dataset. we thoroughly analyze the empirical effectiveness of hoi and demonstrate why it fails to boost performance on the conll 2012 dataset compared to the improvement from encoders. we show that current hoi does not meet up with the original motivation, suggesting that a new perspective of hoi is needed for this task in the era of deep learning-based nlp.