A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution in E nglish (2021)

Abstract: Pronoun Coreference Resolution (PCR) is the task of resolving pronominal expressions to all mentions they refer to. Compared with the general coreference resolution task, the main challenge of PCR is the coreference relation prediction rather than the mention detection. As one important natural language understanding (NLU) component, pronoun resolution is crucial for many downstream tasks and still challenging for existing models, which motivates us to survey existing approaches and think about how to do better. In this survey, we first introduce representative datasets and models for the ordinary pronoun coreference resolution task. Then we focus on recent progress on hard pronoun coreference resolution problems (e.g., Winograd Schema Challenge) to analyze how well current models can understand commonsense. We conduct extensive experiments to show that even though current models are achieving good performance on the standard evaluation set, they are still not ready to be used in real applications (e.g., all SOTA models struggle on correctly resolving pronouns to infrequent objects). All experiment codes will be available upon acceptance.

## introduction the question of how human beings resolve pronouns 1 has long been of interest to both linguistic and natural language processing (nlp) communities, for the reason that a pronoun itself only having weak semantic meaning brings challenges to natural language understanding. to explore solutions for that question, pronoun coreference resolution (pcr) @xcite was proposed. 2 as a challenging yet vital natural language understanding 1 some pronouns may refer to non-nominal antecedents. for example, the pronoun "it" in "it is too cold in the winter here" does not refer to any real object @xcite . but in this survey, we only focus on pronouns that refer to nominal antecedents. to investigate the difference between pcr and the general coreference resolution task, which tries to identify not only the coreference relations between noun phrases (np) and pronouns (p) but also potential coreference relations between noun phrases or coreference relations between pronouns, we conduct experiments with one recent breakthrough model (i.e., end-to-end model @xcite ) on the conll-2012 shard task @xcite under two settings: one without the gold mention and one with the gold mention. in the 'without gold mention' setting, models are required to first identify spans from the documents as the mentions and then predict the coreference relations among these mentions. as a comparison, if gold focus on three kinds of pronouns: third personal pronoun (e.g., she, her, he, him, them, they, it), possessive pronoun (e.g., his, hers, its, their, theirs), and demonstrative pronoun (e.g., this, that, these, those). the first and second personal pronouns are typically not considered as they often refer to the current speakers, which are normally out of the conversation or document. besides that, conventional pcr works @xcite ) mostly focusing on identifying coreference relations between pronouns and noun phrases rather than coreference relation between pronouns. mentions are provided, models only need to predict the coreference relations (i.e., the task of distinguishing between referential and non-referential instances is ignored). from the results in the correct resolution of pronouns typically requires reasoning over both linguistic knowledge (e.g., 'they' typically refers to plural objects foot_0 ) and commonsense knowledge (e.g., in sentence "the fish ate the worm, it was hungry.", 'it' refers to 'fish' because hungry things tend to eat rather than be eaten.). considering that the ordinary pcr task evaluates the inference over both types of knowledge at the same time, the performance on ordinary pcr tasks cannot clearly reflect models' performance regarding different knowledge types. to address this problem, the winograd schema challenge (wsc) @xcite task is proposed. the influence of all commonly used linguistic knowledge is avoided during the creation of wsc such that wsc can be used to reflect how current pcr models can understand commonsense knowledge. in section 2 and 3, we introduce the progress and remaining challenges on the ordinary pcr and wsc tasks respectively. after that, we introduce other pcr tasks that are developed for different research purposes in section 4. in the end, we conclude this survey with section 5. the contribution of this survey is three-fold: ( 1 ) we broadly introduce available pcr tasks, datasets, and models; (2) we summarize the main contribution of recent models; (3) we conduct experiments to analyze the limitations of current models, which can help the community think about how to better solve pcr in the future. ## ordinary pcr ordinary pronoun coreference resolution tasks are often defined over formal textual corpus (e.g., news-paper) and the annotation is usually conducted by domain experts or linguists. the pcr task can be formally defined as follows. given a text d, which contains a pronoun p, the goal is to identify all the mentions that p refers to. we denote the correct mentions p refers to as @xmath0, where c is the correct mention set. similarly, each candidate span is denoted as @xmath1, where s is the set of all candidate spans. note that in the case where no golden mentions are provided, all possible spans in d are used to form s. the task is thus to identify c out of s. in the rest of this section, we introduce the widely used datasets as well as the progress and limitation of current approaches. ## hard pcr as aforementioned, the correct resolution of pronouns requires the inference over both linguistic knowledge and commonsense knowledge. to clearly reflect how models can resolve pronouns that require the inference over commonsense knowledge, the hard pcr task was proposed. as winograd schema challenge (wsc) is one of the most popular hard pcr tasks, we use the task definition in wsc to define the hard pcr task. for each question q, a sentence s is given, which contains a pronoun p and two candidates n 1 , n 2 . the task is to find out which of the candidates p refers to. different from the ordinary pcr task, the influence of all commonly observed features (e.g., gender or plurality) are removed via careful expert design. in wsc, all questions are paired up such that questions in each pair have only minor differences (mostly one-word difference), but the answers are reversed. one pair of the wsc instances is shown in figure 1 . solving these questions typically requires the support of complex commonsense knowledge. for example, human beings can know that the pronoun 'it' in the first sentence refers to 'fish' while the one in the second sentence refers to 'worm' because 'hungry' is a common property of something eating while 'tasty' is a common property of something being eaten. without the support of such commonsense knowledge, answering these questions becomes challenging because both the fish and worm can be hungry or tasty by themselves. ## other pcr tasks besides the ordinary and hard pcr tasks, pcr is also an important research topic for many special purposes (e.g., gender bias) or in some special settings (e.g., visual-aware pcr). in this section, we briefly introduce these tasks: the experiments show that some gender bias (e.g., 'he' is more likely to be predicted to be the doctor rather than the nurse by the machine) indeed exists in pre-trained language representation models. such observation is astonishing and motivates the community to think about how to minimize the influence of such gender bias. ## conclusion in this paper, we survey the progress on the pronoun coreference resolution (pcr) task, and analyze the improvement and limitations of existing approaches. experiments and analysis on both the ordinary and hard pcr tasks demonstrate that even though we have made great progress according to the main evaluation metric, the pcr task is still far away from being solved. for example, all bestperforming ordinary pcr models struggle on the cross-domain setting as well as infrequent objects. also, even though fine-tuning pre-trained language representation models can achieve near-human performance on wsc, it can be sensitive to the hyperparameters. all codes will be released to encourage the research on the pcr task.