Tackling Zero Pronoun Resolution and Non-Zero Coreference Resolution Jointly (2021)

Abstract: Zero pronoun resolution aims at recognizing dropped pronouns and pointing out their anaphoric mentions, while non-zero coreference resolution targets at clustering mentions referring to the same entity. Existing efforts often deal with the two problems separately regardless of their close essential correlations. In this paper, we investigate the possibility of jointly solving zero pronoun resolution and coreference resolution via a novel end-to-end neural model. Specifically, we design a gap-masked self-attention model that encodes gaps and tokens in the same space, where gaps could capture valuable contextual information according to their surrounding tokens while tokens could maintain original sequential information without disturbance. Additionally, we also propose a two-stage interaction mechanism to make full use of the exclusive relationship between zero pronouns and mentions. Our empirical study conducted on the OntoNotes 5.0 Chinese dataset shows that our model could outperform corresponding state-of-the-art approaches on both tasks.

## introduction zero pronoun resolution and non-zero coreference resolution are two fundamental tasks in natural language processing (nlp). zero pronoun resolution, which is only studied in pro-drop languages such as chinese, aims at recognizing dropped pronouns in a given text and pointing out their anaphoric mentions within the text @xcite . coreference resolution, which is studied in all languages, targets at clustering mentions referring to the same real-world entity in the text @xcite . both of the two tasks are vital for many downstream nlp applications including machine translation @xcite , information extraction @xcite ) and text summarization @xcite . given their importance, both tasks have been studied extensively. for zero pronoun resolution, most previous works assume that positions of zero pronouns are given. they encode zero pronouns and their candidate antecedents by lstm @xcite , attention @xcite or bert @xcite , then measure the similarity between embeddings to find out the best antecedents. however, positions of zero pronouns are usually unknown in practice. to solve this problem, some recent works @xcite treat spaces between tokens as candidate zero pronouns, recognize and resolve zero pronouns based on gap embeddings jointly. for non-zero coreference resolution, the state-of-the-art models belong to an end-to-end paradigm or its variants. in these models, all spans in the text are candidate mentions. after representing spans by highway lstm @xcite or transformers @xcite , they calculate mention and antecedent scores using feed forward neural network or machine reading comprehension method @xcite , and select the top k mentions and their antecedents with the top antecedent score. most existing efforts deal with the two problems separately, and someone also state that it is challenging to combine zero pronoun resolution with the resolution of overt mentions @xcite . but according to our observations, there are at least four intuitions motivating us to tackle the two tasks jointly: -firstly, with the joint learning of zero pronoun resolution and non-zero coreference tasks, a more robust and universal representations of tokens could be learned and shared for both tasks, which could potentially benefit both tasks. to strengthen the brand , we will invest 5.5 billion in a renovation plan in the next year . in the future , we will implement such a very major plan , where 𝜙 1 is that 𝜙 2 divide the entire park into seven areas . we will implement the plan as soon as it is approved by the government , and it is expected that next year 𝜙 3 will be executed . while resolving, the contextual information of m 2 , such as "next year", will be incorporated into the representation of m 6 as they are referring to the same entity. therefore, φ 3 next to "next year" will be explicitly resolved to m 6 with reference to the common context. guided by the four intuitions above, this paper proposes a crafted end-to-end neural model for tackling the two tasks jointly, where the positions of zero pronouns are unknown in advance. within the model, both tasks share the same token embedding layer (intuition 1). inspired by @xcite and @xcite , we introduce high-order inference module to enrich mentions' contextual information (intuition 2) and introduce mention detection loss to help mention recognition (intuition 3). additionally, considering the exclusive relations between the position of gaps and spans, we propose a two-stage interaction mechanism to realize the exclusivity in both embedding level and detection scoring level (intuition 4). last but not the least, to learn proper representation for both gaps and tokens in the same space, a gap-masked selfattention model is designed which enables gaps to take advantage of the contextual messages from their surrounding tokens, without polluting the representations of tokens. to summarize, our contributions are as follows: extensive experiments conducted on the widelyused ontonotes 5.0 chinese dataset shows that our model could outperform the state-of-the-art approaches on both tasks. the code is available at https://github.com/cheniison/e2e-joint-coref . ## related work in this section, we first briefly introduce previous works on zero pronoun resolution and non-zero coreference resolution respectively, and then introduce some joint works. according to whether positions of zero pronouns are given, previous works of zero pronoun resolution can be classified into two categories: resolution with gold positions of zero pronouns and resolution without gold positions of zero pronouns. for resolution with gold positions of zero pronouns, previous work concentrates on finding an antecedent for a given zero pronoun. @xcite first use deep learning models to compute embeddings for zero pronouns and candidate antecedents. @xcite proposes a method of generating an amount of pseudo zero pronoun data automatically so that models can be pre-trained on the pseudo data. @xcite introduces reinforcement learning model into zero pronoun resolution to integrate local and global resolution information. @xcite considers bidirectional attention between zero pronouns and candidate antecedents and proposes a pairwise-margin loss and a similarity constraint to optimize their model. for resolution without gold positions of zero pronouns, positions of zero pronouns need to be identified first. @xcite and bouzid and zribi (2020) first find the positions of zero pronouns, after that they filter out non-referential pronouns and determine antecedents. in the work of @xcite , zero pronouns can establish coreference relation with each other to help zero pronouns find overt antecedents far away from them. these works rely on handcraft features and suffer from error propagation. to solve these problems, @xcite tackles context reconstruction in an end-to-end way. it formulates pronouns detection as a sequence labeling task and uses pronoun masking mechanism to combine the detection and the resolution modules. @xcite presents a bert-based multi-task model which handles zero pronoun recovery, zero pronoun detection and zero pronoun resolution jointly. non-zero coreference resolution is an important task in natural language processing in all languages. recently, end-to-end coreference resolution models achieve the state-of-the-art performance in the nonzero coreference task. @xcite proposes the first neural end-to-end coreference resolution model which computes scores of span and mention pairs jointly to detect mentions and predict antecedents. to avoid global inconsistency, @xcite proposes a higher-order model which iteratively updates mention representations. @xcite learns mention detection and mention clustering jointly and proposes a biaffine attention method to compute antecedent scores. instead of using word2vec and elmo, @xcite apply bert to get better span representations. there are also some efforts on solving the two tasks jointly. @xcite proposes an ilp-based model integrating the zero anaphora resolver with a coreference resolver. @xcite exploits zero pronouns to improve non-zero coreference resolution by refining the syntactic parser and training examples. shibata and kurohashi (2018) presents an entity-based joint model for japanese coreference resolution and predicate argument structure analysis. however, these works either rely on artificially designed features and syntactic information, or need the golden positions of the zero pronouns, which are costly and not practical. in this paper, we first propose an end-to-end neural model for solving the two tasks jointly. ## model in this section, we first formally define the zero & non-zero joint coreference resolution task, and then present the details of the joint model. ## experiments in this section, we study the effectiveness of our model. we firstly introduce the datasets and the metrics we use to evaluate models. then we present the details of experimental settings in the model. finally, we display our experimental results. ## conclusions in this paper, we present an end-to-end joint coreference resolution model which tackles zero pronoun resolution and non-zero coreference resolution jointly. to get proper representations for gaps and tokens, we propose a gap-masked self-attention model which puts embeddings of tokens and gaps into the same space without polluting token representations. additionally, to make full use of the exclusive relationship between zero pronoun and mentions, we propose a two-stage interaction mechanism which incorporates information of relevant units while calculating unit scores. extensive experiments on ontonotes 5.0 chinese corpus demonstrate the effectiveness of our model.