ZPR 2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT (2020)

Abstract: Zero pronoun recovery and resolution aim at recovering the dropped pronoun and pointing out its anaphoric mentions, respectively. We propose to better explore their interaction by solving both tasks together, while the previous work treats them separately. For zero pronoun resolution, we study this task in a more realistic setting, where no parsing trees or only automatic trees are available, while most previous work assumes gold trees. Experiments on two benchmarks show that joint modeling significantly outperforms our baseline that already beats the previous state of the arts.

## introduction zero pronoun (zp) is a linguistic phenomenon where a pronoun is dropped for simplicity. figure 1 shows an example, where two pronouns at positions 1 and 2 are omitted. they both refer to "fπ (the police)" in the sentence beginning and their original form is "÷ï (they)". the situation of dropping pronouns happens in most languages. while this phenomenon is not frequent in non-pro-drop languages, such as english, it is extremely severe for pro-drop languages, such as chinese. in addition, dropped pronouns happens more frequently in conversations than in news. our preliminary statistics of chinese shows that 59.2% pronouns are dropped in a corpus of casual dialogues domain, while the number is just 41.6% in another data of broadcast news. in nlp, dropped pronouns can cause loss of important information, such as the subject or object of the central predicate in a sentence, introducing ambiguity to applications such as machine translation @xcite @xcite , question answering @xcite @xcite [ fπ ] ë ÿ / w -™ hˆ 1 ⌃ ™∞ å ae⇧ § ⇥ã 2 â ⇧⌃ h≈ ⇥ [ the police ] suspected that this is a criminal case about illegal guns , 1 brought the guns and bags to the city 2 to deal with the case . figure 1: an zero pronoun example and its english translation, where 1 and 2 are zero pronouns pointing to the span in square brackets. @xcite and dialogue understanding @xcite . as a result, zero pronouns have recently received much research attention @xcite . we study chinese zero pronoun in dialogue settings. there are two long-existing tasks namely zero pronoun recovery, which aims at recovering the original pronoun (such as "÷ (he)" and "y (she)"), and zero pronoun resolution, where the goal is to pinpoint the mention that each dropped pronoun refers to. intuitively, the results of the two tasks highly interact with each other. taking figure 1 as an example, it will be much easier to resolute 1 to "fπ (the police)" rather than "-™ hˆ(criminal case about illegal guns)" if we know 1 corresponds to "÷ï (they)". similarly, it would be more likely to recover 1 as "÷ï (they)" than other candidate pronouns, if we know 1 points to "fπ (the police)". despite their high correlation, previous work considers them as irrelevant tasks, solving them separately by different models. this can waste training resources, as each task has a limited number of labeled instances, and thus data sparsity can limit model performance. besides, we believe that it is unnecessary to keep a specific model for each task, as they can be close enough to be solved together. in addition, most zero pronoun resolution research @xcite @xcite @xcite @xcite as-sumes gold trees being available with the positions of zero pronouns, which is unrealistic in practical applications. during decoding, a zero pronoun resolution model has to rely on automatic trees and zero pronoun detection, thus suffering from error propagation. in this paper, we propose to jointly solve both tasks under a heterogeneous multi-task learning framework, where each data point only has the annotation of one task, to benefit from the supervised data of both tasks. as the result, we enjoy the benefit of more supervised training data. to improve the robustness of heterogeneous training and introduce more supervision, we introduce zero pronoun detection, a common sub-task for both zp resolution and recovery. zero pronoun detection is a binaryclassification task aiming to detect whether a word space has a dropped pronoun. we consider zp recovery as a sequence labeling task, regarding whether each word space has a dropped pronoun and what type the pronoun is. zp resolution is solved as extractive reading comprehension @xcite , where each word space is taken as a query and its anaphoric mentions are treated as the answers. for non-zp spaces where there is no corresponding anaphoric mentions, we assign the sentence beginning (span [0,0]) as the answer. experiments on two benchmarks, ontonotes 5.0 foot_0 (zp resolution) and baiduzhdiao @xcite (zp recovery), show that joint modeling gives us 1.5+ absolute f1-score gains for both tasks over our very strong baselines using bert @xcite . our overall system gives an dramatic improvement of 3.5 f1 points over previous stateof-the-art results on both tasks. ## related work previous work considers zero pronoun resolution and recovery separately. for zero pronoun recovery, existing methods can be classified according to the types of annotations they use. one line of work @xcite simply relies on the human annotations, solving the task as sequence labeling. the other line of work @xcite @xcite mines weak supervision signals from a large bilingual parallel corpus, where the other language is non-prodrop with fewer pronoun drops. the latter requires massive training data, and the mt performance is the primary goal, thus we follow the first line of research using human-annotated data. @xcite studied zero pronoun resolution in multi-turn dialogues, claiming that their model does not rely on parsing trees to extract zp positions and noun phrase as resolution candidates. however, they only consider the dropped pronouns that correspond to one of the dialogue participant. as a result, they only explore a small subset of the entire zp resolution problem, and their task is closer to zero pronoun recovery. most similar to our work, @xcite converted zero pronoun resolution as a machine reading comprehension task @xcite in order to automatically construct a large-scale pseudo dataset for model pretraining. however, their model finetuning and evaluation with benchmark data still rely on human-annotated trees and gold zero pronoun positions. as a result, it is still uncertain what performance a model can achieve without such gold inputs. we address both issues in the joint task. our work is inspired by the recent advances of heterogeneous multi-task learning using bert @xcite , which combines the supervised data of several related tasks to achieve further improvements. in particular, @xcite utilize this framework to jointly solve glue tasks @xcite . but their experiments show that multitask learning does not help across all tasks. our work takes a similar spirit, and our contribution is mainly on the zero pronoun tasks. in addition, we find that it helps the robustness of multi-task learning to add a common sub-task (e.g. zero pronoun detection in our case) for additional supervision and alleviating annotation variances, if such a subtask is available. ## model as shown in figure 2 , we model zp recovery (f rec ), zp resolution (f res ), and the auxiliary zp detection (f det ) task with multi-task learning, where bert @xcite is used to represent each input sentence s 1 . . . s n of n words to provide shared features. ## experiments we study the effectiveness of jointly modeling zp resolution, recovery and detection. ## conclusion we studied the effectiveness of jointly modeling zp recovery and resolution using the recently introduced multi-task learning + bert framework. to alleviate the data magnitude imbalance problem, we introduce zp detection as a common auxiliary sub-task for extra supervision. experiments on two benchmarks show that our model is consistently better than previous results under various settings, and that the auxiliary zp detection sub-task can make the training process more robust.