Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution (2021)

Abstract: Masked language models (MLMs) have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). To further improve this approach, in this study, we made two proposals. The first is a new pretraining task that trains MLMs on anaphoric relations with explicit supervision, and the second proposal is a new finetuning method that remedies a notorious issue, the pretrain-finetune discrepancy. Our experiments on Japanese ZAR demonstrated that our two proposals boost the state-of-the-art performance, and our detailed analysis provides new insights on the remaining challenges.

## introduction in pronoun-dropping languages such as japanese and chinese, the semantic arguments of predicates can be omitted from sentences. as shown in figure 1 , the semantic subject of the predicate used is omitted and represented by φ, which is called zero pronoun. this pronoun refers to the criminal in the first sentence. this way, the task of recognizing the antecedents of zero pronouns is called zero anaphora resolution (zar). this study focuses on japanese zar. zar is a challenging task because it requires reasoning with commonsense knowledge about the semantic associations between zero pronouns and the local contexts of their preceding antecedents. as shown in figure 1 , to identify the omitted semantic subject of used, the model should know the semantic relationship between the criminal's weapon and a hammer, namely, a hammer is likely to be used as a weapon for murder and thus was used by the criminal, is crucial. we hereinafter refer to such knowledge as anaphoric relational knowledge. a conventional approach to acquire anaphoric relational knowledge is to collect predicate-argument *work done while at tohoku university. it seems that ! used a hammer. pairs from large-scale raw corpora and then, use them as features @xcite @xcite , or using selectional preference probability @xcite in machine learning models. a modern approach is to use masked language models (mlms) @xcite , which is effective in implicitly capturing anaphoric relational knowledge. in fact, recent studies used pretrained mlms and achieved drastic performance improvements in the tasks that require anaphoric relational knowledge, including japanese zar @xcite @xcite . to get more out of mlms, in this paper, we propose a new training framework that pretrains and finetunes mlms specialized for zar. the idea is two-fold. first, we design a new pretraining task that trains mlms with explicit supervision on anaphoric relations. many current pretraining tasks adopt a form of the cloze task, where each mlm is trained by predicting the original token filling the [mask] token. although this task provides each mlm with no supervision on anaphoric relations, the mlm implicitly learns about them. in contrast, our new task, called the pseudo zero pronoun resolution (pzero), provides supervision on anaphoric relations. pzero assumes that when the same noun phrases (nps) appear multiple times in a text, they are coreferent. from this assumption, we mask one of such multiple-occurring nps as a pseudo zero the university has surveyed teachers' means of transport and found that most teachers use the train. second, we address the issue called pretrainfinetune discrepancy @xcite . generally, some part of an mlm is changed for finetuning on a target task, e.g., discarding the pretrained parameters at the last layer or adding randomly-initialized new parameters. such changes in the architecture are known to be obstacles to the adaptation of pretrained mlms to target tasks. to solve this issue, we design a new zar model that takes over all the pretrained parameters of an mlm to the zar task with minimal modification. this realizes a smoother adaptation of the anaphoric relational knowledge acquired during pretraining to zar. through experiments on japanese zar, we verify the effectiveness of pzero and the combination of pzero and our new zar model. also, our analysis offers insights into the remaining challenges for japanese zar. to sum up, our main contributions are as follows: ## japanese zero anaphora resolution japanese zar is often treated as a part of the predicate-argument structure analysis, which is the task of identifying semantic arguments for each predicate in a text. in the naist text corpus (ntc) @xcite , a standard benchmark dataset that we used in our experiments, each predicate is annotated with the arguments filling either of the three most common argument roles: the nominative (nom), accusative (acc), or dative (dat) roles. if an argument of a predicate is a syntactic dependant of the predicate, we say that the argument is a syntactically dependent argument (dep) and is relatively easy to identify. if an argument of a predicate is omitted, in contrast, we say that the argument position is filled by zero pronouns. this study is focused on recognizing such zero pronouns and identifying antecedents. the zar task is classified into the following three categories according to the positions of the arguments of a given predicate (i.e., the antecedent of a given zero pronoun): intra-sentential (intra): the arguments within the same sentence where the predicate appears. inter-sentential (inter): the arguments in the sentences preceding the predicate. exophoric: the arguments (entities) that exist outside the text. these are categorized into one of three types: author, reader, and general. 3 the identification of inter-sentential and exophoric arguments is an especially difficult task @xcite . for inter-sentential arguments, a model has to search the whole document. for exophoric arguments, a model has to deal with entities outside the document. because of this difficulty, many previous studies have exclusively focused on the intra-sentential task. in this paper, not only the intra-sentential task, we also treat intersentential and exophoric tasks as the same task formulations, as in previous studies. ## zar models supposing that our model obtains anaphoric relational knowledge for zar by pretraining on pzero, we design a zar model, as it can best utilize such knowledge during finetuning. in this section, we first describe an argument selection model (section 4.1), which is considered the most straightforward adaptation of a pretrained model for zar. then, we propose a novel model that addresses the pretrain-finetune discrepancy (section 4.2). ## experimental settings pzero dataset japanese wikipedia corpus (wikipedia) is the origin of the training data of pzero. 5 all the nps in the corpus are pzero targets. to detect nps, we parsed wikipedia using the japanese dependency parser cabocha @xcite and applied a heuristic rule based on part-of-speech tags. we used @xmath0. from 17.4m sentences in wikipedia, we obtained 17.3m instances as training data for pzero. zar dataset for the zar task, we used naist text corpus 1.5 (ntc) @xcite , which is a standard benchmark dataset of this task @xcite @xcite . we used the training, development, and test splits proposed by @xcite . the numbers of intra-sentential, inter-sentential, and exophoric for the training/test instances were 18068/6159, 11175/4081, and 13676/3826, respectively. the ntc details are shown in appendix a. the evaluation script corresponds to that of @xcite . model our implementation is based on the transformers library @xcite . we used the pretrained parameters of the bert-base-japanese model as the initial parameters of our pretraining models. we trained our model using an adam optimizer @xcite with warm-up steps. as a loss function, we used cross-entropy for the ## results and analysis we have two distinct goals in this experiment, that is, to investigate the effectiveness of (1) pretraining on pzero and (2) finetuning on as-pzero. to achieve these goals, we first compare our as and as-pzero models with previous studies to ensure that our models are strong enough in a conventional experimental setting, i.e., the intra-sentential setting (section 6.1). then we investigate (1) and ( 2 ) based on inter-sentential setting (section 6.2). ## conclusion in this study, we proposed a new pretraining task, pzero, which aims to explicitly teach the model anaphoric relational knowledge necessary for zar. we also proposed a zar model to remedy the pretrain-finetune discrepancy. both the proposed methods improved the performance of japanese zar, leading to a new state-of-the-art performance. our analysis suggests that the hard subcategories of zar; distant arguments and passive predicates are still challenging.