Joint Coreference Resolution for Zeros and non-Zeros in A rabic (2022)

Abstract: Most existing proposals about anaphoric zero pronoun (AZP) resolution regard full mention coreference and AZP resolution as two independent tasks, even though the two tasks are clearly related. The main issues that need tackling to develop a joint model for zero and non-zero mentions are the difference between the two types of arguments (zero pronouns, being null, provide no nominal information) and the lack of annotated datasets of a suitable size in which both types of arguments are annotated for languages other than Chinese and Japanese. In this paper, we introduce two architectures for jointly resolving AZPs and non-AZPs, and evaluate them on Arabic, a language for which, as far as we know, there has been no prior work on joint resolution. Doing this also required creating a new version of the Arabic subset of the standard coreference resolution dataset used for the CoNLL-2012 shared task (Pradhan et al.,2012) in which both zeros and non-zeros are included in a single dataset.

## introduction in pronoun-dropping (pro-drop) languages such as arabic @xcite , chinese @xcite , italian @xcite and other romance languages (e.g., portuguese, spanish), japanese @xcite , and others @xcite , arguments in syntactic positions in which a pronoun is used in english can be omitted. such arguments-sometimes called null arguments, empty arguments, or zeros, and called anaphoric zero pronouns (azp) here when they are anaphoric, are illustrated by the following example: ironically, bush did not show any enthusiasm for the international conference, because bush since the beginning, (he) wanted to attend another conference ... in the example, the '*' is an anaphoric zero pronoun-a gap replacing an omitted pronoun which refers to a previously mentioned noun, i.e. bush. 1 although azps are common in pro-drop languages @xcite , they are typically not considered in standard coreference resolution architectures. existing coreference resolution systems for arabic would cluster the overt mentions of bush, but not the azp position; vice versa, azp resolution systems would resolve the azp, to one of the previous mentions, but not other mentions. the main reason for this is that azps are empty mentions, meaning that it is not possible to encode features commonly used in coreference systems-the head, syntactic and lexical features as in pre-neural systems. as a result, papers such as @xcite have shown that treating the resolution of azps and realized mentions separately is beneficial. however, it has been shown that the more recent language models and end-to-end systems do not suffer from these issues to the same extent. bert, for example, learns surface, semantic and syntactic features of the whole context @xcite and it has been shown that bert encodes sufficient information about azps within its layers to achieve reasonable performance @xcite . however, these findings have not yet led to many coreference resolution models attempting to resolve both types of mentions in a single learning framework (in fact, we are only aware of two, @xcite , the second of which was just proposed) and these have not been evaluated with arabic. in this paper, we discuss two methods for jointly clustering azps and non-azps, that we evaluate on arabic: a pipeline and a joint learning architecture. in order to train and test these two architectures, however, it was also necessary to create a new version of the arabic portion of the conll-2012 shared task corpus in which both zeros and non-zeros are annotated in the same documents. to summarize, the contributions of this paper are as follows: ## related work most existing works regard coreference resolution and azp resolution as two independent tasks. many studies were dedicated to arabic coreference resolution using @xcite @xcite @xcite @xcite @xcite , but azps were excluded from the dataset so no work considered them. @xcite proposed a bert-base approach to resolve azps to their true antecedent, but they did not resolve other mentions. there have been a few proposals on solving the two tasks jointly for other languages. @xcite integrated the azp resolver with a coreference resolution system using an integerlinear-programming model. @xcite employed azps to improve the coreference resolution of non-azps using a syntactic parser. @xcite proposed an entity-based joint coreference resolution and predicate argument structure analysis for japanese. however, these works relied on language-specific features and some assumed the presence of azps. there are two end-to-end neural proposals about learning azps and non-azps together. the first proposal is by @xcite who combined tokens and azp gaps representations using an encoder. the two representations interact in a twostage mechanism to learn their coreference infor-mation, as shown in figure 5 . the second proposal, just published, is by @xcite , who proposed the corefdpr architecture. corefdpr consists of four components: the input representation layer, coreference resolution layer, pronoun recovery layer and general crf layer. in our experiments, we only compared our results with the first proposal because the second system was only evaluated on the chinese conversational speech of ontonotes foot_1 and the model is not publicly available which makes it difficult to compare our results with theirs. ## an extended version of the conll arabic dataset with azps the goal of the conll-2012 coreference shared task is to learn coreference resolution for three languages (english, chinese and arabic). however, azps were excluded from the task even though they are annotated in ontonotes arabic and chinese. this was because considering azps decreased the overall performance on arabic and chinese @xcite , but not on english because it is not a pro-drop language @xcite . so in order to study joint coreference resolution for explicit mentions and zero anaphors, we had to create a novel version of the conll-2012 dataset in which azps and all related information are included. the conll-2012 annotation layers consists of 13 layers and they are in appendix a. existing proposals evaluated their azp systems using ontonotes normal forms (onf) foot_2 . they are annotated with azps and other mentions; however, they are not as well-prepared as conll-2012. to create a conll-like dataset with azps, we extract azps from onf and add them to the alreadyexisting conll files. the goal of the new dataset is to be suitable for clustering azps and non-azps, and can be compared with previous proposals that did not consider azps and as well as with future works that consider them. to include azps and their information (e.g., partof-speech and parse tree) to conll-2012, we can use onf. however, while adding azps to the clusters, we realized that there is one difficulty:some coreference chains only exist in onf, but not in conll-2012. these are clusters consisting of only one mention and one azp, as in the example illustrated in figure 1 . chain 71 has two mentions, an azp (denoted with *) and a mention. since conll-2012 does not consider azps in coreference chains, this cluster would only have a single mention because conll-2012 removed azps (these clusters are known as singletons, contains only one mention). our new dataset includes azps; therefore, such clusters should be included. to add them to the existing conll-2012, we have to assign them a new cluster. we did this by writing a script that automatically extracts azps from onf and adds them in conll-2012 following these steps: adding azps to conl-2012 is beneficial to learn how to resolve them with other mentions or can be useful for future conll-shared tasks and any other related nlp task. of arabic ontonotes will be made available with the next release of ontonotes. the distribution of documents, sentences, words, and azps of this extended dataset are in ## the models earlier proposals resolved azps based on the antecedents that are in the same sentence as the azp or two sentences away @xcite @xcite @xcite @xcite ). however, it has been shown that learning mention coreference in the whole document is beneficial for azp resolution @xcite . therefore, we apply two novel methods for resolving azps using clusters and coreference chains. the pipeline resolves azps based on the output clusters from the coreference resolution model while the joint learning learns how to resolve azps from the coreference chains, we show an example of these two in figure 2 . in the example, the pipeline resolves azps to clusters, instead of mentions and the joint learning finds the coreference chains for mentions, including azps. earlier proposals suffered from two main problems. first, they consider a limited number of candidates (i.e mentions in two sentences away from the azp) as possible true antecedents; however, the true antecedent might be far away from the azp. second, other mentions can share salient context as the true antecedent which can introduce more noise to the learning. our methods mitigate these problems by considering all mentions in the document and employing more relevant information. the pipeline resolves azps based on clusters which decreases dramatically the number of azp candidates. the joint learning resolves azps using coreference chains which incorporates broader context for azps, insufficient contexts results in many errors @xcite . ## results we compare the results of the pipeline and joint learning models with the results of @xcite . we followed chen et al. (2021)'s approach for hyperparameter tunning, but we changed the language model to arabert-base (antoun et al., 2020). we evaluate two tasks. first, we assess the results at joint coreference resolution of both azps and non-azps. second, we evaluate azp resolution only. unlike previous proposals that resolve azps with their antecedents, the azps of our methods and the chen et al.'s (2021) model resolve azps differently. the pipeline uses the output clusters, the joint learning uses the coreference chains and chen et al. (2021) uses two scoring components. ## discussion the main difference between our joint learning approach and @xcite is how azps are detected and learned. in our approach, we detect azps initially before we cluster them with other mentions, while @xcite model learns clustering azps and mentions in an end-to-end system. our results appear to confirm earlier results that considering azp identification end-to-end in the coreference resolution task can negatively affect the performance on the task @xcite one possible explanation might be the overall performance of the mention detection on non-azps is better than azps @xcite . @xcite consider every gap as a candidate azp, which increases the space of possible candidates and affects their detection recall. to mitigate this problem, we use a different neural component for azp detection. the azp identification that we used in the joint learning and pipeline settings only considers gaps that appear after verbs which limits the number of candidates. moreover, the azps in the joint learning have explicit tags which might have resulted in their correct detection, which could be why the approach achieved better results. the main limitation of our proposed approaches is if the azp identification fails to detect many azps in the test phase, it might have dropped the evaluation of the coreference resolution and azp resolution. pre-training bert with azps can be beneficial. existing language models (lms) learn by masking words or perturbing their order @xcite , but this is not applicable to azps. @xcite have shown two approaches to improve lms so they work for azps, first by introducing a new pre-training task and second by a new fine-tuning technique. they showed an increased performance for azp resolution for japanese. in future works, we intend to pre-train a large-scale lm using their methods and see if it can improve the performance of the azp and coreference resolution tasks. ## conclusion we proposed two architectures to resolve azps and non-azps jointly. the first approach is in a pipeline setting and the second in a joint learning representation. the joint learning outperformed the pipeline and another approach @xcite in the joint coreference resolution. we also extended the arabic portion of conll-2012 to include azps which will be suitable for future works and sharedtasks that resolves azps and non-azps together.