Anaphoric Zero Pronoun Identification: A Multilingual Approach (2020)

Abstract: Pro-drop languages such as Arabic, Chinese, Italian or Japanese allow morphologically null but referential arguments in certain syntactic positions, called anaphoric zero-pronouns. Much NLP work on anaphoric zero-pronouns (AZP) is based on gold mentions, but models for their identification are a fundamental prerequisite for their resolution in real-life applications. Such identification requires complex language understanding and knowledge of real-world entities. Transfer learning models, such as BERT, have recently shown to learn surface, syntactic, and semantic information,which can be very useful in recognizing AZPs. We propose a BERT-based multilingual model for AZP identification from predicted zero pronoun positions, and evaluate it on the Arabic and Chinese portions of OntoNotes 5.0. As far as we know, this is the first neural network model of AZP identification for Arabic; and our approach outperforms the stateof-the-art for Chinese. Experiment results suggest that BERT implicitly encode information about AZPs through their surrounding context.

## introduction empty categories provide an important source of syntactic information about the phonetically null arguments in prodrop languages such as arabic @xcite , chinese @xcite , italian (di @xcite , japanese @xcite , and others @xcite . the use of empty categories started with penn treebanks @xcite , followed by arabic treebank @xcite , chinese treebank @xcite and other penn-style series. empty categories are used to represent traces, such as, movement operations in interrogative sentence, also to represent right node raising which is a shared argument in the rightmost constituent of a coordinate structure. another usage of empty categories is zero-pronouns (zp) which are omitted pronouns in places where they are expected to be, and function as overt pronouns. anaphoric zero pronouns (azp) are zps that corefer to one or more noun phrases in a preceding text. the following example of an azp comes from the arabic section of ontonotes: ironically, bush did not show any enthusiasm for the international conference, because since the beginning, (he) wanted to attend another conference ... in the example, the zp indicated with '*' refers to the gap position of an omitted pronoun (in ontonotes 5.0, zps are denoted as * in arabic text, and *pro* in chinese). the omitted pronoun refers to a singular masculine person that has been mentioned previously, in the example "bush/ ‫ﺑ‬ ‫ﻮ‬ ‫ش‬ ". in arabic, we deduce the reference information from the context, especially the verb that precedes the azp, in the example the verb is "wanted/ ‫ﯾ‬ ‫ﺮ‬ ‫ﯾ‬ ‫ﺪ‬ ". since english is not a pro-drop language @xcite , the azp gap position is translated into an overt pronoun (he). the azp problem has inspired much research because it benefits many natural language processing tasks such as machine translation @xcite , and coreference resolution @xcite . recently, there has been a great deal of research on azps for chinese @xcite @xcite @xcite , arabic @xcite , japanese @xcite , korean @xcite , and other languages @xcite . a major drawback of many existing studies is the assumption that azp locations are given; hence, they focus primarily on resolving azps to their correct antecedent. however, such assumption does not reflect real-life applications. another drawback is that current azp identification systems rely on language-dependent features and fail to detect many azp locations. in addition, some languages do not have an azp identification system, one of which is arabic. to alleviate the above-mentioned limitations, we investigate the azp identification task and study if the recently achieved state-of-the-art transfer learning methods, such as bert @xcite , can work well on identifying azps. typically, azp identification task consists of two steps. the first is the extraction step where potential zp locations are extracted. the extraction procedure is based on heuristics and depend on the target language structure. the second step is classification step which determines which of the extracted candidate are azp. the classification step is more challenging because of the varieties and size of the extracted candidates. in this paper, we propose a multilingual approach to azp identification based on bert. we make three main contributions: the rest of the paper is organized as follows. we review arabic and chinese zp-related literature, and other languages as well in section 2. we explain our proposed model in section 3. we discuss the evaluation settings in section 4. we show the results and discuss them in section 5. we conclude in section 6. 2 related work azp identification task has been considered independently, but also as a prerequisite step before azp resolution task because the detection has a heavy impact on the resolution @xcite . arabic: there have been a few studies devoted to azps and empty categories in general. green et al. (2009) proposed a conditional-random-field (crf) sequence classifier to detect arabic noun phrases, and captured zps implicitly. bakr et al. ( 2009 ) applied a statistical approach to detect empty categories. @xcite proposed a pipeline made of maximum entropy classifiers which jointly make a crf to retrieve arabic empty categories. @xcite proposed the first neural model for resolving arabic azp, but they did not consider the azp identification step. as far as we know, no previous work has considered arabic azp identification. chinese: @xcite studied azp resolution and applied a rule-based approach that employed hobbs algorithm @xcite to resolve zps in the chinese treebank; however, did not attempt to automatically identify azp. @xcite is another rule-based approach, for azp resolution and also used a set of handengineered rules to identify azps. @xcite , the first machine learning approach to chinese azps identification and resolution, by applying decision trees incorporated with a set of syntactic and positional features. @xcite employed a tree kernel-based approach to azp identification and resolution. @xcite is an extension of @xcite , they incorporated contextual features for azp resolution and applied a combination of syntactic, lexical and other features for the identification. chen and ng (2014 to detect azps and proposed an improved deep memory network to resolve azps; and @xcite , applied an attention-based neural network to resolve azps and enhanced the performance by training on automatically generated large-scale training data. @xcite focused primarily on azp identification and applied an lstm neural-network with text and part-of-speech information. @xcite , also used an attention-based model, but combined their network with @xcite features to resolve azps. @xcite applied the same heuristics in @xcite to identify azps and applied a collaborative-filtering approach to resolve azps. @xcite identified azps using a learning-based classifier with semantic, lexical and syntactic features, and used coreferential chain information to improve azp resolution. other languages: there has been also a great deal of research on identification and resolution of azps, particularly in japanese @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , but also in other languages, including korean @xcite , spanish @xcite , portuguese @xcite , romanian @xcite , bulgarian @xcite , and sanskrit (gopal and jha, 2017). @xcite proposed the first multilingual approach for azp resolution. current approaches suffer from one (or more) of the following. first, they assume azps are available; so they focus mainly on the resolution part. second, they apply on a private or very small size corpus. third, they rely on an extensive set of features or language-dependent rules to identify azp. ## model to identify azps, context understanding and semantic knowledge of entities are essential in chinese @xcite as well as in arabic which requires, in addition, deep understanding of its complex morphology (alnajadat, 2017). recently, it has been shown that bert @xcite can capture structural properties of a language, such as its surface, semantic, and syntactic aspects @xcite which seems suitable for the azp identification task. therefore, we use bert to produce representations for zp candidates. our model is a binary classifier that takes an automatically predicted zp candidate as input, and classifies it as an azp or not. in this section, we first give an overview of bert and its adaptation modes. we then describe how we generate azp candidates, and how we represent them. finally, we present the training objective and hyperparameter tuning settings. ## results azp identification results for arabic are in prior works @xcite @xcite @xcite ) evaluate azp identification under two settings: gold and system parse because annotation quality can impact the number of recovering candidates in the extraction step. gold annotations are available for both languages and we also automatically parse the data with syntactic trees using the berkeley parser @xcite which is a pre-trained parser using neural networks and self-attention. ## conclusion we proposed a bert-based model for azp identification. our approach is multilingual, and we evaluate on arabic and chinese portions of ontonotes. the model is the first to deal with arabic azp identification and the experiments demonstrated that our method surpasses the state-of-the-art on chinese azps. in addition, our experimental results show that bert learn about anaphoric zero-pronouns through their surrounding context.