J apanese Zero Anaphora Resolution Can Benefit from Parallel Texts Through Neural Transfer Learning (2021)

Abstract: Parallel texts of Japanese and a non-pro-drop language have the potential of improving the performance of Japanese zero anaphora resolution (ZAR) because pronouns dropped in the former are usually mentioned explicitly in the latter. However, rule-based cross-lingual transfer is hampered by error propagation in an NLP pipeline and the frequent lack of transparency in translation correspondences. In this paper, we propose implicit transfer by injecting machine translation (MT) as an intermediate task between pretraining and ZAR. We employ a pretrained BERT model to initialize the encoder part of the encoder-decoder model for MT, and eject the encoder part for fine-tuning on ZAR. The proposed framework empirically demonstrates that ZAR performance can be improved by transfer learning from MT. In addition, we find that the incorporation of the masked language model training into MT leads to further gains.

## introduction figuring out who did what to whom is an essential part of natural language understanding. this is, however, especially challenging for so-called prodrop languages like japanese and chinese because they usually omit pronouns that are inferable from context. the task of identifying the referent of such a dropped element, as illustrated in figure 1(a) , is referred to as zero anaphora resolution (zar). although japanese zar saw a performance boost with the introduction of bert @xcite , there is still a good amount of room for improvement. a major barrier to improvement is the scarcity of training data. the number of annotated sentences is the order of tens of thousands or less @xcite @xcite , and the considerable linguistic expertise required for annotation makes drastic corpus expansion impractical. previous attempts to overcome this limitation exploit orders-of-magnitude larger parallel texts of japanese and english, a non-pro-drop language @xcite . the key idea is that japanese zero pronouns can be recovered from parallel texts because they are usually mentioned explicitly in english, as in figure 1(b) . if translation correspondences are identified and the anaphoric relation in english is identified, then we can identify the antecedent of the omitted argument in japanese. their rule-based transfer from english to japanese had met with limited success, however. it is prone to error propagation due to its dependence on word alignment, parsing, and english coreference resolution. more importantly, the great linguistic differences between the two language often lead to parallel sentences without transparent syntactic correspondences (figure 1(c) ). in this paper, we propose neural transfer learning from machine translation (mt). by generating english translations, a neural mt model should be able to implicitly recover omitted japanese pronouns, thanks to its expressiveness and large training data. we expect the knowledge gained during mt training to be transferred to zar. given that state-of-the-art zar models are based on bert @xcite @xcite , it is a natural choice to explore intermediate task transfer learning @xcite @xcite : a pretrained bert model is first trained on mt and the resultant model is then fine-tuned on zar. 1 a key challenge to this approach is a mismatch in model architectures. while bert is an encoder, the dominant paradigm of neural mt is the encoder-decoder. although both share transformer @xcite as the building block, @xmath0. (c) cable.@xmath1.@xmath2.pst especially 𝜙 ! =nom like.@xmath3.npst @xmath4. because japanese generally avoids having inanimate agents with animate patients, the english inanimate-subject sentence corresponds to two animate-subject clauses in japanese, with two exophoric references to the reader (i.e., you). we use a pretrained bert model to initialize the encoder part of the encoder-decoder model for mt. while this technique was previously used by @xcite and @xcite , they both aimed at improving mt performance. we show that by ejecting the encoder part for use in fine-tuning (figure 2 ), we can achieve performance improvements in zar. we also demonstrate further improvements can be brought by incorporating encoder-side masked language model (mlm) training into the intermediate training on mt. ## proposed method we adopt a zar model of @xcite , which adds a thin layer on top of bert during fine-tuning to solve zar and related tasks (section 3.1). instead of directly moving from mlm pretraining to fine-tuning on zar, we inject mt as an intermediate task (section 3.2). in addition, we introduce the mlm training objective during the intermediate training (section 3.3). ## conclusion in this paper, we proposed to exploit parallel texts for japanese zero anaphora resolution (zar) by inserting machine translation (mt) as an intermediate task between masked language model (mlm) pretraining and fine-tuning on zar. although previous studies reported negative results on the use of mt as an intermediate task, we demonstrated that it did work for japanese zar. our analysis suggests that the intermediate training on mt simultaneously improved the model's ability to translate japanese zero pronouns and the zar performance. we bridged the gap between bert-based zar and the encoder-decoder architecture for mt by initializing the encoder part of the mt model with a pretrained bert. previous studies focusing on mt reported mixed results on this approach, but again, we demonstrated its considerable positive impact on zar. we found that incorporating the mlm objective into the intermediate training was particularly effective. our experimental results were consistent with the speculation that mlm mitigated catastrophic forgetting during intermediate training. with neural transfer learning, we successfully revived the old idea that japanese zar can benefit from parallel texts @xcite . thanks to the astonishing flexibility of neural networks, we would probably be able to connect zar to other tasks through transfer learning. ours baseline (b) 江戸に 剣術修行に 来た 龍馬は 、 定吉の 道場に 入門する 。 @xmath5.pst @xmath6, who came to the edo period [unk] 1603-1867 [unk] to learn swords, entered a training school run by sakakichi. he is left to his son, shigeta. he is a gentle father who watches the growth of ryoma and sana . [null] (a) 第七十四回 全国 高校 ラグビー フットボール 大会 準決勝の 五日 、 近鉄花園ラグビー場の スタンドでは 大阪 朝鮮 高級 学校 ラグビー 部員 二十五人が 青い ウインドブレーカー 姿で 観戦した 。 𝜙 ! -nom 同 ラグビー 場から わずか 一・五キロの 至近 距離に あり ながら 、 same rugby 𝜙!-nom 1.5km=gen @xmath7.ger only twenty-two students of osaka korean pro-pyongyang korean high school students watched the final at kintetsu national high school's hanazono stadium on sunday . although the school is only about five kilometers away from the stadium , 「 各種 学校 」 扱い の ため 大会に 出場できなかった が 、 今年 ようやく 花園への 夢が 実現した 。 it was unable to participate in the tournament because it was treated as a special-needs school. miscellaneous school @xmath8.neg.pst because.of ord 74 clf rugby national high.school football tournament semi @xmath9.member @xmath10.pst gold @xmath11.year finally @xmath12.pst 𝜙!-nom @xmath13.npst @xmath14.npst @xmath15.npst heart kind father be.ger also be.npst proximate school figure 4 : two examples of zar and mt. green, blue, and orange dotted lines represent the output of the baseline model, that of ours, and the gold standard, respectively. english sentences are generated by the corresponding mt (encoder-decoder) model. (a) the example in which mt apparently helped zar. the nominative zero pronoun of "あり" (is) was correctly translated as "the school". the model also succeeded in identifying its antecedent "学 校" (school). (b) the example in which mt was not helpful. the model successfully translated the nominative zero pronoun of the underlined predicate, "で" (be), as "he". it misidentified its antecedent, however. technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. tomomasa furukawa, toshiaki nakazawa, tomohide shibata, daisuke kawahara, and sadao kuroahshi. 2017. automatic construction of a pseudoannotatedzero anaphora corpus using a bilingual corpus. in proceedings of the twenty-third annual meeting of the association for natural language processing. (in japanese). junliang guo, zhirui zhang, linli xu, hao-ran wei, boxing chen, and enhong chen. 2020. incorporating bert into parallel sequence decoding with adapters. in advances in neural information processing systems, volume 33, pages 10843-10854. curran associates, inc. masatsugu hangyo, daisuke kawahara, and sadao kurohashi. 2012. building a diverse document leads corpus annotated with semantic relations. in proceedings of the 26th pacific asia conference on language, information, and computation, pages 535-544, bali, indonesia. faculty of computer science, universitas indonesia. ryu iida, mamoru komachi, naoya inoue, kentaro inui, and yuji matsumoto. 2017. naist text corpus: annotating predicate-argument and coreference relations in japanese. in nancy ide and james puste-jovsky, editors, handbook of linguistic annotation, pages 1177-1196. springer, dordrecht. ryu iida, kentaro torisawa, jong-hoon oh, canasai kruengkrai, and julien kloetzer. 2016. intrasentential subject zero anaphora resolution using multi-column convolutional neural network. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 1244-1254, austin, texas. association for computational linguistics. kenji imamura and eiichiro sumita. 2019. recycling a pre-trained bert encoder for neural machine translation. in proceedings of the 3rd workshop on neural generation and translation, pages 23-31, hong kong. association for computational linguistics. daisuke kawahara, sadao kurohashi, and kôiti hasida. 2002. construction of a japanese relevance-tagged corpus. in proceedings of the third international conference on language resources and evaluation (lrec'02), las palmas, canary islands -spain. european language resources association (elra). youngtae kim, dongyul ra, and soojong lim. 2021. zero-anaphora resolution in korean based on deep language representation model: bert. etri journal, 43(2):299-312.