Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction (2022)

Abstract: We present a study on leveraging multilingual pre-trained generative language models for zero-shot cross-lingual event argument extraction (EAE). By formulating EAE as a language generation task, our method effectively encodes event structures and captures the dependencies between arguments. We design language-agnostic templates to represent the event argument structures, which are compatible with any language, hence facilitating the cross-lingual transfer. Our proposed model finetunes multilingual pre-trained generative language models to generate sentences that fill in the language-agnostic template with arguments extracted from the input passage. The model is trained on source languages and is then directly applied to target languages for event argument extraction. Experiments demonstrate that the proposed model outperforms the current state-of-the-art models on zero-shot cross-lingual EAE. Comprehensive studies and error analyses are presented to better understand the advantages and the current limitations of using generative language models for zero-shot cross-lingual transfer EAE.

## introduction event argument extraction (eae) aims to recognize the entities serving as event arguments and identify their corresponding roles. as illustrated by the english example in figure 1 , given a trigger word "destroyed" for a conflict:attack event, an event argument extractor is expected to identify "commando", "iraq", and "post" as the event arguments and predict their roles as "attacker", "place", and "target", respectively. zero-shot cross-lingual eae has attracted considerable attention since it eliminates the requirement of labeled data for constructing eae models in low-resource languages @xcite @xcite nguyen and nguyen, 2021). in this setting, the model is trained on the examples in the source languages and directly tested on the instances in the target languages. recently, generation-based models foot_0 have shown strong performances on monolingual structured prediction tasks @xcite @xcite , including eae @xcite . these works fine-tune pre-trained generative language models to generate outputs following designed templates such that the final predictions can be easily decoded from the outputs. compared to the traditional classification-based models @xcite @xcite , they better capture the structures and dependencies between entities, as the templates provide additional declarative information. despite the successes, the designs of templates in prior works are language-dependent, which makes it hard to be extended to the zero-shot cross-lingual transfer setting @xcite . naively applying such models trained on the source languages to the target languages usually generates code-switching outputs, yielding poor performance for zero-shot cross-lingual transfer, foot_2 as we will empirically show in section 5.4. how to design languageagnostic generation-based models for zero-shot cross-lingual structured prediction problems is still an open question. in this work, we present a study that leverage multilingual pre-trained generative models for zeroshot cross-lingual event argument extraction and propose x-gear (cross-lingual generative event argument extractor). given an input passage and a carefully designed prompt that contains an event trigger and the corresponding language-agnostic template, x-gear is trained to generate a sentence that fills in a language-agnostic template with arguments. x-gear inherits the strength of generation-based models that captures event structures and the dependencies between entities better than classification-based models. moreover, the pre-trained decoder inherently identifies named entities as candidates for event arguments and does not need an additional named entity recognition module. the language-agnostic templates prevents the model from overfitting to the source language's vocabulary and facilitates cross-lingual transfer. we conduct experiments on two multilingual eae datasets: ace-2005 @xcite and ere @xcite . the results demonstrate that x-gear outperforms the state-of-theart zero-shot cross-lingual eae models. we further perform ablation studies to justify our design and present comprehensive error analyses to understand the limitations of using multilingual generation-based models for zero-shot crosslingual transfer. our code is available at https: //github.com/pluslabnlp/x-gear 2 related work zero-shot cross-lingual structured prediction. zero-shot cross-lingual learning is an emerging research topic as it eliminates the requirement of labeled data for training models in low-resource languages @xcite . various structured prediction tasks have been studied, including named entity recognition @xcite @xcite , dependency parsing @xcite , relation extraction @xcite , and event argument extraction @xcite @xcite . most of them are classification-based models that build classifiers on top of a multilingual pre-trained masked language models. to further deal with the discrepancy between languages, some of them require additional information, such as bilingual dictionaries @xcite , translation pairs @xcite , and dependency parse trees @xcite @xcite . however, as pointed out by previous literature @xcite , classification-based models are less powerful to model dependencies between entities compared to generation-based models. ## zero-shot cross-lingual event argument extraction we focus on zero-shot cross-lingual eae. given an input passage and an event trigger, an eae multilingual generative model input @xmath0, including a woman, were killed monday when their houses were hit by a missile fired by the us -led coalition warplanes, witnesses said. <@xmath1, woman instrument missile place houses template for life:die event training zero-shot cross-lingual transfer given trigger generate output string decode multilingual generative model input @xmath2, 子弹, 实弹 place none template for life:die event testing given trigger generate output string decode figure 2 : the overview of x-gear. given an input passage and a carefully designed prompt containing an event trigger and a language-agnostic template, x-gear fills in the language-agnostic template with event arguments. ## proposed method: x-gear we formulate zero-shot cross-lingual eae as a language generation task and propose x-gear, a cross-lingual generative event argument ex-tractor that is illustrated in figure 2 . there are two challenges raised by this formulation: (1) the input language may vary during training and testing; (2) the generated output strings need to be easily parsed into final predictions. therefore, the output strings have to reflect the change of the input language accordingly while remaining well-structured. we address these challenges by designing language-agnostic templates. specifically, given an input passage x and a designed prompt that contains the given trigger t, its event type e, and a language-agnostic template, x-gear learns to generate an output string that fills in the languageagnostic template with information extracted from input passage. the language-agnostic template is designed in a structured way such that parsing the final argument predictions a and role predictions r from the generated output is trivial. moreover, since the template is language-agnostic, it facilitates cross-lingual transfer. x-gear fine-tunes multilingual pre-trained generative models, such as mbart-50 @xcite or mt5 @xcite , and augments them with a copy mechanism to better adapt to input language changes. we present its details as follows, including the language-agnostic templates, the target output string, the input format, and the training details. ## analysis 6.1 ablation studies copy mechanism. we first study the effect of the copy mechanism. when using mt5-large and mt-base. however, interestingly, adding a copy mechanism is not effective for mbart-50. we conjecture that this is because the pre-trained objective of mbart-50 is denoising autoencoding @xcite , and it has already learned to copy tokens from the input. therefore, adding a copy mechanism is less useful. in contrast, the pre-trained objective of mt5 is to only generate tokens been masked out, resulting in lacking the ability to copy input. thus, the copy mechanism becomes beneficial for mt5. including event type in prompts. in section 4, we mentioned that the designed prompt for x-gear consists of only the input sentence and the language-agnostic template. in this section, we discuss whether explicitly including the event type information in the prompt is helpful. we consider three ways to include the event type information: the performance. one reason is that one word in a language can be mapped to several words in another language. for example, the life event type is related to marry, divorce, born, and die four subevent types. in english, we can use just one word life to cover all four sub-event types. however, in chinese, when talking about marry and divorce, life should be translated to "生活"; when talking about born and die, life should be translated to "生命". this mismatch may cause the performance drop when considering event types in prompts. we leave how to efficiently use event type information in the cross-lingual setting as future work. influence of role order in templates. the order of roles in the designed language-agnostic templates can potentially influence performance. when designing the templates, we intentionally make the order of roles close to the order in natural sentences. 8 to study the effect of different orders, we train x-gear with templates with different random orders and report the results in using english tokens instead of special tokens for roles in templates. in section 4, we mentioned that we use language-agnostic templates model en ⇓ xx ar ⇓ xx zh ⇓ xx xx ⇓ en xx ⇓ ar xx ⇓ zh avg x-gear (mt5-base) 54.3 41.4 51.4 49.4 46.7 51.0 49.1 w/ english tokens 51.4 39.3 49.7 46.6 44.7 49.0 46.8 to facilitate the cross-lingual transfer. to further validate the effectiveness of the language-agnostic template. we conduct experiments using english tokens as the templates. specifically, we set format agent: [none] <@xmath3: [none] <@xmath4: ## conclusion we present the first generation-based models for zero-shot cross-lingual event argument extraction. to overcome the discrepancy between languages, we design language-agnostic templates and propose x-gear, which well capture output dependencies and can be used without additional named entity extraction modules. our experimental results show that x-gear outperforms the current state-of-the-art, which demonstrates the potential of using a language generation framework to solve zero-shot cross-lingual structured prediction tasks.