M ulti MUC : Multilingual Template Filling on MUC -4 (2024)

Abstract: We introduce MultiMUC, the first multilingual parallel corpus for template filling, comprising translations of the classic MUC-4 template filling benchmark into five languages: Arabic, Chinese, Farsi, Korean, and Russian. We obtain automatic translations from a strong multilingual machine translation system and manually project the original English annotations into each target language. For all languages, we also provide human translations for key portions of the dev and test splits. Finally, we present baselines on MultiMUC both with state-of-the-art template filling models for MUC-4 and with ChatGPT. We release MUC-4 and the supervised baselines to facilitate further work on document-level information extraction in multilingual settings.

## introduction the message understanding conferences (mucs) were a series of u.s. government-sponsored competitions that ran from the late 1980s through the late 1990s whose aim was to promote the development of systems for extracting complex relations from text, and which have been credited with inaugurating the field of information extraction (ie; @xcite . the third muc (muc-3) introduced the now classic task of template filling, in which systems must identify events, represented by predefined schemas or templates, in a document, and populate roles or slots in those templates with relevant information extracted or inferred from the text @xcite . muc-3 focused on identifying various forms of terrorism (e.g. bombings, kidnappings) in news reports from numerous countries in latin america. systems were required to extract one template per incident, containing details about the perpetrators, their victims, the weapons used, and damaged guards at the site repelled the attack, which was apparently staged by guerrillas. similarly, it was learned that a [bomb] 1,2 exploded today at the community party headquarters in the capital of caquetà, causing considerable property damage. it was immediately announced that no one had been injured or killed in the [extremist] 1 action. it was also announced that suspected subversives staged another attack…. these terrorist attacks took place 1 day after the serious attack launched at the [2nd army division headquarters] 2 in bucaramanga, which resulted in seven people injured and considerable property damage, affecting nine [homes] 2 . we release multimuc and to help further research in multilingual, document-level ie. 2 2 task and corpus task formally, the template filling task takes the following inputs: given 𝐷, systems must then determine the number of events or template instances (𝑁 𝐷 ≥ 0) attested in 𝐷 (template identification), and populate the slots in each instance based on the information contained in 𝐷 about the event it represents (slot filling). 3 3 note that 𝑁 𝐷 is not given as input and may be zero; thus, part of the task is determining the relevancy of a document given the ontology. supposing template instance 𝑖 𝑗 ∈ {𝑖 1 , . . . , 𝑖 𝑁 𝐷 } has type 𝑡 ∈ t , we can write 𝑖 𝑗 = {𝑠 (1) 𝑡 : 𝑥 (1) , . . . , 𝑠 ( 𝑁 𝑡 ) 𝑡 : 𝑥 ( 𝑁 𝑡 ) }, where 𝑥 (𝑘 ) is a (possibly null) filler of the appropriate type for slot 𝑠 (𝑘 ) 𝑡 . in general, fillers may be of any type, though for muc-4, they are constrained to two types in principle and just one in practice (see below). ## data collection the selection of languages for multimuc was inspired by the five focal languages of the iarpa better program 5 , which introduced the granular template filling task-a spiritual successor to muc-4 (see §6; @xcite . for each language, our data collection process comprised four steps: each step is detailed separately below. figure 2 illustrates steps ( 1 )-(3) for farsi. ## experiments we present three sets of experiments. all make use of the following three variations on training and dev data, designed to assess both the impact of alignment corrections and of parallel data: in all experiments, we report results on the corrected test set. ## discussion here we present some analysis of model errors ( §5.1) and also discuss observations and challenges from annotation ( §5.2).