Massively Multilingual Instruction-Following Information Extraction (2025)

Abstract: The literature on information extraction (IE) has mostly centered around a selected few languages, hindering their applications on multilingual corpora. In this work, we introduce MASSIE - a comprehensive collection for instruction-following multilingual IE that standardizes and unifies 215 manually annotated datasets, covering 96 typologically diverse languages from 18 language families. Based on MASSIE, we conduct empirical studies on few-shot in-context learning and report important factors that either positively or negatively affect LLMs’ performance in multilingual IE, covering 21 LLMs sizing from 0.5B to 72B. Additionally, we introduce LF1 - a structure-aware metric that captures partially matched spans, resolving the conservativeness of standard exact matching scheme which overpenalizes LLMs’ predictions. Overall, our results signify that multilingual IE remains very challenging for existing LLMs, especially on complex tasks involving relations and events. In addition, performance gap is extremely large among high- and low-performing languages, but the group of similar-performing languages largely overlap between different LLMs, suggesting a shared performance bias in current LLMs.

## introduction multilingual information extraction (ie) aims to extract structured insights from unstructured data originating from multiple languages. despite the existence of approximately 7000 languages worldwide @xcite , past ie research has predominantly focused on a selected few highresource languages, such as english or chinese @xcite @xcite , significantly hindering ie applications on multilingual corpora. this tendency persisted in the era of llms, where state-of-the-arts methodologies and evaluations remain centered around these high-resource languages @xcite , further catalyzing unbalance in technology adoptions. concretely, we identify 6 challenges of existing research in multilingual ie: (i) lack of a standardized benchmark: unlike language understanding or text embedding where widely recognized benchmarks such as mmmlu foot_0 @xcite and @xcite exist, there is no standardized benchmark for multilingual ie. as a result, past ie works evaluated on separate sets of datasets and languages with different splitting and demonstrations @xcite @xcite , making results incomparable; (ii) lack of diverse evaluation the set of languages involved in previous ie works, especially llm-related ones @xcite , is often small both in number and language families; (iii) lack of reliable evaluation several ie works report their results on silver-standard datasets @xcite @xcite , which have often been criticized for low-quality ground truths @xcite ; (iv) lack of a suitable metric past ie works rely on exact matching for evaluation @xcite @xcite which overpenalizes llms' predictions, especially in few-shot icl foot_1 or zero-shot sft foot_2 settings where models lack in direct supervisions from the testing domain. in addition, discontinuous metrics such as exact matching have been criticized for obfuscating the measurement of emergent abilities in llms @xcite ; (v) lack of study on prompting techniques past ie works rely on manually crafted prompts @xcite , whereas the influence of latest prompting techniques such as chain-of-thought @xcite or self-improving program @xcite remains un-derexplored; (vi) lack of analysis in large-scale evaluation past ie works have limited evaluation scopes (e.g. datasets, languages), raising questions of whether common characteristics of llms regarding parameter scaling @xcite or demonstrations @xcite would still hold for multilingual ie at scales. furthermore, it remains unclear to which extent have modern llms made progress in conducting multilingual ie. in this paper, we seek to improve the status quo regarding these 6 challenges. particularly for (i, ii, iii), we introduce the massie collection (sec. 2) formed through aggregating and standardizing 215 human-annotated ie datasets featuring 96 languages from 18 language families. based on massie, we construct two benchmarks (sec. 2.3): m-heavy and m-light, with the prior designed for intensive and sparse evaluation while the latter is reserved for iterative development and more balanced evaluation. for (iv), we implement lf1 (sec. 2.4) -a structure-aware soft metric that distinguishes between partially correct prediction spans while taking into account task structures. for (vi), we benchmark 21 llms sizing from 0.5b to 72b and provide high-level analyses in the fewshot icl settings (sec. 3). for (v), we experiment with intermediate reasoning (sec. 4) and prompt searching (sec. 5), and delineate important factors that contribute to llms' prompting performance for multilingual ie. ## conclusion in this paper, we identify and work towards 6 challenges of current research for multilingual ie. in particular, we construct the massie collection and the two associated benchmarks: m-heavy and m-light, both featuring samples from 215 human annotated ie datasets in 96 languages. besides, we introduce lf1 -a structure-aware soft metric for continuous evaluation of multilingual ie. additionally, we conduct an empirical study involving 21 llms on few-shot icl and re-examine important factors influencing models' performance. beyond manual default prompts, we also experiment with integrating intermediate reasoning and automatically tuning prompt components, providing high-level analyses in the process. although our results indicate that existing llms still struggle with multilingual ie, we believe the contained resources and findings would help pave the wave for new frontiers in this area.