KE - MHISTO : Towards a Multilingual Historical Knowledge Extraction Benchmark for Addressing the Long-Tail Problem (2025)

Abstract: Large Language Models (LLMs) face significant challenges when queried about long-tail knowledge, i.e., information that is rarely encountered during their training process. These difficulties arise due to the inherent sparsity of such data. Furthermore, LLMs often lack the ability to verify or ground their responses in authoritative sources, which can lead to plausible yet inaccurate outputs when addressing infrequent subject matter. Our work aims to investigate these phenomena by introducing KE-MHISTO, a multilingual benchmark for Entity Linking and Question Answering in the domain of historical music knowledge, available in both Italian and English. We demonstrate that KE-MHISTO provides significantly broader coverage of long-tail knowledge compared to existing alternatives. Moreover, it poses substantial challenges for state-of-the-art models. Our experiments reveal that smaller, multilingual models can achieve performance comparable to significantly larger counterparts, highlighting the potential of efficient, language-aware approaches for long-tail knowledge extraction. KE-MHISTO is available at: https://github.com/polifonia-project/KE-MHISTO.

## introduction modern knowledge extraction (ke) methods rely on the pre-train-then-finetune paradigm, leveraging large language models (llms) as foundational components. these can be viewed as soft knowledge bases @xcite , making them a cornerstone for ke tasks, including named entity recognition (ner), entity linking (el), relation extraction (re), and question answering (qa) @xcite @xcite @xcite . however, llms' knowledge retention is closely linked to the frequency of information in the pretraining data. llms struggle when probed for so-called long-tail knowledge @xcite @xcite . the long-tail problem involves difficulties in handling queries about lesser-known topics due to limited supporting evidence in pre-training data. according to @xcite , these long-tail queries seek domain-specific factual information, making them particularly challenging for llms. while larger models improve retention and accuracy in performing the task (e.g. in questionanswering), they provide only modest benefits for infrequent knowledge @xcite . benchmarks focused on historical documents provide an opportunity for evaluating sota ke methods on real-world data inherently characterised by long-tail knowledge @xcite . historical documents exemplify the challenges of long-tail ke, as information about historical entities and events is sparse in the pre-training datasets used to develop llms, which are typically based on large-scale general-purpose knowl-edge bases (kb), such as wikipedia. consequently, models relying on llms often fail to perform adequately when performing tasks like ner or el @xcite on such input data, or answering questions about historical entities @xcite . these challenges are even more pronounced in non-english languages @xcite , which are typically underrepresented in both pre-training corpora and downstream nlp resources. however, the insufficient availability of multilingual benchmarks specifically tailored to the long-tail problem hinders progress. in this paper, we present ke-mhisto: a multilingual historical ke benchmark in english and italian comprising high-quality datasets for evaluating el and qa in the long tail. ke-mhisto includes: 1. mmhercl, the multilingual musical heritage named entities recognition, classification and linking benchmark, which extends the existing english-only mhercl-en dataset @xcite by introducing mhercl-ita, a novel italian-language expansion. to the best of our knowledge, mhercl-ita is the first gold standard for historical ner and el in italian. 2. mdy-naknowledge, which expands the free-form historical qa benchmark dynaknowledge @xcite by releasing dynaknowledge-en (dk-en), which increases dynaknowledge's number of english samples by an order of magnitude, and dynaknowledge-ita (dk-ita), a novel italian-language expansion. ke-mhisto shares neither time period nor topical overlap with mainstream benchmarks. we provide an extensive evaluation of ke-mhisto, testing both taskspecialised sota models and llms across all tasks. we find that ke-mhisto presents significant challenges for all tested models, underscoring its effectiveness in evaluating llm limitations on longtail knowledge. as a manually annotated benchmark derived from non-web historical sources, it also exposes biases in widely used knowledge bases (kbs), such as wikidata, offering an opportunity to identify and mitigate them. our experiments further reveal that smaller models trained in a multilingual setting achieve performance comparable to that of significantly larger models, highlighting the potential of efficient, language-aware approaches for long-tail ke, in line with the findings of @xcite . ## methodology the long tail problem presents issues that deserve a dedicated methodological effort towards a highquality, fully supervised benchmark. therefore, we identified four steps: 1. corpus identification, 2. annotators' selection and training, 3. benchmark construction, and 4. evaluation, that we illustrate. corpus identification to maximise long-tail knowledge, we look into resources that include historical entities and facts. however, to accurately measure popularity, the corpus must cover entities that exist on wikipedia and wikidata. furthermore, the corpus should also include resources in languages other than english. we select the polifonia corpus (polifonia corpus), a diachronic, multilingual, and modular resource specialised in the musical heritage (mh) domain. the corpus spans six languages: english, dutch, german, spanish, italian, and french. while resources differ across languages (authoritative 19c music magazines in spanish are different from the same editorial endeavours in german), each module maintains equivalence. we select the periodicals module, consisting of digitised articles from 19th-century musical journals, reflecting contemporary discussions on musical practice and criticism. due to its nature and ocr-derived text, this part of the corpus presents additional challenges related to transcription errors, not contemporary lexicon, and nonstandard syntactic structures, making it an ideal benchmark for evaluating ke methods in long-tail, multilingual settings. annotators should be fluent in the languages and the domain to ensure they can select meaningful samples and questions. in our work, annotations are performed by eight undergraduate students of the foreign languages and literature program. ## the ke-mhisto benchmark ke-mhisto is divided into two parts: one for the el and the other for the qa task. furthermore, each part has a separate module for each language. figure 1 provides an overview of the resource. in what follows, we illustrate each component in detail. mmhercl is an el benchmark comprising sentences from historical periodicals in the music domain. the resource was preliminarily introduced in @xcite based on british periodicals dated from 1823 to 1900. in this work, we add mhercl-ita, a resource constructed from equivalent periodicals in italian: l'arpa: giornale letterario, artistico, teatrale (periodico 8670) and rivista nazionale di musica (periodico 8654). mhercl-ita includes 533 sentences extrapolated from 20 periodicals issues, with a total of 2,431 manually annotated entities. we stress that a significant proportion of entities in both datasets (30% in mhercl-en and 28% in mhercl-ita) lack a corresponding wikidata entry (nil entities), highlighting gaps in existing kbs and demonstrating the nature of mmhercl in covering long-tail knowledge. we report full statistics comparing mhercl-en and mhercl-ita in the iaa for mhercl-en is reported in graciotti et al. @xcite , with a score of 0.82 krippendorff's alpha for nominal-scale data @xcite . to assess annotation quality for mhercl-ita, we sample 200 sentences (12, 775 tokens, 1, 176 ne mentions) and conduct an annotation by two independent annotators of ner and el information. the computed iaa, using the same score, is 0.82, consistent with mhercl-en, indicating high annotation reliability. mdynaknowledge (qa) mdynaknowl-edge is an open-form qa benchmark derived from mmhercl. each question can be answered by taking one or more sentences from the entity's wikipedia page. the first steps towards mdyna-knowledge were introduced by @xcite , where an initial concept of dynaknowledge with 82 samples was presented. dk-en significantly expands upon this by scaling the english dataset to 567 qa pairs and by introducing the novel italian dataset dk-ita. notably, the average provenance length, representing the supporting passage length in wikipedia, is longer in dk-ita, indicating structural differences between italian and english. additionally, the average answer length is significantly higher in dk-ita, which may be attributed to linguistic differences in expressing factual information across languages. ## evaluation in this section, we evaluate ke-mhisto in two ways. first, we perform a long-tail analysis focusing on the popularity distribution of entities, comparing with alternative solutions (section 4.1). next, we report on extensive experiments in zeroshot and retrieval augmented settings, with stateof-the-art (sota) pre-trained llms (section 4.2). we discuss our findings in section 4.3. ## conclusion in this paper we presented a novel multilingual benchmark tailored to the long-tail problem. going beyond existing alternative benchmarks, ke-mhisto consists mostly of entities with low popularity, and it is fully linked to wikipedia and wikidata. our findings indicate that sota models struggle on ke-mhisto, performing significantly worse than on alternative benchmarks. furthermore, models exhibit systematic failures on specific segments of the dataset, revealing biased performance. in addition, smaller models trained in a multilingual setting achieve performance comparable to significantly larger models. this aspect highlights the potential of efficient, language-aware approaches for long-tail ke. in the future, we plan to keep nurturing this resource, adding equivalent datasets for the other languages covered by the polifonia corpus, such as spanish, french, german, and dutch. our contribution is an added milestone towards supporting the systematic evaluation of el and qa methods in a multilingual, long-tail setting.