Time Travel: A Comprehensive Benchmark to Evaluate LMM s on Historical and Cultural Artifacts (2025)

Abstract: Understanding historical and cultural artifacts demands human expertise and advanced computational techniques, yet the process remains complex and time-intensive. While large multimodal models offer promising support, their evaluation and improvement require a standardized benchmark. To address this, we introduce TimeTravel, a benchmark of 10,250 expert-verified samples spanning 266 distinct cultures across 10 major historical regions. Designed for AI-driven analysis of manuscripts, artworks, inscriptions, and archaeological discoveries, TimeTravel provides a structured dataset and robust evaluation framework to assess AI models’ capabilities in classification, interpretation, and historical comprehension. By integrating AI with historical research, TimeTravel fosters AI-powered tools for historians, archaeologists, researchers, and cultural tourists to extract valuable insights while ensuring technology contributes meaningfully to historical discovery and cultural heritage preservation. We evaluate contemporary AI models on TimeTravel, highlighting their strengths and identifying areas for improvement. Our goal is to establish AI as a reliable partner in preserving cultural heritage, ensuring that technological advancements contribute meaningfully to historical discovery. We release the TimeTravel dataset and evaluation suite as open-source resources for culturally and historically informed research.

## introduction in recent years, large multimodal models (lmms) have made significant strides in visual reasoning, 1https ://huggingface.co/datasets/mbzuai/timetravel 2https ://github.com/mbzuai-oryx/timetravel † equal contribution. figure 1 : timetravel taxonomy categorizes artifacts from 10 major civilizations, representing diverse historical and prehistoric periods. it encompasses 266 distinct cultures and over 10k manually verified historical artifact samples, providing a structured framework for comprehensive ai-driven analysis. perception, and multimodal understanding. models such as gpt-4v (openai, 2024) and llava @xcite have excelled in image captioning, visual question answering (vqa), and complex visual reasoning, driving the development of benchmarks @xcite @xcite to assess their capabilities. these benchmarks predominantly focus on modern objects, cultural landmarks, and textual sources, extending multimodal ai applications to domains such as medical imaging, remote sensing, and real-world scene understanding @xcite . however, a critical gap remains: lmms fail to address the historical dimension of visual data, particularly artifacts that have shaped human civilization. examples span multiple artifact categories, including coins, accessories, tools, and statues from ancient civilizations. each artifact is accompanied by a detailed description, providing valuable contextual and historical insights. additional timetravel examples can be found in the appendix (figure 7 and figure 8 ). historical artifacts, from ancient manuscripts and inscriptions to architectural ruins and cultural symbols, offer invaluable insights into the evolution of societies, artistic expression, and technological advancements. these artifacts preserve cultural heritage and serve as primary sources for understanding belief systems, trade networks, and sociopolitical structures of past civilizations. however, interpreting them requires deep contextual knowledge, which current lmms struggle to achieve, particularly in non-english and non-western historical contexts. while some models have been extended to low-resource languages to bridge cultural gaps @xcite , they lack systematic capabilities to analyze artifacts from diverse civilizations. this limitation highlights the urgent need for a specialized benchmark that evaluates the ability of ai to process and understand historical artifacts with cultural and temporal awareness. to address this challenge, we introduce timetravel, an open-source comprehensive benchmark (see ## timetravel benchmark evaluation evaluation metric: to assess the quality, accuracy and relevance of our generated textual descriptions, we employed a combination of traditional and advanced metrics. bleu @xcite and rouge-l @xcite evaluate linguistic fluency and structural similarity, ensuring syntactic alignment with reference texts. meteor @xcite ## conclusion we present the timetravel dataset, a curated collection of historical artifacts from 10 cultural regions, extensively curated by domain experts. we developed a rigorous data collection, filtering, and verification process to ensure accuracy and completeness. using gpt-4o, we generated detailed textual descriptions, making the dataset more accessible and valuable for computational historical research. our evaluation, using bleu, me-teor, rouge-l, spice, bertscore, and llmas-judge, showed that closed-source models outperformed open-source alternatives, although open models are rapidly improving. our analysis highlights the potential of lmms in bridging historical records gaps while maintaining academic integrity. using ai-driven methodologies, this work lays the foundation for advancing cultural heritage preservation and enhancing digital humanities research, thereby ensuring greater accessibility and precision in historical documentation.