Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving (2025)

Abstract: Retrieval-augmented generation (RAG) with foundation models has achieved strong performance across diverse tasks, but their capacity for expert-level reasoning—such as solving Olympiad-level physics problems—remains largely unexplored. Inspired by the way students prepare for competitions by reviewing past problems, we investigate the potential of RAG to enhance physics reasoning in foundation models. We introduce PhoPile , a high-quality multimodal dataset specifically designed for Olympiad-level physics, enabling systematic study of retrieval-based reasoning. PhoPile includes diagrams, graphs, and equations, capturing the inherently multimodal nature of physics problem solving. Using PhoPile, we benchmark RAG-augmented foundation models, covering both large language models (LLMs) and large multimodal models (LMMs) with multiple retrievers. Our results demonstrate that integrating retrieval with physics corpora can improve model performance, while also highlighting challenges that motivate further research in retrieval-augmented physics reasoning.

## introduction physics plays a foundational role in natural sciences and engineering, underpinning progress in fields ranging from construction and aerospace to electronics and materials science @xcite . mastering physics requires not only conceptual understanding of natural laws but also the ability to integrate them with quantitative analysis, diagrams, and symbolic reasoning. recent years have witnessed the exceptional performance of foundation models, including large language models (llms) and large multimodal models * equal contribution. retrieval-augmented generation (rag) has recently emerged as a promising approach to mitigating the limitations of llms by integrating external knowledge sources into their workflows @xcite . yet, its effectiveness for physics reasoning remains largely unexplored. in this context, rag offers a natural solution: just as a student preparing for an exam consults past competition problems to recall formulas and problem-solving strategies, an llm equipped with retrieval can access relevant laws, examples, and reasoning patterns from a curated retrieval corpus. this mechanism not only improves factual accuracy but also helps guide the selection and application of physical principles in diverse contexts. a key obstacle, however, is the absence of suitable benchmarks. whereas mathematics has benefited from a rich ecosystem of high-quality datasets and benchmarks @xcite @xcite @xcite @xcite , physics-specific evaluations are scarce. ## experiments in this section, we introduce the rag pipeline, describe the evaluation workflow, and present the experimental results of foundation models. ## conclusion in this work, we present phopile, a benchmark designed to comprehensively evaluate the ability of foundation models to perform physics reasoning with retrieval-augmented generation (rag) across both text-only and image-based questions. we benchmark a range of mainstream foundation models, including both large language models (llms) and large multimodal models (lmms), together with multiple retrievers. to ensure robust evaluation, we introduce an llm-as-judge framework capable of assessing diverse solution formats. our results demonstrate that combining physics corpora with retrieval can improve performance, while also revealing challenges that motivate further research in retrieval-augmented physics reasoning.