Formula-Text Cross-Retrieval: A Benchmarking Study of Dense Embedding Methods for Mathematical Information Retrieval (2025)

Abstract: Mathematical information retrieval requires understanding the complex relationship between natural language and formulae. This paper presents a benchmarking study on Formula-Text Cross-Retrieval, comparing a sparse baseline (BM25), off-the-shelf dense embeddings (OpenAI, BGE), and a fine-tuned dual-encoder model. Our model, trained with a contrastive objective on the ARQAR dataset, significantly outperforms all baselines, achieving state-of-the-art results. Ablation studies confirm the importance of linearization, a shared-weight architecture, and the Multiple Negatives Ranking loss. The work provides a strong foundation for mathematical NLP applications.

## introduction the articulation of mathematical concepts represents a unique and challenging domain for natural language processing (nlp), characterized by a seamless yet complex interplay between natural language (nl) and formal mathematical expressions. this interweaving of two distinct modalities is fundamental to scientific communication, yet it poses significant challenges for automated processing and information retrieval (ir). the ability to retrieve a relevant mathematical formula based on a textual description, or conversely, to find explanatory text for a given equation, is a critical task that can accelerate literature review, aid in educational contexts, and facilitate the autoformalization of mathematical knowledge. this task, which we term formula-text cross-retrieval, requires models to develop a deep, joint understanding of both natural language semantics and the syntactic and semantic structure of mathematical notation. traditional ir methods, such as lexical term matching algorithms (e.g., bm25 @xcite ), often fall short in this domain. they struggle with the inherent vocabulary mismatch problem; a user's query might describe a concept in words (e.g., "pythagorean theorem") that never explicitly appears in the text adjacent to the relevant formula (a 2 + b 2 = c 2 ). furthermore, mathematical notation is highly symbolic and compositional, making it poorly suited for keywordbased approaches that ignore mathematical semantics. the recent rise of deep learning-based dense embedding models @xcite offers a promising alternative. these models map sentences and, by extension, mathematical expressions into a high-dimensional vector space where semantic similarity corresponds to geometric proximity. this allows for efficient similarity search via nearest-neighbor algorithms, potentially capturing deep semantic relationships beyond lexical overlap (similar to @xcite ). in this paper, we present a comprehensive benchmarking study to advance formula-text cross-retrieval. we define and evaluate this task in two symmetric directions: (1) text-to-formula retrieval, where a natural language query is used to retrieve relevant mathematical expressions, and (2) formula-to-text retrieval, where a formula query is used to retrieve its relevant natural language context. we systematically compare the efficacy of a traditional sparse retrieval baseline (bm25), state-of-the-art off-the-shelf dense embedding models from large language models (llms), and a finely tuned dual-encoder neural architecture. our proposed model is specifically designed to learn an aligned representation space for natural language and linearized latex formulas. through rigorous evaluation on a publicly available benchmark, we demonstrate the superiority of tuned dense embeddings and provide a qualitative analysis of the learned representation space @xcite . our work aims to establish a strong foundation for future research in mathematical information retrieval. ## literature review our work is at the intersection of mathematical information retrieval, dense passage retrieval, and the application of large language models to scientific domains. the challenge of searching within mathematical content has a rich history, most notably explored in the ntcir conference series, which featured dedicated math ir tasks @xcite . these initiatives established standardized evaluation frameworks and highlighted the limitations of traditional symbolic and keyword-based methods, such as matching via formula patterns @xcite or leveraging inverted indices over expanded query terms @xcite . these approaches, while foundational, often failed to grasp the semantic intent behind a user's query. the field of ir was revolutionized by the adoption of neural networks and the concept of dense retrieval @xcite . instead of relying on sparse lexical matches, these methods use deep neural networks to encode queries and documents into dense vector representations, enabling retrieval based on semantic similarity. models like sentence-bert @xcite and dpr @xcite demonstrated the power of bi-encoder architectures trained with contrastive learning objectives, such as multiple negatives ranking loss @xcite , to create high-quality embedding spaces. more recent general-purpose models like bge @xcite and e5 @xcite have pushed the state-of-the-art further. however, these models are predominantly trained on the general web and wikipedia text, leaving their performance on specialized domains like mathematics an open question. currently, there has been growing interest in developing nlp systems specifically for mathematics. this includes work on mathematical word problem solving @xcite , premise selection @xcite , and the creation of large-scale data sets for mathematical reasoning @xcite . a key challenge is the representation of mathematical formulae. early approaches explored encoding formula structure using graph neural networks @xcite or generating embeddings from their latex source @xcite . the rise of large language models pre-trained on code and scientific text, such as minerva @xcite , llemma @xcite , and codex @xcite , has demonstrated remarkable mathematical reasoning capabilities, often accessed via in-context learning. furthermore, the arqar dataset @xcite provides a valuable recent resource with aligned text-formula pairs specifically designed for tasks such as crossretrieval. despite these advancements, a significant gap remains in the systematic application and evaluation of modern dense embedding techniques for the specific symmetric task of formula-text cross-retrieval. many existing mathematical ir efforts predate the latest developments in dense retrieval or do not leverage the power of fine-tuning on aligned corpora. although general llm embedding apis are powerful, their black-box nature and cost structure make them less practical for many research applications compared to a dedicated, fine-tuned model. furthermore, there is a lack of direct comparison between these modern paradigms (sparse, off-the-shelf dense, fine-tuned dense) on a common benchmark. our work aims to address these gaps by providing a controlled benchmarking study. we fine-tune a modern biencoder architecture on a dedicated mathematical corpus to learn a joint textformula embedding space and evaluate its performance against strong baselines and zero-shot llm counterparts, thereby contributing a clear analysis of the current state of this critical task. ## conclusion this paper established a comprehensive benchmark for formula-text cross-retrieval. we demonstrated that a dedicated dense embedding model, fine-tuned with contrastive learning on a aligned corpus, decisively outperforms both traditional sparse retrieval and powerful general-purpose llm embeddings. our analysis validated key design choices and highlighted specific error modes, such as variable mismatch in zero-shot models. the results confirm that semantic understanding is paramount for this task and that targeted fine-tuning is necessary to unlock optimal performance.