Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval (2022)

Abstract: With the recent success of dense retrieval methods based on bi-encoders, studies have applied this approach to various interesting downstream retrieval tasks with good efficiency and in-domain effectiveness.Recently, we have also seen the presence of dense retrieval models in Math Information Retrieval (MIR) tasks,but the most effective systems remain classic retrieval methods that consider hand-crafted structure features.In this work, we try to combine the best of both worlds: a well-defined structure search method for effective formula search and efficient bi-encoder dense retrieval models to capture contextual similarities.Specifically, we have evaluated two representative bi-encoder models for token-level and passage-level dense retrieval on recent MIR tasks.Our results show that bi-encoder models are highly complementary to existing structure search methods, and we are able to advance the state-of-the-art on MIR datasets.

## introduction math information retrieval (mir) is a special information retrieval domain that deals with heterogeneous data. the core task in this field is to retrieve relevant information from documents that contain math formulas. as digitized math content (mostly in l a t e x markup or mathml format) becomes readily available nowadays, being able to index and retrieve formulas (or equations) in those documents effectively is possibly one of the hard nuts left to be cracked before we can search freely for scientific documents, educational materials, and other math content. the need to measure similarities for highly structured formulas with special semantic properties and to model their connections to the surrounding text have a few interesting consequences: (1) heuristic scores like the term-frequency factor in tf-idf scoring variants become less relevant in formula similarity assessment because symbols in a math formula can be interchangeable and similarity may depend on expression structure rather than frequency of co-occurrence. (2) at the same time, the same math content can be expressed differently, e.g., {1, 2, ...} and n + represent the same concept but they are made of totally different tokens. we also need to capture similar math expressions with different structure due to math transformations, e.g., 1 + 1 on the other hand, recent bi-encoder dense retrieval models @xcite @xcite @xcite have been shown to be highly effective for in-domain retrieval while remaining efficient for large corpora in practice. compared to traditional retrieval methods, these models use dual deep encoders, usually built on top of a transformer encoder architecture @xcite , to encode query and document passages separately and eventually output contextual embeddings. similarity scores can be efficiently computed given these embeddings, which limits costly neural inference to indexing time. the effectiveness of these models can be attributed to the encoder's ability to capture contextual connections or even high-level semantics without the necessity for exact lexical matching. this very complementary benefit compared to more rigorous structure search methods motivates us to investigate whether dense retrieval models can improve mir results when combined with existing structure search methods. we summarize the contributions of this work as follows: ## discussion given that linear fusion is able to produce such good results, a natural question to ask is whether other fusion methods can lead to even better results. therefore, we compare popular fusion methods foot_3 on the arqmath-2 datasets and our results are summarized in on the other hand, structure search is helpful to dense retrieval in cases where an obviously relevant document is found by matching a candidate formula perfectly. we illustrate the above statement using the topic where the precision metric has the largest increase after fusion. specifically, topic a.287, which gains the most when the fusion run is compared against the approach0 baseline (p@10 changes from 0 to 0.8), fails for approach0 because structure match does not occur in relevant documents and the query is formula-centered. on the other hand, when compared to the colbert run, the fusion run in topic a.219 shows the most gain in precision (p@10 increases from 0.1 to 0.5). figure 3 shows the ranks of top-10 fusion results and their original positions in topic a.219. by further inspecting in detail, we discover that structure search prevents false positives in dense retrieval. specifically, the top-3 dense retrieval hits in topic a.219 contain binomial coefficient notation, e.g., but is not equivalent mathematically. they are ruled out or get lowered in rank in the top-10 final results because their counterparts in structure search are missing (e.g., rank 1st and 2nd results from colbert run) or out of sight (e.g., rank 3rd result from colbert run), and those colbert hits paired with a structure hit in the approach0 pass stand out. ## conclusions rapid progress in dense retrieval models using deep neural networks has greatly influenced many ir tasks. in this paper, we provide a thorough evaluation of both token-level and passage-level biencoder models in the math information retrieval domain. our dpr and colbert models adapted to this domain in both pretraining and fine-tuning are made publicly accessible to provide stepping stones for future research. our study also highlights the importance of combining structure search with dense retrieval models for better math-aware search. we show that bi-encoder dense retrieval models alone can be less effective than cross encoders, but when combined with strong structure search methods, they can further improve stateof-the-art effectiveness. with the huge modeling capacity of dense retrieval models, we believe it is worth exploring other directions for improvements so that we can unleash the potential of deep models in this domain, for example, to better identify similarities in mathematically transformed expressions with different structures.