First Steps in Building a Knowledge Base of Mathematical Results (2024)

Abstract: This paper explores the initial steps towards extracting information about theorems and proofs from scholarly documents to build a knowledge base of interlinked results. Specifically, we consider two main tasks: extracting results and their proofs from the PDFs of scientific articles and establishing which results are used in the proofs of others across the scientific literature. We discuss the problem statement, methodologies, and preliminary findings employed in both phases of our approach, highlighting the challenges faced.

## introduction the proliferation of academic publications, particularly in theoretical fields, presents a significant challenge in information retrieval and knowledge extraction. as of may 2020 (which is the cut-off date we used to build our dataset), the arxiv foot_0 preprint repository alone hosted approximately 1.7 million academic papers (see figure 1 ), a substantial portion of which include theoretical results such as theorems, lemmas, and propositions (roughly onethird based on cursory search using keyword filtering). and arxiv constitutes a fraction of the entire collection of scientific publications (e.g., the pirate sci-hub web site include 88 million academic papers, and the crossref foot_1 publication database contains metadata about 158 million articles as of may 2024). researchers typically rely on academic search engines like google scholar foot_2 or aminer foot_3 to find relevant literature. while effective for general searches, this method often falls short in mathematical fields such as mathematics or theoretical computer science, where the essence of the work is encapsulated in mathematical results @xcite ) rather than at the textual level. we discuss in this paper first steps in a project whose goal is to build a knowledge base of interlinked mathematical results, directly from the scientific literature in pdf format. the goal of such project would be to enable researchers to perform queries at the level of individual mathematical results, such as: what does the graph of results used to prove other results look like within a given research article? a knowledge base could help visualize these connections, showing how individual results contribute to broader scientific advancements. how many published results depend on a particular theorem, and what would be the impact if this theorem were proven incorrect? understanding the dependency network of a theorem could be crucial for assessing the robustness of a field's knowledge base. our methodology for constructing the knowledge graph involves a two-step process, as illustrated in figure 2: 1. result extraction: the first step focuses on identifying and extracting theoretical results from academic papers, along with their proofs. this involves sophisticated text and data mining techniques to discern and isolate relevant mathematical expressions and their accompanying discussions within a dense and complex document layout. ## related work result extraction. for the extraction task, two main related works are noteworthy. ginev et al. @xcite addressed the classification of mathematical statements as a 13-class problem (where classes are, e.g., proposition, example, remark). however, their evaluation was limited to the first logical paragraph, which is easier to classify, relied solely on the text modality without capturing sequential dependencies among paragraphs, and, crucially, required the availability of an html-ized version of the l a t e x source. in previous work @xcite we explored deep learning-based methods, including object detection and text classification, to extract proofs and theorems. we proposed several unimodal approaches across different modalities; however, there was no clear method for comparing and evaluating these modalities. for instance, our textbased approach used fine-tuned transformers on text lines extracted by the pdfalto tool, while the vision approach utilized bitmap renderings of entire pdf pages. in section 5, we include the line-based approach as a baseline. in contrast, this work proposes a modular multimodal classifier that operates at the paragraph level, incorporating text, vision, and font modalities simultaneously, while also capturing sequential information to predict paragraph labels. pdf dataset extracted results ... result graph ... result connection. a notable task is citation extraction and disambiguation. cohan et al. @xcite proposed a classifier to detect citation intent based on the sentence context. it identifies three primary intents: background, result, and method. they used a bidirectional lstm with an attention layer and three independent mlps for training. the main task was identifying citation intent, while auxiliary tasks included guessing the section title. there are other projects whose goal is to create a structured knowledge base of mathematical results, though not by exploiting the content of pdf articles, including: ## dataset construction our approach relies on training machine learning models, which requires constructing a (sizable) labeled dataset. we explain in this section which tools we use for basic parsing of pdf documents, how we built a labeled dataset using these tools and newly developed ones, and present the characteristics of our dataset. ## methodology in this section, we delve into the specific methodologies adopted for each of the three subtasks outlined in our study: the extraction task and the two connecting tasks -results to paper and results to results. ## experimental results this section presents experimental results derived from implementing our methodologies across three specific subtasks: extraction, connecting papers, and linking specific results. we stress that results from the latter two tasks are preliminary at this stage. ## conclusions this paper presents a comprehensive framework for constructing a knowledge base of mathematical results. it aims to transcend the limitations of traditional search engines through an enhanced, result-oriented navigation system within academic literature. through preliminary experiments on distinct subtasks, we have demonstrated the feasibility of our proposed solution, establishing a proof of concept for a robust and scalable knowledge management system. in data extraction, our experiments have highlighted the efficacy of a multimodal approach, lever-aging a gmu architecture followed by sequential crf modelling. this approach proved adept at capturing sequential and contextual nuances in text, offering a slight advantage over simpler, text-only methods. the ability of this architecture to integrate various data modalities into a cohesive model illustrates its potential for comprehensive information extraction from complex scientific texts, setting a high standard for subsequent refinement and application. the construction of a citation graph detailing the connections between papers through cited results has unveiled a rich tapestry of scholarly communication. notably, the longest path in this citation graph extends to 13 links, underscoring the depth and complexity of academic discourse. however, the aggregated nature of this graph means we must be cautious in interpreting these paths, as they do not guarantee a direct continuation of the topic from one paper to another. our examination of methodologies for linking specific results across papers has underscored the importance of precise terminology in the academic discourse of mathematics. the tf-idf vectorization approach outperformed the semantic capabilities of the distilbert model, suggesting that the reuse of exact terms and phrases is crucial for accurately connecting related results, though our experiments are still preliminary. moving forward, we consider several avenues for exploration and development. enhanced machine learning models, specifically tailored to the nuances of mathematical language and structured problem-solving in academia, could drastically reduce the dependency on manual labelling. ad-ditionally, expanding the knowledge base to include more diverse fields of study and integrating more advanced semantic understanding tools will enhance the breadth and depth of the knowledge graph, providing unparalleled access to interconnected scientific knowledge.