STEM - POM : Evaluating Language Models Math-Symbol Reasoning in Document Parsing (2025)

Abstract: Advances in large language models (LLMs) have spurred research into enhancing their reasoning capabilities, particularly in math-rich STEM (Science, Technology, Engineering, and Mathematics) documents.While LLMs can generate equations or solve math-related queries, their ability to fully understand and interpret abstract mathematical symbols in long, math-rich documents remains limited. In this paper, we introduce STEM-PoM, a comprehensive benchmark dataset designed to evaluate LLMs’ reasoning abilities on math symbols within contextual scientific text. The dataset, sourced from real-world ArXiv documents, contains over 2K math symbols classified as main attributes of variables, constants, operators, and unit descriptors, with additional sub-attributes including scalar/vector/matrix for variables and local/global/discipline-specific labels for both constants and operators. Our extensive experiments demonstrate that state-of-the-art LLMs achieve an average accuracy of 20-60% under in-context learning and 50-60% with fine-tuning, highlighting a substantial gap in their ability to classify mathematical symbols. By improving LLMs’ mathematical symbol classification, STEM-PoM further enhances models’ downstream mathematical reasoning capabilities. The code and data are available at https://github.com/jiaruzouu/STEM-PoM.

## introduction large language models (llms) have demonstrated exceptional reasoning abilities across numerous fields @xcite @xcite @xcite . with the increasing shift towards applying llms to complex tasks @xcite @xcite @xcite , the need for supplementary data beyond the general pre-trained datasets has become † corresponding authors 𝜎 𝜋 𝑑 δ 𝑟 𝑇 𝛼 𝛿 𝑁 𝜙 𝛾 𝑓 𝑛 θ ො 𝑦 figure 1 : total frequency (appearances) of the top-15 mathematical symbols in 1,000 randomly sampled arxiv math-rich documents. each math symbol contemporary belongs to multiple main attribute categories depending on its related context and mathematical expression. this illustrates the contextual polymorphism of a single math symbol. similarly, recent advanced llms, such as chat-gpt @xcite , also face difficulties in understanding and reasoning with abstract mathematical symbols due to their contextual polymorphism @xcite @xcite , as demonstrated in figure 1 . as an intuitive example, in the linear equation: @xmath0, y is categorized as a variable. whereas in the cross-entropy loss function: l(x, y) = -n @xmath1, the symbol y represents the fixed target labels, which is considered a constant for a given dataset. without the corresponding contextual information of a mathematical symbol, llms are unable to distinguish between different attributes of the symbol and cannot effectively process related mathematical reasoning tasks. thus, tagging math symbols within domainspecific contexts is essential for language models. in this paper, we introduce a novel benchmark dataset, stem-pom, designed to evaluate the reasoning capabilities of language models on mathematical symbols across different domains. the stem-pom dataset consists of 2,109 mathematical token instances extracted from a random sampling of 10,000 arxiv manuscripts, which are mathrich documents spanning domains such as mathematics, physics, chemistry, and more. for each dataset instance, we provide a specific mathematical symbol with the corresponding symbol order in the document, main and sub-level attributes, and the related text information from the original arxiv paper. each mathematical symbol in the dataset is classified according to two levels of attributes @xcite . the first-level attribute categorizes the symbol as variable, constant, operator, or unit descriptor. the second-level attribute further classifies the symbol into one of six types based on its first-level category: scalar, vector, matrix, local, global, or discipline-specific. figure 2 illustrates the category distribution of the dataset. to further enrich the stem-pom dataset with additional arxiv manuscripts and other math-rich document resources, we also design the stem-pom labeler, a feasible toolkit to assist dataset generation by automatically searching, extracting, and recording hand-labeled mathematical symbols with their corresponding context from stem articles. we conduct extensive experiments on the stem-pom dataset to assess the mathematical reasoning abilities of a large amount of open-and closed-source vanilla language models. our experimental results indicate that existing language models generally struggle with understanding mathematical symbols, even when provided with relevant context. extending the context length to a certain degree and fine-tuning on symbol-related material can enhance their ability to classify mathematical symbols. we further examine the relationship between the model's mathematical symbol classification and mathematical reasoning capabilities by evaluating its performance on challenging downstream tasks, such as olympiadbench @xcite , after enhancing its classification accuracy through fine-tuning on stem-pom. last, we provide detailed dataset analysis and case studies to investigate the influence factors such as context length on the ability of language models to understand mathematical symbols. in summary, our main contributions are: ## backgrounds part-of-math (pom) tagging. the part-of-math tagging task draws inspiration from similar tagging tasks such as part-of-speech tagging @xcite . in the pom context, the goal is to label individual mathematical tokens or expressions in math formulas with their corresponding mathematical roles. @xcite collected mathematical content, such as formula representation and tagging for specific mathematical formula translations and verifications, including converting formulae into semantic latex or testing with tools like cas (computer algebra systems). however, this focus on structured and narrow formula translations does not align with the broader, more diverse text-based tasks required to assess nlp models, due to the lack of scalability features in the collected math symbols. @xcite recently evaluated the potential of leveraging llms for automated annotation and part-of-math tagging of math symbols conducted on the digital library of mathematical functions (dlmf) @xcite . since the source of math symbols is only one manuscript, the mathematical tokens collected only have a single classification type and are self-consistent. in contrast, our dataset incorporates the inherent messiness of published literature across several stem subjects, where these domainspecific math symbols can have multiple classifications or meanings depending on the discipline and related contextual information. ## stem-pom dataset in this section, we introduce our constructed benchmark dataset, stem-pom. first, we describe the source data used for extracting mathematical symbols and text information in section 3.1. next, we outline the data annotation process in section 3.2 and present the dataset statistics in section 3.2. finally, we provide details on our dataset labeling tool stem-pom labeler in section 3.4. ## conclusion in this paper, we introduce stem-pom, a comprehensive benchmark for evaluating language models' mathematical reasoning abilities to classify math symbols from scientific texts. the dataset includes over 2,000 math instances sourced from arxiv papers. extensive experiments show that the best-performing model, achieves only 73.8% and 60.5% for first and second-level classification accuracy, highlighting the challenge of extracting and categorizing math symbols from large text corpora.