E -Gen: Leveraging E -Graphs to Improve Continuous Representations of Symbolic Expressions (2025)

Abstract: Vector representations have been pivotal in advancing natural language processing (NLP), with prior research focusing on embedding techniques for mathematical expressions using mathematically equivalent formulations. While effective, these approaches are constrained by the size and diversity of training data. In this work, we address these limitations by introducing E-Gen, a novel e-graph-based dataset generation scheme that synthesizes large and diverse mathematical expression datasets, surpassing prior methods in size and operator variety. Leveraging this dataset, we train embedding models using two strategies: (1) generating mathematically equivalent expressions, and (2) contrastive learning to explicitly group equivalent expressions. We evaluate these embeddings on both in-distribution and out-of-distribution mathematical language processing tasks, comparing them against prior methods. Finally, we demonstrate that our embedding-based approach outperforms state-of-the-art large language models (LLMs) on several tasks, underscoring the necessity of optimizing embedding methods for the mathematical data modality. The source code and datasets are available at https://github.com/MLPgroup/E-Gen.

## introduction while large language models (llms) @xcite @xcite have demonstrated effectiveness in processing natural language, these methods still perform suboptimally with math-based content which plays an important role across numerous domains @xcite . for instance, even a state-of-the-art llm like gpt-4v only reaches 49% accuracy in mathvista @xcite , a comprehensive benchmark to evaluate the mathematical reasoning capabilities of llms. in another instance, as tested in @xcite , gpt-4's performance in computing integration is dominated by specialized embedding-based models @xcite . therefore, effective approaches to process semantically rich mathematical notation are necessary. one approach is to compute a semantic representation of mathematical content based on textual context and treat mathematical expressions as a sentence without learning mathematical semantics @xcite . this has been used in various mathematical language processing (mlp) tasks, such as mathematical information retrieval @xcite , identifier definition extraction @xcite @xcite , mathematical reasoning @xcite and theorem proving @xcite . but this approach has two limitations: (1) textual descriptions are lacking in some math content like textbooks or mathematical derivations in academic publications, and (2) don't really explore relations between symbolic operators from a mathematical perspective. to address these problems, recent studies have focused on equivalence and mathematical manipulation between expressions to derive the meaning of mathematical expressions independent of context. however, these approaches remain limited, primarily due to the quality of available data. @xcite introduces eqnet, a treennbased @xcite model to group equivalent expressions together, but their datasets are limited to arithmetic and boolean expressions. se-memb @xcite computes vectorbased semantic representations by learning to generate mathematically equivalent expressions but is significantly limited by the sympy-generated dataset @xcite . sympy functions are designed for simplification and directly derive the most simplified expression without intermedi-ate steps, leading to a dataset that lacks enough rewrites per expression. hence, a more efficient mathematical data generation scheme is required. in this work, we propose e-gen, a highly scalable and efficient corpus generation scheme based on e-graphs @xcite . leveraging a collection of mathematical transformations, e-gen facilitates the creation of synthetic datasets with large clusters of semantically equivalent expressions. this approach significantly improves upon prior sympy-based methods, overcoming limitations in the number of rewrites per expression while improving flexibility and scalability. in summary, we highlight the following contributions of e-gen: ## related work in mlp @xcite , semantic representation shows strong potential across various problems. a representative application is mathematical information retrieval (mir) @xcite @xcite , where expressions or keywords are ranked based on their relevance to a query. @xcite proposes symbol2vec, a mathematical symbol representation generation scheme to group latex symbols having similar contexts together. they further extend this approach to a mir scheme (formula2vec) to prove its effectiveness. @xcite converts symbolic layout trees @xcite of mathematical expressions into token sequences and generates representation based on word2vec. @xcite further improves this approach by combining the symbol layout tree and operator tree representations of an expression. @xcite proposes mathbert, a pretrained model based on mir and formula headline generation task @xcite . identifier definition extraction @xcite is another promising application, which aims to align identifiers found in scientific text with their definitions. @xcite utilizes a transformer-based method to develop an end-to-end joint math entity and relation extraction approach. @xcite provides a mathematical understanding pretrained model by fine-tuning bert on masked mathematical identifier prediction task. in the theorem proving field, @xcite generate theorem proof by retrieving relevant references to a given query theorem. however, as discussed in section 1, mlp studies rely on semantic representation but mostly focus on establishing homomorphism between context and mathematical expressions to construct semantic representation, which is limited by the incapacity to process pure math content and can not really understand mathematical transformations. even though some recent studies @xcite have focused on generating semantic representation based on intrinsic features of mathematical expressions to address this problem, their performance and scope of application are still limited by existing dataset, motivating this work's demonstration of a more effective mathematical corpus generation scheme. ## conclusion in this work, we enhance semantic representations of symbolic expressions by developing e-gen, a novel mathematical corpus generation framework based on the e-graph data structure. e-gen shows strong capability in the generation of a scalable, cluster-based corpus with high diversity in mathematical transformation. to evaluate its effectiveness, two training approaches based on seq2seq and contrastive learning respectively are implemented to capture equivalence relation between expressions. our experimental results demonstrate the efficacy of these embedding models across a variety of downstream tasks, including clustering, semantic understanding beyond syntactic similarity, mistake detection, and mathematical analogies. notably, these semantic representations outperform prior approach and gpt-4o in both quantitative and qualitative tests. this work provides an algorithmic foundation for processing symbolic mathematics, and the vector-based representations can easily integrate with vector embeddings of other data modalities.