Exploring Graph Representations of Logical Forms for Language Modeling (2025)

Abstract: We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the \underline{G}\textit{raph-based }\underline{Fo}\textit{rmal-}\underline{L}\textit{ogical }\underline{D}\textit{istributional }\underline{S}\textit{emantics} (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs (BERT) pretrained on the same data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications.

## introduction although recent advances in llms have led to remarkable performance on a wide variety of benchmarks, the consistent improvements exhibited by sota llms are largely due to corresponding increases in model size @xcite . given the chinchilla scaling laws @xcite and the rate at which sota llms are expanding, @xcite estimate that high-quality english training data will be exhausted at some point between 2026 and 2032: language model expansion is outpacing available natural language production. this suggests that-without models that use significantly less data than current approaches-llms' performance increases will begin to decelerate substantially in the near future. however, there is a considerable amount of evidence in the literature (e.g. @xcite figure 1 : dmrs representation of the sentence "every bored person saw her yesterday." @xcite @xcite @xcite indicating that linguistically-informed lms-models whose inputs and/or architectures are augmented by linguistic knowledge-can improve performance without consuming more text data. in this paper, we argue for the use of lms over logical forms (lflms): lms that take as input semantic representations, rather than text. in particular, we posit the following hypothesis: the linguistic-knowledge catalysis hypothesis (lkch): the (aspects of) linguistic knowledge incorporated into lflms greatly accelerates their learning of elementary linguistic phenomena, in turn accelerating the learning of more complex patterns. one relevant corollary of the lkch is that lflms can learn with less data: the linguistic knowledge built into lflms facilitates more rapid learning of advanced phenomena. we argue that the primary advantage of logical forms (as opposed to other types of linguistic knowledge) with respect to language modeling is the de-noising effect conferred by the functionargument structure inherent to such representations. specifically, the translation of surface text to logical form has an equivalence-classing effect, so that all syntactic paraphrases of the same proposition-for example, an active sentence and its passive counterpart-are mapped to the same representation. the benefit to an lm of the de-noising effect resulting from this equivalence-classing is clear: the model does not need to learn to equate periphrastic structures, and so can immediately begin learning co-occurrence relations between predicates. to make the case empirically for lflms, we introduce the graph-based formal-logical distributional semantics (gfolds; section 3) model as a proof-of-concept of such lms. gfolds is a pretrained, encoder graph transformer @xcite over structures derived from dependency minimal recursion semantics (dmrs; @xcite representations (see figure 1 ): directed, labeled acyclic graphs in which nodes correspond to predicates and edge labels denote relations (i.e. semantic roles) between them, in a similar manner to abstract meaning representation (amr; @xcite structures. unlike amr, which abstracts away from morphosyntactic features such as tense and number, dmrs includes these features (see figure 7 in the appendix). beyond yielding a more faithful representation of linguistic meaning, this additionally has the effect of further de-noising the model's input by offloading the morphological realization of these features to explicitly annotated labels: gfolds does not need to learn the surface patterns corresponding to inflection, as this information is instead explicitly provided through the dmrs representation. for example, the model does not need to understand that the suffix -s has the same effect on meaning as other, irregular realizations of pluralization (e.g. goose ⇒ geese), because plural nouns are directly labeled as such. the contributions of this work are threefold: firstly, we provide experimental support towards the validity of the lkch, demonstrating that-from the start of pretraining-gfolds achieves near-peak performance on tasks designed to evaluate its elementary linguistic knowledge, and that this translates to more rapid learning of complex phenomena (section 4). secondly, we demonstrate the viability of pretrained lflms in section 5, by comparing the performance of gfolds to that of bert (trained on ∼6.5 times more data than gfolds; @xcite on a range of downstream tasks. although the actual bert models outperform gfolds, our model outperforms-by a wide margin-bert models pretrained on the same data as gfolds on all benchmarks, indicating that lflms can learn useful representations with much less data than their textual counterparts. thirdly, we establish that lflms have the potential to compete with textual llms at scale: in section 6, we present evidence indicating that gfolds is likely to scale with respect to parameter count and pretraining dataset size. we make all code for the gfolds model and the experiments conducted in this paper available on github foot_0 . ## related work as discussed in section 1, there exists a body of research indicating that injecting graph representations of linguistic structures into textual lms can improve downstream performance. @xcite fuse dependency parse graphs into pretrainined transformer encoders (e.g. bert and roberta; @xcite , and surpass the then-sota results on relation classification, entity typing, and question answering tasks, demonstrating the general utility of linguistically-informed lms. similarly, @xcite inject syntactic dependency parse graphs into a pretrained bert model, yielding then-sota results on semantic role labeling and relation extraction tasks. these authors additionally compare the respective impacts of syntactic and semantic representations on performance, and find that semantic representations are more beneficial with respect to downstream performance than syntactic structures. @xcite show that linguistic structures can be incorporated into the input of gpt-2 @xcite to improve next word prediction accuracy and entropy. of particular relevance is their finding that elementary dependency structures (eds; @xcite )-a semantic framework that is related to dmrs-yield greater performance improvements than syntactic (or other semantic) representations. all of the models described thus far in this section are hybrid architectures that merge textual and graph representations. furthermore, in these approaches, the textual component of the model is initialized from a pretrained lm such as bert, roberta, or gpt-2. this contrasts with the gfolds model, which takes only graph representations as input and is pretrained from scratch. to the best of our knowledge, functional distributional semantics at scale (fdsas; lo et al., 2023) represents the only foot_1 extant lflm aside from gfolds. fdsas is a variational autoencoder over dmrs graphs that learns probabilistic truthconditional functions for each predicate (node label). fdsas is not, however, a transformer and so does not benefit from the flexibility afforded by such architectures: it is unclear how to scale the model to yield a deeper architecture, and it does not generate usable hidden states that can be passed to (for example) a classification head for downstream tasks. additionally, its rigorous formal-semantic foundations give rise to a degree of inflexibility in this model: for example, lo et al. (2023) discarded prepositions, quantifiers, and modal verbs from fdsas' training dataset out of necessity. ## gfolds in this section, we describe the gfolds model architecture (section 3.1), preprocessing steps that we employed on its dmrs graph inputs (section 3.2), and the model's pretraining procedure (section 3.3). ## evaluating the lkch this section is dedicated to an investigation of the validity of the lkch. the hypothesis can be bro-ken down into two distinct claims: (i) that the (aspects of) linguistic knowledge incorporated into lflms greatly accelerates their learning of elementary linguistic phenomena; and (ii) that this in turn accelerates the learning of more complex patterns. we therefore divided this experiment into two parts, which respectively probe the model's knowledge of elementary and complex linguistic phenomena. due to the claim made in the lkch that linguistically-informed lms' learning of complex patterns is accelerated, we evaluate the model at regular intervals throughout pretraining, in order to measure the rate at which it is learning. ## scalability while the results of section 5 show that gfolds outperforms textual models trained on similar amounts of data, this model is still outperformed by the original bert models. it is therefore crucial to establish the scalability of gfolds: the degree to which we would expect its downstream perfor-mance to scale if it were larger and/or pretrained on more data. to that end, we applied the techniques of @xcite to gfolds, to determine the degree to which our model is under-or over-parameterized-and therefore, by the scaling laws established in @xcite , overor under-trained. as loss is not an exact predictor of downstream performance @xcite @xcite , we follow @xcite and evaluate the impact of pretraining token count on a validation task as well. we again used the relpron dataset for this purpose, as snli introduces potential confounding factors in the form of the fine-tuning procedure. ## conclusion in section 4, we provided direct evidence in support of the lkch: critically, this finding indicates that lflms can learn with less data than textual models. this is supported by our findings in section 6, which indicate gfolds is underparameterized with half of its pretraining data. to the best of our knowledge, the experiments in section 5 represent the first time that a language model pretrained solely over logical forms has been evaluated on a wide range of downstream tasks. gfolds' capacity to be applied to tasks ranging from relpron to snli-and consistently outperform bert models pretrained on the same amount of data-demonstrates this model's versatility. these results therefore represent a significant step towards demonstrating the practical viability of lflms. although much work remains to be done before lflms can reach the same level of performance and utility as their textual counterparts (see section 8), the results of sections 4-6 suggest that such models present an exciting avenue for continuing the improvement of language models at a more sustainable rate of data consumption than llms over text.