DIMSUM : Discourse in Mathematical Reasoning as a Supervision Module (2025)

Abstract: We look at reasoning on GSM8k, a dataset of short texts presenting primary school, math problems. We find, with Mirzadeh et al (2024), that current LLM progress on the data set may not be explained by better reasoning but by exposure to a broader pretraining data distribution. We then introduce a novel information source for helping models with less data or inferior training reason better: discourse structure. We show that discourse structure improves performance for models like Llama2 13b by up to 160%. Even for models that have most likely memorized the data set, adding discourse structural information to the model still improves predictions and dramatically improves large model performance on out of distribution examples.

## introduction recent advancements in large language models (llms) have led to impressive performance on reasoning tasks, particularly on benchmark datasets like gsm8k @xcite . these models exhibit strong problem-solving abilities, often producing solutions that resemble humanlike reasoning. however, recent studies have demonstrated that altering the entities or numerical values can degrade the reasoning capabilities of llms @xcite . other studies have also shown that model reasoning is not robust @xcite @xcite . this raises a critical question: do these models genuinely engage in reasoning, or do they primarily rely on memorization and pattern recognition? these findings suggest a need for a deeper investigation into how llms process logical structures and linguistic variations. prior research @xcite has also investigated the effect of permuting the premise order † equal contribution in gsm8k problems, showing that such modifications affect llms' reasoning capabilities. this observation suggests that, unlike formal logic, where the validity of a conclusion remains unchanged regardless of premise order, natural language reasoning is shaped by structural dependencies within discourse. a linguistic perspective explains this dependency. unlike formal logic, natural language introduces ambiguity and underspecification, which are resolved through contextual and structural cues that tell us how clauses in a text are semantically related. the ordering of clauses within a discourse, for instance, significantly impacts interpretation, and determines, in the absence of other cues, temporal and causal relationships. consider the simple sequences in (1) from @xcite : when we permute the order of the clauses as in (1)b, most native english speakers will draw different conclusions. they will conclude from in (1)a that pat first took off his shoes before getting into bed, whereas in (1)b that pat took his shoes off after he got in bed. ordering in this case determines a semantic relation- @xcite calls it narration that entails that the event in the second clause follows the event in the first. even in seemingly simple narratives, semantic relations such as elaboration, background information, and narration shape how reasoning unfolds. additionally, discourse structure affects anaphora resolution-reordering premises can change how pronouns and referents are interpreted, thereby influencing reasoning outcomes. for example, in (2) we show that information about discourse structure improves performance for all llms on the gsm8k data set, but especially for older, models like llama2 13b with poor reasoning performance by up to 160%. even for models that have most likely memorized the data set, adding discourse structural information to the model improves performance on gsm8k and dramatically improves performance on out of distribution examples. ## hard gsm8k dataset and variants to systematically assess whether llms really reason or just rely on memorized patterns, we use the more difficult portion of the gsm8k data set, hard gsm8k, (with problem statements involving 4 or more premises). we also provide a controlled variant of hard gsm8k to create out-ofdistribution (ood) variants using the transformations defined below (see appendix a.5 for examples). this data will be made publicly available. unlike the data in @xcite , where modifications involves changing entity names and adding or subtracting premises, our variants are more complex in terms of contextual modification and number of substitutions. we investigate whether the model can maintain performance under variations in numerical values and contextual framing. this data enables us to rigorously evaluate the robustness of llms' reasoning, and it serves as a test-bed to study the influence of the discourse structure. ## generating discourse structure to annotate discourse structures systematically within hard gsm8k, we identify 10 complex examples containing multiple events and subjects, with more than eight sentences. we manually annotated these examples according to rules in @xcite , used them as fewshot (brown et al., 2020) exemplars for generating annotations across the entire dataset using the llama 3.1 70b model. ## results we evaluated several llama models @xcite grattafiori et al. ( 2024 )) -llama 3.1 70b, llama 3.1 8b, llama 3 8b, and llama 2 13b-on reasoning with three distinct data sets, hard gsm8k and variants, gsm symbolic and our variants and our gsm mod. we tested the performance of models on problems with and without incorporated discourse structural information. we used a four-shot learning setting. we also tested our approach on mirzadeh et al. ( 2024 )'s data sets with additional premises that were needed (their p1 and p2). to replicate their method, we used eight shot learning. even more noteworthy is the fact that on our altered, unseen examples, even large models like llama3.1 70b saw their prediction success rate drop, by almost 40 percentage points for the contextually shifted stories. once those models had access to the discourse structure their performance improved by sometimes up to 30 percentage points. from the table, it is clear that the contextually shifted stories were much more challenging than the other variants we made. we also tested models on examples with background information from mirzadeh et al. ( 2024 ), and all models correctly avoided using the background information in the computation. in appendix a.4.3, we also show that the llama 3.1 70b with discourse structure are able to predict the correct answers to all the examples provided in chen et al. @xcite . finally, we tested our models on a subset of the @xcite . this is a synthetically generated data set for training, not a benchmark. but we felt that evaluating the usefulness of discourse structure on this data set might show whether our approach overfitted the gsm8k paradigm. we used an 8 shot learning regime, an observed a very significant boost in scores as in ## conclusion we have shown that providing discourse structure, which encodes the semantic relations between premises or clauses in a text, significantly improves the performance of language models on tasks such as those in the gsm8k dataset. our results also demonstrate that even models that perform well on the standard dataset show substantially degraded performance on out-of-distribution variants; however, incorporating discourse structure leads to notable gains in robustness and generalization across these variants. as a future direction, we recognize the importance of scaling the generation of discourse structures. while our current approach relies on llama 3.1 70b, exploring the feasibility and effectiveness of generating discourse structures using smaller language models is a promising avenue. this would broaden the accessibility and applicability of our method, enabling lightweight integration in resource-constrained settings.