Does discourse structure help action prediction? A look at Correction Triangles. (2025)

Abstract: An understanding of natural language corrections is essential for artificial agents that are meant to collaborate and converse with humans. We present some preliminary experiments using language-to-action models investigating whether discourse structure, in particular Correction relations, improves the action prediction capabilities of language-to-action models for simple block world tasks. We focus on scenarios in which a model must correct a previous action, and present a corpus of synthetic dialogues to help explain model performance.

## introduction in order to successfully complete a shared task, such as building a block structure in a shared environment, participants must accumulate a body of shared information about the goal of the task, the changing state of play, and their beliefs and intentions, collectively referred to as common ground @xcite . errors are integral to the process of building common ground, as participants naturally explore and test their strategies through trial and error @xcite . when natural language is among the modes of communication available to participants, corrective speech acts provide an efficient and information-rich mechanism with which they can identify and quickly resolve errors @xcite . for artificial agents that can collaborate with humans using natural language, the ability to understand and use corrections is essential. while most speech acts entail a monotonic update of the common ground (elaborations or acknowledgments, for example), a correction entails a revision to the common ground, and is an example of a divergent speech act @xcite . an agent that understands corrections must: 1. recognize an utterance as an instance of a divergent speech act. the minecraft dialogue corpus (mdc) @xcite features interactions in which two humans, playing the roles of builder and architect, collaborate to construct block structures in a simulated 3d environment. architect and builder communicate via chat window, where architect describes the structure to builder, who may then place and remove blocks on the grid. the mdc provides paradigmatic examples of collaborative conversation situated in a shared environment, wherein the players' linguistic contributions and the non-linguistic builder actions are highly interdependent, particularly when builder performs an erroneous action which the architect must then verbally correct. the minecraft structured dialogue corpus (msdc) @xcite adds an annotation layer to the mdc, drawing semantically-typed relations @xcite between player utter-ances and actions, which elucidate the overarching semantic structure-or discourse structure-of the interaction. in the msdc, correction scenarios are captured in substructures called correction triangles: an architect instruction β stands in the correction relation to the previous erroneous action α, eliciting a new action γ, which is the result of the architect correction and also a correction of the erroneous action (figure 1 ). the msdc discourse structures, including correction triangles, were shown to be automatically retrievable with state-of-the-art accuracy using the llamipa discourse parser @xcite . presumably, an agent utilizing the output of this parser would have at least a partial understanding of correction scenarios, since predicting the relation correction(α, β), amounts to identifying utterance α as a correction ("no, i said add a blue block" (1), and connecting it to the previous context β (builder places red block) (2). however, the question we want to address in this paper is whether an agent can leverage the discourse structure of an interaction to inform its subsequent manipulation of the environment. this goes beyond the parsing task: given a dialogue context and its semantic structure, including correction(α, β), we want to know whether the presence of the structure improves agent predictions of the action sequence γ that would appropriately complete the triangle. in what follows, we briefly discuss current work relevant to our question. we then describe the language-to-action model that will serve as our agent for action prediction experiments using the msdc. we explain how our preliminary results incentivize the creation of synthetic correction dialogues, which allow for more tightly controlled experiments. we then discuss model performance on synthetic correction dialogues and directions for future work. ## related work previous approaches to building agents that understand corrections in collaborative tasks differ with respect to model architectures. @xcite and @xcite build agent models based on pared down cognitive architectures for interactive task learning @xcite , where a corrective utterance generates symbolically encoded, probabilistically weighted hypotheses, and is used to update the agent's belief state and inform an action plan. alternatively, chiyah-garcia et al. (2024) create a language-toaction model by fine-tuning a large vision language model (vlm) on instruction-action pairs from a block world dataset @xcite . they augment the instructions with third position repairs @xcite , and use masking techniques during finetuning to encourage the model to recognize the repair. in our experimental setup, we also use an llm-based language-to-action model: our agent model is based on nebula @xcite , a llama3 architecture fine-tuned on the mdc dialogue-to-action corpus. furthermore, we use the discourse structure annotations from the msdc, which make corrections explicit, whereas the repairs in chiyah-garcia et al. (2024) were unmarked. discourse parsing predicts the semantic relations that hold between the elementary units of a dialogue, and produces a structural representation of the interaction. the most recent approaches to parsing based on llms formulate the structure prediction task as a sequence-to-sequence generation task @xcite , where the parser takes the dialogue units as input, and outputs the discourse structure as a sequence of typed tuples. the llamipa parser @xcite provides state-of-the-art results on the msdc using this approach. our agent model uses the llamipa structure representation, in which the discourse graph is flattened into a string of typed tuples, where each tuple represents a single relation (see figure 2 ). discourse structure has been used to improve performance on various downstream tasks. @xcite leverages discourse information to predict political orientation of news articles, while @xcite uses it to improve extractive meeting summarization. @xcite demonstrate that discourse structure can improve a model's performance on mathematical reasoning tasks. the experiments described in this paper are the first to use discourse structure to improve action prediction in situated collaborative tasks. finally, data synthesis has been increasingly used to provide high-quality training data for llms and llm-based agents @xcite , as well as to perform targeted tests of llm knowledge @xcite . synthetic data has been shown to be especially helpful in determining which concepts an agent trained on the mdc's ambiguous natural language instructions actually learned @xcite navar et al., 2025). this work is the first to create synthetic dialogues with discourse annotations to test the efficacy of discourse structure in a downstream task. ## a structure-aware language-to-action model the nebula language-to-action model @xcite , was trained using the mdc to predict a builder action sequence given the previous dialogue, and was evaluated using the same net action f1 metric as the baseline mdc model @xcite . given a completed action sequence, net action f1 is computed on newly placed blocks that exactly match the color and position of those in the corresponding gold action sequence. nebula leveraged the large context window of the llama3-8b architecture @xcite , which allowed it to predict an action sequence using the entire previous dialogue context, resulting in a contextaware model that doubled the baseline f1 on the mdc. in order to see whether the addition of discourse structure might further improve performance on the action prediction task, we augmented each mdc training sample with the gold foot_0 discourse structure from the msdc. we formatted the structure as a sequence of typed tuples, and appended it to the dialogue context (see figure 2 ). following the nebula training regime, we finetuned llama3-8b using qlora @xcite for 3 epochs on the augmented data (training parameters shown in appendix a). the result was nebulipa, a structureaware language-to-action model. the leftmost column of nebulipa-e ("nebulipa-empty") shows the result of an ablation in which the structure was removed from the test samples, in order to provide some further indication of whether nebulipa, trained with structure, learned to use it. we see that this brute inclusion of structure hinders rather than improves model performance, as f1 drops two points. also, nebulipa-e shows no change with respect to nebulipa. if nebulipa were using discourse information for action prediction, we would expect its removal to result in a drop in f1; yet this result indicates that, overall, training with structure did not lead to the model to exploit it. ## focusing on correction triangles the preliminary result above shows that including full discourse structures, containing many different relation types, does not improve languageto-action model performance on the mdc action prediction task. we note that the relational structure presents each relation uniformly, even though some types are more informative than others, given the discourse context. as mentioned in section 1, corrections describe revisions to the common ground, and so are often more informative than other relation types holding between less salient parts of the context: comment, acknowledge-ment, etc. to test this, we took a subset of mdc samples foot_1 in which the final action sequence to be predicted by the model is the result of an architect correction, i.e. where a correction relation is critical to the final prediction (figure 3 ). the second column of nevertheless, we maintain that this result must not be taken as decisive for three reasons. the first is that the longer dialogue contexts feature a dense relational structure (with uniform relations, as just mentioned), presenting the possibility that the signal provided by more informative relation types, such as correction, is greatly diminished; this is illustrated in appendix b. second, the mdc instructions contain highly context-dependent language, rich in anaphora and ellipsis, which often does not indicate a single correct action sequence. for example, we see in figure 3 that nebulipa performs the correct net action given the previous dialogue, but then places additional blocks-yet there is nothing in the dialogue that prohibits this. the second reason, already mentioned above, is that the contexts are long and can be very dense. lastly, since the net action f1 metric requires action sequences match the exact positions of the gold sequences, it unjustly discounts actions that are the result of instructions that are naturally ambiguous, e.g. put a block in a corner, and thus obscures the model's true performance. taking these factors into consideration, we determined that a set of short dialogues in simple correction scenarios, in which we could zero in on correction triangles, would help us more clearly assess whether structure can be leveraged for action prediction. to this end, we synthetically generated synthcorr300, a set of 300 short dialogues 3 . in each dialogue, architect gives three instructions for simple shapes, one of which builder botches, eliciting a correction in architect's final turn (figure 2 ). the shapes are towers and rows, which nebula was shown to build with high accuracy @xcite , as well as single blocks, which nebula was able to place and remove on rows and towers already present on the grid. for each instruction, a shape and its parameters were chosen randomly: one of six possible colors and a size (for towers and rows) of 3, 4, or 5 blocks. repeated colors, shapes, or sizes occurs in a majority of the dialogues ( ## results there was only a one-point difference between nebula and nebulipa on synthcorr300 ( to consider another angle of comparison, we looked at two ways in which models failed to generate the correct action sequences. action errors occurred when the model correctly identified the shape to be changed (after the first, second, or third instruction), but did not perform the correct actions to do so. site errors occurred when the model changed a shape that was not indicated by the cor-rection, misidentifying the botched sequence. returning to the discussion in section 1 of what is in involved in understanding corrections, we can roughly align site errors with the failure to identify what portion the previous dialogue the correc-tion refers to ( 2 ), e.g. which instruction. we can align action errors with the failure to properly revise the common ground (3), e.g. to perform the correct block placements and removals. ## conclusion in this paper we broach a question in discourse that has gotten relatively little attention-can discourse structure guide action predictions?-and explore a particular llm-based approach for an initial investigation. the results of our synthetic dialogue experiments showed that, overall, access to large contexts overrides the effects of adding explicit discourse representations. however, there was some indication that models trained with structure did learn to exploit corrections, using them to correct relevant parts of the discourse context with higher accuracy. in future work, we will enlarge the synthetic data in order to further investigate the action error results, as well as consider more varied correction scenarios.