Nebula: A discourse aware M inecraft Builder (2024)

Abstract: When engaging in collaborative tasks, humans efficiently exploit the semantic structure of a conversation to optimize verbal and nonverbal interactions. But in recent “language to code” or “language to action” models, this information is lacking. We show how incorporating the prior discourse and nonlinguistic context of a conversation situated in a nonlinguistic environment can improve the “language to action” component of such interactions. We finetune an LLM to predict actions based on prior context; our model, Nebula, doubles the net-action F1 score over the baseline on this task of Jayannavar et al. (2020). We also investigate our model’s ability to construct shapes and understand location descriptions using a synthetic dataset.

## introduction high level building agents use conversation in a collaborative task to combine information about the extant conversation, the world, and prior actions to execute new instructions. such agents interpret messy or vague language, produce actions, then reassess the situation, ask questions or take in corrections from other agents to optimize their actions. successful collaborative conversations are vital for efficiently performing complex interactive tasks. in this paper, we study the messy language of ordinary human collaborative conversation, and how a large language model can learn to execute instructions from such conversations. we isolate several factors that affect this task. the first factor is the interaction between linguistic and nonlinguistic contexts. previous work has shown that at least some context is needed to understand and carry out conversationally given instructions @xcite . we improve on that work by first establishing a baseline by using the entire exchange up to an instruction i as a context for an llm to interpret i. our llm model, nebula (neural builder with llama), trained on the minecraft dialogue corpus (mdc) @xcite , achieves netaction f1 scores that is almost double of @xcite . using the minecraft structured dialogue dataset (msdc) @xcite , which provides semantic relations between mdc dialogue moves and nonlinguistic actions, we show that particular discursive components of the linguistic and nonlinguistic context are necessary and sufficient for the llm to understand an instruction to the degree provided by the baseline. analysing nebula's output reveals two other factors that adversely affect its performance. an instruction in the msdc has two basic components: a description of a shape in terms of four parameters-numbers of components, colors, arrangement and orientation-and the description of a location where the shape should be placed. human architects often use analogies to everyday objects that may be challenging to process; in addition, shape descriptions are often underspecified, meaning that one could perform the instruction correctly in various ways. location descriptions in the minecraft world are also quite difficult to process and highly underspecified. for example, put a tower in a corner could be correctly located in any of the four corners of the minecraft board. we address this problem in two ways: first by further finetuning nebula on a synthetic dataset to improve its performance in building basic shapes and locating them appropriately; and secondly, and more importantly, by revising the evaluation metric used by @xcite to reflect more realistically the semantics of location expressions. we show that, on our synthetic dataset, nebula achieves high accuracy as per our intuitive metric in performing basic instructions. the main contributions of this work are: (i) to show that finetuning llama-3-8b @xcite on the entire prior history of the action-prediction task nebula doubles the net-action f1 score on the minecraft dataset as compared with the neural builder of @xcite ; (ii) to show that training on narrative arcs achieves comparable results with baseline nebula, and that training on narrative arcs performs better than the instruction-action-instruction input template used by @xcite ; (iii) to highlight the drawbacks of the evaluation metric (i.e., net-action f1), and propose and test a new metric on our synthetic datasets. after some preliminaries and discussion of prior work (section 2), we present our model, nebula, and its baseline performance in section 3, and then a necessary and sufficient discourse feature to get scores equivalent to the baseline in section 4. in section 5, we explain several issues associated with the minecraft corpus. we try to address these issues in section 6, where we explain our evaluation metric for underspecified instructions, as well as experiments on our synthetic datasets. we conclude in section 7. instructions to code: neural builder and variants the mdc @xcite incentivizes the development of an algorithm that can predict sequences of actions from instructions. the actions involved basic moves of placing or removing blocks from certain positions in the environment. @xcite train a model consisting of a gru @xcite to handle textual input coupled with a cnn to integrate information from the current state and a gru to predicted an action sequence. although they experiment with several training regimes, the best performance comes from one in which a sequence of conversational moves after some action sequence, assumed to be instructions are given to the model, are followed by the next action sequence of the builder, followed by the next sequence of linguistic moves are input to the model to predict the subsequent action sequence (see figure 1 ). the net-action f1 metric evaluates a model's prediction based on the exact color and coordinate match between the model's predicted sequence and builder's gold action sequence. in general, @xcite show that the problem of predicting action sequences from natural language instructions in naturally occurring dialogue remains extremely challenging. their neural builder has net action f1 of 0.20 on the mdc test set. @xcite propose a somewhat different task from @xcite ; they try to predict when the builder should execute an action and when they should instead ask for a clarification question. to this end, they annotate all builder dialogue moves with a taxonomy of dialogue acts. they then specify a single specific action under the execution label instead of a sequence of actions. thus, their set-up is not directly comparable to that of @xcite . a partial step towards a full discourse structure: they provide labels for various dialogue moves, but the full discourse structure that we propose to use involves relations between moves. these relations are important as they tell us how to link different parts of, for instance, an instruction into a coherent whole. as we aim to demonstrate in this paper, discourse structure can help to clean up datasets for training and thereby improve training. @xcite provide full discourse annotations for the mdc, known as the minecraft structured dialogue corpus (msdc), using the discourse theory and annotation principles of sdrt @xcite extended to a multimodal environment, in which both nonlinguistic actions and discourse moves can enter into semantic relations like elaboration, correction, and narration @xcite . they follow annotation practices given for the stac corpus @xcite . @xcite also adapt the parser from @xcite to predict discourse structures for the mdc with relatively high reliability. statistics on the msdc are in llms in robotics parallel to this work, there has been an increasing amount of research in aiding virtual or real robots with tasks by using llms to provide translations from natural language instruction to code that programs the robot to perform the relevant actions @xcite @xcite . this research is directly relevant to our work, as we use llms to go from natural language to a pseudo-code of pick and place statements. however, whereas liang et al. (2023); singh et al. (2023); yu et al. ( 2023 ) focus on optimizing the translation from instructions, typically one instruction, to various different coding paradigms, we focus on how linguistic and nonlinguistic interactions affect the resulting action sequence. as our results and previous results on the mdc show, producing actions from interactive conversation with frequently underspecified instructions, which are also dependent upon the discourse and nonlinguistic contexts for proper interpretation, is a much more challenging task than translating well crafted unambiguous instructions into code. in addition, we show that to predict a relevant action from the current instruction i n+1 in the mdc environment, it is not sufficient to use a context of just the penultimate instruction i n and previous action sequence a n . ## nebula: an llm for predicting action sequences we've seen that jayannavar et al. (2020)'s neural builder performs poorly as per their evaluation method, i.e., net-action f1. the training scheme of @xcite assumes, in effect, that architect instructions and the builder actions which fulfill these instructions follow one another in a regular succession. a consequence of this assumption is that actions are individuated by the conversational turns that immediately precede and follow them, and likewise that a new action sequence is initiated whenever there is a linguistic move of any kind. neural builder predicts actions from a preceding context containing an instruction, and the instruction just before it, and a representation of the builder moves in between: i n a n i n+1 predicts a n+1 . additionally, neural builder is provided a worldstate representation (see figure 1 ). but this is not realistic, as these bits of text don't always yield a well-formed instruction or even an underspecified one. in addition, often actions and instructions occur simultaneously. we might have a clarification question from the builder in between two action sequences that are in fact carrying out the same action as in figure 2 . builders in the mdc frequently ask questions with respect to the initial instruction about the actions they are currently carrying out; answers to those questions may affect the actions, but it doesn't mean that there are two distinct series of actions pertaining to two distinct instructions, one before the question and its response and one after. in addition, the builder sometimes starts to build before the instruction sequence is complete; intuitively, the initial actions form a coherent action sequence with the actions that are subsequent to the further instruction. these observations show that the assumptions of @xcite about how actions are individuated are too simple. the shape and position of the structure, intended by the initial instruction, can change or be made more precise by different conversational moves. @xcite note that different conversational moves can help conceptualize actions differently. for example, in many minecraft sessions, an initial instruction gives the builder an action type that might be realized in many different ways. something like build a tower of 5 blocks is an action type for which a concrete realization would have to specify the color, and perhaps the nature of the building blocks, as well as a location. as the conversation evolves and unless the architect corrects their instruction, the type of action to be performed becomes more and more specified. a simple baseline alternative to the scheme proposed by @xcite that addresses these difficulties is to see how a model performs with the complete prior conversation and action sequences up to the predicted action. this was not an option for jayannavar et al. (2020)'s model, but more recent llms are capable of doing this. we use llama-2-7b, llama-2-13b and llama-3-8b models to take as context all the conversation and action sequences up to action sequence a n to predict a n . we finetune llama on the mdc's @xcite training set. all the models are finetuned for 3 epochs using qlora method @xcite . ## using discourse structure to improve nebula the ideal way to model collaborative, instructional interactions like those featured in the mdc is to have two simultaneous, interleaved processes that interact with each other. on one hand, there is the evolving conversational structure that conceptualizes the nonlinguistic actions; on the other, there is the sequence of actions that also affects continuations of the conversational structure. this interleaved process is modeled by the discourse structure provided by the msdc of @xcite . the msdc shows a large scale pattern of narrative arcs. these arcs delimit portions of discourse structure linked by a narration relation. each portion begins with an instruction i n from the architect, terminates with an action sequence a m , and involves a negotiation between the architect and the builder about the action sequence to be performed. the negotiation may be extremely short, where the arc just contains a single instruction and resulting (correct) action i n , a m . but it can also contain a complex negotiation involving a number of discourse moves that serve to elaborate or clarify the initial instruction. in this way, the content of a single instruction i n may evolve over many discourse moves. further, if the initial builder actions are incorrect, this will prompt corrective moves by the architect and subsequent revisions by the builder, leading to longer collaborative negotiations until the builder carries out the action sequence that satisfies the architect's initial instruction. figure 3 illustrates a long negotiation enclosed by a narrative arc. the instruction in the first turn results (in green) in a builder action sequence in turn two. the builder then asks a question to confirm that the actions in turn two were correct. the architect replies to the question by correcting (in red) the actions in turn two, which then results in a corrective action sequence in turn four. guided by the assumption that these narrative arcs contain all the contextual information-the initial instruction and all subsequent discourse moves-that is needed to predict the correct actions, we use the narrative arcs provided by the msdc to assign the previous context to each action sequence in the mdc. since the arcs are automatically recoverable to a relatively high degree by the parsers discussed in @xcite , we make the expedient choice to use the gold arcs to test our assumption. we finetune llama-3-8b on the mdc training set using instruction-action pairs, where the instruction is only the conversation within the narrative arc up to the present instruction i n+1 in the pair, not the entire conversation history, which we fed to baseline nebula. since neural builder supplements attenuated contexts with a worldstate representation (see section 3) we also include a "worldstate" at the beginning of the narrative arc in terms of net place actions. we refer to the resulting model as nebula+n (nebula trained on narrative arcs). the majority of the narrative arcs are shorter than or of equivalent length to their i n a n i n+1 counterparts used to train the neural builder (1321 out of 1575 samples in the test set). going on length alone, it is reasonable to assume narrative arcs and i n a n i n+1 are interchangeable. however, if narrative arcs are not only sufficient but also necessary for action prediction, then in the 254 samples where narrative arc is longer than i n a n i n+1 we would expect nebula+n to perform better on those samples when given the narrative arc. and this is the case. in ## problems with the mdc in minecraft, the architect makes use of several location descriptions. these descriptions are often anaphoric to blocks placed in prior instructions, such as place another block next to that one (one that was placed on previous builder turn); locations are also sometimes vaguely designated (towards the centre) or underspecified (in a corner, along an edge, n blocks/spaces in from an edge/from the centre). although the minecraft environment presents (x, y, z) coordinates, the human participants never used them. this could be because, in the minecraft environment, players can move their avatars around the board to get different perspectives, which makes it hard to establish an absolute coordinate system. as a result, the net-action f1 metric, which evaluates a model's action sequence based on whether the block placements match exactly in terms of block color and coordinates with the corresponding gold builder action, is often inappropriate. for instance, if the builder puts down a block at one corner after receiving the instruction in a corner whereas nebula chooses another corner, the metric would give nebula zero credit whereas intuitively it still executed the instruction correctly. to summarize, the net-action f1 evaluation metric treats vague instructions as completely precise ones, and considers one instantiation of an instruction (i.e. the action sequence of builder in the gold data) to be the only ground truth. another related issue is highlighted in figure 2 , where the action sequence for the architect's instruction gets truncated by a question from the builder "there?". in this case, for the aforementioned instruction, only the first three actions (place yellow -1 1 0, pick -1 1 0, place yellow -1 4 0) constitute the ground truth. thus, the underspecified instructions with multiple plausible instantiations, coupled with the strict nature of the metric, puts an upper bound on how much the net-action f1 score can improve on this dataset. more importantly, it doesn't reveal what a model with a high f1 score actually does learn. we attempt to answer this in the next section. ## evaluating nebula on synthetic data given the issues associated with mdc and the evaluation metric, we test baseline nebula (i.e., the one trained on entire conversation history) on simple scenarios using a more just metric. to do so, we construct synthetic datasets at two different levels. the goal of these datasets is to test what basic shapes and location descriptors nebula learns after being trained on mdc. for the first level, we test nebula's ability to construct simple shapes, such as, square, row, rectangle, tower, diagonal, diamond, cube of specific size and understand location (i.e. corner, centre, edge) and orientation descriptions (i.e. horizontal/vertical). we refer to all these shapes as level-1 structures. the resulting dataset, referred to as level-1 dataset, consists of 1368 instructions. some of these instructions simply ask to construct a shape of specific size like "build a 3 × 3 red square.", while others are more detailed, for example, "build a 3 × 3 red horizontal square at the centre." for rows/diagonals/towers, we vary size from 3 to 9. for squares, the size varies from 3×3 to 5×5. for cubes, we only use 3 × 3 × 3. for rectangles, we use sizes m × n, where m ̸ = n, m × @xmath0. for diamonds, we use two variants to describe size "m blocks on a side" and "axes 2m+1 long", where 3 <= @xmath1. we use orientation descriptions (i.e. horizontal/vertical) for squares, rectangles, and diamonds. to evaluate nebula on these instructions, we use simple binary functions is_square(c), is_tower(c) etc. for each shape. these functions take as input the predicted construction c and returns true if c is the desired shape, and false otherwise. for example, is_tower checks whether all the blocks have the same value for x and z (as y is the vertical dimension) and y values are distinct and form a sequence 1, 2, ..., n where n is the number of predicted blocks. for an instruction, we first evaluate if the predicted shape is correct. for correct shapes, we evaluate whether the size/color and we also evaluate the performance of neural builder @xcite on our level-1 dataset. the results are shown in as a second step, we look at nebula's ability to understand location descriptions, in particular ones that are anaphorically specified. to do so, we start with an instantiation (randomly chosen from the set of correct instantiations) for the 1368 instructions in level-1 dataset. so, for a level-1 instruction such as "build a 3 × 3 red square.", we have a 3 × 3 red square already present in the grid. now given a level-1 structure in the grid, we design level-2 instructions which require placing or removal of a specific color block. for place instructions, we use location descriptions like on top of, to the side of, touching, and not touching. so an example of level-2 place instruction is "place a blue block on top of that." where that refers to the level-1 structure in the grid. similarly, for removal instructions, we have the simple instruction "remove a block" and more complex instructions including location descriptions like you just placed. we also have additional location descriptions for certain level-1 structures such as end for rows, diagonals; top, bottom for towers; corner for cube; centre for cube, odd-size squares and towers. an example of level-2 remove instruction is "remove the top block." both level-1 and level-2 datasets were generated in an automated fashion. similar to level-1, we evaluate nebula on level-2 dataset by making use of binary functions like is_ontopof (b, c), is_touching(b, c) where c is the level-1 structure already present in the grid and b is the predicted block. for example, binary function for on top of checks whether there is no block in c which is directly above the block b, and there is a block in c underneath block b. we then examine nebula's errors with on top of. we find that the failure cases mostly were a result of the model placing multiple blocks instead of just one on the given level-1 structure. that is, the model does not always understand a block as a single block. in light of these cases, when we check whether all the blocks in predicted b are on top of c, the accuracy improves from 74.2% to 97.2%. thus, some of the difficulties nebula had with instructions come from what might be a limited understanding of the semantics and pragmatics of indefinite and numerical noun phrases. we provide visual comparisons of neural builder and baseline nebula on level-1 and level-2 dataset examples in tables 9 and 10 of the appendix. ## conclusions and future work we introduce nebula, an llm based action prediction model, for the minecraft dialogue corpus. as a baseline, nebula uses the entire minecraft dialogue up to action a n to predict a n . we show that this baseline doubles the net action f1 scores of @xcite . we then show that certain discourse structures provide necessary and sufficient information for inferring actions at-par with the baseline setting. we also analyze nebula's errors on mdc and provide additional finetuning to improve the model's ability to interpret underspecified shape descriptions and anaphorically-specified locations using our synthetic dataset. this allows us to analyze the shortcomings of the net-action f1 metric, and address them using a more realistic evaluation metric. our evaluation metric captures the notion of relative location, but leaves exact locations typically underspecified, in accordance with our semantic intuitions. for future work, we plan to explore metrics similar to our relative location metric that can be applied more generally, including on the mdc. given the improvement in performance of nebula after finetuning on our synthetic dataset, we hypothesize that in a more controlled collaborative task, with some pedagogical instructions to the architect, nebula could contribute as a useful interface for conversational robots that interact with humans.