Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback (2024)

Abstract: Many approaches to Natural Language Processing tasks often treat them as single-step problems, where an agent receives an instruction, executes it, and is evaluated based on the final outcome. However, language is inherently interactive, as evidenced by the back-and-forth nature of human conversations. In light of this, we posit that human-AI collaboration should also be interactive, with humans monitoring the work of AI agents and providing feedback that the agent can understand and utilize. Further, the AI agent should be able to detect when it needs additional information and proactively ask for help. Enabling this scenario would lead to more natural, efficient, and engaging human-AI collaboration.In this paper, we investigate these directions using the challenging task established by the IGLU competition, an interactive grounded language understanding task in a MineCraft-like world. We delve into multiple types of help players can give to the AI to guide it and analyze the impact of this help on behavior, resulting in performance improvements and an end-to-end interactive system.

## introduction one of the long-lasting goals of ai agents @xcite is the ability to seamlessly interact with humans to assist in solving tasks. to achieve this, the agent must be able to understand human language and respond to it, so it can execute instructions @xcite or ask clarifying questions @xcite . researchers have proposed a large number of tasks aimed at tackling this human-ai collaboration challenge, many based on humans providing instructions to the agent to solve a goal @xcite . an example is the blocks world task, where the agent understands human instructions to move blocks on a grid @xcite . ## related work human-ai interaction tasks the task of humans interacting with ai agents to solve real-world tasks is a long-standing problem @xcite @xcite @xcite . among other challenges, the embodied ai agent needs to understand complex human language @xcite , spatial world orientation, and unseen concepts @xcite . as this problem is still challenging, datasets like iglu @xcite , basalt @xcite and minedojo @xcite have been recently proposed. in this work, we focus on building an agent that understands instructions to place blocks on a grid. iglu task since the iglu task was proposed @xcite , it has been the subject of multiple competitions, such as a rl task (building a rl-based first-person agent to place blocks) @xcite and a nlp task (determining when and what clarification questions to ask) @xcite . in contrast, as we are interested in building a fully interactive agent, we focus on a dialogue only iglu task setup, where an instruction is provided and a model predicts the blocks to be placed. as we do not focus on building a rl agent and we do not use a vision component (the 3d world space is encoded as language in our setup), our work is not directly comparable to the existing iglu baselines. however, we use similar metrics when applicable. further, we hypothesize that our interactive framework can be applied to other iglu-based tasks, by adding a language component that understands help similar to this paper, and leave it for future work. user-feedback as tasks like iglu are difficult, a crucial component of human-ai interactive systems is the ability of the agent to receive direct feedback from humans, to improve performance. this has been studied in active learning @xcite , llm feedback @xcite , robotics @xcite , summarization @xcite , and others (see appendix a). closest to us, mehta and goldwasser show how hints can be provided to the model. we build upon their regional ("top right") and directional ("move left") hints, to enable more forms of user feedback, by proposing additional types of hints. further, compared to mehta and goldwasser, we evaluate on a significantly more challenging task and use a stronger baseline model @xcite . we also propose a novel approach for the agent to identify when it is confused, and then enable it to ask relevant clarification questions. clarifying questions as instructions may be vague or unclear, the ai agent should be able to ask clarification questions @xcite @xcite , to solve the task better. this is often studied, especially in dialogue systems, and is still challenging @xcite @xcite . we use our "help" to determine when the model is confused and should ask a clarification question, and the question is based on what "help" the model needs. ## task-specific models in this section, we first discuss the specific formulation of iglu we use, which is different from other iglu setups @xcite , and unique to us. we then briefly explain the model we use for it. task formulation: the iglu task @xcite involves two players, a builder and an architect, that collaborate to build a target structure in the 3-d minecraft world (fig. 1 ). the builder places blocks based on architect's instructions. in our version of this task, the architect is a human, while the builder is an ai agent. thus, the builder places blocks and subsequently makes mistakes/needs to ask for clarification, while the human architect (which we simulate) helps the builder. further, our task formulation is fully language-based, and there is no vision component. this is because we are primarily interested in how to make agents more interactive, and interactions typically happen via language. hence, we chose this setup for simplicity. thus, our task formulation is as follows: given the architect and builder history complete with the last instruction, and a dialogue representation of the current minecraft world state, predict the coordinate locations of the blocks to place. model architecture we now briefly discuss the baseline system we train for the builder model, based on zholus et al.. later, we will incorporate help into this model. our baseline model is a standard bart-base transformer @xcite , trained for conditional generation using the hugging face package @xcite . as this is a language model, all of its' inputs and outputs are in natural language. thus, we now discuss the method we used to convert (x, y, z) coordinate block locations in the minecraft world grid to language, so they can be passed into the model as textual input. we first determine how far the coordinate is from the origin of the grid (0, 0, 0), for each axis. in language, we define the x-axis as 'left/right', y-axis as 'up/down', and z-axis as 'higher/lower'. we then combine the distance and direction into a sentence, e.g. an x location of -2 would be "2 left" and a z location of 3 would be "3 higher". we ignore model outputs that do not follow this format, as they are invalid. input grids with multiple blocks can also be encoded into language the same way, just with multiple sentences such as "2 left 1 up 3 higher. 4 right 2 down 4 lower.". ## help-specific models while the builder model introduced in sec. 3 achieves competitive iglu performance (sec. 5.3), it still makes a large number of mistakes. thus, in this paper, we propose an interactive setup, where a human can interact to "help" the model when it makes a mistake. rather than telling the model where to place the blocks, which would be difficult to provide and learn from, we propose that humans "help" the model by assisting it with a high-level concept necessary to solve the final task, making it easier. while not only being simpler to provide than solving the final task, this "help" enables the model to learn the task better, to perform better when no help is provided (it can focus on other aspects of the task, different from the concept provided by the help; for results see sec 5.3). for example, through one form of help, "length help", humans assist the model by telling it how many blocks to place. once the model understands this, it can focus and better learn other aspects of the task instruction, such as where actually to place each block. in this paper, we experiment with humans providing help via a natural language sentence. we first introduce 4 different forms of help feedback humans can provide agents, all based on different high-level concepts relevant to the iglu task (sec. 4.1). two were introduced by mehta and goldwasser, and others are novel to this work. detailed ex. of help are in app. b. then, in sec. 4.3, we discuss how this help can be learned and effectively incorporated into the task-specific baseline from sec. 3 @xcite . finally, in sec. 4.4, we explain how agents can leverage their comprehension of various forms of help to aid their own performance, effectively identifying when they are confused and then asking clarification questions. this final step culminates in a genuine interactive scenario, where the agent can receive interactions in the form of help and reciprocate by seeking clarifications. notably, when agents help themselves, they can exhibit improved performance without requiring any human interactions. ## discussion in this section, we analyze our iglu models with help feedback, by asking the following questions: handling inaccurate help help may not always be accurate, especially if provided by humans or someone trying to confuse the model, but the model should be able to adapt. tab. 2 shows the performance of self-generated help that was provided at test time, and it still leads to improvements (tab. 3 ). varying help utterances to simulate the large language variety of humans providing help (specifically restrictive), we first generate a variety of help by prompting llm's to write it, and then ask llm's to determine which region each help corresponds to (details: app. f). once the region is known, we can provide help to the models as normal. results in tab. 6 show that llms can effectively determine help regions, showing our approach can handle real human help. improving mean, but sometimes worsening standard deviation: while our models always improve mean performance, in some of our experiments (but not all), we see results do not improve on std. we hypothesize that this occurs due to our weak baseline model, not because our protocols are ineffective. a stronger baseline should lead to more consistent results. in short, whenever std worsens, it is due to outliers. these are cases where although the help was accurate, the model did not understand the initial iglu instruction, and thus the help confused it, making the initial prediction worse. for example, for restrictive help, if the bart model can't properly search the region provided because it doesn't understand the initial instruction, it may randomly place the block in the region, worsening the prediction. thus, these cases that lead to a worse std are already failure cases, and in fact help does improve performance overall. we discuss this further in sec. h ## conclusion and future work in this paper, we proposed an interactive framework for grounded language understanding tasks, specifically inspired by the iglu task @xcite . our framework enables humans to interact with ai agents through four distinct forms of help feedback, to provide high-level tips based on concepts relevant for the final task. this high-level help is easy to provide and proves beneficial for the ai agent. additionally, we proposed a mechanism for the ai agent to autonomously detect confusion and ask clarification questions. to do this, we leveraged help feedback by developing a model to self-generate help, provide it to the agent, and ask a clarification question if confusion is detected. through this approach, we achieved a fully interactive agent capable of both receiving and providing interactions to humans. our experiments demonstrated performance improvements in these settings. moving forward, our future work will focus on enhancing the performance of clarification questions, and incorporating more types of help. we are also interested in generalizing our contributions to other domains, including tasks that don't require an agent to navigate a 2d/3d space. we believe our approach is directly generalizable to tasks that require an agent to navigate a 3d or 2d space to make decisions (like many robotics tasks). here, the forms of interactions we proposed and how they are used would not change. for tasks that do not have a 3d/2d space, like summarization, we hypothesize that our framework can still be applicable, by modifying help and keeping the rest of the framework the same. for example, for summarization, the agent must read and summarize certain areas of the document, while ignoring other irrelevant areas, in order to produce a successful summary. thus, instead of restrictive help restricting the search space of the agent to an area on a 3d grid, restrictive help can restrict the lines of text in a document that the agent has to read. this help would useful to enable the agent to prioritize the relevant parts of the document, leading to a better summary. similarly, instead of corrective help changing the direction of the grid the agent should search, it can correct the summarization by detailing topics that were missed in the summary. in these ways, our framework can generalize to other tasks, making them end-to-end interactive, and improving performance. investigating this is part of our future work.