LEMON : Language-Based Environment Manipulation via Execution-Guided Pre-training (2022)

Abstract: Language-based environment manipulation requires agents to manipulate the environment following natural language instructions, which is challenging due to the huge space of the environments.To address this challenge, various approaches have been proposed in recent work. Although these approaches work well for their intended environments, they are difficult to generalize across environments. In this work, we propose LEMON, a general framework for language-based environment manipulation tasks. Specifically, we first specify a general approach for language-based environment manipulation tasks, which can deal with various environments using the same generative language model. Then we propose an execution-guided pre-training strategy to inject prior knowledge of environments to the language model with a pure synthetic pre-training corpus. Experimental results on tasks including Alchemy, Scene, Tangrams, ProPara and Recipes demonstrate the effectiveness of LEMON: it achieves new state-of-the-art results on four of the tasks, and the execution-guided pre-training strategy brings remarkable improvements on all experimental tasks.

## introduction building agents that can understand human language and accordingly manipulate the environment around them has been a long-standing goal of artificial intelligence @xcite . various tasks focus on this scene, including collaborative building @xcite , state tracking @xcite ) and instruction following @xcite @xcite . what these tasks have in common is that the agents are required to manipulate the environment based on the natural * work done during internship at microsoft research asia. 1 our code is available at: https://github.com/ microsoft/contextualsp ## lemon framework we now discuss the lemon framework (figure 1 ) in more detail. specifically, we introduce the taskagnostic approach for language-based environment manipulation ( §2.1) and the execution-guided pretraining ( §2.2). as for lemon instantiations for different tasks, we leave the descriptions to §3. ## lemon instantiations to demonstrate the capabilities of lemon, we apply our framework on five exemplary tasks, namely, alchemy, scene, tangrams, propara, and recipes. examples of each task are shown in figure 2 , including visualizations of the initial environment state and the goal environment state, as well as a schematic representation of the program. in this section, for each task, we elaborate the definition of the environment and the applied program to instantiate lemon. ## experiments in this section, we compare lemon with baseline methods on the tasks discussed in §3 to demonstrate its effectiveness. due to space limitation, we do not introduce these baselines below. ## related work language-based environment manipulation the first line of our related work is the previous work on lem tasks. according to the output, existing methods on lem tasks can be mainly divided into two categories: program prediction and state prediction. prior work always treat the lem task as a program prediction problem @xcite @xcite @xcite @xcite . however, these approaches are environment-dependent and cannot be easily adapted to other environments. besides, they either rely on natural language-program pairs as supervision or require complex heuristic rules, which is costly. recent approaches generally treat the lem task as a state prediction problem by predicting the goal state directly @xcite @xcite @xcite . these models can eliminate the data collection issue, but require complex models designed to meet the needs of different kinds of environments. compared with the above work, lemon has the following advantages: 1) the proposed taskagnostic approach does not require additional annotations and is easy to generalize across different environments. 2) the proposed execution-guided pre-training strategy can further improve the model performance with synthetic data only. ## conclusion & future work in this work, we propose lemon, a general framework for language-based environment manipulation tasks that not only models different environments using the same protocol, but also injects prior knowledge of environments into our model. experimental results on five tasks demonstrate the effectiveness of lemon: the execution-guided pretraining strategy brings significant improvements on all of them and lemon achieves the state-ofthe-art performance on four of them. for future work, we hope to extend our approach to more complex environments and tasks such as image editing @xcite ) and text editing @xcite .