Remember what you did so you know what to do next (2023)

Abstract: We explore using the 6B parameter GPT-J language model to create a plan for a simulated robot to achieve 30 classes of goals in ScienceWorld, a text game simulator for elementary science experiments and for which previously published empirical work has shown large language models (LLM)s to be a poor fit (Wang et al., 2022). Using the Markov assumption, the LLM outperforms the state-of-the-art based on reinforcement learning by a factor of 1.4. When we fill the LLM’s input buffer with as many prior steps as will fit, improvement rises to 3.3x. Even when training on only 6.5% of the training data, we observe a 2.3x improvement over the state-of-the-art. Our experiments show that performance varies widely across the 30 classes of actions, indicating that averaging over tasks can hide significant performance issues.

## introduction our research interest is in using a modest-sized, self-hosted large language model (llm) for planning so that a robot can execute the plan(s) to achieve goals. google's innermonologue project @xcite uses palm (540b parameters) and a real robot; @xcite use a small llm (t5-large) but for many decisions relies on the massive model @xcite to achieve goals in scinceworld. our study uses @xcite ) in an opensource, simulated environment, scienceworld @xcite ). we selected scienceworld for this study since it (1) supports several types of primitive actions, many types of entities, and 30 classes of tasks representing significantly differing challenges; (2) provides a simulator for the environment, including natural state changes (e.g., ice melting, a butterfly emerging from a chrysalis); (3) may benefit from applying common sense knowledge; (4) might be improved by external knowledge from a knowledge graph or from text; and (5) had a substantial amount of training data so that we could explore the effect of training set size. our empirical results contrast with those of @xcite , who evaluated five architectures via individually training 30 different models, one for each of the 30 classes of tasks. those models chose the next step given the current text feedback from the simulator and the previous action (i.e., markov assumption). by contrast, we train a single gpt-j model to cover all 30 classes of tasks and explore filling the llm's input buffer with as much game history as fits. a contemporaneous study @xcite uses two models: t5-large makes routine decisions; when needed for some decisions, the architecture calls gpt-4. though performance relying on gpt-4 exceeds gpt-j, when @xcite used turbo, rather than gpt-4, performance is comparable to our results with gpt-j which uses 29-fold fewer parameters than gpt-3.5 turbo. what we have learned is: 1. by including a sequence of previous actions, not just the most recent one, the average score triples compared to drrn. 2. performance of the llm model degrades gracefully with much less training data. with only 6.5% of the training data, the score doubles compared to drrn. 3. if one uses all the training data provided, gpt-j learns enough that adding a pre-condition system text does not help (i.e., overall performance stays the same or drops slightly). language models generally approach the problem as a sequence modeling task, performing the equivalent of offline reinforcement learning @xcite . llms can also store large amounts of world knowledge, giving them a potential advantage over pure rl methods in environments that require prior knowledge, like scienceworld. some recent work has shown how extremely large language models (over 100b parameters) can plan in simulated and real-world environments @xcite @xcite . @xcite and inner monologue @xcite use a huge language model (e.g., a 540b parameter palm, chowdhery et al. ( 2022 )) to create a plan for a robot to convert into actions. in contrast to innermonologue, the system in this paper uses a language model that is 90 times smaller (6b parameter gpt-j, wang (2021)), which both plans and turns its plan into low-level actions, though in a simplified, simulated environment. ## discussion and conclusion llms still exhibit non-common-sensical behaviors, such as emitting redundant actions or trying to interact with nonexistent objects. external components such as the precondition system shown in this work can help alleviate some of these behaviors. other external components could assist llms with structured knowledge. such knowledge could be quickly inspected and updated by humans. our team has yet to succeed in significantly assisting the llm using external knowledge sources. while we have managed to teach the llm to interact with a q&a system that could answer is-a questions for the four classification tasks, such knowledge was already present in the llm, as evidenced by the high scores on classification tasks in appendix d. despite wang et al. ( 2022 )'s findings that llms perform poorly in scienceworld, we find that with a careful formulation of the prompt and access to prior interactions with the game, even a single reasonably sized llm achieves a 3.5x improvement over drrn. the 6b-parameter gpt-j models can match the performance of swiftsage t5 + gpt-3.5-turbo, a more complex architecture that uses a model 29 times larger than gpt-j. even when learning from just a few hundred games (6.5% of the training data), gpt-j achieves a 2.2x improvement over drrn. despite the sizeable overall score improvement, the llm performs poorly on some scienceworld tasks. the strong correlation between the performance of our gpt-j-based models suggests that some tasks are genuinely more difficult for the llm.