A Systematic Survey of Text Worlds as Embodied Natural Language Environments (2022)

Abstract: Text Worlds are virtual environments for embodied agents that, unlike 2D or 3D environments, are rendered exclusively using textual descriptions. These environments offer an alternative to higher-fidelity 3D environments due to their low barrier to entry, providing the ability to study semantics, compositional inference, and other high-level tasks with rich action spaces while controlling for perceptual input. This systematic survey outlines recent developments in tooling, environments, and agent modeling for Text Worlds, while examining recent trends in knowledge graphs, common sense reasoning, transfer learning of Text World performance to higher-fidelity environments, as well as near-term development targets that, once achieved, make Text Worlds an attractive general research paradigm for natural language processing.

## introduction embodied agents offer an experimental paradigm to study the development and use of semantic representations for a variety of real-world tasks, from household tasks @xcite to navigation @xcite to chemical synthesis @xcite . while robotic agents are a primary vehicle for studying embodiment (e.g. @xcite , robotic models are costly to construct, and experiments can be slow or difficult to scale. virtual agents and embodied virtual environments help mitigate many of these issues, allowing large-scale simulations to be run in parallel orders of magnitude faster than real world environments (e.g. @xcite , while controlled virtual environments can be constructed for exploring specific tasks -though this benefit in speed comes at the cost of having to model virtual 3d environments, which can be substantial. text worlds -embodied environments rendered linguistically through textual descriptions instead of graphically through pixels (see ## why use text worlds? for many tasks, text worlds can offer advantages over other embodied environment modelling paradigms -typically in reduced development costs, the ability to model large action spaces, and the ability to study embodied reasoning at a higher level than raw perceptual information. embodied reasoning: embodied agents have been proposed as a solution to the symbol grounding problem @xcite , or the problem of how concepts acquire real-world meaning. humans likely resolve symbol grounding at least partially by assigning semantics to concepts through perceptually-grounded mental simulations @xcite . using embodied agents that take in perceptual data and perform actions in real or virtual environments offers an avenue for studying semantics and symbol grounding empirically @xcite @xcite ). text worlds abstract some of the challenges in perceptual modeling, allowing agents to focus on higher-level semantics, while hybrid worlds that simultaneously render both text and 3d views (e.g. @xcite help control what kind of knowledge is acquired, and better operationalize the study of symbol grounding. ease of development: constructing embodied virtual environments typically has steep development costs, but text worlds are typically easier to construct for many tasks. creating new objects does not require the expensive process of creating new 3d models, or performing visualpercept-to-object-name segmentation or classification (since the scene is rendered linguistically). similarly, a rich action semantics is possible, and comparatively easy to implement -while 3d environments typically have one or a small number of action commands (e.g. @xcite ), text worlds typically implement dozens of action verbs, and thousands of valid verb-nounphrase action combinations @xcite . compositional reasoning: complex reasoning tasks typically require multi-step (or compositional) reasoning that integrates several pieces of knowledge in an action procedure that arrives at a solution. in the context of natural language, compositional reasoning is frequently studied through question answering tasks (e.g. @xcite @xcite or procedural knowledge prediction (e.g. @xcite @xcite . a contemporary challenge is that the number of valid compositional procedures is typically large compared to those that can be tractably annotated as gold, and as such automatically evaluating model performance becomes challenging @xcite . in an embodied environment, an agent's actions have (generally) deterministic consequences for a given environment state, as actions are grounded in an underlying action language (e.g. @xcite or linear logic (e.g. @xcite . embodied environments can offer a more formal semantics to study these reasoning tasks, where correctness of novel procedures could be evaluated directly. ## text world simulators text world simulators render an agent's world view directly into textual descriptions of their environment, rather than into 2d or 3d graphical renderings. similarly, actions the agent wishes to take are provided to the simulator as text (e.g. "read the letter" in zork), requiring agent models to both parse input text from the environment, and generate output text to to interact with that environment. in terms of simulators, the z-machine @xcite is a low-level virtual machine originally designed by infocom for creating portable interactive fiction novels (such as zork). it was paired with a high-level lisp-like domain-specific language (zil) that included libraries for text parsing, and other tools for writing interactive fiction novels. the z-machine standard was reverse-engineered by others (e.g. @xcite in an effort to build their own high-level interactive fiction domain-specific languages, and has since become a standard compilation target due to the proliferation of existing tooling and legacy environments. 1 inform7 @xcite is a popular high-level language designed for interactive fiction novels that allows environment rules to be directly specified in a simplified natural language, substantially lowering the barrier to entry for creating text worlds. the text generation engine allows substantial variation in the way the environments are described, from dry formulaic text to more natural, varied, conversational descriptions. inform7 is compiled to inform6, an earlier object-oriented scripting language with c-like syntax, which itself is compiled to z-machine code. ceptre @xcite is a linear-logic simulation engine developed with the goal of specifying more generic tooling for operational logics than inform 7. textworld @xcite ) adapt ceptre's linear logic state transitions for environment descriptions, and add tooling for generative environments, visualization, and rl agent coupling, all of which is compiled into inform7 source code. parallel to this, the jericho environment @xcite allows inferring relevant vocabulary and template-based object interactions for z-machine-based interactive fiction games, easing action selection for agents. 1 a variety of text adventure tooling, including the adventure game toolkit (agt) and text adventure development system (tads), was developed starting in the late 1980s, but these simulators have generally not been adopted by the nlp ## text world environments environments are worlds implemented in simulators, that agents explore to perform tasks. environments can be simple or complex, evaluate task-specific or domain-general competencies, be static or generative, and have small or large action spaces compared to higher-fidelity simulators (see the appendix for a comparison of action space sizes across environments and simulators). ## contemporary focus areas world generation: generating detailed environments with complex tasks is labourious, while randomly generating environments currently provides limited task complexity and environment cohesiveness. world generation aims to support the generation of complex, coherent environments, either through better tooling for human authors (e.g. @xcite , or automated generation systems that may or may not have a humanin-the-loop. @xcite explore creating cohesive game worlds in the light environment using a variety of embedding models including starspace @xcite and bert @xcite . automatic evaluations show performance of between 36-47% in world building, defined as cohesively populating an environment with locations, objects, and characters. similarly, hu-man evaluation shows that users prefer starspacegenerated environments over those generated by a random baseline. in a more restricted domain, @xcite show that two models, one markov chain model, the other a generative language model (gpt-2), are capable of generating quests in a cooking environment, while there is a tradeoff between human ratings of quest creativity and coherence. @xcite propose a largescale end-to-end solution to world generation that automatically constructs interactive fiction environments based on a story (such as sherlock holmes) provided as input. their system first builds a knowledge graph of the story by framing kg construction as a question answering task, using their model (askbert) to populate this graph. the system then uses either a rule-based baseline or a generative model (gpt-2) to generate textual descriptions of the world from this knowledge graph. user studies show that humans generally prefer these neural-generated worlds to the rule-generated worlds (measured in terms of interest, coherence, and genre-resemblance), but that neural-generated performance still substantially lags behind that of human-generated worlds. hybrid 3d-text environments: hybrid simulators that can simultaneously render worlds both graphically (2d or 3d) as well as textually offer a mechanism to quickly learn high-level tasks without having to first solve grounding or perceptual learning challenges. the alfworld simulator @xcite combines the alfred 3d home environment @xcite with a simultaneous textworld interface to that same environment, and introduce the butler agent, which shows increased task generalization on the 3d environment when first trained on the text world. prior to alfworld, @xcite showed that a language model (gpt-2) was able to successfully generate detailed step-by-step textual descriptions of alfred task trajectories for up to 58% of unseen cases using task descriptions alone, without visual input. building on this, micheli (2021) confirmed gpt-2 also performs well on the text world rendering of alfworld, and is able to successfully complete goals in 95% of unseen cases. taken together, these results show the promise of quickly learning complex tasks at a high-level in a text-only environment, then transferring this performance to agents grounded in more complex environments. ## contemporary limitations and challenges environment complexity is limited, and it's currently difficult to author complex worlds. two competing needs are currently at odds: the desire for complex environments to learn complex skills, and the desire for environment variation to encourage robustness in models. current tooling emphasizes creating varied procedural environments, but those environments have limited complexity, and require agents to complete straightforward tasks. economically creating complex, interactive environments that simulate a significant fraction of real world interactions is still well beyond current simulators or libraries -but required for higher-fidelity interactive worlds that have multiple meaningful paths toward achieving task goals. generating these environments semi-automatically (e.g. @xcite ) may offer a partial solution. independent of tooling, libraries and other middleware offer near-term solutions to more complex environment modeling, much in the same way 3d game engines are regularly coupled with physics engine middleware to dramatically reduce the time required to implement forces, collisions, lighting, and other physics-based modeling. currently, few analogs exist for text worlds. the addition of a chemistry engine that knows ice warmed above the freezing point will change to liquid water, or a generator engine that knows the sun is a source of sunlight during sunny days, or an observation engine that knows tools (like microscopes or thermometers) can change the observation model of a pomdp -may offer tractability in the form of modularization. efforts using large-scale crowdsourcing to construct knowledge bases of commonsense knowledge (e.g., atomic, @xcite may be required to support these efforts. current planning languages offer a partial solution for environment modelling. while simulators partially implement facilities for world modeling, some (e.g. @xcite suggest using mature planning languages like strips @xcite or pddl @xcite for more full-featured modeling. this would not be without significant development effort -existing implementations of planning languages typically assume full-world observability (in conflict with pomdp modelling), and primarily agent-directed state-space changes, making complex world modeling with partial ob-servability, and complex environment processes (such as plants that require water and light to survive, or a sun that rises and sets causing different items to be observable in day versus night) outside the space of being easily implemented with off-the-shelf solutions. in the near-term, it is likely that a domain-specific language specific to complex text world modeling would be required to address these needs while simultaneously reducing the time investment and barrier-to-entry for end users. analyses of environment complexity can inform agent design and evaluation. text world articles frequently emphasize agent modeling contributions over environment, methodological, or analysis contributions -but these contributions are critical, especially in the early stages of this subfield. agent performance in easy environments has increased incrementally, while medium-to-hard environments have seen comparatively modest improvements. agent performance is typically reported as a distribution over a large number of environments, and the methodological groundwork required to understand when different models exceed others in time or performance over these environment distributions is critical to making forward progress. transfer learning in the form of training on one set of environments and testing on others has become a standard feature of benchmarks (e.g. @xcite , but focused contributions that work to precisely characterize the limits of what can be learned from (for example) omniquest and transferred to zork, and what capacities must be learned elsewhere, will help inform research programs in agent modeling and environment design. transfer learning between text world and 3d environments. tasks learned at a high-level in text worlds help speed learning when those same models are transferred to more complex 3d environments @xcite . this framing of transfer learning may resemble how humans can converse about plans for future actions in locations remote from those eventual actions (as when we apply knowledge learned in classrooms to the real world). as such, text-plus-3d environment rendering shows promise as a manner of controlling for different sources of complexity in multi-modal task learning (from high-level task-specific knowledge to low-level perceptual knowledge), and appears a promising research methodology for imparting complex task knowledge on agents that are able to navigate high-fidelity virtual environments.