Automatic Exploration of Textual Environments with Language-Conditioned Autotelic Agents (2022)

Abstract: The purpose of this extended abstract is to discuss the possible fruitful interactions between intrinsically-motivated language-conditioned agents and textual environments. We define autotelic agents as agents able to set their own goals. We identify desirable properties of textual nenvironments that makes them a good testbed for autotelic agents. We them list drivers of exploration for such agents that would allow them to achieve large repertoires of skills in these environments, enabling such agents to be repurposed for solving the benchmarks implemented in textual environments. We then discuss challenges and further perspectives brought about by this interaction.

## a bridge between autotelic agents and text environments the main point of this essay is the relevance of studying language autotelic agents in textual environments @xcite @xcite , both for testing exploration methods in a context that is at once simple experimentally and rich from the perspective of environment interactions; and for transferring the skills of general-purpose agents trained to explore in an autonomous way to the predefined tasks of textual environment benchmarks. we identify three key properties, plus one additional benefit, of text worlds: ## drivers of exploration in autotelic agents we identify three main drivers of exploration in autotelic agents. environments we use should support exploration algorithms that implement these principles; the resulting agents then have a chance to acquire a diverse set of skills that can be repurposed for solving the benchmarks proposed by textual environments. ## challenges for autotelic textual agents text worlds bring a set of unique challenges for autotelic agents, among which we foresee: agents trained in such environments will learn a form of language use, not by predicting the most likely sequence of words from a large-scale dataset @xcite but by learning to use it pragmatically to effect changes in the environment. of course, the limits of the autotelic agent's world will mean the limits of its language; an interesting development is to build agents that explore textual environments to refine external linguistic knowledge provided by a pretrained language model. this external knowledge repository can be seen as culturallyaccumulated common sense, a perspective that is related to so-called vygotskian ai @xcite in which a developmental agent learns by interacting with an external social partner that imparts outside language knowledge and organizes the world so as to facilitate the autotelic agent's exploration. to conclude, textual environments are ideal testbeds for autotelic language-conditioned agents, and conversely such agents can bring progress on text world benchmarks. there is also promise in the interaction between exploratory agents and large language models encoding exterior linguis-tic knowledge. preliminary steps have been taken in this direction @xcite but the full breadth of drivers of exploration we identify has yet to be studied. we hope to foster discussion, define concrete implementations and identify challenges by bringing together the developmental perspective on ai and the textual environment community.