Language Models are Few-Shot Butlers (2021)

Abstract: Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. Hence, these autoregressive models constitute ideal agents to operate in text-based environments where language understanding and generative capabilities are essential. Nonetheless, collecting expert demonstrations in such environments is a time-consuming endeavour. We introduce a two-stage procedure to learn from a small set of demonstrations and further improve by interacting with an environment. We show that language models fine-tuned with only 1.2% of the expert demonstrations and a simple reinforcement learning algorithm achieve a 51% absolute improvement in success rate over existing methods in the ALFWorld environment.

## introduction over the past few years, successive generations of language models @xcite @xcite have reshaped the way we approach natural language processing problems. these transformer-based @xcite networks scale to ever-increasing amounts of parameters, data and compute @xcite while demonstrating impressive transfer-learning capabilities across a wide variety of benchmarks @xcite @xcite . however, the application of modern language models to action generation in text-based environments @xcite @xcite remains largely unexplored @xcite . intuitively, large-scale models leveraging strong linguistic priors should thrive in settings where observations and actions are textual @xcite , but several issues arise. indeed, agents must conform to an environment's generative grammar and gameplay specificities making off the shelf transfer fail in most games. a way to overcome this problem is to acquire expert demonstrations and resort to the widely used paradigm of fine-tuning on task-specific data @xcite @xcite . nevertheless, collecting demonstrations in text-based environments requires far more time and expert knowledge than for most nlp tasks. a single demonstration includes tens of actions taken over a long time horizon to solve multiple subgoals. in this work, we propose a two-stage procedure to address these issues and develop language models acting as agents in text-based environments. first, we train language models to imitate a few dozens of expert demonstrations in order to respect an environment's grammar and acquire basic gamesense. second, we let the models interact with the environment and iteratively treat successful trajectories as additional expert demonstrations for further fine-tuning. we demonstrate the effectiveness of our approach in the recently introduced alfworld environment @xcite 1 , which was designed with an extensive set of tasks and expert demonstrations. in summary, our contributions are the following: ## experiments experiments were implemented with the transformers @xcite and pytorch @xcite libraries and were conducted on an nvidia rtx 3090. 2 ## conclusion we developed new agents for text-based environments with pretrained language models. these agents acquired game knowledge through demonstrations and interactions to drastically outperform current methods in the alfworld environment. while we investigated learning under the standard fine-tuning paradigm, more sophisticated ap-proaches could be explored @xcite and recent works @xcite even suggest that scaled-up and carefully calibrated models achieve great downstream results without requiring any parameter updates. thus, in the near future one can imagine language models solving text-based environments with only a few demonstrations for priming.