Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use (2024)

Abstract: In real-world scenarios, it is desirable for embodied agents to have the ability to leverage human language to gain explicit or implicit knowledge for learning tasks. Despite recent progress, most previous approaches adopt simple low-level instructions as language inputs, which may not reflect natural human communication. We expect human language to be informative (i.e., providing feedback on agents’ past behaviors and offering guidance on achieving their future goals) and diverse (i.e., encompassing a wide range of expressions and style nuances). To enable flexibility of language use in teaching agents tasks, this paper studies different types of language inputs in facilitating reinforcement learning (RL) embodied agents. More specifically, we examine how different levels of language informativeness and diversity impact agent learning and inference. Our empirical results based on four RL benchmarks demonstrate that agents trained with diverse and informative language feedback can achieve enhanced generalization and fast adaptation to new tasks. These findings highlight the pivotal role of language use in teaching embodied agents new tasks in an open world.

## introduction developing embodied agents that can understand and communicate with humans in natural language to learn and accomplish tasks is a long-standing goal in artificial intelligence. in recent years, the integration of human language and reinforcement learning (rl) has seen significant advancements. unlike traditional rl methods that typically rely on numerical reward signals to guide agent learning, recent works @xcite explore using language as a rich and intuitive signal to shape an agent's behaviors. for example, when the agent is making mistakes during the task completion, providing language feedback can largely improve the instantaneous performance thus enhancing the overall agent learning efficiency and effectiveness @xcite . however, existing methods generally employ simple instructions, such as "turn left" and "put the apple to the table" to teach/control an agent @xcite @xcite @xcite . while useful, these instructions may not fully reflect the flexibility of language use in task learning and collaboration @xcite @xcite @xcite . in the real world, humans often express complex language instructions that are more informative. for instance, when a student makes a mistake, a teacher may help them to retrospect on what went wrong (i.e., hindsight instructions) and then guide them on what should be done next to finish the goal (i.e., foresight instructions). in addition, humans are likely to engage in conversations with more diverse language patterns, describing the same goal with different expressions and styles. therefore, we ask the following question: how do the informativeness and diversity of natural language used during rl training affect an agent's ability to learn tasks? we take a popular offline rl model -decision transformer (dt) @xcite -as a backbone architecture and conduct a comprehensive study to examine how informativeness and diversity of language use may impact agents' learning ability. to control informativeness, we leverage expert agents' actions as a reference to generate hindsight reflection and foresight guidance, using hand-crafted language templates. to increase diversity, we construct a gpt-augmented language pool, where @xcite is used to augment hand-crafted templates into much more natural and richer expressions. we further extended dt into a multi-modal language-teachable dt (ltdt) and demonstrated that ltdt agents that are trained with diverse and informative language significantly outperform the counterpart agents that are trained either with simple language alone or with no language inputs. notably, we found that even with just one language template, combining hindsight and foresight feedback together improves agents' performance by an average of 9.86 points ( from 37.95% to 47.81%) on four popular offline rl benchmarks. when more language diversity is incorporated into training, an additional 10.14 points (from 47.81% to 57.95%) are obtained. the contributions of this paper can be summarized as follows: offline reinforcement learning offline reinforcement learning (rl) has become a focal point of research due to its ability to utilize pre-existing datasets for training agents without real-time interactions. several algorithms address the unique challenges of offline rl, such as mitigating extrapolation errors and ensuring robust policy evaluation. a survey by @xcite outlines the field's taxonomy and open problems. benchmarking efforts by @xcite assess various batch deep rl algorithms. key approaches include conservative q-learning (cql) @xcite , implicit q-learning (iql) @xcite , and the decision transformer (dt) @xcite , which treats rl as a sequence modeling problem @xcite . recent work also explores generalization across tasks @xcite @xcite , the use of exploratory data @xcite , and integrating large language models (llms) @xcite . efficient online rl leveraging offline data is also a focus @xcite . our research builds on the decision transformer (dt) by integrating language feedback, creating the language-teachable decision transformer (ltdt). this novel approach incorporates rich, human-like language instructions, improving agent learning and decision-making through enhanced informativeness and diversity of language inputs. ## problem setting in this section, we outline the problem setting by defining the offline reinforcement learning problem (sec. ## data generation to train an agent that can understand language feedback in an offline reinforcement learning manner, algorithm 1 offline data collection 1: initialize d ← ∅ 2: for each episode with seedi do 3: initialize di ← ∅ 4: initialize environment env with seedi. 5: append task description t d to di 6: initialize the non-expert agent with a sub-optimal policy π. 7: initialize the expert agent with policy π * . 8: algorithm 1 outlines the data generation process, and we explain the algorithm in detail in the following sections. ## experiment in this section, we design experiments to answer the following two research questions (rqs): for rq1, we control agents trained with hindsight information, foresight information, or both to investigate the function of informativeness. we compare agents trained with language from handcrafted templates and from the gpt-augmented language pool to examine the function of language diversity. for rq2, agents are taught in languages from the gpt-augmented language pool. we compare different pre-trained agents' performance on an unseen task after fine-tuning with few-shot adaptation samples. ## conclusion in this paper, we investigate how the informativeness and diversity of language feedback affect embodied agents. we introduce the language-teachable decision transformer (ltdt), which makes decisions based on human language feedback. to facilitate the training of ltdt agents, we propose an easy-to-use pipeline for collecting offline hindsight and foresight gpt templates. we compare the performance of agents by varying the informativeness and diversity of the training languages across four reinforcement learning environments and evaluate the agents' ability to understand real-world human language using online gpt as a proxy. our results demonstrate that training with more informative and diverse language feedback significantly enhances agent performance and enables fast adaptation to unseen tasks.