Language Model Adaption for Reinforcement Learning with Natural Language Action Space (2024)

Abstract: Reinforcement learning with natural language action space often suffers from the curse of dimensionality due to the combinatorial nature of the natural language. Previous research leverages pretrained language models to capture action semantics and reduce the size of the action space. However, since pretrained models are typically trained on general corpora, there can be an unpredictable mismatch between the priors encoded in pretrained models and the characteristics of the specific RL environment. To address this issue, we propose Mutual-Information Regularized Policy Optimization, MIPO. MIPO enables implicit and dynamic reduction of the action space. Starting from the prior provided by the pretrained language model, our method dynamically adjusts the prior during the learning process based on the guidance of mutual information regularization. Theoretically, we demonstrate that this policy optimization process leads to the monotonic improvement on the mutual-information regularized RL objective. Empirically, we conduct experiments in various environments and demonstrate the effectiveness of MIPO.

## introduction deep reinforcement learning (rl) has gained great success from mastering the game of go @xcite to training state-of-the-art large language models @xcite . despite its success in various domains, the low sample efficiency of rl has consistently hindered its wide adoption in real-world applications where the cost of interaction is not negligible @xcite . furthermore, as suggested in @xcite , the sample complexity of rl algorithms will be inevitably influenced by the size of the action space. this issue becomes even more pronounced in scenarios with enormous action spaces, such as multi-agent rein- † corresponding author. one natural solution to tackle the challenges of large action space is to eliminate irrelevant actions accordingly to avoid the exploration of unnecessary state-action pairs. while a general principle of action space reduction is hard to find, for reinforcement learning problems with natural language action space @xcite , leveraging action semantics provides a means to infer the potential usefulness of actions. previous work has explored the use of pretrained language models for action space reduction @xcite . despite being a viable solution, pretrained models encode prior knowledge derived from general corpora, which may differ from the specifics of the actual environment. to alleviate such a mismatch, some studies @xcite utilize the domain-specific dataset to finetune the pretrained model for better adaptation. others @xcite @xcite assume the feedback of action admissibility from the environment, which assesses the relevance of each action to the current task, and use it to prune the action space. however, neither the domain-specific dataset nor the admissibility signal is always available. this leads to the question of whether it is possible to adjust the prior knowledge without incurring additional costs or making assumptions. to address this challenge, we propose mutual-information regularized policy optimization, namely mipo. unlike previous work where the hard action mask is used to prune the action space, we use the pretrained language model to generate a prior policy over the action space of the agent. the agent is subsequently trained to optimize the environmental reward as well as being close to this prior. then, based on the marginalization of the agent's policy, we dynamically adjust the general prior provided by the pretrained language model to adapt to the specific environment. such a learning process can be unified as a mutual-information (mi) regularized reinforcement learning problem, where the standard rl objective is augmented with the mutual information between the agent's policy and non-text state information. in theory, we prove that mipo achieves the monotonic improvement on the mi-regularized objective. to validate our method in practice, we perform a series of experiments on text-based games. the experimental results show that mipo outperforms baselines in terms of training speed and final performance. our contribution can be summarized as follows. first, we propose mipo, an mi-regularized policy optimization method that achieves implicit dynamic action space reduction without requiring additional datasets or assumptions. then, we prove that our method achieves monotonic improvement in the mi-regularized objective. finally, we empirically validate our method and demonstrate its effectiveness. ## method in this section, we present mutual-information regularized policy optimization (mipo). mipo leverages a pretrained language model conditioned on the state variable s text to generate a prior policy π prior (a| s text ). then in section 4.1, we incorporate the kl divergence between the current policy π(a|s) and the prior policy into the policy iteration process. in this way, we encourage the policy to optimize the environmental reward, as well as being close to the prior policy. inspired by the mutual-information (mi) regularization, we exploit the marginalization of the agent's policy to adapt the prior policy to the environment. as shown in section 4.2, this alternative optimization process yields a monotonic improvement on the mi-regularized rl objective. finally, in section 4.3, we show how to implement this optimization process in practice. ## conclusion in this paper, we propose mipo, which achieves dynamic language model adaption and implicit action space reduction for reinforcement learning with natural language action space. we show the problem can be formulated as the optimization of the mi-regularized rl objective, and we theoretically prove that mipo achieves monotonic improvement on the mi-regularized rl objective. we perform experiments on text-based games, and our mipo outperforms all baselines, which justifies its effec-tiveness.