Natural Language-based State Representation in Deep Reinforcement Learning (2024)

Abstract: This paper investigates the potential of using natural language descriptions as an alternative to direct image-based observations for learning policies in reinforcement learning. Due to the inherent challenges in managing image-based observations, which include abundant information and irrelevant features, we propose a method that compresses images into a natural language form for state representation. This approach allows better interpretability and leverages the processing capabilities of large-language models. We conducted several experiments involving tasks that required image-based observation. The results demonstrated that policies trained using natural language descriptions of images yield better generalization than those trained directly from images, emphasizing the potential of this approach in practical settings.

## introduction directly learning policies from images holds great promise for practical reinforcement learning applications. however, managing image-based observations is challenging due to their potential abundance of information and irrelevant features. furthermore, the learned policy can often be a black box, as the action corresponding to an image observation is difficult to comprehend. this scenario makes interpretability challenging, and the policies often fail to generalize to the slightest changes to the environments. these can hinder the ability to leverage these policies in real-world tasks. on the other hand, language has been humans' primary mode of communication. a situation can be precisely described through language, and conversely, a situation can be constructed from a language description. for instance, movies are often produced based on narratives found in books (e.g., game of thrones, lord of the rings). ultimately, language is a primary source through which humans reason and understand others' reasoning. in reinforcement learning, the ability to learn a policy that generalizes well is essential for realworld system deployment. specifically, agents should be adept at operating in scenarios distinct from their training environments. several strategies have been proposed to address these challenges. these encompass data augmentation methods like random cropping and the addition of jitter to imagebased observations @xcite @xcite @xcite , the injection of random noise @xcite , network randomization @xcite @xcite , and regularization techniques @xcite @xcite . these methods have consistently demonstrated their potential in boosting generalization. the core principle underlying these techniques is the amplification of training data diversity, which aids in crafting a more universally applicable policy. however, such perturbations are often introduced without due regard for task semantics. this oversight can modify critical observation elements, potentially diminishing the efficacy of policy learning. furthermore, random perturbations through various observation manipulations-such as cropping, blocking, or combining two random images from different environment levels-may yield unrealistic observations that the agent is unlikely to encounter during testing. therefore, these techniques might underperform in settings where agents rely on realistic observations for policy learning. nevertheless, these methods modify the image space, and the policy learning happens from pixel images. thus, the learned policy can still be non-interpretable, and it is unclear how the policy behaves when a particular assumption, such as color information, is not held for a particular task. the autoencoder-based approach takes the image our method is to compress image pixels into natural language descriptions, serving as the state information of reinforcement learning. this language-based approach is advantageous as it is easy for humans to understand and provides a clearer insight into how the computer perceives visuals. our findings demonstrate that policies trained using natural language descriptions of images showcase enhanced generalization capabilities towards unobserved scenarios, surpassing the performance of policies directly trained from raw images. and represents it in a lower-dimensional space (e.g., ae, vae) @xcite @xcite , which is then used as state information for reinforcement learning. however, such an approach can still suffer from the black-box policy issue, and the intermediate representation might lose information about the original image observation. these methods modify the image space or the lower-dimensional projection where policy learning occurs. consequently, the learned policy can remain non-interpretable, and it is unclear how the policy will behave when certain assumptions, such as color information, do not apply to a specific task. with the existing approaches, the resulting learned policy can be challenging to interpret and may fail when minor environmental changes occur. recent natural language processing and computer vision advancements have enabled a more detailed, accurate understanding of image content. these advances are typically driven by large-scale models, often referred to as foundational models, which contain billions of parameters and are trained on internet-scale datasets with substantial computational resources. in this paper, we primarily focus on decisionmaking derived from language descriptions of visuals (e.g., images). we first compress the visual information (i.e., pixels) into natural language and use this language as state information to learn policy with reinforcement learning (figure 1 ). this approach has several advantages. for instance, the language representation is inherently interpretable, providing a more accurate indication of what the agent understands from the visual scene. in this setup, the agent can learn from a natural language description of the image. this approach provides multiple benefits. primarily, the representation is easily interpretable by humans, unlike raw pixel data from the image. moreover, it paves the way for harnessing the immense processing power of large language models (llms) to handle natural language state information. for instance, unnecessary features, such as color information, can be filtered out by directing the llm to ignore them (e.g., prompt: rewrite the description excluding color information) in particular, we utilize the vision-language model (vlm) (i.e., llava @xcite ) to generate a natural language description of the im-age observation. the resulting language is then passed to a large language model (llm) (i.e., llama @xcite ) for further preprocessing. finally, it is converted into a text embedding vector using pre-trained embedding models (i.e., sentence transformer @xcite ). we conducted experiments to evaluate the effectiveness of a vision and language model (vlm) in learning from text in reinforcement learning contexts. these experiments encompassed tasks that required image-based observation. specifically, we conducted experiments on openai gym @xcite (gymnasium @xcite ) environment, frozenlake. the rendered image was used as the observation, with the task being to learn a policy from this image observation. we compared our text-based learning approach with learning directly from the raw pixel information. our results indicated that policies trained using natural language descriptions of images exhibited superior generalization compared to those trained directly from images. moreover, our language-based state representation is inherently interpretable compared to directly learning from pixels, indicating a strong use case for languagebased state representation. in particular, in the frozen lake environment, training results show that all baselines learn the task efficiently, achieving optimal performance. when tested in a new ice environment variation, the ppo-lang method maintains its performance, highlighting the strength of language-based learning. in contrast, policies based on image observations tend to overfit and fail to generalize in new environments, proving ineffective for the intended task. ## preliminaries and problem settings markov decision process (mdp). an mdp can be described by the tuple @xmath0, a, p, r). within this framework, an agent at a discrete timestep t interacts with its environment from a current state s @xmath1, selecting an action a @xmath2. subsequently, the environment transitions to a new state s t+1 ∈ s, governed by the transition probabilities p(s t+1 |s t , a t ). the agent then receives a reward r t , determined by the reward function r. reinforcement learning. within the context of reinforcement learning, the agent operates within an mdp and aims to discover a policy π ∈ π that leads to the maximization of the cumulative reward. here, π represents the space of all feasible policies. based on the current state, the agent selects an action in line with policy π, and the optimal policy π * ∈ π is the one that yields the greatest total rewards over time. deep reinforcement learning incorporates deep learning to handle more complex, highdimensional input spaces. by utilizing deep neural networks, it can represent policy or value functions with greater flexibility and sophistication. drl is suitable for applications that require processing raw pixel data or controlling intricate systems and has become instrumental in advancing various fields, from gaming to autonomous robotics. the integration of deep learning enables more precise function approximation, allowing agents to learn optimal policies in more challenging environments. ## methodology this section provides a breakdown of our approach to producing natural language descriptions from visual data, specifically images. a pivotal component in this appraoch is the vision-language model (vlm). notably, we employ the vlm variant known as llava @xcite . the llava vlm initially processes the input image when employed in our method. during this phase, the model identifies and understands various features and objects contained within the visual data. upon extracting and understanding these elements, the vlm subsequently crafts a well-structured, coherent, and descriptive narration in natural language form. the assumption is that the resultant description pinpoints the image's most prominent and defining characteristics, ensuring that agent receives an accurate and detailed understanding of the visual content. after extracting the natural language description from the vlm, our method allows pre-processing text with a large language model (llm) (e.g., llama @xcite , chatgpt). when the language description, as generated by the vlm, is inputted into the llm model, the latter performs complicated processing tasks. these tasks aim to enhance the language output by refining its structure and improving its coherence and information content. furthermore, in scenarios where generalization is the goal, this pre-processing step undertaken by the llm is valuable. the model inspects the description to identify and eliminate superfluous or irrelevant details. such action is required, especially when we consider the need for an agent to develop a consistent and invariant representation of an image observation within a given environment. by removing unnecessary details, we ensure that the agent focuses only on the most vital aspects of the environment, thereby optimizing its learning process. after processing the natural language description, the subsequent step in our pipeline focuses on text representation through embedding. for this purpose, we utilize pre-trained models, specifically the sentence transformer architecture, as outlined by @xcite . the sentence transformer is designed to convert textual data into dense vectors of fixed dimensions, known as text embeddings. the primary objective of these embeddings is to encapsulate the semantic information and context inherent in the original text. by converting the refined natural language description into this vector format, we aim for an efficient representation for computational processing and to maintain the semantic properties of the input data. by integrating a sequence of models-the vision-language model (vlm) for initial image description generation, the large language model (llm) for subsequent description refinement, and the sentence transformer for text embedding transformation, we have developed a methodology that efficiently extracts and represents pertinent information from images in a structured, semantic format. this systematic approach facilitates a cohesive fusion of visual and textual data. ## related work the integration of language with reinforcement learning has been a subject of growing interest in the research community. language, being one of the most remarkable human achievements, plays a pivotal role in our ability to learn, teach, reason, and interact with others. however, the current state-of-the-art reinforcement learning agents have shown limitations in understanding or utilizing human language. the potential benefits of integrating language with reinforcement learning are manifold. agents that can harness language in conjunction with rewards and demonstrations have the potential to enhance their generalization capabilities, scope, and sample efficiency. the linking language to actions and observations has been explored with methods aiming to effectively associate language with actions and observations in various environments @xcite @xcite . from a generalization perspective, various strategies have been proposed; these include data augmentation techniques such as random cropping and noise addition, as well as network randomization to augment training diversity @xcite @xcite @xcite @xcite @xcite @xcite . however, the effectiveness of these methods can diminish if the semantics of the task are overlooked. moreover, these methods often lack interpretability and can fail when certain assumptions do not hold. the autoencoder-based method, which reduces images into a lower-dimensional space @xcite @xcite , can also face challenges like unclear policy behavior due to its black-box nature. these methods modify the image space or the lowerdimensional projection where policy learning occurs, which can result in non-interpretable policies. in contrast, our work emphasizes decisionmaking through language descriptions of visual content. this language-centered approach addresses the challenges present in image-based policy learning. we aim to improve transparency and generalization in reinforcement learning tasks by focusing on language. while the integration of language with reinforcement learning has been a topic of interest, the majority of existing research has primarily focused on direct associations between language and actions or observations. these methods often rely on data augmentation techniques, network randomization, or image manipulations to enhance generalization. however, these strategies can sometimes lead to unrealistic outcomes during testing or lack interpretability, especially when certain assumptions do not hold true. for instance, autoencoder-based methods, which condense images into a lowerdimensional space, might grapple with ambiguous policy behavior due to their opaque nature. such methods can also risk losing essential image information, leading to policies that are hard to interpret. the method in paper @xcite requires a semantic representation and a semantic parser, whereas our method utilizes textual descriptions generated by a vision-language model. the proposed method in the paper @xcite begins by describing a target task in natural language. then, a pre-trained language model (lm) translates this task description into a state abstraction that filters out irrelevant features, necessitating user demonstration. in contrast, our method depends on a vision-language model to generate descriptions and a large language model (llm) to filter out irrelevant features, identified through prompts (e.g., 'remove color information'). these prompts can originate from either the environment or the user. overall, our work takes a fundamentally different approach. we emphasize decision-making through language descriptions of visual content. instead of relying heavily on visual cues, which can be susceptible to overfitting or misinterpretation, our method harnesses the power of language to provide a more robust and transparent representation. this language-centered approach addresses the challenges inherent in image-based policy learning and offers a more interpretable and generalizable solution. we aim to create models better equipped to handle diverse scenarios by grounding reinforcement learning in linguistic descriptions. ## conclusion in reinforcement learning, while directly learning policies from images offers potential, it also presents challenges due to the abundance of information and irrelevant features in image-based observations. such policies often lack interpretability and struggle to generalize across varying environments. language, a primary mode of human communication, offers a precise way to describe and construct situations, serving as a foundation for human reasoning. we introduce an approach that leverages language descriptions of visuals for decision-making. the resulting policy is more interpretable by converting visual information into natural language and using this language as state information. it offers a clearer insight into the agent's understanding of the visual scene. utiliz-ing vision-language models and large language models, we present a method to generate natural language descriptions of image observations, preprocess them, and convert them into text embedding vectors. experiments conducted on the ope-nai gym frozen lake environment demonstrate the superiority of policies trained using natural language descriptions over those trained directly from images. such language-based state representations offer enhanced interpretability and generalization, underscoring the potential of language as a powerful tool in reinforcement learning.