Improving Reinfocement Learning Agent Training using Text based Guidance: A study using Commands in D ravidian Languages (2023)

Abstract: Reinforcement learning (RL) agents have achieved remarkable success in various domains, such as game-playing and protein structure prediction. However, most RL agents rely on exploration to find optimal solutions without explicit guidance. This paper proposes a methodology for training RL agents using text-based instructions in Dravidian Languages, including Telugu, Tamil, and Malayalam along with using the English language. The agents are trained in a modified Lunar Lander environment, where they must follow specific paths to successfully land the lander. The methodology involves collecting a dataset of human demonstrations and textual instructions, encoding the instructions into numerical representations using text-based embeddings, and training RL agents using state-of-the-art algorithms. The results demonstrate that the trained Soft Actor-Critic (SAC) agent can effectively understand and generalize instructions in different languages, outperforming other RL algorithms such as Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG).

## introduction reinforcement learning (rl) has developed by leaps and bounds in the past few years, there have been agents capable of beating world champions in games like go @xcite , there have been agents capable of predicting protein structures @xcite and more recently there has also been an rl agent capable of optimizing computer code @xcite . however, most of the rl agents optimally find the best solution by themselves through exploration of the environment and there lacks a technique through which these agents can be guided so that we can control the path or trajectory that the agent takes while reaching the optimal solution. there has been little work done to guide or help the rl agents get to the goal state through text-based instructions, especially in the dravidian languages. the current literature @xcite and @xcite provide a basis for this approach where they have constructed a bimodal embedding network to guide the rl agent on text-based instructions. however, existing literature doesn't compare various reinforcement learning algorithms and they also don't consider the possibility of training the agents to understand the text instructions in multiple languages. this paper aims to address this literature gap by proposing a methodology for training reinforcement learning agents in the lunar lander game using text-based embeddings in four languages: english, @xcite . by encoding instructions into meaningful numerical representations @xcite , the agents can effectively understand and respond to natural language instructions, leading to more immersive and intuitive user-agent interactions. the proposed methodology for training rl agents with natural language guidance in the lunar lander game involves: 1) collecting a dataset of human demonstrations and textual instructions in multiple languages, 2) encoding the instructions into numerical representations using text-based embeddings, 3) employing rl algorithms with the embeddings as input to train the rl agent, optimizing for successful landings, 4) evaluating the effectiveness of the methodology for unseen paths. we show that the trained soft actor-critic agent is capable of generalizing well to act according to the instruction given in any language. the organization of the remainder of the paper is as follows: section 2 details the related works. section 3 provides a detailed description of the environment used for training the rl agents. section 4 presents the proposed methodology and the results are discussed in section 5. finally, we conclude in section 6 while providing future directions for research. ## related work existing work that combines reinforcement learning and natural language can be categorized into two tasks. in the first task, rl agents are trained in environments where the environment is rendered using only text descriptions, unlike the standard 2d or 3d environments that we traditionally see @xcite . the second task focuses on helping or guiding reinforcement learning agents through natural language, which the present work focuses on. in @xcite , a methodology was presented to use natural language to train a reinforcement learning agent to beat "mon-tezumas revenge", a game that standard rl agents like a3c fail to solve. to prepare a dataset, games were played manually by humans, and snapshots of the game state were taken. the snapshots along with text instructions were used to train cnn and rnn networks using cosine similarity loss to produce text and image embeddings of the game state. these embeddings were given to the rl agent as observations and a new reward function was designed which incorporated a similarity measure reward based on the text and image embeddings. the agent trained obtained a score of 3500 which outperformed the best model at that time by a score of 1000. the authors in @xcite used a similar methodology to @xcite but they replaced the rnn-based network for text embedding with a pre-trained bert model. this model supports giving instructions using synonyms of original instructions and the model would still be able to understand the instructions. through their experiments, the authors say that the agent is able to get to the goal state 24 times out of 100 test tasks with a bertdistance model and 17 times with a bert-cosine model. in recent times, work has focused on using large language models (llms) as rl agents. in @xcite , the authors have presented voyager which uses llm as an rl agent. the llm harnesses the world language learned to generate consistent action plans or executable policies. the methodology presented in voyager relies on using a black-box llm (gpt-4) and skips any need to train or finetune the model. the methodology is comprised of three components: an automatic curriculum which is based on the goal of "discovering as many diverse things as possible". a skill library to store and retrieve code generated by the llm based on embeddings of the generated function descriptions. an iterative prompting mechanism that generates code for various tasks. finally, there is also a self-evaluation component where the llm acts as a critic to evaluate the generated function. though the voyager agent performs well when compared with other similar agents, it, however, has its own downsides like significant cost incurred through gpt-4 api, hallucinations, and inaccuracies in generating a new skill. most of the drawbacks can be improved by employing a multimodal llm that can benefit from both text and visual data or finetuning an llm with knowledge about various aspects of the environment to reduce inaccuracies and hallucinations. we still require advancements in research in the domain to tackle other problems like the high inference time of llms and computational costs of finetuning. existing literature did not explore the possibility of using multiple languages to guide reinforcement learning agents, and there has not been a comparative study on various reinforcement learning algorithms for natural languageguided learning. the present work aims to tackle these research gaps by presenting a methodology to train reinforcement learning agents through text guidance in various languages including dravidian languages and also perform a comparative study on the employability of various state-of-the-art reinforcement learning algorithms for text guidance. 3 environment the environment used in this study is a modified version of the lunar lander environment which is a part of the box2d environments from the open-source python library "gymnasium". the environment is a typical rocket trajectory optimization problem and the goal of the environment is to land the lander on the landing pad. the environment supports both continuous state configuration and discrete state configuration. the continuous state version of the original environment is considered in this study which contains eight observations: the coordinates of the lander in x & y, its linear velocities in x & y, its angle, its angular velocity, two booleans that represent whether each leg is in contact with the ground or not and the action space consists of two continuous actions: the first coordinate of an action determines the throttle of the main engine, while the second coordinate specifies the throttle of the lateral boosters. the original environment is partitioned into 9 regions as shown in figure 1 to construct the modified environment. the new goal of the environment is to trace the lander along the path that is given (an example path can be: top center, top right, middle right, and bottom center) and finally land on the landing pad. a random path from the preconfigured list of paths is automatically assigned by the environment every time the environment is reset. the path given to the environment can contain locations of the environment in the following languages: english, telugu, tamil, and malayalam. refer to appendix a for a detailed list of the paths that can be generated. a newly shaped reward function is defined through which the lander receives a reward of figure 2 : high-level overview of methodology 0 if it moves closer to the target location along the path, -1 if it moves away or deviates from the path, a negative reward based on the angle tilted if its more than 45 degrees, 100 for each leg in contact with the landing pad, and -20 if it tries to land without going along the path. this will effectively punish the agent if it is not following the trajectory given and will reward the agent if it follows along the trajectory and successfully lands. with the mentioned changes, the modified environment still has 8 observations and 2 actions but there is an added feature to generate to path and a new shaped reward function that we employ. ## methodology a high-level overview of the methodology is presented in figure 2 . the 8 sensor observations from the lander along with image embedding of the current state and text embedding of the current target location will go in as input to the reinforcement learning agent which gives an action to be taken. the text instruction to the rnn can be in the following languages: english, telugu, tamil, and malayalam. the training of these networks can be split into two stages, in the first stage the embedding networks cnn and rnn are trained together @xcite using a cosine similarity loss, and then in the second stage, the rl agent is trained. 4.1 bimodal embedding networks to capture the current state of the environment a cnn-based network is used and to capture the text instruction an rnn-based network is used. both of these networks are trained together based on cosine embedding loss. each of the networks outputs an embedding of length 10. the architecture of the two networks is presented in figure 3. ## results and discussion figure 8 shows a visualization of the input text embeddings plotted using umap @xcite . the plot depicts similar instructions being plotted together in the low dimensional space indicating that our embedding network has learned to build connections among the vocabulary used for training it. instruction cosine similarity to ఎగువ ఎడమ (top left) మధయ్ ఎడమ 0.98 (middle left) ఎగువ కుడి 0.06 (top right) దిగువ ఎడమ 0.05 (bottom left) ఎగువ కేంద ం 0.02 (top center) మధయ్ కుడి -0.08 (middle right) as discussed before, 9 unique paths were used for training and a total of 36 paths are made from the 9 paths by translating them to various dravidian languages. appendix a provides an overview of the paths used in training. for the first experiment, only 9 paths in the telugu language were considered and 3 rl agents were trained. the sac agent obtained an average reward of 192.94, the ppo agent obtained an average reward of -184.53 and the ddpg agent obtained an average reward of -344.46. in order to test the ability of rl agents to generalize to unseen paths, a few experiments are conducted as presented in the second set of experiments consisted of training the rl agents using instructions from all 4 languages: english, telugu, tamil, and malayalam. the total number of instructions considered is 36 (9 from each language). the trained sac agent obtained an average reward of 187.19, the ppo agent obtained -262.8, and the ddpg agent obtained a -419.37 reward. we can observe that the performance of the agent trained on multi-language instructions is lower compared with the agent trained on a single language. though the sac agent received 20m additional training steps for multiple languages, it obtained less average reward than a single language agent. ppo and ddpg showed similar performance to single language agents, failing to converge. for the agents trained on instructions from multiple languages first a test was performed to evaluate the agents on using combinations of languages. presented in the final set of tests as presented in ## conclusion and future works we have successfully demonstrated a methodology to guide the reinforcement learning agents through text instructions in multiple dravidian languages. ppo, ddpg, and sac algorithms were put to use for the task and results showed that sac generalized well even for unseen paths. when unseen instructions from a mix of dravidian languages were given sac ppo ddpg s.no path average average average reward reward reward 1 top center, మధయ్ కేంద ం, கீழ ் ைமயம ் 190.6 -185.6 -453.1 (top center, middle center, bottom center) 2 മുകളിൽ മധ്യം, நடு ைமயம ் , దిగువ కేంద ం 195.1 -263.5 -159.1 (top center, middle center, bottom center) 3 ேமல ் ைமயம ் , മുകളിൽ വലത് middle right, దిగువ కేంద ం 181.5 -423.6 -218.8 (top center, top right, middle right, bottom center) 4 ఎగువ కేంద ం, top left நடு இடது, താെഴ മധ്യം 180.7 -375.4 -476.4 (top center, top left, middle left, bottom center) to the sac agent, it obtained an average reward of 119.8 while an sac agent trained on telugu language instructions alone obtained an average reward of 183.2 for unseen telugu instructions. the correctness of the embeddings was also verified through the umap plot and cosine similarity. the work presented in this paper can be extended by using various architectures for embedding networks and making them more efficient. another direction that can be explored is the use of multimodal multilanguage large language models which are capable of understanding images and text in multiple languages, providing access to good computational infrastructure one can try training these llms to understand dravidian languages and also act as reinforcement learning agents.