Original: Prompts and Pre-Trained Language Models for Offline Reinforcement Learning
Revised: Prompts and Pre-Trained Language Models for Offline Reinforcement Learning

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: In this preliminary study, we introduce a simpleway to leverage pre-trained language modelsin deep offline RL settings that are not naturally suited for textual representation.
Revised: In this preliminary study, we introduce a simple way to leverage pre-trained language models in deep offline RL settings that are not naturally suited for textual representation.

Original: We propose using a state transformation into a humanreadable text and a minimal fine-tuning of thepre-trained language model when training withdeep offline RL algorithms.
Revised: We propose using a state transformation into a humanreadable text and a minimal fine-tuning of the pre-trained language model when training with deep offline RL algorithms.

Original: This approachshows consistent performance gains on the NeoRL MuJoCo datasets.
Revised: This approach shows consistent performance gains on the NeoRL MuJoCo datasets.

Original: Our experiments suggest that LM fine-tuning is crucial for goodperformance on robotics tasks.
Revised: Our experiments suggest that LM fine-tuning is crucial for good performance on robotics tasks.

Original: However, wealso show that it is not necessary when workingwith finance environments in order to retain significant improvement in the final performance.
Revised: However, we also show that it is not necessary when working with finance environments in order to retain significant improvement in the final performance.

Original:  Introduction
Revised:  Introduction

Original: The interest in applying natural language to reinforcement learning is rapidly growing.
Revised: The interest in applying natural language to reinforcement learning is rapidly growing.

Original: One of thissphere’s underexplored directions is the incorporation of pre-trained language models into deep RLalgorithms where the environments are not naturally suited for textual representation.
Revised: One of this sphere’s underexplored directions is the incorporation of pre-trained language models into deep RL algorithms where the environments are not naturally suited for textual representation.

Original: Recently, Reid et al.
Revised: Recently, Reid et al.

Original: (2022) took a step in thisdirection by taking an earlier proposed Decision
Revised: (2022) took a step in this direction by taking an earlier proposed Decision

Original: Transformer (Chen et al., 2021) and matching itwith a pre-trained language model, demonstrating remarkable results on several deep offline RLbenchmarks.
Revised: Transformer (Chen et al., 2021) and matching it with a pre-trained language model, demonstrating remarkable results on several deep offline RL benchmarks.

Original: However, this method is designedspecifically for the Decision Transformer, and thereis a plethora of other algorithms available.
Revised: However, this method is designed specifically for the Decision Transformer and there is a plethora of other algorithms available.

Original: In this work, we demonstrate that there is a simpler way to combine pre-trained language models(P-LMs) with deep offline RL algorithms for environments with vector-like observations.
Revised: In this work we demonstrate that there is a simpler way to combine pre-trained language models (P-LMs) with deep offline RL algorithms for environments with vector-like observations.

Original: We propose using prompt engineering (Liu et al., 2021a)by presenting the vector state as a string, embedding it via P-LM, and then concatenating the resulting encoding with the original observation.
Revised: We propose using prompt engineering (Liu et al., 2021a) by presenting the vector state as a string, embedding it via P-LM, and then concatenating the resulting encoding with the original observation.

Original: Thisresults in state-of-the-art performance on a subsetof the NeoRL benchmark (Qin et al., 2021) acrossmodel-free algorithms.
Revised: This results in state-of-the-art performance on a subset of the NeoRL benchmark (Qin et al., 2021) across model-free algorithms.

Original:  Background 
Revised:  Background 2. Offline RL

Original: 2. Offline RL

Original: Reinforcement Learning (RL) is commonly defined by Markov Decision Process (MDP) with a tuple ( S, A, P, r, γ ) , where S is a state space, A is an action space, P ( s ′ | s, a ) is a transition function, r ( s, a ) is a reward function, and γ is a discount factor.
Revised: Reinforcement Learning (RL) is commonly defined by Markov Decision Process (MDP) with a tuple ( S, A, P, r, γ ) , where S is a state space, A is an action space, P ( s ′ | s, a ) is a transition function, r ( s, a ) is a reward function, and γ is a discount factor.

Original: The goal of the agent is to obtain a policy π ( s, a ) that maximizes E π (cid:2)(cid:80)
Revised: The goal of the agent is to obtain a policy π ( s, a ) that maximizes E π (cid:2)(cid:80)

Original: ∞ t =0 γ t r t +1 (cid:3) by interacting with the MDP.
Revised: ∞ t =0 γ t r t +1 (cid:3) by interacting with the MDP.

Original: Offline RL is different from RL, as its agent cannot interact with the environment and the policy must be obtained by learning from the dataset D collected with some behavior policy π β or a set of such policies.
Revised: Offline RL is different from RL, as its agent cannot interact with the environment and the policy must be obtained by learning from the dataset D collected with some behavior policy π β or a set of such policies.

Original: The main challenge of offline RL is that D usually does not contain all possible transactions while the agent does not have any opportunity to explore.
Revised: The main challenge of offline RL is that D usually does not contain all possible transactions while the agent does not have any opportunity to explore.

Original: This is one of the reasons why this problem is considered challenging, as the agent cannot improve on incorrect estimations of states or actions if they are not presented in D (Levine et al., 2020).
Revised: This is one of the reasons why this problem is considered challenging, as the agent cannot improve on incorrect estimations of states or actions if they are not presented in D (Levine et al., 2020).

Original: 2. Language Models in RL
Revised: 2. Language Models in RL

Original: There is a considerable amount of recent work that employs pre-trained LMs for solving RL tasks.
Revised: There is a considerable amount of recent work that employs pre-trained LMs for solving RL tasks.

Original: For example, Majumdar et al.
Revised: For example, Majumdar et al.

Original: (2020) use pre-trained
Revised: (2020) use pre-trained

Original: ViLBERT (Lu et al., 2019) to solve vision-andlanguage navigation tasks, and Yao et al.
Revised: ViLBERT (Lu et al., 2019) to solve vision-andlanguage navigation tasks, and Yao et al.

Original: (2020) show that fine-tuning GPT-

Revised: (2020) show that fine-tuning GPT-

Original: (Radford et al., 2019) for action generation boosts performance in textbased games.
Revised: (Radford et al., 2019) for action generation boosts performance in textbased games.

Original: In another work, Singh et al.
Revised: In another work, Singh et al.

Original: (2021) fine-tune DistilBERT (Sanh et al., 2019) to encode states and actions in text-based games, and this approach obtains higher scores comparing to those that do not use pre-trained LMs.
Revised: (2021) fine-tune DistilBERT (Sanh et al., 2019) to encode states and actions in text-based games, and this approach obtains higher scores comparing to those that do not use pre-trained LMs.

Original: These studies show that knowledge stored in pre-trained LMs might be useful for solving RL tasks with natural  language.
Revised: These studies show that knowledge stored in pre-trained LMs might be useful for solving RL tasks with natural [Figure] language.

Original: For a more comprehensive overview, see
Revised: For a more comprehensive overview, see

Original: Luketina et al.
Revised: Luketina et al.

Original:  Madureira and Schlangen [Figure]
Revised: ; Madureira and Schlangen (2020).

Original: While the aforementioned works focus on problems involving natural language, there is a recentinterest in incorporating pre-trained LMs for problems not defined via natural language by default.
Revised: While the aforementioned works focus on problems involving natural language, there is a recent interest in incorporating pre-trained LMs for problems not defined via natural language by default.

Original: In particular, Li et al.
Revised: In particular, Li et al.

Original: (2022) demonstrates that utilizing pre-trained LM for encoding both the goal,history, and parts of the observation for imitationlearning leads to improved systematic generalization.
Revised: (2022) demonstrates that utilizing pre-trained LM for encoding both the goal, history, and parts of the observation for imitation learning leads to improved systematic generalization.

Original: To the best of our knowledge, Reid et al.
Revised: To the best of our knowledge, Reid et al.

Original: (2022)is the only existing work in offline reinforcementlearning that adapts a pre-trained LM model fortext-free RL tasks.
Revised: (2022) is the only existing work in offline reinforcement learning that adapts a pre-trained LM model for text-free RL tasks.

Original: The authors propose using a pretrained LM to improve the convergence speed andthe final performance of the Decision Transformer(Chen et al., 2021) by making the reward, action,and state encodings closer to the language modelembeddings.
Revised: The authors propose using a pretrained LM to improve the convergence speed and the final performance of the Decision Transformer (Chen et al., 2021) by making the reward, action, and state encodings closer to the language model embeddings.

Original:  Proposed Approach
Revised:  Proposed Approach

Original: Inspired by the recent success in informing deep
Revised: Inspired by the recent success in informing deep

Original: RL algorithms with pre-trained language models,we propose an alternative approach to leveragethem in environments that are not obviously suitedfor the textual representation, similar to Reid et al.(2022);
Revised: RL algorithms with pre-trained language models, we propose an alternative approach to leverage them in environments that are not obviously suited for the textual representation, similar to Reid et al.

Original:  Li et al.
Revised: ; Li et al.

Original: Here, we suggest a methodbased on prompt-engineering (Liu et al., 2021a)depicted in Figure 1: (1) re-write an observationinto a human-readable textual representation (2)encode it using a pre-trained language model, and(3) concatenate the resulting embedding with theoriginal observation.
Revised: Here, we suggest a method based on prompt-engineering (Liu et al., 2021a) depicted in Figure 1: (1) re-write an observation into a human-readable textual representation (2) encode it using a pre-trained language model, and (3) concatenate the resulting embedding with the original observation.

Original: The first step can be an arduous task, requiringsemantic parsing and bringing more structure tothe problem.
Revised: The first step can be an arduous task, requiring semantic parsing and bringing more structure to the problem.

Original: To alleviate this, we propose usinga simple transformation that only requires givinga human-readable description of the dimensionsof the state vector.
Revised: To alleviate this, we propose using a simple transformation that only requires giving a human-readable description of the dimensions of the state vector.

Original: For example, proprioceptiveinputs are common in deep RL literature, and eachdimension can usually be mapped to "Position X: 10.0 meters" or "Left Thigh Angle: 0.4 rad".
Revised: For example, proprioceptive inputs are common in deep RL literature, and each dimension can usually be mapped to "Position X: 10.0 meters" or "Left Thigh Angle: 0.4 rad".

Original: So the procedure is as follows: given a vector state s , we apply an environment-specific transformation f :
Revised: So the procedure is as follows: given a vector state s , we apply an environment-specific transformation f :

Original: S → String that converts each dimension value into a string, prepends a human-readable label to it, and then concatenates all of these strings into one.
Revised: S → String that converts each dimension value into a string, prepends a human-readable label to it, and then concatenates all of these strings into one.

Original: In general, while pre-trained language models seem to handle human-readable texts well, token embeddings provided by pre-trained language models may not transfer between different domains: e.g., the corpus used for training the LM does not contain a lot of text with numbers representing robot joints or velocities.
Revised: In general, while pre-trained language models seem to handle human-readable texts well, token embeddings provided by pre-trained language models may not transfer between different domains: e.g., the corpus used for training the LM does not contain a lot of text with numbers representing robot joints or velocities.

Original: Therefore, in addition to re-writing the observation, we also perform minimal fine-tuning of the LM when training an offline
Revised: Therefore, in addition to re-writing the observation, we also perform minimal fine-tuning of the LM when training an offline

Original: RL algorithm.
Revised: RL algorithm.

Original: This is done in a way similar to Lu et al.
Revised: This is done in a way similar to Lu et al.

Original: (2021): instead of tuning the entire model, we only tune the input embeddings and layer norm parameters.
Revised: (2021): instead of tuning the entire model, we only tune the input embeddings and layer norm parameters.

Original: The proposed approach can be applied to any off-the-shelf offline RL algorithm, given that the length of the textual representation is suitable for the utilized LM.
Revised: The proposed approach can be applied to any off-the-shelf offline RL algorithm, given that the length of the textual representation is suitable for the utilized LM.

Original: Note that it is not necessary to parse the observation space for meaningful entities, as it is enough to simply provide the labels for each dimension.
Revised: Note that it is not necessary to parse the observation space for meaningful entities, as it is enough to simply provide the labels for each dimension.

Original:  Experimental Evaluation
Revised:  Experimental Evaluation

Original: Concrete Realization The proposed method allows for various choices of language models, observation encodings, and deep offline RL algorithms.
Revised: Concrete Realization The proposed method allows for various choices of language models, observation encodings, and deep offline RL algorithms.

Original: We use MPNET (Song et al., 2020) trained with
Revised: We use MPNET (Song et al., 2020) trained with

Original: Sentence Transformer (Reimers and Gurevych, 2019) as a backbone for the LM.
Revised: Sentence Transformer (Reimers and Gurevych, 2019) as a backbone for the LM.

Original: For the textual representation, we round the numbers to 5 decimal places to speed up LM inference and reduce the memory space used by the model.
Revised: For the textual representation, we round the numbers to 5 decimal places to speed up LM inference and reduce the memory space used by the model.

Original: A batch sizeof 32 is employed in order for the model to fit on a single NVIDIA Tesla V100 GPU with 32 GB
Revised: A batch size

Original: [Table]
Revised: [Table]

Original: of memory
Revised: of 32 is employed in order for the model to fit on a single NVIDIA Tesla V100 GPU with 32 GB of memory.

Original: We rely on TD3+BC (Fujimoto and
Revised: We rely on TD3+BC (Fujimoto and

Original: Gu, 2021) as a deep offline RL algorithm in ourexperiments.
Revised: Gu, 2021) as a deep offline RL algorithm in our experiments.

Original: Benchmarks We probe our approach using asubset of NeoRL benchmark (Qin et al., 2021),running the experiments on the MuJoCo tasks, andreport the best performance after a hyperparametersearch.
Revised: Benchmarks We probe our approach using a subset of NeoRL benchmark (Qin et al., 2021), running the experiments on the MuJoCo tasks, and report the best performance after a hyperparameter search.

Original: Following the original evaluation protocolby Qin et al.
Revised: Following the original evaluation protocol by Qin et al.

Original: (2021), we train each policy for 3 · 10 5iterations.
Revised: (2021), we train each policy for 3 · 10iterations.

Original: However, we evaluate the policies withonly 100 episodes instead of 1000, as we observedno difference in the final results.
Revised: However, we evaluate the policies with only 100 episodes instead of 1000, as we observed no difference in the final results.

Original: We also run thevanilla TD3+BC algorithm, as its results are notreported in the original NeoRL paper.
Revised: We also run the vanilla TD3+BC algorithm, as its results are not reported in the original NeoRL paper.

Original: The results of our preliminary set of experimentsare presented in Table 1.
Revised: The results of our preliminary set of experiments are presented in Table 1.

Original: We observe significantperformance gains on two out of three settings using the proposed method (TD3+BC, LM-Finetune),and a slight drop on the third one.
Revised: We observe significant performance gains on two out of three settings using the proposed method (TD3+BC, LM-Finetune), and a slight drop on the third one.

Original: Sensitivity to Online Evaluation Budget Thepreference between deep offline RL algorithms wasnoted to be dependent on the online evaluation budget available (Kurenkov and Kolesnikov, 2022).
Revised: Sensitivity to Online Evaluation Budget The preference between deep offline RL algorithms was noted to be dependent on the online evaluation budget available (Kurenkov and Kolesnikov, 2022).

Original: Toanalyze whether conclusions about the proposedmethod change under the varied budget, we alsoreport the Expected Online Performance graphsin Figure 2.
Revised: To analyze whether conclusions about the proposed method change under the varied budget, we also report the Expected Online Performance graphs in Figure 2.

Original: Our results indicate that, on environments where an improvement over the base modelis possible, said improvement is achieved across allonline budgets.
Revised: Our results indicate that, on environments where an improvement over the base model is possible, said improvement is achieved across all online budgets.

Original: Can We Avoid Fine-Tuning?
Revised: Can We Avoid Fine-Tuning?

Original: To assess theimportance of fine-tuning the input embeddings,we also run an ablation reported in Table 1 (denotedas LM-No-Tune).
Revised: To assess the importance of fine-tuning the input embeddings, we also run an ablation reported in Table 1 (denoted as LM-No-Tune).

Original: We observe that improvement ispossible even without fine-tuning, but only on oneof the environments and with a smaller magnitude.
Revised: We observe that improvement is possible even without fine-tuning, but only on one of the environments and with a smaller magnitude.

Original: We hypothesize that this is due to a domain shift(Lu et al., 2021), i.e., it is possible that the trainingdata for the LM did not contain much informationabout robot velocities and angles, and therefore
Revised: We hypothesize that this is due to a domain shift (Lu et al., 2021), i [Figure] e., it is possible that the training

Original: [Figure]
Revised: [Figure]

Original:  some amount of fine-tuning is helpful.
Revised: data for the LM did not contain much information about robot velocities and angles, and therefore some amount of fine-tuning is helpful.

Original: To test this further, we run the ablation in another environment – FinRL (Liu et al., 2021c).
Revised: To test this further, we run the ablation in another environment – FinRL (Liu et al., 2021c).

Original: The observation space is also vector-based, but the values represent different financial indicators (e.g., price of a certain stock, an amount of a specific stock, etc.).
Revised: The observation space is also vector-based, but the values represent different financial indicators (e [Figure] g., price of a certain stock, an amount of a specific stock, etc.).

Original: We suppose that these indicators are more common than the proprioceptive inputs when training the
Revised: We suppose that these indicators are more common than the proprioceptive inputs when training the

Original: LMs, and therefore a non-tuned LM should result in consistent improvements.
Revised: LMs, and therefore a non-tuned LM should result in consistent improvements.

Original: The results for various dataset levels and behavioral policies expertise can be found in Table 6.
Revised: The results for various dataset levels and behavioral policies expertise can be found in Table 6.

Original: And indeed, we can see that no fine-tuning can be  sufficient to achieve substantial improvements over the base model in most settings.
Revised: And indeed, we can see that no fine-tuning can be [Table] sufficient to achieve substantial improvements over the base model in most settings

Original: [Table]
Revised: [Table]

Original:  Future Work and Limitations
Revised:  Future Work and Limitations

Original: Despite the encouraging results we observed inthis preliminary study, we believe there are severalimportant directions that should be explored andexperiments to be done to substantiate the claimsfurther:
Revised: Despite the encouraging results we observed in this preliminary study, we believe there are several important directions that should be explored and experiments to be done to substantiate the claims further:

Original: • Probe on a larger scale : ablations with other deep offline RL algorithms, more environment and dataset sizes.
Revised: • Probe on a larger scale : ablations with other deep offline RL algorithms, more environment and dataset sizes.

Original: • Systematic generalization : while we per- formed testing on a common set of offline RL benchmarks, the most appealing side of the pre-trained language models is their abil-
Revised: • Systematic generalization : while we performed testing on a common set of offline RL benchmarks, the most appealing side of the pre-trained language models is their ability to generalize.

Original: ity to generalize.

Original: And while deep online RL algorithms were studied for systematic gen- eralization (Jiang et al., 2021), their offline counterparts were not.
Revised: And while deep online RL algorithms were studied for systematic generalization (Jiang et al., 2021), their offline counterparts were not.

Original: We expect that the use of language models can substantially benefit deep offline RL algorithms in this aspect.
Revised: We expect that the use of language models can substantially benefit deep offline RL algorithms in this aspect.

Original: • Explore different tuning approaches : recent success in prompt-tuning (Liu et al., 2021b) for downstream NLP tasks suggests that the prompt engineering can be bypassed.
Revised: • Explore different tuning approaches : recent success in prompt-tuning (Liu et al., 2021b) for downstream NLP tasks suggests that the prompt engineering can be bypassed.

Original: We be- lieve that exploring this method type in the context of deep RL can also be of interest.
Revised: We believe that exploring this method type in the context of deep RL can also be of interest.

Original: • Vision-based policies : the proposed algo- rithm is not well-suited for vision-based poli-
Revised: • Vision-based policies : the proposed algorithm is not well-suited for vision-based policies, and studying how it can be adapted

Original: cies, and studying how it can be adapted forthis problem type is also a potential directionfor future research.
Revised: for this problem type is also a potential direction for future research