Original: On the Effect of Pre-training for Transformer in Different Modality on Ofﬂine Reinforcement Learning
Revised: On the Effect of Pre-training for Transformer in Different Modality on Ofﬂine Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We empirically investigate how pre-training on data of different modalities, such aslanguage and vision, affects ﬁne-tuning of Transformer-based models to Mujocoofﬂine reinforcement learning tasks.
Revised: We empirically investigate how pre-training on data of different modalities, such aslanguage and vision, affects ﬁne-tuning of Transformer-based models to Mujocoofﬂine reinforcement learning tasks.

Original: Analysis of the internal representation revealsthat the pre-trained Transformers acquire largely different representations beforeand after pre-training, but acquire less information of data in ﬁne-tuning thanthe randomly initialized one.
Revised: Analysis of the internal representation revealsthat the pre-trained Transformers acquire largely different representations beforeand after pre-training, but acquire less information of data in ﬁne-tuning thanthe randomly initialized one.

Original: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much, andthe catastrophic performance of the model pre-trained with image data comes fromtoo large gradients.
Revised: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much andthat the bad performance of the model pre-trained with image data could partiallycome from large gradients and gradient clipping.

Original: To study what information the Transformer pre-trained withlanguage data utilizes, we ﬁne-tune this model with no context provided, ﬁndingthat the model learns efﬁciently even without context information.
Revised: To study what informationthe Transformer pre-trained with language data utilizes, we ﬁne-tune this modelwith no context provided, ﬁnding that the model learns efﬁciently even withoutcontext information.

Original: Subsequentfollow-up analysis supports the hypothesis that pre-training with language data islikely to make the Transformer get context-like information and utilize it to solvethe downstream task.
Revised: Subsequent follow-up analysis supports the hypothesis thatpre-training with language data is likely to make the Transformer get context-likeinformation and utilize it to solve the downstream task.

Original:  Introduction
Revised:  Introduction

Original: The past few years have witnessed the tremendous success of pre-trained Transformer-based models[1] on versatile natural language processing (NLP) tasks.
Revised: The past few years have witnessed the tremendous success of pre-trained Transformer-based models[1] on versatile natural language processing (NLP) tasks.

Original: This is because pre-training with a largecorpus enables Transformer-based models to gain rich universal language representations transferableto vast downstream tasks [2, 
Revised: This is because pre-training with a largecorpus enables Transformer-based models to gain rich universal language representations transferableto vast downstream tasks [2, 3, 4].

Original: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efﬁciently solve even non-NLP tasks [5, 6, 7, 8, 9, 7, 8, 10].
Revised: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efﬁciently solve even non-NLP tasks [5, 6, 7, 8, 9, 7, 8, 10].

Original: Becauseof its versatility, it has been suggested that Transformers pre-trained with a large corpus could becalled a kind of universal computation engines [5].
Revised: Becauseof its versatility, it has been suggested that Transformers pre-trained with a large corpus could becalled a kind of universal computation engines [5].

Original: One area worthwhile attempting to study the beneﬁts of pre-trained Transformer is reinforcementlearning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].
Revised: One area worthwhile attempting to study the beneﬁts of pre-trained Transformer is reinforcementlearning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].

Original: In this vein of research Reid et al. studied the effect of pre-training of Transformers on ofﬂinereinforcement learning (ofﬂine RL) tasks [9].
Revised: In this vein of research, Reid et al. studied the effect of pre-training of Transformers on ofﬂinereinforcement learning (ofﬂine RL) tasks [9].

Original: Their study demonstrates a surprising contrast thatpre-training with image signiﬁcantly deteriorates performance, while that with language data doesnot negatively affect downstream tasks, rather it even improves performance for some datasets.
Revised: Their study demonstrates a surprising contrast thatpre-training with image signiﬁcantly deteriorates performance, while that with language data doesnot negatively affect downstream tasks, rather it even improves performance for some datasets.

Original: However, it is not yet known how the presence or absence of pre-training and its content leads todifferences in the performance of downstream tasks.
Revised: However, it is not yet known how the presence or absence of pre-training and its content leads todifferences in the performance of downstream tasks.

Original: What information does/doesn’t the modelpre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, andwhy does the model pre-trained on image data ( image-pre-trained model ) result in catastrophicperformance?
Revised: What information does/doesn’t the modelpre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, andwhy does the model pre-trained on image data ( image-pre-trained model ) result in catastrophicperformance?

Original: In this paper, we approach these questions by analyzing the inside of the Transformerthat is ﬁne-tuning to Mujoco [14] ofﬂine RL tasks.
Revised: In this paper, we approach these questions by analyzing the inside of the Transformerthat is ﬁne-tuning to Mujoco [14] ofﬂine RL tasks.

Original: Analyzing the internals of Transformer, westudy how randomly initialized, language-pre-trained, and image-pre-trained models acquire differentrepresentations, and what the differences could be attributed to.
Revised: Analyzing the internals of Transformer, westudy how randomly initialized, language-pre-trained, and image-pre-trained models acquire differentrepresentations, and what the differences could be attributed to.

Original: Our contributions are as follows
Revised: Our contributions are as follows:

Original: • We study the internal representation of pre-trained and randomly initialized Transformersand ﬁnd that the pre-trained models largely change their representation, while they encodeless information of data during ﬁne-tuning.•
Revised: • We study the internal representation of pre-trained and randomly initialized Transformersand ﬁnd that the pre-trained models largely change their representation, while they encodeless information of data during ﬁne-tuning.•

Original: The analysis of parameter change and gradient shows that the pre-trained models do notchange parameters much  due to gradient confusion, andin addition, that the catastrophicperformance of the image-pre-trained model is probably caused by  thetoo large gradient.• We examine context-dependence by ﬁne-tuning models with giving no context information,ﬁnding that the language-pre-trained Transformer efﬁciently learns even without context andthis could be the case by replacing just a single Transformer block with a pre-trained one.• Subsequent analysis of the change in how far away the model process the tokens of inputsupports the possibility that some contextual equivalent information has been positivelytransferred from language pre-training.
Revised: The analysis of parameter change and gradient shows that the pre-trained models do notchange parameters much perhaps due to gradient confusion and that the performance of theimage-pre-trained model might be partly affected by gradient clipping on the large gradient.• We examine context-dependence by ﬁne-tuning models with giving no context information,ﬁnding that the language-pre-trained Transformer efﬁciently learns even without context andthis could be the case by replacing just a single Transformer block with a pre-trained one.• Subsequent analysis of the change in how far away the model process the tokens of inputsupports the possibility that some contextual equivalent information has been positivelytransferred from language pre-training.

Original:  Related Works
Revised:  Related Works

Original: The study of transferability of neural representation has been one of the most important research topicsin machine learning research.
Revised: The study of transferability of neural representation has been one of the most important research topicsin machine learning research.

Original: A bunch of studies have investigated how pre-training inﬂuences thedownstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].
Revised: A bunch of studies have investigated how pre-training inﬂuences thedownstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].

Original: Recently,several studies have investigated the transferability of Transformers to various NLP tasks sinceunderstanding its high transferability is of great interest.
Revised: Recently,several studies have investigated the transferability of Transformers to various NLP tasks sinceunderstanding its high transferability is of great interest.

Original: Previous research reveals that Transformerrepresentation encodes some syntax [22], semantics [23], cross-lingual information [24], and worldknowledge [25].
Revised: Previous research reveals that Transformerrepresentation encodes some syntax [22], semantics [23], cross-lingual information [24], and worldknowledge [25].

Original: Also, it has been pointed out that Transformers obtains not only language-speciﬁcinformation but also more abstract inter-token dependencies [26, 27, 28].
Revised: Also, it has been pointed out that Transformers obtains not only language-speciﬁcinformation but also more abstract inter-token dependencies [26, 27, 28].

Original: This ﬁnding suggests thatpre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.
Revised: This ﬁnding suggests thatpre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.

Original: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks,reporting the versatility of the Transformer even in such multi-modal cases.
Revised: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks,reporting the versatility of the Transformer even in such multi-modal cases.

Original: Such prior workshave shown that language-pre-trained Transformers can efﬁciently learn image classiﬁcation [29, 5],symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and moregenerally, sequence modeling problems [5].
Revised: Such prior workshave shown that language-pre-trained Transformers can efﬁciently learn image classiﬁcation [29, 5],symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and moregenerally, sequence modeling problems [5].

Original: Our study is in line with the research that examines theapplicability of pre-trained Transformers on reinforcement learning.
Revised: Our study is in line with the research that examines theapplicability of pre-trained Transformers on reinforcement learning.

Original: Most prior works applying pre-trained Transformer on reinforcement learning explicitly expressgoals, observations, or actions in a language [30, 7, 8, 10, 31].
Revised: Most prior works applying pre-trained Transformer on reinforcement learning explicitly expressgoals, observations, or actions in a language [30, 7, 8, 10, 31].

Original: On the other hand, the work of [9]applies pre-trained Transformers on the ofﬂine RL tasks that language is unrelated to (e.g. Mujoco[14]) and still reports that pre-training is helpful.
Revised: On the other hand, the work of [9]applies pre-trained Transformers on the ofﬂine RL tasks that language is unrelated to (e.g. Mujoco[14]) and still reports that pre-training is helpful.

Original: This study is interesting because it suggests thatlanguage-pre-trained Transformers seem to exploit some similarity between the structure of naturallanguage data and trajectory data, which is more abstract and fundamental than linguistic information.
Revised: This study is interesting because it suggests thatlanguage-pre-trained Transformers seem to exploit some similarity between the structure of naturallanguage data and trajectory data, which is more abstract and fundamental than linguistic information.

Original: Also, exploring the applicability of pre-trained Transformers, which have already been successful, isof great engineering signiﬁcance since pre-training in ofﬂine RL has room for improvement comparedto other modalities.
Revised: Also, exploring the applicability of pre-trained Transformers, which have already been successful, isof great engineering signiﬁcance since pre-training in ofﬂine RL has room for improvement comparedto other modalities.

Original: Examining the internals of Transformers, we further delve into the ﬁndings fromthis study by Reid et al.
Revised: Examining the internals of Transformers, we further delve into the ﬁndings fromthis study by Reid et al.

Original: [9] and explores the applicability of Transformers to ofﬂine RL.
Revised: [9] and explores the applicability of Transformers to ofﬂine RL.

Original:  Background3.1 Transformer
Revised:  Background3.1 Transformer

Original: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .
Revised: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .

Original: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [32],and layer normalization (layer norm) [33].
Revised: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [32],and layer normalization (layer norm) [33].

Original: Denoting the linear projection of input sequence of lengthn to query, key, values of length m by Q 2 R n ⇥ d Q , K 2 R m ⇥ d K , and V 2 R m ⇥ d V , respectively,the self-attention is Attention ( Q, K, V ) = softmax ⇣
Revised: Denoting the linear projection of input sequence of lengthn to query, key, values of length m by Q 2 R n ⇥ d Q , K 2 R m ⇥ d K , and V 2 R m ⇥ d V , respectively,the self-attention is Attention ( Q, K, V ) = softmax ⇣

Original: QK > p d K ⌘
Revised: QK > p d K ⌘

Original: V .
Revised: V .

Original: We use GPTs [34], and hence theautoregressive language model, where queries head to only keys before their position.
Revised: We use GPTs [34], and hence theautoregressive language model, where queries head to only keys before their position.

Original: 3.2 Ofﬂine Reinforcement Learning and Decision Transformer
Revised: 3.2 Ofﬂine Reinforcement Learning and Decision Transformer

Original: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.
Revised: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.

Original: For ease of mathematical handling, the environment is usually described as a Markov decision process
Revised: For ease of mathematical handling, the environment is usually described as a Markov decision process

Original: (MDP).
Revised: (MDP).

Original: The MDP consists of states s 2 S , actions a 2 A , transition probability P
Revised: The MDP consists of states s 2 S , actions a 2 A , transition probability P

Original: ( s 0 | s , a ) , and 86 reward function r ( s , a ) :
Revised: ( s 0 | s , a ) , and 86 reward function r ( s , a ) :

Original: S ⇥ A !
Revised: S ⇥ A !

Original: R .
Revised: R .

Original: The interaction of the agent in the environment is represented as 87 K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) ,where s t , a t , and r t are state, action, and reward at time step t .
Revised: The interaction of the agent in the environment is represented as 87 K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) ,where s t , a t , and r t are state, action, and reward at time step t .

Original: The goal is to ﬁnd an optimal policy⇡ ( a | s ) to maximize the expected cumulative rewards E [
Revised: The goal is to ﬁnd an optimal policy⇡ ( a | s ) to maximize the expected cumulative rewards E [

Original: P
Revised: P

Original: Kt =

Revised: Kt =

Original: r t ] for the trajectory.
Revised: r t ] for the trajectory.

Original: Ofﬂine RL aims to achieve objectives of reinforcement learning from the trajectory data collected bysome policy [35].
Revised: Ofﬂine RL aims to achieve objectives of reinforcement learning from the trajectory data collected bysome policy [35].

Original: Because ofﬂine RL is purely characterized by trajectory data, several studies haveproposed formulating ofﬂine RL as a sequence modeling problem [36, 37].
Revised: Because ofﬂine RL is purely characterized by trajectory data, several studies haveproposed formulating ofﬂine RL as a sequence modeling problem [36, 37].

Original: Decision Transformer isa seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoderof BERT [2]) to solve such a sequence modeling problem [37].
Revised: Decision Transformer isa seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoderof BERT [2]) to solve such a sequence modeling problem [37].

Original: In Decision Transformer, inputdata is a sequence ⌧ =
Revised: In Decision Transformer, inputdata is a sequence ⌧ =

Original: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-goˆ R t =
Revised: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-goˆ R t =

Original: P t  1 i =

Revised: P t  1 i =

Original: r i is the cumulative reward until time step t .
Revised: r i is the cumulative reward until time step t .

Original: The previous studies [9, 37] employ theabove problem set up to discuss the applicability of Transformer for ofﬂine RL.
Revised: The previous studies [9, 37] employ theabove problem set up to discuss the applicability of Transformer for ofﬂine RL.

Original: We also follow thesestudies and use the setup for our analysis.
Revised: We also follow thesestudies and use the setup for our analysis.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Unless otherwise noted, all of our experimental setups, including model conﬁguration, dataset, andhyperparameters follow the previous work [9] 1 .
Revised: Unless otherwise noted, all of our experimental setups, including model conﬁguration, dataset, andhyperparameters follow the previous work [9] 1 .

Original: The sanity check that our trained models achievecomparable performance with the previous work is described in Appendix A.
Revised: The sanity check that our trained models achievecomparable performance with the previous work is shown in Table 1. Details are in Appendix A.

Original: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initializedmodel ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect ofpre-training.
Revised: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initializedmodel ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect ofpre-training.

Original: Pre-trained models are from the Transformers library from HuggingFace [38].
Revised: Pre-trained models are from the Transformers library from HuggingFace [38].

Original: Like theprevious study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .
Revised: Like theprevious study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .

Original: For ofﬂine RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [39] provided by D4RL [40] 108, an ofﬂine RL dataset library.
Revised: For ofﬂine RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [39] provided by D4RL [40] 108, an ofﬂine RL dataset library.

Original: Speciﬁcally, we employ medium datasets of HalfCheetah , Walker2d ,and Hopper environments.
Revised: Speciﬁcally, we employ medium datasets of HalfCheetah , Walker2d ,and Hopper environments.

Original: Unless otherwise speciﬁed, we show the result for Hopper-medium .
Revised: Unless otherwise speciﬁed, we show the result for Hopper-medium .

Original: Weput the results for other environments in Appendix J.
Revised: Weput the results for other environments in Appendix J.

Original: Following the previous study [9], we train the models for 40 epochs, each of which consists of 2500steps.
Revised: Following the previous study [9], we train the models for 40 epochs, each of which consists of 2500steps.

Original: In the following sections, we refer to the 40th epoch checkpoint as the post-ﬁne-tuning modeland the 0th epoch model (the initial state) as the pre- ﬁne-tuning model .
Revised: In the following sections, we refer to the 40th epoch checkpoint as the post-ﬁne-tuning modeland the 0th epoch model (the initial state) as the pre- ﬁne-tuning model .

Original: The training details, includinghyperparameters and the optimizer, are detailed in Appendix B.
Revised: The training details, includinghyperparameters and the optimizer, are detailed in Appendix B.

Revised: [Table]

Original:  Results and Analysis
Revised:  Results and Analysis

Original: 5.1 Activation Similarity
Revised: 5.1 Activation Similarity

Original: We ﬁrst study how the pre-trained and randomly initialized models shape the representation in ﬁnetuning.
Revised: We ﬁrst study how the pre-trained and randomly initialized models shape the representation in ﬁnetuning.

Original: To that end, we compare the centered kernel alignment (CKA) [41] of each layer’s activationbetween pre and post-ﬁne-tuning, investigating how each layer’s representation changes by learningofﬂine RL data 3 .
Revised: To that end, we compare the centered kernel alignment (CKA) [41] of each layer’s activationbetween pre and post-ﬁne-tuning, investigating how each layer’s representation changes by learningofﬂine RL data 3 .

Original: The CKA has been used in various studies and has provided numerous insights intothe understanding of neural networks [42, 43, 44, 45, 46].
Revised: The CKA has been used in various studies and has provided numerous insights intothe understanding of neural networks [42, 43, 44, 45, 46].

Original: In particular, we compute the linear CKA,following previous studies [41, 47], for a subset of the dataset.
Revised: In particular, we compute the linear CKA,following previous studies [41, 47], for a subset of the dataset.

Original: We use the activation of each layerfrom the last time step of the context for return-to-go, state, and action, respectively.
Revised: We use the activation of each layerfrom the last time step of the context for return-to-go, state, and action, respectively.

Original: We show theresult of state, putting the results for the others in Appendix J.1.
Revised: We show the We result of state, putting the results for the others in Appendix J.1.

Original: Following the previous study [45],we compute CKA of activation not only for Transformer block output but for all internal activation ofthe Transformer blocks.
Revised: Following the previous study [45],we compute CKA of activation not only for Transformer block output but for all internal activation ofthe Transformer blocks.

Original: The deﬁnition of the CKA and the details are in Appendices D.2 and D.1.
Revised: The deﬁnition of the CKA and the details are in Appendices D.2 and D.1.

Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1 shows the result.
Revised: 1 shows the result.

Original: The y-axis is each layer’s CKA between representation at pre and post-ﬁnetuning.
Revised: The y-axis is each layer’s CKA between representation at pre and post-ﬁnetuning.

Original: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twice [Figure] the others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPTand align the x-axis for different models for comparison. 
Revised: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twicethe others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPTand align the x-axis for different models for comparison. Higher CKA means higher similarity.

Original: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layersexcept for a few layers, indicating that pre-trained models largely change representation.
Revised: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layersexcept for a few layers, indicating that pre-trained models largely change representation.

Original: On the otherhand, for the randomly initialized model (Random Init), the representations have a higher similarityto the initial state from the middle to the shallow layers.
Revised: On the otherhand, for the randomly initialized model (Random Init), the representations have a higher similarityto the initial state from the middle to the shallow layers.

Original: Also, the CKAs of the randomly initializedmodel seem to be higher than pre-trained models across layers.
Revised: Also, the CKAs of the randomly initializedmodel seem to be higher than pre-trained models across layers.

Original: These observations indicate that therandomly initialized and pre-trained models structure their representations in very different ways.
Revised: These observations indicate that therandomly initialized and pre-trained models structure their representations in very different ways.

Revised: Further support for this conclusion is shown in Appendix D.5.1.

Original: To identify which layers have relatively higher CKA for the randomly initialized model, we obtainlayer names with CKA values above a threshold (dashed line at 0.23 in Fig.
Revised: To identify which layers have relatively higher CKA for the randomly initialized model, we obtainlayer names with CKA values above a threshold (dashed line at 0.23 in Fig.

Original: 1 (a)).
Revised: 1 (a)).

Original: Then, we ﬁndthat most of them are layer normalization layers (the list of the layer names are put in Appendix
Revised: Then, we ﬁndthat most of them are layer normalization layers (the list of the layer names are put in Appendix

Original: D.3).
Revised: D.3).

Original: Therefore, we can speculate that the difference in CKA similarity of layer normalization can berelated to the effect of pre-training.
Revised: Therefore, we can speculate that the difference in CKA similarity of layer normalization can berelated to the effect of pre-training.

Original: This possibility could be supported by previous research [5] thatreports that layer norm parameters are the most important to ﬁne-tune.
Revised: This possibility could be supported by previous research [5] thatreports that layer norm parameters are the most important to ﬁne-tune.

Original: To check if pre-trained models and the randomly initialized model converge to a similar representationor not, we compute the CKA between representations of ﬁne-tuned these models.

Original: The result is shownin Fig.

Original: 2, where (a) is the CKA between GPT2 and iGPT, (b) is that between GPT2 and randominitialization, and (c) is that between iGPT and random initialization.

Original: We observe that CKA values inthese heat maps are small, conﬁrming that different models learn different representations.

Original: In uni-modal cases, where the statistical nature of the input data does not change signiﬁcantly,previous studies have a consensus that the shallow layer acquires a more universal representationand the deep layer acquires a task-speciﬁc representation for a wide range of architectures and tasks[48, 49, 50, 51, 52, 53, 54, 44, 46].
Revised: In uni-modal cases, where the statistical nature of the input data does not change signiﬁcantly,previous studies have a consensus that the shallow layer acquires a more universal representationand the deep layer acquires a task-speciﬁc representation for a wide range of architectures and tasks[48, 49, 50, 51, 52, 53, 54, 44, 46].

Original: The ﬁnding that the language-pre-trained Transformer performs aswell as or better than the baseline but largely changes the representation suggests that in multi-modalcases, something different may be happening in forming the representation than in prior studies.[Figure]
Revised: The ﬁnding that the language-pre-trained Transformer performs aswell as or better than the baseline but largely changes the representation suggests that in multi-modalcases, something different may be happening in forming the representation than in prior studies.

Original: 5.2 Mutual Information Between Hidden Representation and Data
Revised: 5.2 Mutual Information Between Hidden Representation and Data

Original: In the previous section, we observe that pre-trained models drastically change their internal representation during ﬁne-tuning, attaining largely different representations.
Revised: In the previous section, we observe that pre-trained models drastically change their internal representation during ﬁne-tuning, attaining largely different representations.

Original: One possible consequence ofthis observation is that the pre-trained model may have adapted better to the data in the downstreamtasks.
Revised: One possible consequence ofthis observation is that the pre-trained model may have adapted better to the data in the downstreamtasks.

Original: To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models.
Revised: To explore this possibility, we calculate estimated values of mutual information betweenhidden representation and data and compare them among different models.

Original: Mutual information is awell-known measure of the mutual dependence between the two random variables.
Revised: Mutual information is awell-known measure of the mutual dependence between the two random variables.

Original: Many previousstudies have used this metric to investigate the extent to which neural representation encodes input andlabel information and provided profound insight into how neural networks work [55, 56, 57, 58, 59].
Revised: Many previousstudies have used this metric to investigate the extent to which neural representation encodes input andlabel information and provided profound insight into how neural networks work [55, 56, 57, 58, 59].

Original: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .
Revised: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .

Original: We employ Mutual Information
Revised: We employ Mutual Information

Original: Neural Estimation (MINE) [60] to estimate mutual information since this method can estimatemutual information even when random vectors are continuous and have different dimensions 4 .
Revised: Neural Estimation (MINE) [60] to estimate mutual information since this method can estimatemutual information even when random vectors are continuous and have different dimensions 4 .

Original: Since
Revised: Since

Original: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual actiona t , we deﬁne the label as Y = a t .
Revised: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual action a t , we deﬁne the label as Y = a t .

Original: Unlike non-sequential models, the causal Transformer exploitspast context information.
Revised: Unlike non-sequential models, the causal Transformer exploitspast context information.

Original: Thus, denoting the input token except for a K by x t and the internalactivation of Transformer block l of the input by T l ( x t ) , we compute ˆ
Revised: Thus, denoting the input token except for a K by x t and the internalactivation of Transformer block l of the input by T l ( x t ) , we compute ˆ

Original: I ( x t ; T l ( s K ))
Revised: I ( x t ; T l ( s K ))

Original: as ˆ I ( X ; T ) andˆ I ( a K ; T l ( x t ))
Revised: as ˆ I ( X ; T ) andˆ I ( a K ; T l ( x t ))

Original: as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .
Revised: as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .

Original: We calculate these values for theshallow, middle, and deep Transformer blocks, respectively.
Revised: We calculate these values for theshallow, middle, and deep Transformer blocks, respectively.

Original: We show the result of the middle blockhere and leave the remaining in Appendix J.2.
Revised: We show the result of the middle blockhere and leave the remaining in Appendix J.2.

Original: The other details are described in Appendix E.1.
Revised: The other details are described in Appendix E.1.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 3 is the result, where Fig.
Revised: 2 is the result, where Fig.

Original: 3 (a) and (b) are results of ˆ
Revised: 2 (a) and (b) are results of ˆ

Original: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.
Revised: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.

Original: Themarkers triangle , circle , and cross represent return-to-go, state, and action, respectively.
Revised: Themarkers triangle , circle , and cross represent return-to-go, state, and action, respectively.

Original: The x-axis isthe position of tokens in the context and the y-axis is the estimated value of mutual information.
Revised: The x-axis isthe position of tokens in the context and the y-axis is the estimated value of mutual information.

Original: We observe that the randomly initialized model consistently has higher mutual information bothfor input and label than pre-trained models.
Revised: We observe that the randomly initialized model consistently has higher mutual information bothfor input and label than pre-trained models.

Original: Given the catastrophic performance of iGPT, it isunderstandable that iGPT representations have almost no information.
Revised: Given the catastrophic performance of iGPT, it isunderstandable that iGPT representations have almost no information.

Original: The result comparing GPT2to random initialization indicates that the changed representation observed in Section 5.1 does notencode more information about the input or the label than the randomly initialized model.
Revised: The result comparing GPT2to random initialization indicates that the changed representation observed in Section 5.1 does notencode more information about the input or the label than the randomly initialized model.

Original: That is, thelanguage-pre-trained model does well probably not because it acquires as much or more informationabout the data as the random initialization.
Revised: That is, thelanguage-pre-trained model does well probably not because it acquires as much or more informationabout the data as the random initialization.

Original: This result suggests that although the representation haschanged, there remains something unchanged, and the language-pre-trained model may leverage it tosolve the ofﬂine RL task.
Revised: This result suggests that although the representation haschanged, there remains something unchanged, and the language-pre-trained model may leverage it tosolve the ofﬂine RL task.

Revised: Note that this might be affected by limitations of mutual information.

Original: We also notice that for label the language-pre-trained model has consistently higher mutual information for a speciﬁc token type (return-to-go), while that is not always the case for the randomlyinitialized model.
Revised: We also notice that for label the language-pre-trained model has consistently higher mutual information for a speciﬁc token type (return-to-go), while that is not always the case for the randomlyinitialized model.

Original: This result implies that the language-pre-trained model may process the informationof a speciﬁc token type at each layer for prediction.
Revised: This result implies that the language-pre-trained model may process the informationof a speciﬁc token type at each layer for prediction.

Original: 5.3 Parameter Similarity
Revised: 5.3 Parameter Similarity

Original: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploitssome pre-acquired information.
Revised: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploitssome pre-acquired information.

Original: To investigate this issue on a ﬁner scale, we turn our eye fromrepresentation to parameter.
Revised: To investigate this issue on a ﬁner scale, we turn our eye fromrepresentation to parameter.

Original: We conduct parameter-level similarity analysis and investigate to whatextent parameter change between pre and post-ﬁne-tuning.
Revised: We conduct parameter-level similarity analysis and investigate to whatextent parameter change between pre and post-ﬁne-tuning.

Original: In concrete, we compute l 2 distance andcosine similarity of parameters between pre and post-ﬁne-tuning.
Revised: In concrete, we compute l 2 distance andcosine similarity of parameters between pre and post-ﬁne-tuning.

Original: The details are in Appendix F.1.
Revised: The details are in Appendix F.1.

Original: The result for l 2 distance is shown in Fig.
Revised: The result for l 2 distance is shown in Fig.

Original: 4 and for cosine similarity in Fig.
Revised: 3 and for cosine similarity in Fig.

Original: 5, where the top is theresult for GPT2, the middle is for iGPT, and the bottom is for random initialization.
Revised: 4, where the top is theresult for GPT2, the middle is for iGPT, and the bottom is for random initialization.

Original: The x-axis is theparameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layerparameters.
Revised: The x-axis is theparameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layerparameters.

Original: For visibility, we do not show the labels for ticks and put them in Appendix F.2.
Revised: For visibility, we do not show the labels for ticks and put them in Appendix F.2.

Original: Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models,meaning that the parameters of pre-trained models do not change that much.
Revised: Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models,meaning that the parameters of pre-trained models do not change that much.

Original: At ﬁrst glance, thisappears to contradict the results of Section 5.1.
Revised: At ﬁrst glance, thisappears to contradict the results of Section 5.1.

Original: However, prior research has reported that most of theinformation in Transformers propagates through skip connections [45].
Revised: However, prior research has reported that most of theinformation in Transformers propagates through skip connections [45].

Original: This means that, as long asthe parameters of the shallow layers change, the representation of each layer may change signiﬁcantlywithout changing the parameters of each layer.
Revised: This means that, as long asthe parameters of the shallow layers change, the representation of each layer may change signiﬁcantlywithout changing the parameters of each layer.

Original: We indeed observe that the shallow layer l 2 distanceof the pre-trained model shows relatively large changes.
Revised: We indeed observe that the shallow layer l 2 distanceof the pre-trained model shows relatively large changes.

Original: The existence of unchanged parametersmeans that knowledge acquired through pre-training could be stored in those parameters.
Revised: The existence of unchanged parametersmeans that knowledge acquired through pre-training could be stored in those parameters.

Original: 5.4 Gradient Analysis
Revised: 5.4 Gradient Analysis

Revised: [Figure]

Original: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.
Revised: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.

Original: We dig intothis issue, investigating why GPT2 and iGPT do not change parameters.
Revised: We dig intothis issue, investigating why GPT2 and iGPT do not change parameters.

Original: To this end, we compare thegradient norm and gradient confusion [61] at the early phase of the training ( 1 st epoch), discussingthe training difﬁculty.
Revised: To this end, we compare thegradient norm and gradient confusion [61] at the early phase of the training ( 1 st epoch), discussingthe training difﬁculty.

Original: When parameter gradients of loss for two different samples are negativelycorrelated, we say that there is gradient confusion.
Revised: When parameter gradients of loss for two different samples are negativelycorrelated, we say that there is gradient confusion.

Original: Empirically, we can measure this by the minimumgradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusionis low.
Revised: Empirically, we can measure this by the minimumgradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusionis low.

Original: The previous study showed that training is easier when gradient confusion is low [61].
Revised: The previous study showed that training is easier when gradient confusion is low [61].

Original: Becausegradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), wecompute the values for clipped gradients.
Revised: Becausegradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), wecompute the values for clipped gradients.

Original: The details are described in Appendix G.1.
Revised: The details are described in Appendix G.1.

Original: Fig.
Revised: Fig.

Original: 6 is the result for the gradient confusion and Fig.
Revised: 5 is the result for the gradient confusion and Fig.

Original: 7 is that for gradient norm.
Revised: 6 is that for gradient norm.

Original: Fig.
Revised: Fig.

Original: 6 shows thatthe minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initializedmodel.
Revised: 5 shows thatthe minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initializedmodel.

Original: This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model.
Revised: This high gradient confusion probably makes pre-trained models harder to train, whichexplains why they change parameters less than the randomly initialized model.

Original: Fig.
Revised: Fig.

Original: 7, on the otherhand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25),while that of the others is below the bound.
Revised: 6, on the otherhand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of the others is below the bound.

Original: That is, the differences in the magnitude of the gradientare collapsed for iGPT, reducing the informational value of the gradients.
Revised: That is, the differences in the magnitude of the gradientare collapsed for iGPT, reducing the informational value of the gradients.

Original: This result indicates thatthe catastrophic performance of iGPT could come from the too-large gradient.

Original: To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm ofeach parameter set.
Revised: To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm ofeach parameter set.

Original: We calculate the gradient norm ratio of weights and biases of all parameters inall Transformer blocks.
Revised: We calculate the gradient norm ratio of weights and biases of all parameters inall Transformer blocks.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 8: for visibility, we show the parameters of theﬁrst layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remainingresults together as others .
Revised: 7: for visibility, we show the parameters of theﬁrst layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remainingresults together as others .

Original: The full result is shown in Appendix G.2.
Revised: The full result is shown in Appendix G.2.

Original: We observe that weights andbiases of the ﬁrst layer norm module dominate the norm.
Revised: We observe that weights andbiases of the ﬁrst layer norm module dominate the norm.

Original: Thus, we conclude that the difﬁculty oftraining iGPT probably comes from the too-large gradient of the ﬁrst layer norm module.
Revised: This result implies that the difﬁculty oftraining iGPT might partially come from the use of gradient clipping on large gradients dominated byfew parameters. A complementary analysis is described in Appendix G.2.

Original: 5.5 Fine-Tuning with No Context Information
Revised: 5.5 Fine-Tuning with No Context Information

Original: In the sections up to this point, we ﬁnd the possibility that the language-pre-trained model may exploitsome pre-acquired information.
Revised: In the sections up to this point, we ﬁnd the possibility that the language-pre-trained model may exploitsome pre-acquired information.

Original: Then what information does the language-pre-trained model exploitto solve the tasks?
Revised: Then what information does the language-pre-trained model exploitto solve the tasks?

Original: One possibility is the ability to handle context because Transformer is skilledat handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].
Revised: One possibility is the ability to handle context because Transformer is skilledat handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].

Original: Therefore, we train the language-pre-trained and the randomly initialized model with no access to thecontext ( K = 1 ) to dig into this possibility.
Revised: Therefore, we train the language-pre-trained and the randomly initialized model with no access to thecontext ( K = 1 ) to dig into this possibility.

Original: If a model learns efﬁciently without context, the model isalready likely to have context-equivalent information.
Revised: If a model learns efﬁciently without context, the model isalready likely to have context-equivalent information.

Original: Details of experiments are in Appendix H.1.
Revised: Details of experiments are in Appendix H.1.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The mean return throughout training time is shownin Fig.
Revised: The mean return throughout training time is shownin Fig.

Original: 9 and the normalized scores of return 5 areshown in Table 1.
Revised: 8 and the normalized scores of return 5 areshown in Table 4.

Original: The solid lines in Fig.
Revised: The solid lines in Fig.

Original: 9 are resultswithout context and the dashed lines for referenceare those with context.
Revised: 8 are resultswithout context and the dashed lines for referenceare those with context.

Original: From Fig.
Revised: From Fig.

Original: 9, we observe thatwhen
Revised: 8, we observe thatwhen

Original: the context is not provided ( K = 1 ), the GPT2reaches the high mean return much faster than therandomly initialized model.
Revised: the context is not provided ( K = 1 ), the GPT2reaches the high mean return much faster than therandomly initialized model.

Original: Furthermore, Table 1reveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), whilethe language-pre-trained model consistently achieves a comparable performance with that of thebaseline result ( K = 20 ).
Revised: Furthermore, Table 4reveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), whilethe language-pre-trained model consistently achieves a comparable performance with that of thebaseline result ( K = 20 ).

Original: That is, we observe that the language-pre-trained Transformer consistently [Table] performs more efﬁciently when no context is provided.
Revised: That is, we observe that the language-pre-trained Transformer consistentlyperforms more efﬁciently when no context is provided.

Original: This is somewhat surprising, given thatprevious studies have pointed out that Decision Transformer performs signiﬁcantly worse with nocontext [37].
Revised: This is somewhat surprising, given thatprevious studies have pointed out that Decision Transformer performs signiﬁcantly worse with nocontext [37].

Original: The result that the language-pre-trained Transformer learns efﬁciently as well withoutcontext as it does with context suggests that the model may have already acquired some kind ofcontext-like information through pre-training and is utilizing it to solve the task.
Revised: The result that the language-pre-trained Transformer learns efﬁciently as well withoutcontext as it does with context suggests that the model may have already acquired some kind ofcontext-like information through pre-training and is utilizing it to solve the task.

Original: 5.6 More In-Depth Analysis of Context Dependence5.6.1 Replacement by the Pre-Trained Block
Revised: 5.6 More In-Depth Analysis of Context Dependence5.6.1 Replacement by the Pre-Trained Block

Original: In the previous sections, we ﬁnd that the language-pre-trained Transformer exploits context-relatedinformation to solve the task.
Revised: In the previous sections, we ﬁnd that the language-pre-trained Transformer exploits context-relatedinformation to solve the task.

Original: To identify which block contains the useful information, we conduct ablock replacement experiment, where we replace a Transformer block of the randomly initializedmodel with that of the language-pre-trained model and then ﬁne-tune the model.
Revised: To identify which block contains the useful information, we conduct ablock replacement experiment, where we replace a Transformer block of the randomly initialized [Table] model with that of the language-pre-trained model and then ﬁne-tune the model without context.

Original: If replacement of ablock does not help training, the block probably did not learn effective information in pre-training.
Revised: Ifreplacement of a block does not help training, the block probably did not learn effective informationin pre-training.

Original: Because we observe in Section 5.5 that the early phase characterizes the difference of the models, wefocus on the epochs up to 10th epoch.
Revised: Because we observe in Section 5.5 that the early phase characterizes the difference ofthe models, we focus on the epochs up to 10th epoch.

Original: We explain the other details in Appendix I.1.1.
Revised: We explain the other details in Appendix I.1.1.

Original: [Figure]
Revised: [Figure]

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 10, where (a) is the prediction error of action and (b) is the mean return.
Revised: 9, where (a) is the prediction error of action and (b) is the mean return 263.

Original: To get a rough trend, we take the exponential moving average.
Revised: To get a rough trend, we take the exponential moving average.

Original: We observe that only when block 1is replaced with the pre-trained block, does the action error (Fig.
Revised: We observe that only when block1 is replaced with the pre-trained block, does the action error (Fig.

Original: 10 (a)) and the mean return (Fig.10 (b)) behave similarly to the randomly initialized model.
Revised: (a)) and the mean return (Fig.9 (b)) behave similarly to the randomly initialized model.

Original: Thus, we conclude that all blocks otherthan block 1 contain context-equivalent useful information to solve tasks.
Revised: Thus, we conclude that all blocks otherthan block 1 contain context-equivalent useful information to solve tasks.

Original: The ﬁndings that replacinga single block with the pre-trained one improves learning efﬁciency and that effective informationis distributed across layers alone are surprising enough.
Revised: The ﬁndings that replacinga single block with the pre-trained one improves learning efﬁciency and that effective informationis distributed across layers alone are surprising enough.

Original: This could be related to the dominant roleof skip connections for information propagation [45] or the nature of non-monotonic change in thecontent each layer processes [20] in Transformers.
Revised: This could be related to the dominant roleof skip connections for information propagation [45] or the nature of non-monotonic change in thecontent each layer processes [20] in Transformers.

Original: We show the result for Hopper since the trendexplained above is most pronounced.
Revised: We show the result for Hopper since the trendexplained above is most pronounced.

Original: For other environments, although the similarity between therandom model and block1 is much weaker, the observation thatthe performance of block 1 is theworst is consistent.
Revised: For other environments, although the similarity between therandom model and block1 is much weaker, the observation that block 1 is not good is consistent.

Original: Results for other environments are in Appendix J.5.1.
Revised: Results for other environments are in Appendix J.6.1.

Original: 5.6.2 Attention Distance Analysis
Revised: 5.6.2 Attention Distance Analysis

Original: In previous sections, we ﬁnd that the information improving learning efﬁciency is preserved in allblocks other than a particular block and that is probably context-related.
Revised: In previous sections, we ﬁnd that the information improving learning efﬁciency is preserved in allblocks only in the varying amount and that is probably context-related.

Original: To further conﬁrm thatthe context-like information is indeed preserved, we investigate how the model processes contextby calculating the attention distance [62] of each Transformer block for the result of Section 5.5.
Revised: To further conﬁrm thatthe context-like information is indeed preserved, we investigate how the model processes contextby calculating the attention distance [62] of each Transformer block for the result of Section 5.5.

Original: Attention distance is the average distance between key and query weighted by attention weight.
Revised: Attention distance is the average distance between key and query weighted by attention weight.

Original: Previous studies successfully used this measure to reveal how locally Transformers process context[45, 62].
Revised: Previous studies successfully used this measure to reveal how locally Transformers process context[45, 62].

Original: Attention distance will elucidate how far away each model processes the tokens of input.
Revised: Attention distance will elucidate how far away each model processes the tokens of input.

Original: In particular, we compute the gap ( | d att ( epoch )
Revised: In particular, we compute the gap ( | d att ( epoch )

Original:  d att (0) | ) between attention distance at the initialstate ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves theway to utilize context.
Revised:  d att (0) | ) between attention distance at the initialstate ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves theway to utilize context.

Original: If the gap is smaller, it means that the attention distance is more preserved.
Revised: If the gap is smaller, it means that the attention distance is more preserved.

Original: To clarify the difference between the language-pre-trained model and the randomly initialized model, wecompare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initializedmodel still produces a low return, while the GPT2 achieves a high return (Fig.9).
Revised: Toclarify the difference between the language-pre-trained model and the randomly initialized model, wecompare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initializedmodel still produces a low return, while the GPT2 achieves a high return (Fig.

Original: The detail of thesetup and results for other epochs are shown in Appendices I.2.1 and J.5.2.
Revised: The detail of thesetup and results for other epochs are shown in Appendices I.2.1 and J.6.2.

Original: [Figure]

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 11, where the x-axis is the Transformer block index and the y-axis isthe gap between attention distances.
Revised: 10, where the x-axis is the Transformer block index and the y-axis isthe gap between attention distances.

Original: Fig.
Revised: Fig.

Original: 11 (a), shows that the attention gaps of GPT2 except forblock 1 are small, indicating that attention distances are kept from the pre-training in these blocks.
Revised: 10 (a), shows that the attention gaps of GPT2 except forblock 1 are small, indicating that attention distances are kept from the pre-training in these blocks.

Original: Taken together with the ﬁnding of Section 5.6.1 that blocks other than block 1 improve learningefﬁciency, these preserved attention distances are likely to positively affect ﬁne-tuning.
Revised: Taken together with the ﬁnding of Section 5.6.1 that blocks other than block 1 improve learningefﬁciency, these preserved attention distances are likely to positively affect ﬁne-tuning.

Original: This resultfurther supports the hypothesis that language-pre-training brings the model context-like informationand the use of this information allows efﬁcient learning.
Revised: This resultfurther supports the hypothesis that language-pre-training brings the model context-like informationand the use of this information allows efﬁcient learning.

Original:  We also notice the outstanding contrast that gaps of language-pre-trained models are almost zeroexcept for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model arelarge.
Revised: [Figure] We also notice the outstanding contrast that gaps of language-pre-trained models are almost zeroexcept for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model arelarge.

Original: This trend is clearer from middle to deep blocks (3 to 10) for all environments.
Revised: This trend is clearer from middle to deep blocks (3 to 10) for all environments.

Original: Turning an eyeto Fig.
Revised: Turning an eyeto Fig.

Original: 10 (a), we notice that the action error of these blocks decreases faster than the other block (0to 2 and 11).
Revised: 9 (a), we notice that the action error of these blocks decreases faster than the other block (0 to2 and 11).

Original: Thus, we conclude that the pre-trained context-like information could help the trainingby enabling the model to predict action more accurately.
Revised: Thus, we conclude that the pre-trained context-like information could help the training byenabling the model to predict action more accurately.

Revised: We summarize ﬁndings of Section 5 with particular attention to the implication for performance:

Revised: • 5.1 ⇢ Re-using representation is not the cause of the good performance of GPT2.

Revised: • 5.2 ⇢ Fitting to data better is not the cause of the good performance of GPT2.• 5.3 ⇢ The good performance of GPT2 might come from some unchanged parameters.• 5.4 ⇢ Gradient clipping and large gradient might be a cause of the bad performance of iGPT.• 5.5 ⇢ GPT2 can learn efﬁciently even without the context provided.

Revised: • 5.6.1 ⇢ Even a single pre-trained Transformer block makes training more efﬁcient.• 5.6.2 ⇢

Revised: A cause of the good performance of GPT2 is a pre-acquired way to use context.

Original:  Discussion
Revised:  Discussion

Original: Conclusion We have examined how pre-training on data of different modalities inﬂuences ﬁnetuning to ofﬂine RL.
Revised: Conclusion We have examined how pre-training on data of different modalities inﬂuences ﬁnetuning to ofﬂine RL.

Original: Internal representation analysis shows that pre-trained Transformer largelychanges its representation while attaining less knowledge of the downstream task.
Revised: Internal representation analysis shows that pre-trained Transformer largelychanges its representation while attaining less knowledge of the downstream task.

Original: By the analysis ofchange in parameters, we ﬁnd that pre-trained Transformers do not change parameters that much,and the image-pre-trained model fails probably because of large gradients.
Revised: By the analysisof change in parameters, we ﬁnd that pre-trained Transformers do not change parameters that muchand that the bad performance of the image-pre-trained model might partially come from gradientclipping on the large gradient dominated by a few parameters.

Original: Fine-tuning models withno context, we ﬁnd that the language-pre-trained model can efﬁciently solve ofﬂine RL tasks evenwithout context.
Revised: Fine-tuning models with no context,we ﬁnd that the language-pre-trained model can efﬁciently solve ofﬂine RL tasks even without context.

Original: Follow-up analysis supports the hypothesis that language pre-training probably givesthe Transformer context-like information and the model exploits it to tackle the ofﬂine RL tasks.
Revised: Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformercontext-like information and the model exploits it to tackle the ofﬂine RL tasks.

Original: Discussion The ﬁnding in Section 5.6.2 of the usefulness of middle blocks is interesting, given thatthe contextual model’s middle layers contain syntactic information [63, 64] and are most transferable[20].
Revised: Discussion The ﬁnding in Section 5.6.2 of the usefulness of middle blocks is interesting, given thatthe contextual model’s middle layers contain syntactic information [63, 64] and are most transferable[20].

Original: This implies that the syntactic capability of Transformers could be related to the performanceon ofﬂine RL.
Revised: This implies that the syntactic capability of Transformers could be related to the performanceon ofﬂine RL.

Original: The role of layer norm is also noteworthy as we observe that it may characterize thebehavior in ﬁne-tuning (Sections 5.1 and 5.4).
Revised: The role of layer norm is also noteworthy as we observe that it may characterize thebehavior in ﬁne-tuning (Sections 5.1 and 5.4).

Revised: Studying the pre-training on other RL tasks is alsocritical to understanding the beneﬁt of pre-training in RL.

Original: Many studies of human evolution and developmentpoint to a close relationship between language and behavior [65, 66].
Revised: Many studies of human evolution anddevelopment point to a close relationship between language and behavior [65, 66].

Original: Hence, studying the relationshipbetween these modalities matters to create human-like intelligence that handles language.
Revised: Hence, studyingthe relationship between these modalities matters to create human-like intelligence that handleslanguage.

Original: A moredetailed analysis of this relationship is a promising research direction.
Revised: A more detailed analysis of this relationship is a promising research direction.

Original: Limitation and Impact
Revised: Limitation and Impact

Original: We consider only a random seed and a few datasets from OpenAI Gymfor analysis.
Revised: We consider only a random seed and a few datasets from OpenAI Gymfor analysis.

Original: Thus, using diverse data and more random seeds would enhance the generality of ourconclusion.
Revised: Thus, using diverse data and more random seeds would enhance the generality of ourconclusion.

Original: Also, the validity of the metrics we used should be critically examined since differentresults could be obtained by using more carefully designed ones.
Revised: Also, the validity of the metrics we used should be critically examined since differentresults could be obtained by using more carefully designed ones.

Original: Because this work is fundamentalresearch, no immediate serious negative societal impact is expected.
Revised: Because this work is fundamentalresearch, no immediate serious negative societal impact is expected.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Section
Revised: [Yes] Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Section
Revised: [Yes] Section

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Appendix
Revised: [Yes] Appendix

Original: K
Revised: K

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Appendix B
Revised: [Yes] Appendix B

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Appendix L
Revised: [Yes] Appendix L

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] All existing assets
Revised: [Yes] All existing assets

Original: we
Revised: we

Original: used are mentioned in footnote or References.
Revised: used are mentioned in footnote or References.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Appendix M
Revised: [Yes] Appendix M

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] Appendix M
Revised: [Yes] Appendix M

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did youinclude the full text of instructions given to participants and screenshots, if
Revised: you

Original: [Table]
Revised: the text

Original: applicable? [N/A]
Revised: given

Original: Did you describe any potential participant risks, with links to Institutional Review 
Revised: participants and screenshots Institutional Review participants

Original: Board ) approvals, if applicable
Revised: the

Original: [N/A]
Revised: [Table]

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: compensation

Original: [N/A]
Revised: [Table]