Original: On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
Revised: On the Effect of Pre-training for Transformer in Different Modality on Ofﬂine Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks.
Revised: We empirically investigate how pre-training on data of different modalities, such as language and vision, affects ﬁne-tuning of Transformer-based models to Mujoco ofﬂine reinforcement learning tasks.

Original: Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one.
Revised: Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in ﬁne-tuning than the randomly initialized one.

Original: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping.
Revised: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping.

Original: To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information.
Revised: To study what information the Transformer pre-trained with language data utilizes, we ﬁne-tune this model with no context provided, ﬁnding that the model learns efﬁciently even without context information.

Original: Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.
Revised: Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.

Original:  Introduction
Revised:  Introduction

Original: The past few years have witnessed the tremendous success of pre-trained Transformer-based models [1] on versatile natural language processing (NLP) tasks.
Revised: The past few years have witnessed the tremendous success of pre-trained Transformer-based models [1] on versatile natural language processing (NLP) tasks.

Original: This is because pre-training with a large corpus enables Transformer-based models to gain rich universal language representations transferable to vast downstream tasks [2, 3, 4].
Revised: This is because pre-training with a large corpus enables Transformer-based models to gain rich universal language representations transferable to vast downstream tasks [2, 3, 4].

Original: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efficiently solve even non-NLP tasks [5, 6, 7, 8, 9, 10].
Revised: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efﬁciently solve even non-NLP tasks [5, 6, 7, 8, 9, 10].

Original: Because of its versatility, it has been suggested that Transformers pre-trained with a large corpus could be called a kind of universal computation engines [5].
Revised: Because of its versatility, it has been suggested that Transformers pre-trained with a large corpus could be called a kind of universal computation engines [5].

Original: One area worthwhile attempting to study the benefits of pre-trained Transformer is reinforcement learning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].
Revised: One area worthwhile attempting to study the beneﬁts of pre-trained Transformer is reinforcement learning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].

Original: In this vein of research
Revised: In this vein of research

Original: Reid et al.
Revised: Reid et al.

Original: studied the effect of pre-training of Transformers on offline reinforcement learning (offline RL) tasks [9].
Revised: studied the effect of pre-training of Transformers on ofﬂine reinforcement learning (ofﬂine RL) tasks [9].

Original: Their study demonstrates a surprising contrast that pre-training with image significantly deteriorates performance, while that with language data does not negatively affect downstream tasks, rather it even improves performance for some datasets.
Revised: Their study demonstrates a surprising contrast that pre-training with image signiﬁcantly deteriorates performance, while that with language data does not negatively affect downstream tasks, rather it even improves performance for some datasets.

Original: However, it is not yet known how the presence or absence of pre-training and its content leads to differences in the performance of downstream tasks.
Revised: However, it is not yet known how the presence or absence of pre-training and its content leads to differences in the performance of downstream tasks.

Original: What information does/doesn’t the model pre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, and why does the model pre-trained on image data ( image-pre-trained model ) result in catastrophic performance?
Revised: What information does/doesn’t the model pre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, and why does the model pre-trained on image data ( image-pre-trained model ) result in catastrophic performance?

Original: In this paper, we approach these questions by analyzing the inside of the Transformer[Table] [Table] that is fine-tuning to Mujoco [14] offline RL tasks.
Revised: In this paper, we approach these questions by analyzing the inside of the Transformer that is ﬁne-tuning to Mujoco [14] ofﬂine RL tasks.

Original: Analyzing the internals of Transformer, we study how randomly initialized, language-pre-trained, and image-pre-trained models acquire different representations, and what the differences could be attributed to.
Revised: Analyzing the internals of Transformer, we study how randomly initialized, language-pre-trained, and image-pre-trained models acquire different representations, and what the differences could be attributed to.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We study the internal representation of pre-trained and randomly initialized Transformers and find that the pre-trained models largely change their representation, while they encode less information of data during fine-tuning.
Revised: • We study the internal representation of pre-trained and randomly initialized Transformers and ﬁnd that the pre-trained models largely change their representation, while they encode less information of data during ﬁne-tuning.

Original: • The analysis of parameter change and gradient shows that the pre-trained models do not change parameters much perhaps due to gradient confusion and that the performance of the image-pre-trained model might be partly affected by gradient clipping on the large gradient.
Revised: • The analysis of parameter change and gradient shows that the pre-trained models do not change parameters much perhaps due to gradient confusion and that the performance of the image-pre-trained model might be partly affected by gradient clipping on the large gradient.

Original: • We examine context-dependence by fine-tuning models with giving no context information, finding that the language-pre-trained Transformer efficiently learns even without context and this could be the case by replacing just a single Transformer block with a pre-trained one.
Revised: • We examine context-dependence by ﬁne-tuning models with giving no context information, ﬁnding that the language-pre-trained Transformer efﬁciently learns even without context and this could be the case by replacing just a single Transformer block with a pre-trained one.

Original: • Subsequent analysis of the change in how far away the model process the tokens of input supports the possibility that some contextual equivalent information has been positively transferred from language pre-training.
Revised: • Subsequent analysis of the change in how far away the model process the tokens of input supports the possibility that some contextual equivalent information has been positively transferred from language pre-training.

Original:  Related Works
Revised:  Related Works

Original: The study of transferability of neural representation has been one of the most important research topics in machine learning research.
Revised: The study of transferability of neural representation has been one of the most important research topics in machine learning research.

Original: A bunch of studies have investigated how pre-training influences the downstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].
Revised: A bunch of studies have investigated how pre-training inﬂuences the downstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].

Original: Recently, several studies have investigated the transferability of Transformers to various NLP tasks since understanding its high transferability is of great interest.
Revised: Recently, several studies have investigated the transferability of Transformers to various NLP tasks since understanding its high transferability is of great interest.

Original: Previous research reveals that Transformer representation encodes some syntax [22], semantics [23], cross-lingual information [24], and world knowledge [25].
Revised: Previous research reveals that Transformer representation encodes some syntax [22], semantics [23], cross-lingual information [24], and world knowledge [25].

Original: Also, it has been pointed out that Transformers obtains not only language-specific information but also more abstract inter-token dependencies [26, 27, 28].
Revised: Also, it has been pointed out that Transformers obtains not only language-speciﬁc information but also more abstract inter-token dependencies [26, 27, 28].

Original: This finding suggests that pre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.
Revised: This ﬁnding suggests that pre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.

Original: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks, reporting the versatility of the Transformer even in such multi-modal cases.
Revised: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks, reporting the versatility of the Transformer even in such multi-modal cases.

Original: Such prior works have shown that language-pre-trained Transformers can efficiently learn image classification [5], symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and more generally, sequence modeling problems [5].
Revised: Such prior works have shown that language-pre-trained Transformers can efﬁciently learn image classiﬁcation [5], symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and more generally, sequence modeling problems [5].

Original: Our study is in line with the research that examines the applicability of pre-trained Transformers on reinforcement learning.
Revised: Our study is in line with the research that examines the applicability of pre-trained Transformers on reinforcement learning.

Original: Most prior works applying pre-trained Transformer on reinforcement learning explicitly express goals, observations, or actions in a language [29,7, 8, 10].
Revised: Most prior works applying pre-trained Transformer on reinforcement learning explicitly express goals, observations, or actions in a language [29, 8, 10].

Original: On the other hand, the work of [9] applies pre-trained Transformers on the offline RL tasks that language is unrelated to (e.g. Mujoco [14]) and still reports that pre-training is helpful.
Revised: On the other hand, the work of [9] applies pre-trained Transformers on the ofﬂine RL tasks that language is unrelated to (e.g. Mujoco [14]) and still reports that pre-training is helpful.

Original: This study is interesting because it suggests that language-pre-trained Transformers seem to exploit some similarity between the structure of natural language data and trajectory data, which is more abstract and fundamental than linguistic information.
Revised: This study is interesting because it suggests that languagepre-trained Transformers seem to exploit some similarity between the structure of natural language data and trajectory data, which is more abstract and fundamental than linguistic information.

Original: Also, exploring the applicability of pre-trained Transformers, which have already been successful, is of great engineering significance since pre-training in offline RL has room for improvement compared to other modalities.
Revised: Also, exploring the applicability of pre-trained Transformers, which have already been successful, is of great engineering signiﬁcance since pre-training in ofﬂine RL has room for improvement compared to other modalities.

Original: Examining the internals of Transformers, we further delve into the findings from this study by Reid et al.
Revised: Examining the internals of Transformers, we further delve into the ﬁndings from this study by Reid et al.

Original: [9] and explores the applicability of Transformers to offline RL.
Revised: [9] and explores the applicability of Transformers to ofﬂine RL.

Original:  Background 3.1 Transformer
Revised:  Background 3.1 Transformer

Original: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .
Revised: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .

Original: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [30], and layer normalization (layer norm) [31].
Revised: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [30], and layer normalization (layer norm) [31].

Original: Denoting the linear projection of input sequence of length n to query, key, values of length m by Q ∈ R n × d Q , K ∈ R m × d K , and V ∈ R m × d V , respectively, the self-attention is Attention ( Q, K, V ) = softmax (cid:16)
Revised: Denoting the linear projection of input sequence of length n to query, key, values of length m by Q 2 R n ⇥ d Q , K 2 R m ⇥ d K , and V 2 R m ⇥ d V , respectively, the self-attention is Attention ( Q, K, V ) = softmax ⇣

Original: QK ⊤ √ d K (cid:17) V .
Revised: QK > p d K ⌘ V .

Original: We use GPTs [32], and hence the autoregressive language model, where queries head to only keys before their position.
Revised: We use GPTs [32], and hence the autoregressive language model, where queries head to only keys before their position.

Original: 3.2 Offline Reinforcement Learning and Decision Transformer
Revised: 3.2 Ofﬂine Reinforcement Learning and Decision Transformer

Original: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.
Revised: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.

Original: For ease of mathematical handling, the environment is usually described as a Markov decision process (MDP).
Revised: For ease of mathematical handling, the environment is usually described as a Markov decision process (MDP).

Original: The MDP consists of states s ∈ S , actions a ∈ A , transition probability P ( s ′ | s , a ) , and reward function r ( s , a ) : S×A → R .
Revised: The MDP consists of states s 2 S , actions a 2 A , transition probability P ( s 0 | s , a ) , and reward function r ( s , a ) : S ⇥ A !

Original:  The interaction of the agent in the environment is represented as K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) , where s t , a t , and r t are state, action, and reward at time step t .
Revised: R . The interaction of the agent in the environment is represented as K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) , where s t , a t , and r t are state, action, and reward at time step t .

Original: The goal is to find an optimal policy π ( a | s ) to maximize the expected cumulative rewards E [ (cid:80) Kt =0{+=
+} r t ] for the trajectory.
Revised: The goal is to ﬁnd an optimal policy ⇡ ( a | s ) to maximize the expected cumulative rewards E [ P Kt {+=
+} r t ] for the trajectory.

Original: Offline RL aims to achieve objectives of reinforcement learning from the trajectory data collected by some policy [33].
Revised: Ofﬂine RL aims to achieve objectives of reinforcement learning from the trajectory data collected by some policy [33].

Original: Because offline RL is purely characterized by trajectory data, several studies have proposed formulating offline RL as a sequence modeling problem [34, 35].
Revised: Because ofﬂine RL is purely characterized by trajectory data, several studies have proposed formulating ofﬂine RL as a sequence modeling problem [34, 35].

Original: Decision Transformer is a seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoder of BERT [2]) to solve such a sequence modeling problem [35].
Revised: Decision Transformer is a seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoder of BERT [2]) to solve such a sequence modeling problem [35].

Original: In Decision Transformer, input data is a sequence τ =
Revised: In Decision Transformer, input data is a sequence ⌧ =

Original: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-go ˆ R t =[-(cid:80) t − 1 i =
-]
Revised: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-go ˆ R t =[-(cid:80) t − 1 i =
-]

Original:  r i is the cumulative reward until time step t .
Revised: P Ki = t r i is the cumulative reward from time step t .

Original: The previous studies [9, 35] employ the above problem set up to discuss the applicability of Transformer for offline RL.
Revised: The previous studies [9, 35] employ the above problem set up to discuss the applicability of Transformer for ofﬂine RL.

Original: We also follow these studies and use the setup for our analysis.
Revised: We also follow these studies and use the setup for our analysis.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Unless otherwise noted, all of our experimental setups, including model configuration, dataset, and hyperparameters follow the previous work [9] 2 .
Revised: Unless otherwise noted, all of our experimental setups, including model conﬁguration, dataset, and hyperparameters follow the previous work [9] 2 .

Original: The sanity check that our trained models achieve comparable performance with the previous work is shown in Table 1.
Revised: The sanity check that our trained models achieve comparable performance with the previous work is shown in Table 1.

Original: Details are in Appendix A.
Revised: Details are in Appendix A.

Original: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initialized model ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect of pre-training.
Revised: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initialized model ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect of pre-training.

Original: Pre-trained models are from the Transformers library from HuggingFace [36].
Revised: Pre-trained models are from the Transformers library from HuggingFace [36].

Original: Like the previous study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .
Revised: Like the previous study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .

Original: For offline RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [37] provided by D4RL [38], an offline RL dataset library.
Revised: For ofﬂine RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [37] provided by D4RL [38], an ofﬂine RL dataset library.

Original: Specifically, we employ medium datasets of HalfCheetah , Walker2d , and Hopper environments.
Revised: Speciﬁcally, we employ medium datasets of HalfCheetah , Walker2d , and Hopper environments.

Original: Unless otherwise specified, we show the result for Hopper-medium .
Revised: Unless otherwise speciﬁed, we show the result for Hopper-medium .

Original: We put the results for other environments in Appendix J.
Revised: We put the results for other environments in Appendix J.

Original: Following the previous study [9], we train the models for 40 epochs, each of which consists ofsteps.
Revised: Following the previous study [9], we train the models for 40 epochs, each of which consists ofsteps.

Original: In the following sections, we refer to the 40th epoch checkpoint as the post-fine-tuning model and the 0th epoch model (the initial state) as the pre- fine-tuning model .
Revised: In the following sections, we refer to the 40th epoch checkpoint as the post-ﬁne-tuning model and the 0th epoch model (the initial state) as the pre- ﬁne-tuning model .

Original: The training details, including hyperparameters and the optimizer, are detailed in Appendix B.
Revised: The training details, including hyperparameters and the optimizer, are detailed in Appendix B.

Original: [Table]
Revised: [Table]

Original:  Results and Analysis5.1 Activation Similarity
Revised:  Results and Analysis

Revised: Activation Similarity

Original: We first study how the pre-trained and randomly initialized models shape the representation in finetuning.
Revised: We ﬁrst study how the pre-trained and randomly initialized models shape the representation in ﬁnetuning.

Original: To that end, we compare the centered kernel alignment (CKA) [39] of each layer’s activation between pre and post-fine-tuning, investigating how each layer’s representation changes by learning offline RL data 4 .
Revised: To that end, we compare the centered kernel alignment (CKA) [39] of each layer’s activation between pre and post-ﬁne-tuning, investigating how each layer’s representation changes by learning ofﬂine RL data 4 .

Original: The CKA has been used in various studies and has provided numerous insights into the understanding of neural networks [40, 41, 42, 43, 44].
Revised: The CKA has been used in various studies and has provided numerous insights into the understanding of neural networks [40, 41, 42, 43, 44].

Original: In particular, we compute the linear CKA, following previous studies [39, 45], for a subset of the dataset.
Revised: In particular, we compute the linear CKA, following previous studies [39, 45], for a subset of the dataset.

Original: We use the activation of each layer from the last time step of the context for return-to-go, state, and action, respectively.
Revised: We use the activation of each layer from the last time step of the context for return-to-go, state, and action, respectively.

Original: We show the result of state, putting the results for the others in Appendix J.1.
Revised: We show the result of state, putting the results for the others in Appendix J.1.

Original: Following the previous study [43], we compute CKA of activation not only for Transformer block output but for all internal activation of the Transformer blocks.
Revised: Following the previous study [43], we compute CKA of activation not only for Transformer block output but for all internal activation of the Transformer blocks.

Original: The definition of the CKA and the details are in Appendices D.2 and D.1.
Revised: The deﬁnition of the CKA and the details are in Appendices D.2 and D.1.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1 shows the result.
Revised: 1 shows the result.

Original: The y-axis is each layer’s CKA between representation at pre and post-finetuning.
Revised: The y-axis is each layer’s CKA between representation at pre and post-ﬁnetuning.

Original: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twice the others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPT and align the x-axis for different models for comparison.
Revised: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twice the others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPT and align the x-axis for different models for comparison.

Original: Higher CKA means higher similarity.
Revised: Higher CKA means higher similarity.

Original: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layers except for a few layers, indicating that pre-trained models largely change representation.
Revised: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layers except for a few layers, indicating that pre-trained models largely change representation.

Original: On the other hand, for the randomly initialized model (Random Init), the representations have a higher similarity to the initial state from the middle to the shallow layers.
Revised: On the other hand, for the randomly initialized model (Random Init), the representations have a higher similarity to the initial state from the middle to the shallow layers.

Original: Also, the CKAs of the randomly initialized model seem to be higher than pre-trained models across layers.
Revised: Also, the CKAs of the randomly initialized model seem to be higher than pre-trained models across layers.

Original: These observations indicate that the randomly initialized and pre-trained models structure their representations in very different ways.
Revised: These observations indicate that the randomly initialized and pre-trained models structure their representations in very different ways.

Original: Further support for this conclusion is shown in Appendix D.4.
Revised: Further support for this conclusion is shown in Appendix D.4.

Original: To identify which layers have relatively higher CKA for the randomly initialized model, we obtain layer names with CKA values above a threshold (dashed line at 0.23 in Fig.
Revised: To identify which layers have relatively higher CKA for the randomly initialized model, we obtain layer names with CKA values above a threshold (dashed line at 0.23 in Fig.

Original: 1 (a)).
Revised: 1 (a)).

Original: Then, we find that most of them are layer normalization layers (the list of the layer names are put in Appendix
Revised: Then, we ﬁnd that most of them are layer normalization layers (the list of the layer names are put in Appendix

Original: D.3).
Revised: D.3).

Original: Therefore, we can speculate that the difference in CKA similarity of layer normalization can be related to the effect of pre-training.
Revised: Therefore, we can speculate that the difference in CKA similarity of layer normalization can be related to the effect of pre-training.

Original: This possibility could be supported by previous research [5] that reports that layer norm parameters are the most important to fine-tune.
Revised: This possibility could be supported by previous research [5] that reports that layer norm parameters are the most important to ﬁne-tune.

Original: In uni-modal cases, where the statistical nature of the input data does not change significantly, previous studies have a consensus that the shallow layer acquires a more universal representation and the deep layer acquires a task-specific representation for a wide range of architectures and tasks [46, 47, 48, 49, 50, 39, 51, 42, 44].
Revised: In uni-modal cases, where the statistical nature of the input data does not change signiﬁcantly, previous studies have a consensus that the shallow layer acquires a more universal representation and the deep layer acquires a task-speciﬁc representation for a wide range of architectures and tasks [46, 47, 48, 49, 50, 39, 51, 42, 44].

Original: The finding that the language-pre-trained Transformer performs as well as or better than the baseline but largely changes the representation suggests that in multi-modal cases, something different may be happening in forming the representation than in prior studies.
Revised: The ﬁnding that the language-pre-trained Transformer performs as well as or better than the baseline but largely changes the representation suggests that in multi-modal cases, something different may be happening in forming the representation than in prior studies.

Original: 5.2 Mutual Information Between Hidden Representation and Data
Revised: 5.2 Mutual Information Between Hidden Representation and Data

Original: In the previous section, we observe that pre-trained models drastically change their internal representation during fine-tuning, attaining largely different representations.
Revised: In the previous section, we observe that pre-trained models drastically change their internal representation during ﬁne-tuning, attaining largely different representations.

Original: One possible consequence of this observation is that the pre-trained model may have adapted better to the data in the downstream tasks.
Revised: One possible consequence of this observation is that the pre-trained model may have adapted better to the data in the downstream tasks.

Original: To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models.
Revised: To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models.

Original: Mutual information is a well-known measure of the mutual dependence between the two random variables.
Revised: Mutual information is a well-known measure of the mutual dependence between the two random variables.

Original: Many previous studies have used this metric to investigate the extent to which neural representation encodes input and label information and provided profound insight into how neural networks work [52, 53, 54, 55, 56]
Revised: Many previous studies have used this metric to investigate the extent to which neural representation encodes input and label information and provided profound insight into how neural networks work [52, 53, 54, 55, 56].

Original: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .
Revised: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .

Original: We employ Mutual Information
Revised: We employ Mutual Information

Original: Neural Estimation (MINE) [57] to estimate mutual information since this method can estimate mutual information even when random vectors are continuous and have different dimensions 5 .
Revised: Neural Estimation (MINE) [57] to estimate mutual information since this method can estimate mutual information even when random vectors are continuous and have different dimensions 5 .

Original: Since
Revised: Since

Original: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual action a t , we define the label as Y = a t .
Revised: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual action a t , we deﬁne the label as Y = a t .

Original: Unlike non-sequential models, the causal Transformer exploits past context information.
Revised: Unlike non-sequential models, the causal Transformer exploits past context information.

Original: Thus, denoting the input token except for a K by x t and the internal activation of Transformer block l of the input by T l ( x t ) , we compute ˆ
Revised: Thus, denoting the input token except for a K by x t and the internal activation of Transformer block l of the input by T l ( x t ) , we compute ˆ

Original: I ( x t ; T l ( s K ))
Revised: I ( x t ; T l ( s K ))

Original: as ˆ I ( X ; T ) and ˆ I ( a K ; T l ( x t )) as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .
Revised: as ˆ I ( X ; T ) and ˆ I ( a K ; T l ( x t )) as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .

Original: We calculate these values for the shallow, middle, and deep Transformer blocks, respectively.
Revised: We calculate these values for the shallow, middle, and deep Transformer blocks, respectively.

Original: We show the result of the middle block here and leave the remaining in Appendix J.2.
Revised: We show the result of the middle block here and leave the remaining in Appendix J.2.

Original: The other details are described in Appendix E.1.
Revised: The other details are described in Appendix E.1.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 is the result, where Fig.
Revised: 2 is the result, where Fig.

Original: 2 (a) and (b) are results of ˆ
Revised: 2 (a) and (b) are results of ˆ

Original: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.
Revised: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.

Original: The markers triangle , circle , and cross represent return-to-go, state, and action, respectively.
Revised: The markers triangle , circle , and cross represent return-to-go, state, and action, respectively.

Original: The x-axis is the position of tokens in the context and the y-axis is the estimated value of mutual information.
Revised: The x-axis is the position of tokens in the context and the y-axis is the estimated value of mutual information.

Original: We observe that the randomly initialized model consistently has higher mutual information both for input and label than pre-trained models.
Revised: We observe that the randomly initialized model consistently has higher mutual information both for input and label than pre-trained models.

Original: Given the catastrophic performance of iGPT, it is understandable that iGPT representations have almost no information.
Revised: Given the catastrophic performance of iGPT, it is understandable that iGPT representations have almost no information.

Original: The result comparing GPT to random initialization indicates that the changed representation observed in Section 5.1 does not encode more information about the input or the label than the randomly initialized model.
Revised: The result comparing GPT to random initialization indicates that the changed representation observed in Section 5.1 does not encode more information about the input or the label than the randomly initialized model.

Original: That is, the language-pre-trained model does well probably not because it acquires as much or more information about the data as the random initialization.
Revised: That is, the language-pre-trained model does well probably not because it acquires as much or more information about the data as the random initialization.

Original: This result suggests that although the representation has changed, there remains something unchanged, and the language-pre-trained model may leverage it to solve the offline RL task.
Revised: This result suggests that although the representation has changed, there remains something unchanged, and the language-pre-trained model may leverage it to solve the ofﬂine RL task.

Original: Note that this might be affected by limitations of mutual information.
Revised: Note that this might be affected by limitations of mutual information.

Original: We also notice that for label the language-pre-trained model has consistently higher mutual information for a specific token type (return-to-go), while that is not always the case for the randomly initialized model.
Revised: We also notice that for label the language-pre-trained model has consistently higher mutual information for a speciﬁc token type (return-to-go), while that is not always the case for the randomly initialized model.

Original: This result implies that the language-pre-trained model may process the information of a specific token type at each layer for prediction.
Revised: This result implies that the language-pre-trained model may process the information of a speciﬁc token type at each layer for prediction.

Original: 5.3 Parameter Similarity
Revised: 5.3 Parameter Similarity

Original: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploits some pre-acquired information.
Revised: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploits some pre-acquired information.

Original: To investigate this issue on a finer scale, we turn our eye from representation to parameter.
Revised: To investigate this issue on a ﬁner scale, we turn our eye from representation to parameter.

Original: We conduct parameter-level similarity analysis and investigate to what extent parameter change between pre and post-fine-tuning.
Revised: We conduct parameter-level similarity analysis and investigate to what extent parameter change between pre and post-ﬁne-tuning.

Original: In concrete, we compute l 2 distance and cosine similarity of parameters between pre and post-fine-tuning.
Revised: In concrete, we compute l 2 distance and cosine similarity of parameters between pre and post-ﬁne-tuning.

Original: The details are in Appendix F.1.
Revised: The details are in Appendix F.1.

Original: The result for l 2 distance is shown in Fig.
Revised: The result for l 2 distance is shown in Fig.

Original: 3 and for cosine similarity in Fig.
Revised: 3 and for cosine similarity in Fig.

Original: 4, where the top is the result for GPT2, the middle is for iGPT, and the bottom is for random initialization.
Revised: 4, where the top is the result for GPT2, the middle is for iGPT, and the bottom is for random initialization.

Original: The x-axis is the parameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layer parameters.
Revised: The x-axis is the parameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layer parameters.

Original: For visibility, we do not show the labels for ticks and put them in Appendix F.2.[Figure] || [Figure]
Revised: For visibility, we do not show the labels for ticks and put them in Appendix F.2.

Original: [Figure] Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models, meaning that the parameters of pre-trained models do not change that much.
Revised: Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models, meaning that the parameters of pre-trained models do not change that much.

Original: At first glance, this appears to contradict the results of Section 5.1.
Revised: At ﬁrst glance, this appears to contradict the results of Section 5.1.

Original: However, prior research has reported that most of the information in Transformers propagates through skip connections [43].
Revised: However, prior research has reported that most of the information in Transformers propagates through skip connections [43].

Original: This means that, as long as the parameters of the shallow layers change, the representation of each layer may change significantly without changing the parameters of each layer.
Revised: This means that, as long as the parameters of the shallow layers change, the representation of each layer may change signiﬁcantly without changing the parameters of each layer.

Original: We indeed observe that the shallow layer l 2 distance of the pre-trained model shows relatively large changes.
Revised: We indeed observe that the shallow layer l 2 distance of the pre-trained model shows relatively large changes.

Original: The existence of unchanged parameters means that knowledge acquired through pre-training could be stored in those parameters
Revised: The existence of unchanged parameters means that knowledge acquired through pre-training could be stored in those parameters.

Original: [Figure] 5.4 Gradient Analysis
Revised: 5.4 Gradient Analysis

Original: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.
Revised: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.

Original: We dig into this issue, investigating why GPT2 and iGPT do not change parameters.
Revised: We dig into this issue, investigating why GPT2 and iGPT do not change parameters.

Original: To this end, we compare the gradient norm and gradient confusion [58] at the early phase of the training ( 1 st epoch), discussing the training difficulty.
Revised: To this end, we compare the gradient norm and gradient confusion [58] at the early phase of the training ( 1 st epoch), discussing the training difﬁculty.

Original: When parameter gradients of loss for two different samples are negatively correlated, we say that there is gradient confusion.
Revised: When parameter gradients of loss for two different samples are negatively correlated, we say that there is gradient confusion.

Original: Empirically, we can measure this by the minimum gradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusion is low.
Revised: Empirically, we can measure this by the minimum gradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusion is low.

Original: The previous study showed that training is easier when gradient confusion is low [58].
Revised: The previous study showed that training is easier when gradient confusion is low [58].

Original: Because gradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), we compute the values for clipped gradients.
Revised: Because gradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), we compute the values for clipped gradients.

Original: The details are described in Appendix G.1.
Revised: The details are described in Appendix G.1.

Original: Fig.
Revised: Fig.

Original: 5 is the result for the gradient confusion and Fig.
Revised: 5 is the result for the gradient confusion and Fig.

Original: 6 is that for gradient norm.
Revised: 6 is that for gradient norm.

Original: Fig.
Revised: Fig.

Original: 5 shows that the minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initialized model.
Revised: 5 shows that the minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initialized model.

Original: This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model.
Revised: This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model.

Original: Fig.
Revised: Fig.

Original: 6, on the other hand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of the others is below the bound.That is, the differences in the magnitude of the gradient are collapsed for iGPT, reducing the informational value of the gradients. GPT iGPT Random Init 1.
Revised: 6, on the other hand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of the others is below the bound.

Original: M i n o f c o ss i m ( 1 , 2 )

Original: GPT iGPT Random Init 0. C li pped G r ad i en t
Revised: That is, the differences in the magnitude of the gradient are collapsed for iGPT, reducing the informational value of the gradients.

Original: N o r m [Figure] others To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm of each parameter set.
Revised: To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm of each parameter set.

Original: We calculate the gradient norm ratio of weights and biases of all parameters in all Transformer blocks.
Revised: We calculate the gradient norm ratio of weights and biases of all parameters in all Transformer blocks.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 7: for visibility, we show the parameters of the first layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remaining results together as others .
Revised: 7: for visibility, we show the parameters of the ﬁrst layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remaining results together as others .

Original: The full result is shown in Appendix G.2.
Revised: The full result is shown in Appendix G.2.

Original: We observe that weights and biases of the first layer norm module dominate the norm.
Revised: We observe that weights and biases of the ﬁrst layer norm module dominate the norm.

Original: This result implies that the difficulty of training iGPT might partially come from the use of gradient clipping on large gradients dominated by few parameters.
Revised: This result implies that the difﬁculty of training iGPT might partially come from the use of gradient clipping on large gradients dominated by few parameters.

Original: A complementary analysis is described in Appendix G.3.
Revised: A complementary analysis is described in Appendix G.3.

Original: 5.5 Fine-Tuning with No Context Information
Revised: 5.5 Fine-Tuning with No Context Information

Original: In the sections up to this point, we find the possibility that the language-pre-trained model may exploit some pre-acquired information.
Revised: In the sections up to this point, we ﬁnd the possibility that the language-pre-trained model may exploit some pre-acquired information.

Original: Then what information does the language-pre-trained model exploit to solve the tasks?
Revised: Then what information does the language-pre-trained model exploit to solve the tasks?

Original: One possibility is the ability to handle context because Transformer is skilled at handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].
Revised: One possibility is the ability to handle context because Transformer is skilled at handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].

Original: Therefore, we train the language-pre-trained and the randomly initialized model with no access to the context ( K = 1 ) to dig into this possibility.
Revised: Therefore, we train the language-pre-trained and the randomly initialized model with no access to the context ( K = 1 ) to dig into this possibility.

Original: If a model learns efficiently without context, the model is already likely to have context-equivalent information.
Revised: If a model learns efﬁciently without context, the model is already likely to have context-equivalent information.

Original: Details of experiments are in Appendix H.1.
Revised: Details of experiments are in Appendix H.1.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The mean return throughout training time is shown in Fig.
Revised: The mean return throughout training time is shown in Fig.

Original: 8 and the normalized scores of return 6 are shown in Table 2.
Revised: 8 and the normalized scores of return 6 are shown in Table 2.

Original: The solid lines in Fig.
Revised: The solid lines in Fig.

Original: 8 are results without context and the dashed lines for reference are those with context.
Revised: 8 are results without context and the dashed lines for reference are those with context.

Original: From Fig.
Revised: From Fig.

Original: 8, we observe that when the context is not provided ( K = 1 ), the GPT reaches the high mean return much faster than the randomly initialized model.
Revised: 8, we observe that when the context is not provided ( K = 1 ), the GPT reaches the high mean return much faster than the randomly initialized model.

Original: Furthermore, Tablereveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), while the language-pre-trained model consistently achieves a comparable performance with that of the baseline result ( K = 20 ).
Revised: Furthermore, Tablereveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), while the language-pre-trained model consistently achieves a comparable performance with that of the baseline result ( K = 20 ).

Original: That is, we observe that the language-pre-trained Transformer consistently performs more efficiently when no context is provided.
Revised: That is, we observe that the language-pre-trained Transformer consistently performs more efﬁciently when no context is provided.

Original: This is somewhat surprising, given that previous studies have pointed out that Decision Transformer performs significantly worse with no context [35].
Revised: This is somewhat surprising, given that previous studies have pointed out that Decision Transformer performs signiﬁcantly worse with no context [35].

Original: The result that the language-pre-trained Transformer learns efficiently as well withoutFollowing the previous studies [38, 35, 9], we report normalized scores: 100 ×
Revised: The result that the language-pre-trained Transformer learns efﬁciently as well without

Original: [Table]
Revised: [Table]

Original: context as it does with context suggests that the model may have already acquired some kind of context-like information through pre-training and is utilizing it to solve the task.
Revised: context as it does with context suggests that the model may have already acquired some kind of context-like information through pre-training and is utilizing it to solve the task.

Original: 5.6 More In-Depth Analysis of Context Dependence 5.6.1 Replacement by the Pre-Trained Block
Revised: 5.6 More In-Depth Analysis of Context Dependence 5.6.1 Replacement by the Pre-Trained Block

Original: In the previous sections [Figure] we find that the language-pre-trained Transformer exploits context-related information to solve the task.
Revised: In the previous sections, we ﬁnd that the language-pre-trained Transformer exploits context-related information to solve the task.

Original: To identify which block contains the useful information [Figure] we conduct a block replacement experiment, where we replace a Transformer block of the randomly initialized model with that of the language-pre-trained model and then fine-tune the model without context.
Revised: To identify which block contains the useful information, we conduct a block replacement experiment, where we replace a Transformer block of the randomly initialized model with that of the language-pre-trained model and then ﬁne-tune the model without context.

Original: If replacement of a block does not help training, the block probably did not learn effective information in pre-training.
Revised: If replacement of a block does not help training, the block probably did not learn effective information in pre-training.

Original: Because we observe in Section 5.5 that the early phase characterizes the difference of the models, we focus on the epochs up to 10th epoch.
Revised: Because we observe in Section 5.5 that the early phase characterizes the difference of the models, we focus on the epochs up to 10th epoch.

Original: We explain the other details in Appendix I.1.
Revised: We explain the other details in Appendix I.1.1.

Original: [Figure]
Revised: [Figure]

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 9, where (a) is the prediction error of action and (b) is the mean return.
Revised: 9, where (a) is the prediction error of action and (b) is the mean return.

Original: To get a rough trend, we take the exponential moving average.
Revised: To get a rough trend, we take the exponential moving average.

Original: We observe that only when block 1 is replaced with the pre-trained block, does the action error (Fig.
Revised: We observe that only when block 1 is replaced with the pre-trained block, does the action error (Fig.

Original: (a)) and the mean return (Fig.
Revised: (a)) and the mean return (Fig.

Original: 9 (b)) behave similarly to the randomly initialized model.
Revised: 9 (b)) behave similarly to the randomly initialized model.

Original: Thus, we conclude that all blocks other than block 1 contain context-equivalent useful information to solve tasks.
Revised: Thus, we conclude that all blocks other than block 1 contain context-equivalent useful information to solve tasks.

Original: The findings that replacing a single block with the pre-trained one improves learning efficiency and that effective information is distributed across layers alone are surprising enough.
Revised: The ﬁndings that replacing a single block with the pre-trained one improves learning efﬁciency and that effective information is distributed across layers alone are surprising enough.

Original: This could be related to the dominant role of skip connections for information propagation [43] or the nature of non-monotonic change in the content each layer processes [20] in Transformers.
Revised: This could be related to the dominant role of skip connections for information propagation [43] or the nature of non-monotonic change in the content each layer processes [20] in Transformers.

Original: We show the result for Hopper since the trend explained above is most pronounced.
Revised: We show the result for Hopper since the trend explained above is most pronounced.

Original: For other environments, although the similarity between the random model and block1 is much weaker, the observation that block 1 is not good is consistent.
Revised: For other environments, although the similarity between the random model and block1 is much weaker, the observation that block 1 is not good is consistent.

Original: Results for other environments are in Appendix J.6.1.
Revised: Results for other environments are in Appendix J.6.1.

Original: 5.6.2 Attention Distance Analysis
Revised: 5.6.2 Attention Distance Analysis

Original: In previous sections, we find that the information improving learning efficiency is preserved in all blocks only in the varying amount and that is probably context-related.
Revised: In previous sections, we ﬁnd that the information improving learning efﬁciency is preserved in all blocks only in the varying amount and that is probably context-related.

Original: To further confirm that the context-like information is indeed preserved, we investigate how the model processes context by calculating the attention distance [59] of each Transformer block for the result of Section 5.5.
Revised: To further conﬁrm that the context-like information is indeed preserved, we investigate how the model processes context by calculating the attention distance [59] of each Transformer block for the result of Section 5.5.

Original: Attention distance is the average distance between key and query weighted by attention weight.
Revised: Attention distance is the average distance between key and query weighted by attention weight.

Original: Previous studies successfully used this measure to reveal how locally Transformers process context [43, 59].
Revised: Previous studies successfully used this measure to reveal how locally Transformers process context [43, 59].

Original: Attention distance will elucidate how far away each model processes the tokens of input.
Revised: Attention distance will elucidate how far away each model processes the tokens of input.

Original: In particular, we compute the gap ( | d att ( epoch )
Revised: In particular, we compute the gap ( | d att ( epoch )

Original: − d att (0) | ) between attention distance at the initial state ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves the way to utilize context.
Revised:  d att (0) | ) between attention distance at the initial state ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves the way to utilize context.

Original: If the gap is smaller, it means that the attention distance is more preserved.
Revised: If the gap is smaller, it means that the attention distance is more preserved.

Original: To clarify the difference between the language-pre-trained model and the randomly initialized model, we compare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initialized model still produces a low return, while the GPT2 achieves a high return (Fig.
Revised: To clarify the difference between the language-pre-trained model and the randomly initialized model, we compare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initialized model still produces a low return, while the GPT2 achieves a high return (Fig.

Original: The detail of the setup and results for other epochs are shown in Appendices I.2.1 and J.6.2.
Revised: The detail of the setup and results for other epochs are shown in Appendices I.2.1 and J.6.2.

Original: [Figure]
Revised: [Figure]

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 10, where the x-axis is the Transformer block index and the y-axis is the gap between attention distances.
Revised: 10, where the x-axis is the Transformer block index and the y-axis is the gap between attention distances.

Original: Fig.
Revised: Fig.

Original: 10 (a), shows that the attention gaps of GPT2 except for block 1 are small, indicating that attention distances are kept from the pre-training in these blocks.
Revised: 10 (a), shows that the attention gaps of GPT2 except for block 1 are small, indicating that attention distances are kept from the pre-training in these blocks.

Original: Taken together with the finding of Section 5.6.1 that blocks other than block 1 improve learning efficiency, these preserved attention distances are likely to positively affect fine-tuning.
Revised: Taken together with the ﬁnding of Section 5.6.1 that blocks other than block 1 improve learning efﬁciency, these preserved attention distances are likely to positively affect ﬁne-tuning.

Original: This result further supports the hypothesis that language-pre-training brings the model context-like information and the use of this information allows efficient learning.
Revised: This result further supports the hypothesis that language-pre-training brings the model context-like information and the use of this information allows efﬁcient learning.

Original: We also notice the outstanding contrast that gaps of language-pre-trained models are almost zero except for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model are large.
Revised: We also notice the outstanding contrast that gaps of language-pre-trained models are almost zero except for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model are large.

Original: This trend is clearer from middle to deep blocks (3 to 10) for all environments.
Revised: This trend is clearer from middle to deep blocks (3 to 10) for all environments.

Original: Turning an eye to Fig.
Revised: Turning an eye to Fig.

Original: 9 (a), we notice that the action error of these blocks decreases faster than the other block (0 to 2 and 11).
Revised: 9 (a), we notice that the action error of these blocks decreases faster than the other block (0 to 2 and 11).

Original: Thus, we conclude that the pre-trained context-like information could help the training by enabling the model to predict action more accurately.
Revised: Thus, we conclude that the pre-trained context-like information could help the training by enabling the model to predict action more accurately.

Original: We summarize findings of Section 5 with particular attention to the implication for performance:
Revised: We summarize ﬁndings of Section 5 with particular attention to the implication for performance:

Original: • 5.1: Re-using representation is not the cause of the good performance of GPT2.
Revised: • 5.1: Re-using representation is not the cause of the good performance of GPT2.

Original: • 5.2: Fitting to data better is not the cause of the good performance of GPT2.
Revised: • 5.2: Fitting to data better is not the cause of the good performance of GPT2.

Original: • 5.3: The good performance of GPT2 might come from some unchanged parameters.
Revised: • 5.3: The good performance of GPT2 might come from some unchanged parameters.

Original: • 5.4: Gradient clipping and large gradient might be a cause of the bad performance of iGPT.
Revised: • 5.4: Gradient clipping and large gradient might be a cause of the bad performance of iGPT.

Original: • 5.5: GPT2 can learn efficiently even without the context provided.
Revised: • 5.5: GPT2 can learn efﬁciently even without the context provided.

Original: Even a single pre-trained Transformer block makes training more efficient.
Revised: Even a single pre-trained Transformer block makes training more efﬁcient.

Original: A cause of the good performance of GPT2 is a pre-acquired way to use context.
Revised: A cause of the good performance of GPT2 is a pre-acquired way to use context.

Original:  Discussion
Revised:  Discussion

Original: Conclusion We have examined how pre-training on data of different modalities influences finetuning to offline RL.
Revised: Conclusion We have examined how pre-training on data of different modalities inﬂuences ﬁnetuning to ofﬂine RL.

Original: Internal representation analysis shows that pre-trained Transformer largely changes its representation while attaining less knowledge of the downstream task.
Revised: Internal representation analysis shows that pre-trained Transformer largely changes its representation while attaining less knowledge of the downstream task.

Original: By the analysis of change in parameters, we find that pre-trained Transformers do not change parameters that much and that the bad performance of the image-pre-trained model might partially come from gradient clipping on the large gradient dominated by a few parameters.
Revised: By the analysis of change in parameters, we ﬁnd that pre-trained Transformers do not change parameters that much and that the bad performance of the image-pre-trained model might partially come from gradient clipping on the large gradient dominated by a few parameters.

Original: Fine-tuning models with no context , we find that the language-pre-trained model can efficiently solve offline RL tasks even without context.
Revised: Fine-tuning models with no context, we ﬁnd that the language-pre-trained model can efﬁciently solve ofﬂine RL tasks even without context.

Original: Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformer context-like information and the model exploits it to tackle the offline RL tasks.
Revised: Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformer context-like information and the model exploits it to tackle the ofﬂine RL tasks.

Original: Discussion The finding in Section 5.6.2 of the usefulness of middle blocks is interesting, given that the contextual model’s middle layers contain syntactic information [60, 61] and are most transferable [20].
Revised: Discussion The ﬁnding in Section 5.6.2 of the usefulness of middle blocks is interesting, given that the contextual model’s middle layers contain syntactic information [60, 61] and are most transferable [20].

Original: This implies that the syntactic capability of Transformers could be related to the performance on offline RL.
Revised: This implies that the syntactic capability of Transformers could be related to the performance on ofﬂine RL.

Original: The role of layer norm is also noteworthy as we observe that it may characterize the behavior in fine-tuning (Sections 5.1 and 5.4).
Revised: The role of layer norm is also noteworthy as we observe that it may characterize the behavior in ﬁne-tuning (Sections 5.1 and 5.4).

Original: Studying the pre-training on other RL tasks is also critical to understanding the benefit of pre-training in RL.
Revised: Studying the pre-training on other RL tasks is also critical to understanding the beneﬁt of pre-training in RL.

Original: Many studies of human evolution and development point to a close relationship between language and behavior [62, 63].
Revised: Many studies of human evolution and development point to a close relationship between language and behavior [62, 63].

Original: Hence, studying the relationship between these modalities matters to create human-like intelligence that handles language.
Revised: Hence, studying the relationship between these modalities matters to create human-like intelligence that handles language.

Original: A more detailed analysis of this relationship is a promising research direction.
Revised: A more detailed analysis of this relationship is a promising research direction.

Original: Limitation and Impact
Revised: Limitation and Impact

Original: We consider only a random seed and a few datasets from OpenAI Gym for analysis.
Revised: We consider only a random seed and a few datasets from OpenAI Gym for analysis.

Original: Thus, using diverse data is important and checking if the claim holds with the average results with many more seeds is crucial.Also, the validity of the metrics we used should be critically examined since different results could be obtained by using more carefully designed ones. Because this work is fundamental research, no immediate serious negative societal impact is expected. Acknowledgments and Disclosure of Funding
Revised: Thus, using diverse data is important and checking if the claim holds with the average results with many more seeds is crucial.

Original: We thank DEEPCORE Inc. for providing the computational resource.

Original: G Gradient Analysis

Original: G.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Gradient Norm for Each Parameter . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.3 Analysis of the Effect of Gradient Clipping . . . . . . . . . . . . . . . . . . . .

Original: [Table]

Original: H.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context .

Original: I More

Original: [Table]

Original: -Depth Analysis

Original: [Table]

Original: Context Dependence

Original: I.

Original: Replacement by the Pre-Trained Block . . . . . . . . . . . . . . . . . . . . . .

Original: I.

Original: Attention Distance Analysis . .

Original: [Table]

Original: Results for Other Conditions

Original: J.

Original: Activation Similarity . .

Original: J.

Original: Mutual Information Between Hidden Representation and Data . . . . . . . . . .

Original: J.

Original: Parameter Similarity . .

Original: J.

Original: Gradient Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: J.

Original: Fine-Tuning with No Context Information . . . . . . . . . . . . . . . . . . . .

Original: J.

Original: More In-Depth Analysis of Context Dependence . . . . . . . . . . . . . . . . .

Original: K Details for Computation

Original: [Table]

Original: Used for

Original: [Table]

Original: L.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: L.2 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: M

Original: Sections in Section

Original: N

Original: Relationship Between Parameter Change and Representation Change

Original: O

Original: Return

Original: A Fine-Tuning Result for Sanity Check

Original: We compare our performance results to those of previous studies to ensure that they are not far off from the results of the prior study [9], the result of which is shown in Table 1.

Original: Following the previous studies [38, 35, 9], we report the normalized score of mean return: 100 × score − random score expert score − random score , where random score is mean return generated by a random policy and expert score is generated by a policy trained by Soft Actor-Critic [64].

Original: Mean return is the sum of rewards averaged over trajectory.

Original: For further details of the datasets and metrics, please refer to the paper that proposes D4RL [38].

Original: Although the previous work [9] used several techniques to improve the performance, e.g. language model co-training, the extension of positional embedding, and similarity encouragement, we do not employ any of these techniques so that we study the pure effect of pre-training.

Original: The training details are described in Appendix B

Original: The result for the previous work is the average and standard deviation of three random seeds, while our result is those of two random seeds.

Original: The aim of this comparison is just to confirm that our result is not too pathological, checking soundness with two seeds would be valid enough.

Original: For reference, we also include the results of the Decision Transformer (DT) since the randomly initialized model (Random Init) is a large Decision Transformer.

Original: Following the previous study [9], which says that it conducted early stopping, we show the result of the best checkpoint in 40 epoch checkpoints.

Original: The result shows that our result is not too far away from the previous work, ensuring that the models of the subject of our analysis are valid enough.

Original: B Training Details

Original: Hyperparameters are determined following previous study [9].

Original: The hyperparameters and other setups for fine-tuning the model are summarized in Table 3.

Original: The number of layers is 12 for GPT2 and randomly initialized model and 24 for iGPT.

Original: The other setup not specified, including β of Adam optimizer [65] and model initialization scheme follows Pytorch default ones or the default values of optional arguments in the scripts that our experiment is based on 8 .

Original: We do not do any pre-training ourselves, but use publicly available pre-trained models explained in Section 3.

Original: Each model is trained to minimize the mean squared loss between its predicted actions ˆ a t and the true actions a t .

Original: To get the big picture of how fine-tuning of GPT-based models to offline RL data is conducted, please refer to the pseudo-code of the previous work [35].

Original: The D4RL mujoco task used as offline RL data for our analysis has expert , medium , medium-replay , medium-expert , and random datasets.

Original: The expert dataset is the one trained by Soft Actor-Critic [64], medium is the one partially trained by Soft Actor-Critic and was stopped early, medium-replay is the one accumulated in the replay buffer before the model reached medium ’s level, and medium-expert is a mixture of medium and expert results.

Original: The random dataset is the trajectory collected by the random policy.

Original: We conducted our experiments on the medium data set since it was used by the previous study [9] and our analysis is based on the observations of this previous study.

Original: The medium dataset we used is 1 million time steps collected by the policy explained above.

Original: For the detail of the dataset, please refer to the original paper [38].

Original: [Table]

Original: C Model Architecture and Module Names

Original: In the following sections, we use the module names of our models ( gpt2 and openai/imagegptsmall ) to describe the experimental details.

Original: Thus, for the reference, we show the model architecture we use and the names of the modules of the model.

Original: We show that of gpt2 but the configuration is the same for openai/imagegpt-small except for the number of Transformer blocks.

Original: For the detailed configurations of pre-trained models, please refer to the following links:

Original: • gpt2 : https://huggingface.co/gpt • openai/imagegpt-small : https://huggingface.co/openai/imagegpt-small

Original: We show the model architecture below, following the Pytorch format: 1 DecisionTransformer( (

Original: transformer):

Original: GPT2Model(

Original: wte): Embedding (50257,

Original: ) (wpe): Embedding (1024, 768) (drop): Dropout(p=0.1, inplace=

Original: False)

Original: (h): ModuleList( (0): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=

Original: False)

Original: (resid_dropout): Dropout(

Original: (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.2, inplace=False) )

Original: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) )

Original: (2): GPT2Block( .

Original: ) (11): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=

Original: False) (resid_dropout): Dropout(p=0.2, inplace=False) )

Original: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) ) )

Original: (ln_f): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) )

Original: (embed_timestep): Embedding (1000, 768) (embed_return): Linear in_features

Original: 1, out features =768, bias=

Original: True) (embed_state): Linear in features=11, out_features =768, bias=

Original: True)

Original: (embed_action): Linear features=3,

Original: _features =768, bias=

Original: True) (embed_ln): LayerNorm (

Original: 768,), eps=1e-05, elementwise_affine=

Original: True) 68 )

Original: Listing 1: Model architecture.

Original: Among the above modules, for our analysis, we focus on the modules in DecisionTransformer.

Original: transformer.h .

Original: We summarize the module-related notations used in this paper below for reference:

Original: • Transformer blocks : modules directly under DecisionTransformer.transformer.h – e.g. DecisionTransformer.transformer.h[0] .

Original: • Outputs of blocks : outputs of mlp.dropout of blocks.

Original: – e.g. DecisionTransformer.transformer.h[0].mlp.dropout .

Original: • Layers : modules under Transformer blocks.

Original: – e.g. DecisionTransformer.transformer.h[0].attn.c_attn .

Original: • Parameter set : parameters of each layer.

Original: – e.g. DecisionTransformer.transformer.h[0].ln_1.weight .

Original: Note that these notations are sometimes used interchangeably as long as it doesn’t significantly deteriorate the readability. D Activation Similarity D.1 Details of Experiments We randomly sample 100 samples and compute unbiased estimators [39] of linear CKA for these 100 activation vectors. The activation to be analyzed are outputs from all layers .
Revised: Also, the validity of the metrics we used should be critically examined since different results could be obtained by using more carefully designed ones.

Original: In the Decision

Original: Transformer, the effective total input length is context length K times the number of token types ( ˆ R, s , a ) .

Original: As a result, the shape of the activation obtained at each layer is (batch size, 3x context length, embedding dimension) .

Original: In this study, we compute the CKA for the activity of shape (batch size, embedding dimension) that corresponds to the last time step in the context of return-to-go, state, and action, respectively.

Original: In other words, noting activation as in the form of python NumPy array activation , we obtain activation[:, -1, :] , activation[:, -2, :] , and activation[:, -3, :] .

Original: In the previous work analyzing the internal representation of Transformer [41], the dimension of context and representation seem to be concatenated: activation.contiguous().view(activation.

Original: shape[0] -1) .

Original: Since the representation obtained in this way is hard to interpret, we decide to use the way we described above to obtain activation.

Original: A descriptive diagram for activation that we compute

Original: CKA about is shown in Fig.

Original: The design of the diagram is based on a previous study [35].

Original: [Figure]

Original: Each rectangle colored in red is an activation we consider.

Original: The left-most rectangles are the activation with return-to-go as input, the middle rectangles are the that with status as input, and the right-most rectangles are that with action as input.

Original: Since we consider only the last in the context, the rectangles are drawn only where the tokens of the K-th step are received as input.

Original: In Section 5.1, we show the results of activation that s K is fed in the diagram.

Original: D.2 Centered Kernel Alignment (CKA)

Original: We show the definition of Centered Kernel Alignment (CKA).

Original: Given two activation vectors X ∈ R m × p 1 and Y ∈ R m × p 1 , where m is data size and p 1 and p 2 are dimensions of hidden layers, the

Original: CKA of the two representations is as follows:

Original: [Equation]

Original: where HSCI is the Hilbert-Schmidt independence criterion [66] and K ij =

Original: k ( x i , x j ) and L ij = l

Original: ( x i , x j ) are kernels.

Original: Particularly, linear CKA is defined as follows:

Original: [Equation]

Original: where || · || F is the Frobenius norm.

Original: For further details, please refer to the previous work [39].

Original: D.3 Layer Names Whose CKA is Above a Threshold

Original: In Section 5.1, we say we observe that most of the high CKAs of Fig.

Original: 1 (b), which are greater than 0.38, are those of layer normalization.

Original: The reason we set this seemingly arbitrary threshold is that we observe higher layers in a block than other layers in Fig.

Original: 1 and want to identify these layers.

Original: By eye inspection, we find the threshold over which all these points are included in Fig.

Original: 1 is around 0.38 and so we select this value as a threshold.

Original: Here, we show the raw result of the observation.

Original: The layer name list is shown below:

Original: 1 0.ln_1 , 0.attn.c_attn , 0.ln_2 , 0.mlp.c_fc , 0.mlp.c_proj , 0.mlp.dropout 2 1.ln_1 , 1.attn.c_attn , 1.ln_ 3 2.ln_1 , 2.ln_ 4 3.ln_1 , 3.ln_2 , 3.mlp.c_fc 5 4.ln_1 , 4.attn.c_attn , 4.ln_ 6 5.ln_1 ,

Original: [Table]

Original: .attn.c_attn ,

Original: [Table]

Original: .ln

Original: _ 7 6.ln_1 , 6.attn.c_attn ,

Original: [Table]

Original: .ln

Original: [Table]

Original: .ln_1 ,

Original: [Table]

Original: .ln

Original: [Table]

Original: .ln_1 ,

Original: [Table]

Original: .ln

Original: [Table]

Original: [Table]

Original: .ln_1 ,

Original: [Table]

Original: .ln

Original: [Table]

Original: .ln_1 ,

Original: [Table]

Original: .attn.c_attn ,

Original: [Table]

Original: .ln

Original: [Table]

Original: .ln

Original: [Table]

Original: [Table]

Original: ln_

Original: [Table]

Original: Listing

Original: [Table]

Original: Note that, for example, 0.ln_1 is the first layer normalization layer ( ln_1 ) of the first transformer block ( 0 ).

Original: We observe that 24 of 34 layers are layer norm modules ( ln_1 and ln_2 ).

Original: D.4 CKA Between Different Models

Original: To check if pre-trained models and the randomly initialized model converge to a similar representation or not, we compute the CKA between representations of fine-tuned these models.

Original: The result is shown in Fig.

Original: 12, where (a) is the CKA between GPT2 and iGPT, (b) is that between GPT2 and random initialization, and (c) is that between iGPT and random initialization.

Original: We observe that CKA values in these heat maps are small, confirming that different models learn different representations.

Original: [Figure]

Original: D.5 CKA Between Different Layers in a Model D.5.1 Post-Fine-Tuning

Original: In Appendix D.4, we find that each layer of the pre-trained model and randomly initialized model learn representation differently.

Original: In this section, we delve into what representation these models acquire.

Original: For that purpose, we compute CKA values of the different layers in the same model after fine-tuning.

Original: The result for the state input token with the Hopper-medium dataset is shown in Fig.

Original: The results for other environments and input tokens are in Appendix J.1.

Original: [Figure]

Original: Comparing the CKA matrix of pre-trained models (Fig. 13 (a) and Fig.

Original: 13 (b)) with that of the randomly initialized model (Fig.

Original: 13 (c))

Original: , we observe that the randomly initialized model has an almost equally divided lattice structure, while pre-trained models have some block-like structure.

Original: In particular, the shallow blocks and the middle to top blocks of language-pre-trained model have a bit similar representation, respectively.

Original: This result indicates that probably layers of random initialization process information separately, while some layers of language-pre-trained models coordinate to represent information.

Original: D.5.2 Pre-Fine-Tuning

Original: For just reference, we put the CKA matrix of pre-trained models before fine-tuning as well.

Original: The result is shown in Fig.

Original: We observe the grid structure at the initial state as well.

Original: [Figure]

Original: E Mutual Information Between Hidden Representation and Data E.1 Details of Experiments

Original: MINE, the definition of which is explained in Appendix E.2, estimates mutual information by training a neural network.

Original: The neural network we use is a feed-forward ReLU network with two hidden layers of width 400.

Original: We train the neural network for 1000 iterations by Adam optimizer with a learning rate of 1 e − 4 .

Original: Same as in Appendix D.1, we randomly sample 100 trajectories and use them to obtain the activation.

Original: Thus, the dataset for mutual information calculation is the pair between theseactivation vectors and 100 trajectories for ˆ

Original: I ( X ; T ) and 100 last action vectors for ˆ

Original: I

Original: ( Y ; T ) .

Original: If the estimated value is NaN , we exclude the point from the figure.

Original: The descriptive diagram that shows hidden activation and data we consider to compute estimated mutual information is Fig.

Original: 15, where (a) is for ˆ I ( X ; T ) and (b) is for ˆ

Original: I

Original: ( Y ; T ) .

Original: This is an example of mutual information for the deep Transformer block.

Original: For the middle

Original: [Figure]

Original: E.2 Mutual Information Neural Estimation (MINE)

Original: Given random variables X and T , whose joint probability is P XT and marginal distributions are P

Original: X = (cid:82) T d P XT and P T = (cid:82) X d P XT , the mutual information of X and T are as follows:

Original: [Equation]

Original: [Equation]

Original: Denoting neural network with parameters θ ∈ Θ by NN θ and the empirical distribution of P characterized by n i.i.d.

Original: samples by ˆ P ( n ) , MINE is defined as follows [57]:

Original: [Equation]

Original: Please refer to the previous study [57] for the details.

Original: E.3 Mutual Information Without Considering Context

Original: As a complementary analysis, we also calculate the estimated mutual information between hidden representation and s t and a t .

Original: In other words, we compute ˆ

Original: I ( s t ; T l ( s t )) and ˆ I ( a t ; T l ( s t ))

Original: as ˆ I ( X ; T ) and ˆ I

Original: ( Y ; T )

Original: for all time steps in the context t = 1 , ..., K .

Original: Then, we take an average of them over the context.

Original: The descriptive diagram is shown in Fig.

Original: [Figure]

Original: [Figure]

Original: The result for Hopper-medium is shown in Fig.

Original: Each point indicates the estimated mean mutual information and the shaded area is the standard deviation.

Original: Just as we did in Section 5.1, we take an average over two adjacent elements for iGPT for comparison.

Original: We again observe that the randomly initialized model generally has more information on input and label, and iGPT has almost nothing about label-related information.

Original: F Parameter Similarity F.1 Details of Experiments

Original: The parameters considered in the analysis are those in Transformer blocks ( DecisionTransformer. transformer.h ).

Original: We concatenate all these parameters into a single vector and compute l 2 distance and cosine similarity for this vector.

Original: For the post-fine-tuning model in Section 5.3, we use the model trained with the Hopper-medium dataset.

Original: The results for other environments are in Appendix J.3.

Original: F.2 Ticks’ Labels of Figures 3 and

Original: The labels of ticks of Figs.

Original: 3 and 4 in Section 5.3 correspond to all parameter sets .

Original: For example, one of the parameter sets is 0.ln_1.weight :

Original: this is the weight vector of the layer normalization in the first ( 0 ) transformer block ( h ).

Original: The labels for ticks of GPT2 and the randomly initialized model are summarized below:

Original: ’0.ln_1.weight’, ’0.ln_1.bias’, ’0.attn.c_attn.weight’, ’0.attn.c_attn.bias’, ’0.attn.c_proj.weight’, ’

Original: .attn.c_proj.bias’, ’0.ln_2.weight’, ’0.ln_2.bias’, ’0.mlp.c_fc.weight’, ’0.mlp.c_fc.bias’, ’0.mlp.c_proj.weight’, ’0.mlp.c_proj.bias’, ’1.ln_1.weight’, ’1.ln_1.bias’, ’1.attn.c_attn.weight’, ’1.attn.c_attn.bias’, ’1.attn.c_proj.weight’, ’1.attn.c_proj.bias’, ’1.ln_2.weight’, ’1.ln_2.bias’, ’1.mlp.c_fc.weight’, ’1.mlp.c_fc.bias’, ’1.mlp.c_proj.weight’, ’1.mlp.c_proj.bias’,

Original: ’2.ln_1.weight’, ’2.ln_1.bias’, ’2.attn.c_attn.weight’, ’2.attn.c_attn.bias’, ’2.attn.c_proj.weight’, ’2.attn.c_proj.bias’, ’2.ln_2.weight’, ’2.ln_2.bias’, ’2.mlp.c_fc.weight’, ’2.mlp.c_fc.bias’, ’2.mlp.c_proj.weight’, ’2.mlp.c_proj.bias’, ’3.ln_1.weight’, ’3.ln_1.bias’, ’3.attn.c_attn.weight’, ’3.attn.c_attn.bias’, ’3.attn.c_proj.weight’, ’3.attn.c_proj.bias’, ’3.ln_2.weight’, ’3.ln_2.bias’, ’3.mlp.c_fc.weight’, ’3.mlp.c_fc.bias’, ’3.mlp.c_proj.weight’, ’3.mlp.c_proj.bias’, ’4.ln_1.weight’, ’4.ln_1.bias’, ’4.attn.c_attn.weight’, ’4.attn.c_attn.bias’, ’4.attn.c_proj.weight’, ’4.attn.c_proj.bias’, ’4.ln_2.weight’, ’4.ln_2.bias’, ’4.mlp.c_fc.weight’, ’4.mlp.c_fc.bias’, ’4.mlp.c_proj.weight’, ’4.mlp.c_proj.bias’, ’5.ln_1.weight’, ’5.ln_1.bias’, ’5.attn.c_attn.weight’, ’5.attn.c_attn.bias’, ’5.attn.c_proj.weight’, ’5.attn.c_proj.bias’, ’5.ln_2.weight’, ’5.ln_2.bias’, ’5.mlp.c_fc.weight’, ’5.mlp.c_fc.bias’, ’5.mlp.c_proj.weight’, ’5.mlp.c_proj.bias’, ’6.ln_1.weight’, ’6.ln_1.bias’, ’6.attn.c_attn.weight’, ’6.attn.c_attn.bias’, ’6.attn.c_proj.weight’, ’6.attn.c_proj.bias’, ’6.ln_2.weight’, ’6.ln_2.bias’, ’6.mlp.c_fc.weight’, ’6.mlp.c_fc.bias’, ’6.mlp.c_proj.weight’, ’6.mlp.c_proj.bias’, ’7.ln_1.weight’, ’7.ln_1.bias’, ’7.attn.c_attn

Original: ’7.attn.c_proj.bias’, ’7.ln_2.weight’, ’7.ln_2.bias’, ’7.mlp.c_fc.weight’, ’7.mlp.c_fc.bias’, ’7.mlp.c_proj.weight’, ’7.mlp.c_proj.bias’, ’8.ln_1.weight’, ’8.ln_1.bias’, ’8.attn.c_attn.weight’, ’8.attn.c_attn.bias’, ’8.attn.c_proj.weight’, ’8.attn.c_proj.bias’, ’8.ln_2.weight’, ’8.ln_2.bias’, ’8.mlp.c_fc.weight’, ’8.mlp.c_fc.bias’, ’8.mlp.c_proj.weight’, ’8.mlp.c_proj.bias’, ’9.ln_1.weight’, ’9.ln_1.bias’, ’9.attn.c_attn.weight’, ’9.attn.c_attn.bias’, ’9.attn.c_proj.weight’, ’9.attn.c_proj.bias’, ’9.ln_2.weight’, ’9.ln_2.bias’, ’9.mlp.c_fc.weight’, ’9.mlp.c_fc.bias’, ’9.mlp.c_proj.weight’, ’9.mlp.c_proj.bias’, ’10.ln_1.weight’, ’10.ln_1.bias’, ’10.attn.c_attn.weight’, ’10.attn.c_attn.bias’, ’10.attn.c_proj.weight’, ’10.attn.c_proj.bias’, ’10.ln_2.weight’, ’10.ln_2.bias’, ’10.mlp.c_fc.weight’, ’10.mlp.c_fc.bias’, ’10.mlp.c_proj.weight’, ’10.mlp.c_proj.bias’, ’11.ln_1.weight’, ’11.ln_1.bias’, ’11.attn.c_attn.weight’, ’11.attn.c_attn.bias’, ’11.attn.c_proj.weight’, ’11.attn.c_proj.bias’, ’11.ln_2.weight’, ’11.ln_2.bias’, ’11.mlp.c_fc.weight’, ’11.mlp.c_fc.bias’, ’11.mlp.c_proj.weight’, ’11.mlp.c_proj.bias’

Original: Label list of

Original: For iGPT, the number of transformer block is twice (from 0-11 to 0-23) but the configuration is the same.

Original: G Gradient Analysis G.1 Details of Experiments

Original: We randomly sample 100 samples for gradient norm in Fig.

Original: 6 and 50 samples for gradient confusion in Fig.

Original: 5 and compute the gradient of the loss on each of these samples for all parameters of the models fine-tuned after 1 epoch.

Original: The bar plot of Fig.

Original: 5 is the minimum of the 50 × 50 =cosine similarities.

Original: Each point of Fig.

Original: 6 corresponds to the gradient norm of each data sample ingradient norms.

Original: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.

Original: The whiskers extend from the box by 1.5 times the inter-quartile range.

Original: For gradient clipping, we use a method in Pytorch [67] and set the maximum norm to be 0 .

Original: 25 , following the previous work [9].

Original: The process of creating Fig.

Original: 7 is as follows.

Original: We first compute gradient norms per parameter set and examine the bar plot that we will explain in Appendix G.2.

Original: Noticing that two peaks exist in the figure, we check the label of the parameter sets and find that they are 0.ln_1.weight and 0.ln_1.bias .

Original: Thus, we lump the remaining as others and only show the ratio in the main body of the paper to tell readers the main finding from the observation in an easy-to-understand manner.

Original: The parameter sets considered to compute the parameter set-wise gradient norms are the same as those in Appendix F.2.

Original: G.2 Gradient Norm for Each Parameter

Original: For visibility, we show only the ratio in Section 5.4 as Fig.

Original: The full results for gradient norms of each parameter are shown in Fig.

Original: The labels of the ticks on the x-axis are the same as that in

Original: Appendix F.2.

Original: We immediately notice that the first two parameters ( 0.ln_1.weight and 0.ln_1.bias ) are much larger than the remains.

Original: [Figure]

Original: G.3 Analysis of the Effect of Gradient Clipping

Original: In Section 5.4, we noted that large gradient and gradient clipping could be a cause of the bad performance of iGPT.

Original: To further explore this point, we conducted a simple experiment.

Original: The experiments of the previous study [9] and our experiments both used a Pytorch function called torch.nn.utils.clip_grad_norm_ 9 for gradient clipping.

Original: This function divides each gradient by the total norm of all gradients and multiplies the clipping constant (gradient norm clipping).

Original: However, as we pointed out in Section 5.4, the gradient norm values are dominated by only a few parameters (the layer normalization layer in the first block).

Original: This large norm affects the normalization of all gradients, decreasing the informational value of most gradients.

Original: Therefore, we hypothesized that one of the possible reasons why iGPT is difficult to learn can be the use of torch.nn

Original: .utils.clip_grad_norm_

Original: function .

Original: Thus we trained the image-pre-trained model without using gradient clipping.

Original: The basic experimental setting is the same as that of Section 5.4 except that we trained the model for 10 epochs, instead of 40 epochs.

Original: The results are shown in Figs 19 and 20 for action error and mean return, respectively.

Original: The x-axis is the epoch.

Original: These figures show that eliminating gradient clipping does not immediately solve the catastrophic performance at least up to 10 epochs.

Original: However, we did find that action error is much smaller for no gradient clipping condition, indicating that the learning process of it seems to be more stable and efficient than the clipping was applied.

Original: We also found that the return means seem to improve, albeit very slightly.

Original: Thus, we can conclude that gradient clipping is a cause of the bad performance of iGPT, though it does not seem to be critical. Exploring the critical factor that makes iGPT performance catastrophic is left a future work. [Figure] [Figure]
Revised: Because this work is fundamental research, no immediate serious negative societal impact is expected.

Original: H Fine-Tuning with No Context Information H.1 Details of Experiments

Original: The configuration of the training is the same as that described in Appendix B except that the context is 1 (no context).

Original: We evaluate the model per epoch and the normalized score in Table 4 is the result of the best checkpoint in the 40 checkpoints.

Original: The mean return reported in Fig.

Original: 8 and Table 4 is the average of the returns over the trajectory.

Original: Table 4 is the result of one random seed (seed = 666).

Original: The mean and standard deviation of the normalized score for the two seeds (seed = 666 & 42) and the mean return of another seed (seed = 42) are reported in Appendix J.5.

Original: In this paper, we mean context by the number of accessible time steps for a model to predict, following the previous study [35].

Original: If the context length is K , the model can use inputs in past K steps to predict the action at the current step, while the context length is 1 (no context), the model has to predict only from the current time step.

Original: [Figure]

Original: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context

Original: In Section 5.5, we observe that the performance of randomly initialized models for the Hopper environment is particularly worse than that for other environments (Table 4).

Original: In this Section, we further explore a possible cause of this observation.

Original: We speculate that this may be because how much of the range of context that needs to be looked at changes depending on the data set.

Original: For example, prior studies on decision transformer reported that the effect of context varied depending on the task [35].

Original: Improvement by context means that the context is important information for solving the task.

Original: Thus, it could be possible that Hopper is the dataset that requires more information from context than the other two data sets.

Original: As a test of the hypothesis, we randomly sampled a batch sample and calculated the mutual information between action and state or return-to-go at the same time step and compare them between different environments; note that this is the mutual information between the data, not between the data and representation.

Original: The higher the value of the mutual information, the higher the mutual dependence between state or return-to-go and action at the same time step.

Original: Hence, higher mutual information suggests that the model could predict action better only from the information at the current time step.

Original: In particular, we sample 100 samples of context length K = 20 .

Original: Then, for all time step t , we compute the estimated mutual information ˆ I ( s t ; a t ) between state s t and action a t and that ˆ

Original: I ( ˆ R t ; a t ) between return-to-go ˆ R t and action.

Original: Other setup for mutual information estimation is the same as that of

Original: Section 5.2.

Original: The result is shown in Fig 22, where (a) is the box plot for the state and action and (b) is that for the return-to-go and action.

Original: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.

Original: The whiskers extend from the box by 1.5 times the inter-quartile range.

Original: Each point corresponds to different time step in the context.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: We observe that the estimated mutual information between return-to-go and action is smaller on average for Hopper than for other environments (Fig.

Original: [-22 (b)), though that between action and state does not differ that much (Fig. 
-]
Revised: [-22 (b)), though that between action and state does not differ that much (Fig. 
-]

Original: (a)).

Original: Prior research has indicated that return-to-go information seems to be important for prediction [9].

Original: Hence, we can say that models have to use more information from the other steps in the context to solve the Hopper task than models do for other environments.

Original: This result supports our hypothesis above.

Original: I More In-Depth Analysis of Context Dependence I.1 Replacement by the Pre-Trained Block I.1.1 Details of Experiments

Original: The training configuration is the same as that of Section 5.5.

Original: The smoothing factor of the exponential moving average for Fig.

Original: 9 is 0.8.

Original: The action error is the mean square loss between the true action a t and the predicted action ˆ a t .

Original: Considering that the goal of this experiment is to highlight the effect of pre-training, we train the model for 10 epochs for Hopper and 5 epochs for HalfCheetah and

Original: Walker2D because Fig.

Original: 8 shows that the difference between the randomly initialized and pre-trained model is evident during these epochs.

Original: I.2 Attention Distance Analysis I.2.1 Details of Experiments

Original: In the main body of this paper, we mean attention distance by average/mean attention distance in the previous work [59, 43].

Original: Because we use only one attention head in this study, there is a single point per sample in Fig.

Original: We randomly sample 100 trajectory samples and compute attention for these trajectories.

Original: When calculating attention distance, we allow the model to access the context of

Original: K = 20 length and compute the attention distance up to the length.

Original: Each point of the box plot (Fig.

Original: 10) corresponds to the attention distance of each sample.

Original: The configuration for the box plot is the same as that in Appendix G.

Original: J Results for Other Conditions J.1 Activation Similarity J.1.1 CKA Between Pre and Post-Fine-Tuning

Original: [Figure]

Original: [Figure]

Original: J

Original: CKA

Original: Between Pre and Post-Fine-Tuning (Seed = 42)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.1.3 CKA Between Different Models

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.1.4 CKA Between Different Layers in a Model

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.2 Mutual Information Between Hidden Representation and Data

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: I ( Y ;

Original: [Figure]

Original: [Figure]

Original: Init GPT

Original: [Figure]

Original: ˆ

Original: I ( Y ; T )

Original: J.3 Parameter Similarity J.3.1 Parameter Similarity (Other Environment)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.3.2 Parameter Similarity (Seed = 42)

Original: [Figure]

Original: [Figure]

Original: ||

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: T

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.4 Gradient Analysis J.4.1 Gradient Analysis (Other Environments)

Original: [Figure]

Original: [Figure]

Original: M i n o f c o ss i m ( 1 ,

Original: [Figure]

Original: [Figure]

Original: iGPT

Original: Random Init 0.

Original: M i n o f c o ss i m ( 1 , 2 )

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: li pped

Original: G r ad i en t N [Figure] r m [Figure]
Revised: Acknowledgments and Disclosure of Funding

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: C li pped

Original: G r ad i en t N o r m

Original: [Figure]

Original: [Figure]

Original: r ad i en t N o r m

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J

Original: -Tuning with No Context Information

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.6.2 Attention Distance Analysis

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: K Details for Computation

Original: The computational environment for our experiments is as follows:

Original: • GPU: NVIDIA GeForce GTX TITAN

Original: X × 2GPU.

Original: • CPU: Intel Xeon E5-2620 v3 2.4 GHz, 6 Cores × 2CPU.

Original: software: Ubuntu 20.04.4, GCC 9.4.0, CUDA 11.6.

Original: The computational time of fine-tuning for 40 epochs is around from 24 to 72 hours for context length

Original: K = 20 and around from 7 to 48 hours for context length K = 1 per run.

Original: We use the offline RL dataset of Mujoco in D4RL [38]. The license of the dataset in D4RL is a Creative Commons Attribution 4.0 License (CC BY).
Revised: We thank DEEPCORE Inc. for providing the computational resource.

Original: Thus, we can use this dataset with no consent as long as we follow the term of use.

Original: L.2 Code

Original: The code we used does not require special consent from authors as long as we follow the terms of use.

Original: Their licenses are as follows:

Original: • https://github.com/machelreid/can-wikipedia-help-offline-rl: MIT Licence.

Original: • https://github.com/rail-berkeley/d4rl:

Original: Apache License 2.0.

Original: • https://github.com/google-research/google-research/tree/master/representation_similarity: ApacheLicense2.0.

Original: • https://github.com/gtegner/mine-pytorch: MIT License.

Original: M Note on the Role of Each Sub-Sections in Section

Original: Sections 5.1 and 5.2 show what may not be a cause of the good performance.

Original: In other words, these sections eliminate some seemingly possible causes of good performance of language-pre-trained models, respectively.

Original: These analyses further highlight the importance of attention distance for performance.

Original: Without the analysis of Section 5.1, we cannot exclude the possibility that good performance comes from re-using pre-trained representation as well , which is common in the unimodal case as we explained in Section 5.1.

Original: Similarly, without the analysis of Section 5.2, the performance can benefit from fitting better into training data as well .

Original: In that case, it would be less clear whether utilizing some prior knowledge is solely important.

Original: Sections 5.4, 5.5, 5.6.1, and 5.6.2 discuss what may be a cause of the good performance, as summarized at the end of Section 5.

Original: Section 5.3 provides complementary findings for these sections.

Original: For example, the existence of some unchanged parameters implies the possibility that some information is re-used, while the relatively larger change in only shallower layers of pre-trained models partially supports that these unchanged parameters do not seem to contradict with changed representation observed in Section 5.1, as explained in Appendix N.

Original: N Note on Relationship Between Parameter Change and Representation Change

Original: In Section 5.1 we say that representation in some layers of the pre-trained model changes, while in

Original: Section 5.3 we say that parameters of them in some layers do not change that much, or vice versa for the randomly initialized model.

Original: These observations of Sections 5.1 and 5.3 are not necessarily contradictory because the change of parameters in layer ℓ does not correspond one-to-one with that in representation in the layer.

Original: First, the representation of layer ℓ is affected by all parameters and input data prior to layer ℓ .

Original: So, even if the parameters of layer ℓ have not changed, if the parameters before layer ℓ have changed, the representation of layer ℓ may change.

Original: Second, since not all of the parameters of a neural network necessarily contribute to the output, the output of a layer may not change even if some parameters of that layer change.

Original: For example, if the input value to ReLU is negative, the output will remain 0 no matter how much the parameters involved change.

Original: So, for example, it is possible that even if the parameters of the ℓ -layer change, the representation of the ℓ -layer remains the same.

Original: Another example is when the values of two weights w ℓ 1 and w ℓ 2 in layer ℓ do not change the output of the vanilla feed-forward network because of its symmetry.

Original: Another possible cause of this observation unique to the current analysis is that we use CKA to measure representational similarity.

Original: CKA is designed to be invariant to some transformations on the representation matrix [39].

Original: Thus, different representation matrix is regarded as the same under these transformations even when parameters are changed.

Original: O Note on Why Better Action Prediction Does Not Always Result in Better Return

Original: In Section 5.2, we explained that randomly initialized models seem to predict action better, while in

Original: Table 1, they perform worse than GPT2.

Original: We will add a possible explanation of why better action prediction not necessarily comes to a better return.

Original: The most likely cause of this observation is the current problem setup.

Original: As mentioned in Appendix B, the medium dataset is early stopped and thus does not necessarily converge to an optimal policy.

Original: This means that if the model accurately learns trajectory and predicts the next action, it does not always mean that it is the best action.

Original: Hence, we can see that a low action error does not necessarily mean a large mean return.

Original: However, We do not believe that the use of medium will hurt the validity of the analysis using this data since this dataset is not that pathological.

Original: Another cause might be related to mutual information.

Original: The current analysis of mutual information shows that the representation of the hidden layer of the randomly initialized model holds more information as well of the input as well as the labels.

Original: Since the neural net representation is a vector representation if a single vector contains both types of information, these types of information might have been mixed.

Original: This may make it difficult to properly utilize only the label information, making it harder to accurately predict the action.

Original: Another possibility is that the pre-trained model and the randomly initialized model differ in terms of which input token type (return-to-go, state, or action) information they encode.

Original: For example, in Fig.

Original: 2 (b), the language pre-trained model encodes information uniformly from the same token type: the hidden representation with high mutual information is the representation corresponding to the input from the triangles, i.e., the part corresponding to the return-to-go.

Original: On the other hand, in the randomly initialized models, there are variations (triangle, circle, and cross).

Original: It was noted in a previous study that attention weight was strong between the same token types [9].

Original: Therefore, perhaps the pre-trained model might encode less total information, but the amount of information effectively utilized is not that different.

Original: Finally, as pointed out in the explanation about the limitation, the limitations of mutual information as a metric might be a cause.