Original: On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
Revised: On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks.
Revised: We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks.

Original: Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one.
Revised: Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one.

Original: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping.
Revised: A closer look at the parameter changes of the pretrained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping.

Original: To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information.
Revised: To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information.

Original: Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.
Revised: Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.

Original:  Introduction
Revised:  Introduction

Original: The past few years have witnessed the tremendous success of pre-trained Transformer-based models [1] on versatile natural language processing (NLP) tasks.
Revised: The past few years have witnessed the tremendous success of pre-trained Transformer-based models [1] on versatile natural language processing (NLP) tasks.

Original: This is because pre-training with a large corpus enables Transformer-based models to gain rich universal language representations transferable to vast downstream tasks [2, 3, 4].
Revised: This is because pre-training with a large corpus enables Transformer-based models to gain rich universal language representations transferable to vast downstream tasks [2, 3, 4].

Original: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efficiently solve even non-NLP tasks [5, 6, 7, 8, 9, 10].
Revised: Remarkably, recent studies demonstrate that Transformers pretrained with language data can efficiently solve even non-NLP tasks [5, 6, 7, 8, 9, 10].

Original: Because of its versatility, it has been suggested that Transformers pre-trained with a large corpus could be called a kind of universal computation engines [5].
Revised: Because of its versatility, it has been suggested that Transformers pre-trained with a large corpus could be called a kind of universal computation engines [5].

Original: One area worthwhile attempting to study the benefits of pre-trained Transformer is reinforcement learning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].
Revised: One area worthwhile attempting to study the benefits of pre-trained Transformer is reinforcement learning because the applicability of pre-training to this area still has room to be explored [11, 12, 13].

Original: In this vein of research
Revised: In this vein of research

Original: Reid et al.
Revised: Reid et al.

Original: studied the effect of pre-training of Transformers on offline reinforcement learning (offline RL) tasks [9].
Revised: studied the effect of pre-training of Transformers on offline reinforcement learning (offline RL) tasks [9].

Original: Their study demonstrates a surprising contrast that pre-training with image significantly deteriorates performance, while that with language data does not negatively affect downstream tasks, rather it even improves performance for some datasets.
Revised: Their study demonstrates a surprising contrast that pre-training with image significantly deteriorates performance, while that with language data does not negatively affect downstream tasks, rather it even improves performance for some datasets.

Original: However, it is not yet known how the presence or absence of pre-training and its content leads to differences in the performance of downstream tasks.
Revised: However, it is not yet known how the presence or absence of pre-training and its content leads to differences in the performance of downstream tasks.

Original: What information does/doesn’t the model pre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, and why does the model pre-trained on image data ( image-pre-trained model ) result in catastrophic performance?
Revised: What information does/doesn’t the model pre-trained on language data ( language-pre-trained model ) leverage to solve downstream tasks, and why does the model pre-trained on image data ( image-pre-trained model ) result in catastrophic performance?

Original: In this paper, we approach these questions by analyzing the inside of the Transformer that is fine-tuning to Mujoco [14] offline RL tasks.
Revised: In this paper, we approach these questions by analyzing the inside of the Transformer that is fine-tuning to Mujoco [14] offline RL tasks.

Original: Analyzing the internals of Transformer, we study how randomly initialized, language-pre-trained, and image-pre-trained models acquire different representations, and what the differences could be attributed to.
Revised: Analyzing the internals of Transformer, we study how randomly initialized, language-pre-trained, and image-pre-trained models acquire different representations, and what the differences could be attributed to.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We study the internal representation of pre-trained and randomly initialized Transformers and find that the pre-trained models largely change their representation, while they encode less information of data during fine-tuning.
Revised: • We study the internal representation of pre-trained and randomly initialized Transformers and find that the pre-trained models largely change their representation, while they encode less information of data during fine-tuning.

Original: • The analysis of parameter change and gradient shows that the pre-trained models do not change parameters much perhaps due to gradient confusion and that the performance of the image-pre-trained model might be partly affected by gradient clipping on the large gradient.
Revised: • The analysis of parameter change and gradient shows that the pre-trained models do not change parameters much perhaps due to gradient confusion and that the performance of the image-pre-trained model might be partly affected by gradient clipping on the large gradient.

Original: • We examine context-dependence by fine-tuning models with giving no context information, finding that the language-pre-trained Transformer efficiently learns even without context and this could be the case by replacing just a single Transformer block with a pre-trained one.
Revised: • We examine context-dependence by fine-tuning models with giving no context information, finding that the language-pre-trained Transformer efficiently learns even without context and this could be the case by replacing just a single Transformer block with a pre-trained one.

Original: • Subsequent analysis of the change in how far away the model process the tokens of input supports the possibility that some contextual equivalent information has been positively transferred from language pre-training.
Revised: • Subsequent analysis of the change in how far away the model process the tokens of input supports the possibility that some contextual equivalent information has been positively transferred from language pre-training.

Original:  Related Works
Revised:  Related Works

Original: The study of transferability of neural representation has been one of the most important research topics in machine learning research.
Revised: The study of transferability of neural representation has been one of the most important research topics in machine learning research.

Original: A bunch of studies have investigated how pre-training influences the downstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].
Revised: A bunch of studies have investigated how pre-training influences the downstream tasks, such as computer vision tasks [15, 16, 17, 18] and NLP tasks [19, 20, 21].

Original: Recently, several studies have investigated the transferability of Transformers to various NLP tasks since understanding its high transferability is of great interest.
Revised: Recently, several studies have investigated the transferability of Transformers to various NLP tasks since understanding its high transferability is of great interest.

Original: Previous research reveals that Transformer representation encodes some syntax [22], semantics [23], cross-lingual information [24], and world knowledge [25].
Revised: Previous research reveals that Transformer representation encodes some syntax [22], semantics [23], cross-lingual information [24], and world knowledge [25].

Original: Also, it has been pointed out that Transformers obtains not only language-specific information but also more abstract inter-token dependencies [26, 27, 28].
Revised: Also, it has been pointed out that Transformers obtains not only language-specific information but also more abstract inter-token dependencies [26, 27, 28].

Original: This finding suggests that pre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.
Revised: This finding suggests that pre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.

Original: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks, reporting the versatility of the Transformer even in such multi-modal cases.
Revised: In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks, reporting the versatility of the Transformer even in such multi-modal cases.

Original: Such prior works have shown that language-pre-trained Transformers can efficiently learn image classification [29, 5], symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and more generally, sequence modeling problems [5].
Revised: Such prior works have shown that language-pre-trained Transformers can efficiently learn image classification [5], symbolic mathematics [6], simple arithmetic [4], reinforcement learning tasks [7, 8, 10, 9], and more generally, sequence modeling problems [5].

Original: Our study is in line with the research that examines the applicability of pre-trained Transformers on reinforcement learning.
Revised: Our study is in line with the research that examines the applicability of pre-trained Transformers on reinforcement learning.

Original: Most prior works applying pre-trained Transformer on reinforcement learning explicitly express goals, observations, or actions in a language [30, 7, 8, 10].
Revised: Most prior works applying pre-trained Transformer on reinforcement learning explicitly express goals, observations, or actions in a language [29, 7, 8, 10].

Original: On the other hand, the work of [9] applies pre-trained Transformers on the offline RL tasks that language is unrelated to (e.g. Mujoco [14]) and still reports that pre-training is helpful.
Revised: On the other hand, the work of [9] applies pre-trained Transformers on the offline RL tasks that language is unrelated to (e.g. Mujoco [14]) and still reports that pre-training is helpful.

Original: This study is interesting because it suggests that language-pre-trained Transformers seem to exploit some similarity between the structure of natural language data and trajectory data, which is more abstract and fundamental than linguistic information.
Revised: This study is interesting because it suggests that language-pre-trained Transformers seem to exploit some similarity between the structure of natural language data and trajectory data, which is more abstract and fundamental than linguistic information.

Original: Also, exploring the applicability of pre-trained Transformers, which have already been successful, is of great engineering significance since pre-training in offline RL has room for improvement compared to other modalities.
Revised: Also, exploring the applicability of pre-trained Transformers, which have already been successful, is of great engineering significance since pre-training in offline RL has room for improvement compared to other modalities.

Original: Examining the internals of Transformers, we further delve into the findings from this study by Reid et al.
Revised: Examining the internals of Transformers, we further delve into the findings from this study by Reid et al.

Original: [9] and explores the applicability of Transformers to offline RL.
Revised: [9] and explores the applicability of Transformers to offline RL.

Original:  Background 3.1 Transformer
Revised:  Background 3.1 Transformer

Original: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .
Revised: Transformers are sequence-to-sequence models composed of stacked identical blocks, called Transformer blocks .

Original: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [31], and layer normalization (layer norm) [32].
Revised: Each block consists of self-attention [1], multi-layer perceptron, skip-connection [30], and layer normalization (layer norm) [31].

Original: Denoting the linear projection of input sequence of length n to query, key, values of length m by Q ∈ R n × d Q , K ∈ R m × d K , and V ∈ R m × d V , respectively, the self-attention is Attention ( Q, K, V ) = softmax (cid:16)
Revised: Denoting the linear projection of input sequence of length n to query, key, values of length m by Q ∈ R n × d Q , K ∈ R m × d K , and V ∈ R m × d V , respectively, the self-attention is Attention ( Q, K, V ) = softmax (cid:16)

Original: QK ⊤ √ d K (cid:17) V .
Revised: QK ⊤ √ d K (cid:17) V .

Original: We use GPTs [33], and hence the autoregressive language model, where queries head to only keys before their position.
Revised: We use GPTs [32], and hence the autoregressive language model, where queries head to only keys before their position.

Original: 3.2 Offline Reinforcement Learning and Decision Transformer
Revised: 3.2 Offline Reinforcement Learning and Decision Transformer

Original: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.
Revised: Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment.

Original: For ease of mathematical handling, the environment is usually described as a Markov decision process (MDP).
Revised: For ease of mathematical handling, the environment is usually described as a Markov decision process (MDP).

Original: The MDP consists of states s ∈ S , actions a ∈ A , transition probability P ( s ′ | s , a ) , and reward function r ( s , a ) : S×A → R .
Revised: The MDP consists of states s ∈ S , actions a ∈ A , transition probability P ( s ′ | s , a ) , and reward function r ( s , a ) : S×A → R .

Original: The interaction of the agent in the environment is represented as
Revised: The interaction of the agent in the environment is represented as

Original: K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) , where s t , a t , and r t are state, action, and reward at time step t .
Revised: K length trajectory, or sequence of state, action, and reward ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ..., s K , a K , r K ) , where s t , a t , and r t are state, action, and reward at time step t .

Original: The goal is to find an optimal policy π ( a | s ) to maximize the expected cumulative rewards E [ (cid:80) Kt =0 r t ] for the trajectory.
Revised: The goal is to find an optimal policy π ( a | s ) to maximize the expected cumulative rewards E [ (cid:80) Kt =0 r t ] for the trajectory.

Original: Offline RL aims to achieve objectives of reinforcement learning from the trajectory data collected by some policy [34].
Revised: Offline RL aims to achieve objectives of reinforcement learning from the trajectory data collected by some policy [33].

Original: Because offline RL is purely characterized by trajectory data, several studies have proposed formulating offline RL as a sequence modeling problem [35, 36].
Revised: Because offline RL is purely characterized by trajectory data, several studies have proposed formulating offline RL as a sequence modeling problem [34, 35].

Original: Decision Transformer is a seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoder of BERT [2]) to solve such a sequence modeling problem [36].
Revised: Decision Transformer is a seminal work that attempts to use a causal transformer (in other words, GPT architecture or decoder of BERT [2]) to solve such a sequence modeling problem [35].

Original: In Decision Transformer, input data is a sequence τ =
Revised: In Decision Transformer, input data is a sequence τ =

Original: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-go ˆ R t = (cid:80) t − 1 i =

Revised: ( ˆ R 0 , s 0 , a 0 , ˆ R 1 , s 1 , a 1 , ..., ˆ R K , s K , a K ) of length K , where return-to-go ˆ R t = (cid:80) t − 1 i =

Original: r i is the cumulative reward until time step t .
Revised: r i is the cumulative reward until time step t .

Original: The previous studies [9, 36] employ the above problem set up to discuss the applicability of Transformer for offline RL.
Revised: The previous studies [9, 35] employ the above problem set up to discuss the applicability of Transformer for offline RL.

Original: We also follow these studies and use the setup for our analysis.
Revised: We also follow these studies and use the setup for our analysis.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Unless otherwise noted, all of our experimental setups, including model configuration, dataset, and hyperparameters follow the previous work [9] 2 .
Revised: Unless otherwise noted, all of our experimental setups, including model configuration, dataset, and hyperparameters follow the previous work [9] 2 .

Original: The sanity check that our trained models achieve comparable performance with the previous work is shown in Table 1.
Revised: The sanity check that our trained models achieve comparable performance with the previous work is shown in Table 1.

Original: Details are in Appendix A.
Revised: Details are in Appendix A.

Original: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initialized model ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect of pre-training.
Revised: We compare the model with GPT2 architecture [3] that is randomly initialized ( randomly initialized model ), pre-trained with language data ( GPT2 ), and with image data ( iGPT ) to study the effect of pre-training.

Original: Pre-trained models are from the Transformers library from HuggingFace [37].
Revised: Pre-trained models are from the Transformers library from HuggingFace [36].

Original: Like the previous study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .
Revised: Like the previous study, the model code for GPT2 and iGPT are gpt2 and openai/imagegpt-small .

Original: For offline RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [38] provided by D4RL [39], an offline RL dataset library.
Revised: For offline RL tasks, we use some of Mujoco tasks [14] in OpenAI Gym [37] provided by D4RL [38], an offline RL dataset library.

Original: Specifically, we employ medium datasets of HalfCheetah , Walker2d , and Hopper environments.
Revised: Specifically, we employ medium datasets of HalfCheetah , Walker2d , and Hopper environments.

Original: Unless otherwise specified, we show the result for Hopper-medium .
Revised: Unless otherwise specified, we show the result for Hopper-medium .

Original: We put the results for other environments in Appendix J.
Revised: We put the results for other environments in Appendix J.

Original: Following the previous study [9], we train the models for 40 epochs, each of which consists ofsteps.
Revised: Following the previous study [9], we train the models for 40 epochs, each of which consists ofsteps.

Original: In the following sections, we refer to the 40th epoch checkpoint as the post-fine-tuning model and the 0th epoch model (the initial state) as the pre- fine-tuning model .
Revised: In the following sections, we refer to the 40th epoch checkpoint as the post-fine-tuning model and the 0th epoch model (the initial state) as the pre- fine-tuning model .

Original: The training details, including hyperparameters and the optimizer, are detailed in Appendix B.
Revised: The training details, including hyperparameters and the optimizer, are detailed in Appendix B.

Original: [Table]
Revised: [Table]

Original:  Results and Analysis 5.1 Activation Similarity
Revised:  Results and Analysis 5.1 Activation Similarity

Original: We first study how the pre-trained and randomly initialized models shape the representation in finetuning.
Revised: We first study how the pre-trained and randomly initialized models shape the representation in finetuning.

Original: To that end, we compare the centered kernel alignment (CKA) [40] of each layer’s activation between pre and post-fine-tuning, investigating how each layer’s representation changes by learning offline RL data 4 .
Revised: To that end, we compare the centered kernel alignment (CKA) [39] of each layer’s activation between pre and post-fine-tuning, investigating how each layer’s representation changes by learning offline RL data 4 .

Original: The CKA has been used in various studies and has provided numerous insights into the understanding of neural networks [41, 42, 43, 44, 45].
Revised: The CKA has been used in various studies and has provided numerous insights into the understanding of neural networks [40, 41, 42, 43, 44].

Original: In particular, we compute the linear CKA, following previous studies [40, 46], for a subset of the dataset.
Revised: In particular, we compute the linear CKA, following previous studies [39, 45], for a subset of the dataset.

Original: We use the activation of each layer from the last time step of the context for return-to-go, state, and action, respectively.
Revised: We use the activation of each layer from the last time step of the context for return-to-go, state, and action, respectively.

Original: We show the result of state, putting the results for the others in Appendix J.1.
Revised: We show the result of state, putting the results for the others in Appendix J.1.

Original: Following the previous study [44], we compute CKA of activation not only for Transformer block output but for all internal activation of the Transformer blocks.
Revised: Following the previous study [43], we compute CKA of activation not only for Transformer block output but for all internal activation of the Transformer blocks.

Original: The definition of the CKA and the details are in Appendices D.2 and D.1.
Revised: The definition of the CKA and the details are in Appendices D.2 and D.1.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1 shows the result.
Revised: 1 shows the result.

Original: The y-axis is each layer’s CKA between representation at pre and post-finetuning.
Revised: The y-axis is each layer’s CKA between representation at pre and post-finetuning.

Original: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twice the others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPT and align the x-axis for different models for comparison.
Revised: The x-axis is the normalized layer index: because the number of layers of iGPT (216) is twice the others (108), we average CKA over two adjacent elements starting from the 0th layer for iGPT and align the x-axis for different models for comparison.

Original: Higher CKA means higher similarity.
Revised: Higher CKA means higher similarity.

Original: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layers except for a few layers, indicating that pre-trained models largely change representation.
Revised: We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layers except for a few layers, indicating that pre-trained models largely change representation.

Original: On the other hand, for the randomly initialized model (Random Init), the representations have a higher similarity to the initial state from the middle to the shallow layers.
Revised: On the other hand, for the randomly initialized model (Random Init), the representations have a higher similarity to the initial state from the middle to the shallow layers.

Original: Also, the CKAs of the randomly initialized model seem to be higher than pre-trained models across layers.
Revised: Also, the CKAs of the randomly initialized model seem to be higher than pre-trained models across layers.

Original: These observations indicate that the randomly initialized and pre-trained models structure their representations in very different ways.
Revised: These observations indicate that the randomly initialized and pre-trained models structure their representations in very different ways.

Original: Further support for this conclusion is shown in Appendix D.5.1.
Revised: Further support for this conclusion is shown in Appendix D.4.

Original: To identify which layers have relatively higher CKA for the randomly initialized model, we obtain layer names with CKA values above a threshold (dashed line at 0.23 in Fig.
Revised: To identify which layers have relatively higher CKA for the randomly initialized model, we obtain layer names with CKA values above a threshold (dashed line at 0.23 in Fig.

Original: 1 (a)).
Revised: 1 (a)).

Original: Then, we find that most of them are layer normalization layers (the list of the layer names are put in Appendix
Revised: Then, we find that most of them are layer normalization layers (the list of the layer names are put in Appendix

Original: D.3).
Revised: D.3).

Original: Therefore, we can speculate that the difference in CKA similarity of layer normalization can be related to the effect of pre-training.
Revised: Therefore, we can speculate that the difference in CKA similarity of layer normalization can be related to the effect of pre-training.

Original: This possibility could be supported by previous research [5] that reports that layer norm parameters are the most important to fine-tune.
Revised: This possibility could be supported by previous research [5] that reports that layer norm parameters are the most important to fine-tune.

Original: In uni-modal cases, where the statistical nature of the input data does not change significantly, previous studies have a consensus that the shallow layer acquires a more universal representation and the deep layer acquires a task-specific representation for a wide range of architectures and tasks [47, 48, 49, 50,  51, 40, 52, 43, 45].
Revised: In uni-modal cases, where the statistical nature of the input data does not change significantly, previous studies have a consensus that the shallow layer acquires a more universal representation and the deep layer acquires a task-specific representation for a wide range of architectures and tasks [46, 47, 48, 49, 50, 39, 51, 42, 44].

Original: The finding that the language-pre-trained Transformer performs as well as or better than the baseline but largely changes the representation suggests that in multi-modal cases, something different may be happening in forming the representation than in prior studies.
Revised: The finding that the language-pre-trained Transformer performs as well as or better than the baseline but largely changes the representation suggests that in multi-modal cases, something different may be happening in forming the representation than in prior studies.

Original: 5.2 Mutual Information Between Hidden Representation and Data
Revised: 5.2 Mutual Information Between Hidden Representation and Data

Original: In the previous section, we observe that pre-trained models drastically change their internal representation during fine-tuning, attaining largely different representations.
Revised: In the previous section, we observe that pre-trained models drastically change their internal representation during fine-tuning, attaining largely different representations.

Original: One possible consequence of this observation is that the pre-trained model may have adapted better to the data in the downstream tasks.
Revised: One possible consequence of this observation is that the pre-trained model may have adapted better to the data in the downstream tasks.

Original: To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models.
Revised: To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models.

Original: Mutual information is a well-known measure of the mutual dependence between the two random variables.
Revised: Mutual information is a well-known measure of the mutual dependence between the two random variables.

Original: Many previous studies have used this metric to investigate the extent to which neural representation encodes input and label information and provided profound insight into how neural networks work [53, 54, 55, 56, 57]
Revised: Many previous studies have used this metric to investigate the extent to which neural representation encodes input and label information and provided profound insight into how neural networks work [52, 53, 54, 55, 56]

Original: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .
Revised: In particular, we calculate the estimated mutual information ˆ I ( X ; T ) between some layer representation T and input data X , and that ˆ I ( Y ; T ) between T and label Y .

Original: We employ Mutual Information
Revised: We employ Mutual Information

Original: Neural Estimation (MINE) [58] to estimate mutual information since this method can estimate mutual information even when random vectors are continuous and have different dimensions 5 .
Revised: Neural Estimation (MINE) [57] to estimate mutual information since this method can estimate mutual information even when random vectors are continuous and have different dimensions 5 .

Original: Since
Revised: Since

Original: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual action a t , we define the label as Y = a t .
Revised: Decision Transformers are trained to minimize the gap between predicted action ˆ a t and actual action a t , we define the label as Y = a t .

Original: Unlike non-sequential models, the causal Transformer exploits past context information.
Revised: Unlike non-sequential models, the causal Transformer exploits past context information.

Original: Thus, denoting the input token except for a K by x t and the internal activation of Transformer block l of the input by T l ( x t ) , we compute ˆ
Revised: Thus, denoting the input token except for a K by x t and the internal activation of Transformer block l of the input by T l ( x t ) , we compute ˆ

Original: I ( x t ; T l ( s K ))
Revised: I ( x t ; T l ( s K ))

Original: as ˆ I ( X ; T ) and ˆ I ( a K ; T l ( x t )) as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .
Revised: as ˆ I ( X ; T ) and ˆ I ( a K ; T l ( x t )) as ˆ I ( Y ; T ) for all token positions, t = 0 , ..., K .

Original: We calculate these values for the shallow, middle, and deep Transformer blocks, respectively.
Revised: We calculate these values for the shallow, middle, and deep Transformer blocks, respectively.

Original: We show the result of the middle block here and leave the remaining in Appendix J.2.
Revised: We show the result of the middle block here and leave the remaining in Appendix J.2.

Original: The other details are described in Appendix E.1.
Revised: The other details are described in Appendix E.1.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 is the result, where Fig.
Revised: 2 is the result, where Fig.

Original: 2 (a) and (b) are results of ˆ
Revised: 2 (a) and (b) are results of ˆ

Original: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.
Revised: I ( X ; T ) and ˆ I ( Y ; T ) , respectively.

Original: The markers triangle , circle , and cross represent return-to-go, state, and action, respectively.
Revised: The markers triangle , circle , and cross represent return-to-go, state, and action, respectively.

Original: The x-axis is the position of tokens in the context and the y-axis is the estimated value of mutual information.
Revised: The x-axis is the position of tokens in the context and the y-axis is the estimated value of mutual information.

Original: We observe that the randomly initialized model consistently has higher mutual information both for input and label than pre-trained models.
Revised: We observe that the randomly initialized model consistently has higher mutual information both for input and label than pre-trained models.

Original: Given the catastrophic performance of iGPT, it is understandable that iGPT representations have almost no information.
Revised: Given the catastrophic performance of iGPT, it is understandable that iGPT representations have almost no information.

Original: The result comparing GPT to random initialization indicates that the changed representation observed in Section 5.1 does not encode more information about the input or the label than the randomly initialized model.
Revised: The result comparing GPT to random initialization indicates that the changed representation observed in Section 5.1 does not encode more information about the input or the label than the randomly initialized model.

Original: That is, the language-pre-trained model does well probably not because it acquires as much or more information about the data as the random initialization.
Revised: That is, the language-pre-trained model does well probably not because it acquires as much or more information about the data as the random initialization.

Original: This result suggests that although the representation has changed, there remains something unchanged, and the language-pre-trained model may leverage it to solve the offline RL task.
Revised: This result suggests that although the representation has changed, there remains something unchanged, and the language-pre-trained model may leverage it to solve the offline RL task.

Original: Note that this might be affected by limitations of mutual information.
Revised: Note that this might be affected by limitations of mutual information.

Original: We also notice that for label the language-pre-trained model has consistently higher mutual information for a specific token type (return-to-go), while that is not always the case for the randomly initialized model.
Revised: We also notice that for label the language-pre-trained model has consistently higher mutual information for a specific token type (return-to-go), while that is not always the case for the randomly initialized model.

Original: This result implies that the language-pre-trained model may process the information of a specific token type at each layer for prediction.
Revised: This result implies that the language-pre-trained model may process the information of a specific token type at each layer for prediction.

Original: 5.3 Parameter Similarity
Revised: 5.3 Parameter Similarity

Original: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploits some pre-acquired information.
Revised: In the previous section, we discuss the possibility that the language-pre-trained Transformer exploits some pre-acquired information.

Original: To investigate this issue on a finer scale, we turn our eye from representation to parameter.
Revised: To investigate this issue on a finer scale, we turn our eye from representation to parameter.

Original: We conduct parameter-level similarity analysis and investigate to what extent parameter change between pre and post-fine-tuning.
Revised: We conduct parameter-level similarity analysis and investigate to what extent parameter change between pre and post-fine-tuning.

Original: In concrete, we compute l 2 distance and cosine similarity of parameters between pre and post-fine-tuning.
Revised: In concrete, we compute l 2 distance and cosine similarity of parameters between pre and post-fine-tuning.

Original: The details are in Appendix F.1.
Revised: The details are in Appendix F.1.

Original: The result for l 2 distance is shown in Fig.
Revised: The result for l 2 distance is shown in Fig.

Original: 3 and for cosine similarity in Fig.
Revised: 3 and for cosine similarity in Fig.

Original: 4, where the top is the result for GPT2, the middle is for iGPT, and the bottom is for random initialization.
Revised: 4, where the top is the result for GPT2, the middle is for iGPT, and the bottom is for random initialization.

Original: The x-axis is the parameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layer parameters.
Revised: The x-axis is the parameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layer parameters.

Original: For visibility, we do not show the labels for ticks and put them in Appendix F.2.
Revised: For visibility, we do not show the labels for ticks and put them in Appendix F.2.

Original: [Figure]
Revised: [Figure]

Original: ||
Revised: ||

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models, meaning that the parameters of pre-trained models do not change that much.
Revised: Overall, We observe that l 2 distance is small and cosine similarity is large for pre-trained models, meaning that the parameters of pre-trained models do not change that much.

Original: At first glance, this appears to contradict the results of Section 5.1.
Revised: At first glance, this appears to contradict the results of Section 5.1.

Original: However, prior research has reported that most of the information in Transformers propagates through skip connections [44].
Revised: However, prior research has reported that most of the information in Transformers propagates through skip connections [43].

Original: This means that, as long as the parameters of the shallow layers change, the representation of each layer may change significantly without changing the parameters of each layer.
Revised: This means that, as long as the parameters of the shallow layers change, the representation of each layer may change significantly without changing the parameters of each layer.

Original: We indeed observe that the shallow layer l 2 distance of the pre-trained model shows relatively large changes.
Revised: We indeed observe that the shallow layer l 2 distance of the pre-trained model shows relatively large changes.

Original: The existence of unchanged parameters means that knowledge acquired through pre-training could be stored in those parameters
Revised: The existence of unchanged parameters means that knowledge acquired through pre-training could be stored in those parameters

Original: [Figure]
Revised: [Figure]

Original: 5.4 Gradient Analysis
Revised: 5.4 Gradient Analysis

Original: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.
Revised: In Section 5.3, we observe that pre-trained models’ parameters don’t change that much.

Original: We dig into this issue, investigating why GPT2 and iGPT do not change parameters.
Revised: We dig into this issue, investigating why GPT2 and iGPT do not change parameters.

Original: To this end, we compare the gradient norm and gradient confusion [59] at the early phase of the training ( 1 st epoch), discussing the training difficulty.
Revised: To this end, we compare the gradient norm and gradient confusion [58] at the early phase of the training ( 1 st epoch), discussing the training difficulty.

Original: When parameter gradients of loss for two different samples are negatively correlated, we say that there is gradient confusion.
Revised: When parameter gradients of loss for two different samples are negatively correlated, we say that there is gradient confusion.

Original: Empirically, we can measure this by the minimum gradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusion is low.
Revised: Empirically, we can measure this by the minimum gradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusion is low.

Original: The previous study showed that training is easier when gradient confusion is low [59].
Revised: The previous study showed that training is easier when gradient confusion is low [58].

Original: Because gradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), we compute the values for clipped gradients.
Revised: Because gradient clipping is used in the previous work [9] and this study (the maximum norm is 0 . 25 ), we compute the values for clipped gradients.

Original: The details are described in Appendix G.1.
Revised: The details are described in Appendix G.1.

Original: Fig.
Revised: Fig.

Original: 5 is the result for the gradient confusion and Fig.
Revised: 5 is the result for the gradient confusion and Fig.

Original: 6 is that for gradient norm.
Revised: 6 is that for gradient norm.

Original: Fig.
Revised: Fig.

Original: 5 shows that the minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initialized model.
Revised: 5 shows that the minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initialized model.

Original: This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model.
Revised: This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model.

Original: Fig.
Revised: Fig.

Original: 6, on the other hand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of the others is below the bound.
Revised: 6, on the other hand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of the others is below the bound.

Original: That is, the differences in the magnitude of the gradient are collapsed for iGPT, reducing the informational value of the gradients.
Revised: That is, the differences in the magnitude of the gradient are collapsed for iGPT, reducing the informational value of the gradients.

Original: GPT iGPT
Revised: GPT iGPT

Original: Random Init 1.
Revised: Random Init 1.

Original: M i n o f c o ss i m ( 1 , 2 )
Revised: M i n o f c o ss i m ( 1 , 2 )

Original: GPT iGPT
Revised: GPT iGPT

Original: Random Init 0.
Revised: Random Init 0.

Original: C li pped
Revised: C li pped

Original: G r ad i en t
Revised: G r ad i en t

Original: N o r m
Revised: N o r m

Original: [Figure]
Revised: [Figure]

Original: others
Revised: others

Original: To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm of each parameter set.
Revised: To identify which parameter of iGPT dominates the gradient norm, we compare the gradient norm of each parameter set.

Original: We calculate the gradient norm ratio of weights and biases of all parameters in all Transformer blocks.
Revised: We calculate the gradient norm ratio of weights and biases of all parameters in all Transformer blocks.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 7: for visibility, we show the parameters of the first layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remaining results together as others .
Revised: 7: for visibility, we show the parameters of the first layer norm module (weights 0.ln_1.weight and biases 0.ln_1.bias ) and lump the remaining results together as others .

Original: The full result is shown in Appendix G.2.
Revised: The full result is shown in Appendix G.2.

Original: We observe that weights and biases of the first layer norm module dominate the norm.
Revised: We observe that weights and biases of the first layer norm module dominate the norm.

Original: This result implies that the difficulty of training iGPT might partially come from the use of gradient clipping on large gradients dominated by few parameters.
Revised: This result implies that the difficulty of training iGPT might partially come from the use of gradient clipping on large gradients dominated by few parameters.

Original: A complementary analysis is described in Appendix G.3.
Revised: A complementary analysis is described in Appendix G.3.

Original: 5.5 Fine-Tuning with No Context Information
Revised: 5.5 Fine-Tuning with No Context Information

Original: In the sections up to this point, we find the possibility that the language-pre-trained model may exploit some pre-acquired information.
Revised: In the sections up to this point, we find the possibility that the language-pre-trained model may exploit some pre-acquired information.

Original: Then what information does the language-pre-trained model exploit to solve the tasks?
Revised: Then what information does the language-pre-trained model exploit to solve the tasks?

Original: One possibility is the ability to handle context because Transformer is skilled at handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].
Revised: One possibility is the ability to handle context because Transformer is skilled at handling the relations of tokens, which enables it to perform well in a variety of NLP tasks [22].

Original: Therefore, we train the language-pre-trained and the randomly initialized model with no access to the context ( K = 1 ) to dig into this possibility.
Revised: Therefore, we train the language-pre-trained and the randomly initialized model with no access to the context ( K = 1 ) to dig into this possibility.

Original: If a model learns efficiently without context, the model is already likely to have context-equivalent information.
Revised: If a model learns efficiently without context, the model is already likely to have context-equivalent information.

Original: Details of experiments are in Appendix H.1.
Revised: Details of experiments are in Appendix H.1.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The mean return throughout training time is shown in Fig.
Revised: The mean return throughout training time is shown in Fig.

Original: 8 and the normalized scores of return 6 are shown in Table 4.
Revised: 8 and the normalized scores of return 6 are shown in Table 2.

Original: The solid lines in Fig.
Revised: The solid lines in Fig.

Original: 8 are results without context and the dashed lines for reference are those with context.
Revised: 8 are results without context and the dashed lines for reference are those with context.

Original: From Fig.
Revised: From Fig.

Original: 8, we observe that when the context is not provided ( K = 1 ), the GPT reaches the high mean return much faster than the randomly initialized model.
Revised: 8, we observe that when the context is not provided ( K = 1 ), the GPT reaches the high mean return much faster than the randomly initialized model.

Original: Furthermore, Tablereveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), while the language-pre-trained model consistently achieves a comparable performance with that of the baseline result ( K = 20 ).
Revised: Furthermore, Tablereveals that the randomly initialized model fails to learn the trajectory for a dataset ( Hopper ), while the language-pre-trained model consistently achieves a comparable performance with that of the baseline result ( K = 20 ).

Original: That is, we observe that the language-pre-trained Transformer consistently performs more efficiently when no context is provided.
Revised: That is, we observe that the language-pre-trained Transformer consistently performs more efficiently when no context is provided.

Original: This is somewhat surprising, given that previous studies have pointed out that Decision Transformer performs significantly worse with no context [36].
Revised: This is somewhat surprising, given that previous studies have pointed out that Decision Transformer performs significantly worse with no context [35].

Original: The result that the language-pre-trained Transformer learns efficiently as well without
Revised: The result that the language-pre-trained Transformer learns efficiently as well without

Original: Following the previous studies [39, 36, 9], we report normalized scores: 100 ×
Revised: Following the previous studies [38, 35, 9], we report normalized scores: 100 ×

Original: [Table]
Revised: [Table]

Original: context as it does with context suggests that the model may have already acquired some kind of context-like information through pre-training and is utilizing it to solve the task.
Revised: context as it does with context suggests that the model may have already acquired some kind of context-like information through pre-training and is utilizing it to solve the task.

Original: 5.6 More In-Depth Analysis of Context Dependence 5.6.1 Replacement by the Pre-Trained Block
Revised: 5.6 More In-Depth Analysis of Context Dependence 5.6.1 Replacement by the Pre-Trained Block

Original: In the previous sections
Revised: In the previous sections

Original: [Figure]
Revised: [Figure]

Original: we find that the language-pre-trained Transformer exploits context-related information to solve the task.
Revised: we find that the language-pre-trained Transformer exploits context-related information to solve the task.

Original: To identify which block contains the useful information
Revised: To identify which block contains the useful information

Original: [Figure]
Revised: [Figure]

Original: we conduct a block replacement experiment, where we replace a Transformer block of the randomly initialized model with that of the language-pre-trained model and then fine-tune the model without context.
Revised: we conduct a block replacement experiment, where we replace a Transformer block of the randomly initialized model with that of the language-pre-trained model and then fine-tune the model without context.

Original: If replacement of a block does not help training, the block probably did not learn effective information in pre-training.
Revised: If replacement of a block does not help training, the block probably did not learn effective information in pre-training.

Original: Because we observe in Section 5.5 that the early phase characterizes the difference of the models, we focus on the epochs up to 10th epoch.
Revised: Because we observe in Section 5.5 that the early phase characterizes the difference of the models, we focus on the epochs up to 10th epoch.

Original: We explain the other details in Appendix I.1.
Revised: We explain the other details in Appendix I.1.

Original: [Figure]
Revised: [Figure]

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 9, where (a) is the prediction error of action and (b) is the mean return.
Revised: 9, where (a) is the prediction error of action and (b) is the mean return.

Original: To get a rough trend, we take the exponential moving average.
Revised: To get a rough trend, we take the exponential moving average.

Original: We observe that only when block 1 is replaced with the pre-trained block, does the action error (Fig.
Revised: We observe that only when block 1 is replaced with the pre-trained block, does the action error (Fig.

Original: (a)) and the mean return (Fig.
Revised: (a)) and the mean return (Fig.

Original: 9 (b)) behave similarly to the randomly initialized model.
Revised: 9 (b)) behave similarly to the randomly initialized model.

Original: Thus, we conclude that all blocks other than block 1 contain context-equivalent useful information to solve tasks.
Revised: Thus, we conclude that all blocks other than block 1 contain context-equivalent useful information to solve tasks.

Original: The findings that replacing a single block with the pre-trained one improves learning efficiency and that effective information is distributed across layers alone are surprising enough.
Revised: The findings that replacing a single block with the pre-trained one improves learning efficiency and that effective information is distributed across layers alone are surprising enough.

Original: This could be related to the dominant role of skip connections for information propagation [44] or the nature of non-monotonic change in the content each layer processes [20] in Transformers.
Revised: This could be related to the dominant role of skip connections for information propagation [43] or the nature of non-monotonic change in the content each layer processes [20] in Transformers.

Original: We show the result for Hopper since the trend explained above is most pronounced.
Revised: We show the result for Hopper since the trend explained above is most pronounced.

Original: For other environments, although the similarity between the random model and block1 is much weaker, the observation that block 1 is not good is consistent.
Revised: For other environments, although the similarity between the random model and block1 is much weaker, the observation that block 1 is not good is consistent.

Original: Results for other environments are in Appendix J.6.1.
Revised: Results for other environments are in Appendix J.6.1.

Original: 5.6.2 Attention Distance Analysis
Revised: 5.6.2 Attention Distance Analysis

Original: In previous sections, we find that the information improving learning efficiency is preserved in all blocks only in the varying amount and that is probably context-related.
Revised: In previous sections, we find that the information improving learning efficiency is preserved in all blocks only in the varying amount and that is probably context-related.

Original: To further confirm that the context-like information is indeed preserved, we investigate how the model processes context by calculating the attention distance [60] of each Transformer block for the result of Section 5.5.
Revised: To further confirm that the context-like information is indeed preserved, we investigate how the model processes context by calculating the attention distance [59] of each Transformer block for the result of Section 5.5.

Original: Attention distance is the average distance between key and query weighted by attention weight.
Revised: Attention distance is the average distance between key and query weighted by attention weight.

Original: Previous studies successfully used this measure to reveal how locally Transformers process context [44, 60].
Revised: Previous studies successfully used this measure to reveal how locally Transformers process context [43, 59].

Original: Attention distance will elucidate how far away each model processes the tokens of input.
Revised: Attention distance will elucidate how far away each model processes the tokens of input.

Original: In particular, we compute the gap ( | d att ( epoch )
Revised: In particular, we compute the gap ( | d att ( epoch )

Original: − d att (0) | ) between attention distance at the initial state ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves the way to utilize context.
Revised: − d att (0) | ) between attention distance at the initial state ( d att (0) ) and that at an epoch ( d att ( epoch ) ) to study to what extent the model preserves the way to utilize context.

Original: If the gap is smaller, it means that the attention distance is more preserved.
Revised: If the gap is smaller, it means that the attention distance is more preserved.

Original: To clarify the difference between the language-pre-trained model and the randomly initialized model, we compare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initialized model still produces a low return, while the GPT2 achieves a high return (Fig.
Revised: To clarify the difference between the language-pre-trained model and the randomly initialized model, we compare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initialized model still produces a low return, while the GPT2 achieves a high return (Fig.

Original: The detail of the setup and results for other epochs are shown in Appendices I.2.1 and J.6.2.
Revised: The detail of the setup and results for other epochs are shown in Appendices I.2.1 and J.6.2.

Original: [Figure]
Revised: [Figure]

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 10, where the x-axis is the Transformer block index and the y-axis is the gap between attention distances.
Revised: 10, where the x-axis is the Transformer block index and the y-axis is the gap between attention distances.

Original: Fig.
Revised: Fig.

Original: 10 (a), shows that the attention gaps of GPT2 except for block 1 are small, indicating that attention distances are kept from the pre-training in these blocks.
Revised: 10 (a), shows that the attention gaps of GPT2 except for block 1 are small, indicating that attention distances are kept from the pre-training in these blocks.

Original: Taken together with the finding of Section 5.6.1 that blocks other than block 1 improve learning efficiency, these preserved attention distances are likely to positively affect fine-tuning.
Revised: Taken together with the finding of Section 5.6.1 that blocks other than block 1 improve learning efficiency, these preserved attention distances are likely to positively affect fine-tuning.

Original: This result further supports the hypothesis that language-pre-training brings the model context-like information and the use of this information allows efficient learning.
Revised: This result further supports the hypothesis that language-pre-training brings the model context-like information and the use of this information allows efficient learning.

Original: We also notice the outstanding contrast that gaps of language-pre-trained models are almost zero except for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model are large.
Revised: We also notice the outstanding contrast that gaps of language-pre-trained models are almost zero except for block 1 (and block 2 for HalfCheetah), while those for the randomly initialized model are large.

Original: This trend is clearer from middle to deep blocks (3 to 10) for all environments.
Revised: This trend is clearer from middle to deep blocks (3 to 10) for all environments.

Original: Turning an eye to Fig.
Revised: Turning an eye to Fig.

Original: 9 (a), we notice that the action error of these blocks decreases faster than the other block (0 to 2 and 11).
Revised: 9 (a), we notice that the action error of these blocks decreases faster than the other block (0 to 2 and 11).

Original: Thus, we conclude that the pre-trained context-like information could help the training by enabling the model to predict action more accurately.
Revised: Thus, we conclude that the pre-trained context-like information could help the training by enabling the model to predict action more accurately.

Original: We summarize findings of Section 5 with particular attention to the implication for performance:
Revised: We summarize findings of Section 5 with particular attention to the implication for performance:

Original: • 5.1 ： Re-using representation is not the cause of the good performance of GPT2.
Revised: • 5.1 ： Re-using representation is not the cause of the good performance of GPT2.

Original: • 5.2 ： Fitting to data better is not the cause of the good performance of GPT2.
Revised: • 5.2 ： Fitting to data better is not the cause of the good performance of GPT2.

Original: • 5.3 ： The good performance of GPT2 might come from some unchanged parameters.
Revised: • 5.3 ： The good performance of GPT2 might come from some unchanged parameters.

Original: • 5.4 ： Gradient clipping and large gradient might be a cause of the bad performance of iGPT.
Revised: • 5.4 ： Gradient clipping and large gradient might be a cause of the bad performance of iGPT.

Original: • 5.5 ： GPT2 can learn efficiently even without the context provided.
Revised: • 5.5 ： GPT2 can learn efficiently even without the context provided.

Original: • 5.6.1 ： Even a single pre-trained Transformer block makes training more efficient.
Revised: • 5.6.1 ： Even a single pre-trained Transformer block makes training more efficient.

Original: A cause of the good performance of GPT2 is a pre-acquired way to use context.
Revised: A cause of the good performance of GPT2 is a pre-acquired way to use context.

Original:  Discussion
Revised:  Discussion

Original: Conclusion We have examined how pre-training on data of different modalities influences finetuning to offline RL.
Revised: Conclusion We have examined how pre-training on data of different modalities influences finetuning to offline RL.

Original: Internal representation analysis shows that pre-trained Transformer largely changes its representation while attaining less knowledge of the downstream task.
Revised: Internal representation analysis shows that pre-trained Transformer largely changes its representation while attaining less knowledge of the downstream task.

Original: By the analysis of change in parameters, we find that pre-trained Transformers do not change parameters that much and that the bad performance of the image-pre-trained model might partially come from gradient clipping on the large gradient dominated by a few parameters.
Revised: By the analysis of change in parameters, we find that pre-trained Transformers do not change parameters that much and that the bad performance of the image-pre-trained model might partially come from gradient clipping on the large gradient dominated by a few parameters.

Original: Fine-tuning models with no context
Revised: Fine-tuning models with no context

Original: , we find that the language-pre-trained model can efficiently solve offline RL tasks even without context.
Revised: , we find that the language-pre-trained model can efficiently solve offline RL tasks even without context.

Original: Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformer context-like information and the model exploits it to tackle the offline RL tasks.
Revised: Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformer context-like information and the model exploits it to tackle the offline RL tasks.

Original: Discussion The finding in Section 5.6.2 of the usefulness of middle blocks is interesting, given that the contextual model’s middle layers contain syntactic information [61, 62] and are most transferable [20].
Revised: Discussion The finding in Section 5.6.2 of the usefulness of middle blocks is interesting, given that the contextual model’s middle layers contain syntactic information [60, 61] and are most transferable [20].

Original: This implies that the syntactic capability of Transformers could be related to the performance on offline RL.
Revised: This implies that the syntactic capability of Transformers could be related to the performance on offline RL.

Original: The role of layer norm is also noteworthy as we observe that it may characterize the behavior in fine-tuning (Sections 5.1 and 5.4).
Revised: The role of layer norm is also noteworthy as we observe that it may characterize the behavior in fine-tuning (Sections 5.1 and 5.4).

Original: Studying the pre-training on other RL tasks is also critical to understanding the benefit of pre-training in RL.
Revised: Studying the pre-training on other RL tasks is also critical to understanding the benefit of pre-training in RL.

Original: Many studies of human evolution and development point to a close relationship between language and behavior [63, 64].
Revised: Many studies of human evolution and development point to a close relationship between language and behavior [62, 63].

Original: Hence, studying the relationship between these modalities matters to create human-like intelligence that handles language.
Revised: Hence, studying the relationship between these modalities matters to create human-like intelligence that handles language.

Original: A more detailed analysis of this relationship is a promising research direction.
Revised: A more detailed analysis of this relationship is a promising research direction.

Original: Limitation and Impact
Revised: Limitation and Impact

Original: We consider only a random seed and a few datasets from OpenAI Gym for analysis.
Revised: We consider only a random seed and a few datasets from OpenAI Gym for analysis.

Original: Thus, using diverse data is important and checking if the claim holds with the average results with many more seeds is crucial.
Revised: Thus, using diverse data is important and checking if the claim holds with the average results with many more seeds is crucial.

Original: Also, the validity of the metrics we used should be critically examined since different results could be obtained by using more carefully designed ones.
Revised: Also, the validity of the metrics we used should be critically examined since different results could be obtained by using more carefully designed ones.

Original: Because this work is fundamental research, no immediate serious negative societal impact is expected
Revised: Because this work is fundamental research, no immediate serious negative societal impact is expected

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank DEEPCORE Inc. for providing the computational resource.
Revised: We thank DEEPCORE Inc. for providing the computational resource.

Original: Gradient Analysis
Revised: Gradient Analysis

Original: G.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Gradient Norm for Each Parameter . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.2 Gradient Norm for Each Parameter . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.3 Analysis of the Effect of Gradient Clipping . . . . . . . . . . . . . . . . . . . .
Revised: G.3 Analysis of the Effect of Gradient Clipping . . . . . . . . . . . . . . . . . . . .

Original: H Fine-Tuning with
Revised: H Fine-Tuning with

Original: No
Revised: No

Original: Context Information
Revised: Context Information

Original: H.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.1 Details of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context .
Revised: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context .

Original: I More In-Depth Analysis of Context Dependence
Revised: I More In-Depth Analysis of Context Dependence

Original: I.
Revised: I.

Original: Replacement by the Pre-Trained Block . . . . . . . . . . . . . . . . . . . . . .
Revised: Replacement by the Pre-Trained Block . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Attention Distance Analysis . .
Revised: Attention Distance Analysis . .

Original: J
Revised: J

Original: Results for Other Conditions
Revised: Results for Other Conditions

Original: J.
Revised: J.

Original: Activation Similarity . .
Revised: Activation Similarity . .

Original: J.
Revised: J.

Original: Mutual Information Between Hidden Representation and Data . . . . . . . . . .
Revised: Mutual Information Between Hidden Representation and Data . . . . . . . . . .

Original: J.
Revised: J.

Original: Parameter Similarity . .
Revised: Parameter Similarity . .

Original: J.
Revised: J.

Original: Gradient Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Gradient Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: J.
Revised: J.

Original: Fine-Tuning with No Context Information . . . . . . . . . . . . . . . . . . . .
Revised: Fine-Tuning with No Context Information . . . . . . . . . . . . . . . . . . . .

Original: J.
Revised: J.

Original: More In-Depth Analysis of Context Dependence . . . . . . . . . . . . . . . . .
Revised: More In-Depth Analysis of Context Dependence . . . . . . . . . . . . . . . . .

Original: K
Revised: K

Original: Link to the Anonymized Repository
Revised: Link to the Anonymized Repository

Original: L
Revised: L

Original: Details for
Revised: Details for

Original: Computation
Revised: Computation

Original: M Licences of Assets Used for Our Experiments
Revised: M Licences of Assets Used for Our Experiments

Original: M.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: M.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: M.2 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: M.2 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: N Note on The Role of Each Sub-Sections in Section
Revised: N Note on The Role of Each Sub-Sections in Section

Original: O Note on Relationship Between Parameter Change and Representation Change
Revised: O Note on Relationship Between Parameter Change and Representation Change

Original: P Note on Why Better Action Prediction Not Always Result in Better Return
Revised: P Note on Why Better Action Prediction Not Always Result in Better Return

Original: A Fine-Tuning Result for Sanity Check
Revised: A Fine-Tuning Result for Sanity Check

Original: We compare our performance results to those of previous studies to ensure that they are not far off from the results of the prior study [9], the result of which is shown in Table 1.
Revised: We compare our performance results to those of previous studies to ensure that they are not far off from the results of the prior study [9], the result of which is shown in Table 1.

Original: Following the previous studies [39, 36, 9], we report the normalized score of mean return: 100 × score − random score expert score − random score , where random score is mean return generated by a random policy and expert score is generated by a policy trained by Soft Actor-Critic [65].
Revised: Following the previous studies [38, 35, 9], we report the normalized score of mean return: 100 × score − random score expert score − random score , where random score is mean return generated by a random policy and expert score is generated by a policy trained by Soft Actor-Critic [64].

Original: Mean return is the sum of rewards averaged over trajectory.
Revised: Mean return is the sum of rewards averaged over trajectory.

Original: For further details of the datasets and metrics, please refer to the paper that proposes D4RL [39].
Revised: For further details of the datasets and metrics, please refer to the paper that proposes D4RL [38].

Original: Although the previous work [9] used several techniques to improve the performance, e.g. language model co-training, the extension of positional embedding, and similarity encouragement, we do not employ any of these techniques so that we study the pure effect of pre-training.
Revised: Although the previous work [9] used several techniques to improve the performance, e.g. language model co-training, the extension of positional embedding, and similarity encouragement, we do not employ any of these techniques so that we study the pure effect of pre-training.

Original: The training details are described in Appendix B
Revised: The training details are described in Appendix B

Original: The result for the previous work is the average and standard deviation of three random seeds, while our result is those of two random seeds.
Revised: The result for the previous work is the average and standard deviation of three random seeds, while our result is those of two random seeds.

Original: The aim of this comparison is just to confirm that our result is not too pathological, checking soundness with two seeds would be valid enough.
Revised: The aim of this comparison is just to confirm that our result is not too pathological, checking soundness with two seeds would be valid enough.

Original: For reference, we also include the results of the Decision Transformer (DT) since the randomly initialized model (Random Init) is a large Decision Transformer.
Revised: For reference, we also include the results of the Decision Transformer (DT) since the randomly initialized model (Random Init) is a large Decision Transformer.

Original: Following the previous study [9], which says that it conducted early stopping, we show the result of the best checkpoint in 40 epoch checkpoints.
Revised: Following the previous study [9], which says that it conducted early stopping, we show the result of the best checkpoint in 40 epoch checkpoints.

Original: The result shows that our result is not too far away from the previous work, ensuring that the models of the subject of our analysis are valid enough.
Revised: The result shows that our result is not too far away from the previous work, ensuring that the models of the subject of our analysis are valid enough.

Original: B Training Details
Revised: B Training Details

Original: Hyperparameters are determined following previous study [9].
Revised: Hyperparameters are determined following previous study [9].

Original: The hyperparameters and other setups for fine-tuning the model are summarized in Table 3.
Revised: The hyperparameters and other setups for fine-tuning the model are summarized in Table 3.

Original: The number of layers is 12 for GPT2 and randomly initialized model and 24 for iGPT.
Revised: The number of layers is 12 for GPT2 and randomly initialized model and 24 for iGPT.

Original: The other setup not specified, including β of Adam optimizer [66] and model initialization scheme follows Pytorch default ones or the default values of optional arguments in the scripts that our experiment is based on 8 .
Revised: The other setup not specified, including β of Adam optimizer [65] and model initialization scheme follows Pytorch default ones or the default values of optional arguments in the scripts that our experiment is based on 8 .

Original: We do not do any pre-training ourselves, but use publicly available pre-trained models explained in Section 3.
Revised: We do not do any pre-training ourselves, but use publicly available pre-trained models explained in Section 3.

Original: Each model is trained to minimize the mean squared loss between its predicted actions ˆ a t and the true actions a t .
Revised: Each model is trained to minimize the mean squared loss between its predicted actions ˆ a t and the true actions a t .

Original: To get the big picture of how fine-tuning of GPT-based models to offline RL data is conducted, please refer to the pseudo-code of the previous work [36].
Revised: To get the big picture of how fine-tuning of GPT-based models to offline RL data is conducted, please refer to the pseudo-code of the previous work [35].

Original: The D4RL mujoco task used as offline RL data for our analysis has expert , medium , medium-replay , medium-expert , and random datasets.
Revised: The D4RL mujoco task used as offline RL data for our analysis has expert , medium , medium-replay , medium-expert , and random datasets.

Original: The expert dataset is the one trained by Soft Actor-Critic [65], medium is the one partially trained by Soft Actor-Critic and was stopped early, medium-replay is the one accumulated in the replay buffer before the model reached medium ’s level, and medium-expert is a mixture of medium and expert results.
Revised: The expert dataset is the one trained by Soft Actor-Critic [64], medium is the one partially trained by Soft Actor-Critic and was stopped early, medium-replay is the one accumulated in the replay buffer before the model reached medium ’s level, and medium-expert is a mixture of medium and expert results.

Original: The random dataset is the trajectory collected by the random policy.
Revised: The random dataset is the trajectory collected by the random policy.

Original: We conducted our experiments on the medium data set since it was used by the previous study [9] and our analysis is based on the observations of this previous study.
Revised: We conducted our experiments on the medium data set since it was used by the previous study [9] and our analysis is based on the observations of this previous study.

Original: The medium dataset we used is 1 million time steps collected by the policy explained above.
Revised: The medium dataset we used is 1 million time steps collected by the policy explained above.

Original: For the detail of the dataset, please refer to the original paper [39].
Revised: For the detail of the dataset, please refer to the original paper [38].

Original: [Table]
Revised: [Table]

Original: C Model Architecture and Module Names
Revised: C Model Architecture and Module Names

Original: In the following sections, we use the module names of our models ( gpt2 and openai/imagegptsmall ) to describe the experimental details.
Revised: In the following sections, we use the module names of our models ( gpt2 and openai/imagegptsmall ) to describe the experimental details.

Original: Thus, for the reference, we show the model architecture we use and the names of the modules of the model.
Revised: Thus, for the reference, we show the model architecture we use and the names of the modules of the model.

Original: We show that of gpt2 but the configuration is the same for openai/imagegpt-small except for the number of Transformer blocks.
Revised: We show that of gpt2 but the configuration is the same for openai/imagegpt-small except for the number of Transformer blocks.

Original: For the detailed configurations of pre-trained models, please refer to the following links:
Revised: For the detailed configurations of pre-trained models, please refer to the following links:

Original: • gpt2 : https://huggingface.co/gpt • openai/imagegpt-small : https://huggingface.co/openai/imagegpt-small
Revised: • gpt2 : https://huggingface.co/gpt • openai/imagegpt-small : https://huggingface.co/openai/imagegpt-small

Original: We show the model architecture below, following the Pytorch format: 1 DecisionTransformer( (
Revised: We show the model architecture below, following the Pytorch format: 1 DecisionTransformer( (

Original: transformer):
Revised: transformer):

Original: GPT2Model(
Revised: GPT2Model(

Original: wte): Embedding (50257,
Revised: wte): Embedding (50257,

Original: ) (wpe): Embedding (1024, 768) (drop): Dropout(p=0.1, inplace=
Revised: ) (wpe): Embedding (1024, 768) (drop): Dropout(p=0.1, inplace=

Original: False)
Revised: False)

Original: (h): ModuleList( (0): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=
Revised: (h): ModuleList( (0): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=
Revised: True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=

Original: False)
Revised: False)

Original: (resid_dropout): Dropout(
Revised: (resid_dropout): Dropout(

Original: (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.2, inplace=False) )
Revised: (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.2, inplace=False) )

Original: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=
Revised: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) )
Revised: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) )

Original: (2): GPT2Block( .
Revised: (2): GPT2Block( .

Original: ) (11): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=
Revised: ) (11): GPT2Block( (ln_1): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=

Original: False) (resid_dropout): Dropout(p=0.2, inplace=False) )
Revised: False) (resid_dropout): Dropout(p=0.2, inplace=False) )

Original: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=
Revised: (ln_2): LayerNorm ((768,), eps=1e-05, elementwise_affine=

Original: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) ) )
Revised: True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation () (dropout): Dropout(p=0.2, inplace=False) ) ) )

Original: (ln_f): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) )
Revised: (ln_f): LayerNorm ((768,), eps=1e-05, elementwise_affine=True) )

Original: (embed_timestep): Embedding (1000, 768) (embed_return): Linear in_features
Revised: (embed_timestep): Embedding (1000, 768) (embed_return): Linear in_features

Original: 1, out features =768, bias=
Revised: 1, out features =768, bias=

Original: True) (embed_state): Linear in features=11, out_features =768, bias=
Revised: True) (embed_state): Linear in features=11, out_features =768, bias=

Original: True)
Revised: True)

Original: (embed_action): Linear features=3,
Revised: (embed_action): Linear features=3,

Original: _features =768, bias=
Revised: _features =768, bias=

Original: True) (embed_ln): LayerNorm (
Revised: True) (embed_ln): LayerNorm (

Original: 768,), eps=1e-05, elementwise_affine=
Revised: 768,), eps=1e-05, elementwise_affine=

Original: True) 68 )
Revised: True) 68 )

Original: Listing 1: Model architecture.
Revised: Listing 1: Model architecture.

Original: Among the above modules, for our analysis, we focus on the modules in DecisionTransformer.
Revised: Among the above modules, for our analysis, we focus on the modules in DecisionTransformer.

Original: transformer.h .
Revised: transformer.h .

Original: We summarize the module-related notations used in this paper below for reference:
Revised: We summarize the module-related notations used in this paper below for reference:

Original: • Transformer blocks : modules directly under DecisionTransformer.transformer.h – e.g. DecisionTransformer.transformer.h[0] .
Revised: • Transformer blocks : modules directly under DecisionTransformer.transformer.h – e.g. DecisionTransformer.transformer.h[0] .

Original: • Outputs of blocks : outputs of mlp.dropout of blocks.
Revised: • Outputs of blocks : outputs of mlp.dropout of blocks.

Original: – e.g. DecisionTransformer.transformer.h[0].mlp.dropout .
Revised: – e.g. DecisionTransformer.transformer.h[0].mlp.dropout .

Original: • Layers : modules under Transformer blocks.
Revised: • Layers : modules under Transformer blocks.

Original: – e.g. DecisionTransformer.transformer.h[0].attn.c_attn .
Revised: – e.g. DecisionTransformer.transformer.h[0].attn.c_attn .

Original: • Parameter set : parameters of each layer.
Revised: • Parameter set : parameters of each layer.

Original: – e.g. DecisionTransformer.transformer.h[0].ln_1.weight .
Revised: – e.g. DecisionTransformer.transformer.h[0].ln_1.weight .

Original: Note that these notations are sometimes used interchangeably as long as it doesn’t significantly deteriorate the readability.
Revised: Note that these notations are sometimes used interchangeably as long as it doesn’t significantly deteriorate the readability.

Original: D Activation Similarity D.1 Details of Experiments
Revised: D Activation Similarity D.1 Details of Experiments

Original: We randomly sample 100 samples and compute unbiased estimators [40] of linear CKA for these 100 activation vectors.
Revised: We randomly sample 100 samples and compute unbiased estimators [39] of linear CKA for these 100 activation vectors.

Original: The activation to be analyzed are outputs from all layers .
Revised: The activation to be analyzed are outputs from all layers .

Original: In the Decision
Revised: In the Decision

Original: Transformer, the effective total input length is context length K times the number of token types ( ˆ R, s , a ) .
Revised: Transformer, the effective total input length is context length K times the number of token types ( ˆ R, s , a ) .

Original: As a result, the shape of the activation obtained at each layer is (batch size, 3x context length, embedding dimension) .
Revised: As a result, the shape of the activation obtained at each layer is (batch size, 3x context length, embedding dimension) .

Original: In this study, we compute the CKA for the activity of shape (batch size, embedding dimension) that corresponds to the last time step in the context of return-to-go, state, and action, respectively.
Revised: In this study, we compute the CKA for the activity of shape (batch size, embedding dimension) that corresponds to the last time step in the context of return-to-go, state, and action, respectively.

Original: In other words, noting activation as in the form of python NumPy array activation , we obtain activation[:, -1, :] , activation[:, -2, :] , and activation[:, -3, :] .
Revised: In other words, noting activation as in the form of python NumPy array activation , we obtain activation[:, -1, :] , activation[:, -2, :] , and activation[:, -3, :] .

Original: In the previous work analyzing the internal representation of Transformer [42], the dimension of context and representation seem to be concatenated: activation.contiguous().view(activation.
Revised: In the previous work analyzing the internal representation of Transformer [41], the dimension of context and representation seem to be concatenated: activation.contiguous().view(activation.

Original: shape[0] -1) .
Revised: shape[0] -1) .

Original: Since the representation obtained in this way is hard to interpret, we decide to use the way we described above to obtain activation.
Revised: Since the representation obtained in this way is hard to interpret, we decide to use the way we described above to obtain activation.

Original: A descriptive diagram for activation that we compute
Revised: A descriptive diagram for activation that we compute

Original: CKA about is shown in Fig.
Revised: CKA about is shown in Fig.

Original: The design of the diagram is based on a previous study [36].
Revised: The design of the diagram is based on a previous study [35].

Original: [Figure]
Revised: [Figure]

Original: Each rectangle colored in red is an activation we consider.
Revised: Each rectangle colored in red is an activation we consider.

Original: The left-most rectangles are the activation with return-to-go as input, the middle rectangles are the that with status as input, and the right-most rectangles are that with action as input.
Revised: The left-most rectangles are the activation with return-to-go as input, the middle rectangles are the that with status as input, and the right-most rectangles are that with action as input.

Original: Since we consider only the last in the context, the rectangles are drawn only where the tokens of the K-th step are received as input.
Revised: Since we consider only the last in the context, the rectangles are drawn only where the tokens of the K-th step are received as input.

Original: In Section 5.1, we show the results of activation that s K is fed in the diagram.
Revised: In Section 5.1, we show the results of activation that s K is fed in the diagram.

Original: D.2 Centered Kernel Alignment (CKA)
Revised: D.2 Centered Kernel Alignment (CKA)

Original: We show the definition of Centered Kernel Alignment (CKA).
Revised: We show the definition of Centered Kernel Alignment (CKA).

Original: Given two activation vectors X ∈ R m × p 1 and Y ∈ R m × p 1 , where m is data size and p 1 and p 2 are dimensions of hidden layers, the
Revised: Given two activation vectors X ∈ R m × p 1 and Y ∈ R m × p 1 , where m is data size and p 1 and p 2 are dimensions of hidden layers, the

Original: CKA of the two representations is as follows:
Revised: CKA of the two representations is as follows:

Original: [Equation]
Revised: [Equation]

Original: where HSCI is the Hilbert-Schmidt independence criterion [67] and K ij =
Revised: where HSCI is the Hilbert-Schmidt independence criterion [66] and K ij =

Original: k ( x i , x j ) and L ij = l
Revised: k ( x i , x j ) and L ij = l

Original: ( x i , x j ) are kernels.
Revised: ( x i , x j ) are kernels.

Original: Particularly, linear CKA is defined as follows:
Revised: Particularly, linear CKA is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where || · || F is the Frobenius norm.
Revised: where || · || F is the Frobenius norm.

Original: For further details, please refer to the previous work [40].
Revised: For further details, please refer to the previous work [39].

Original: D.3 Layer Names Whose CKA is Above a Threshold
Revised: D.3 Layer Names Whose CKA is Above a Threshold

Original: In Section 5.1, we say we observe that most of the high CKAs of Fig.
Revised: In Section 5.1, we say we observe that most of the high CKAs of Fig.

Original: 1 (b), which are greater than 0.38, are those of layer normalization.
Revised: 1 (b), which are greater than 0.38, are those of layer normalization.

Original: The reason we set this seemingly arbitrary threshold is that we observe higher layers in a block than other layers in Fig.
Revised: The reason we set this seemingly arbitrary threshold is that we observe higher layers in a block than other layers in Fig.

Original: 1 and want to identify these layers.
Revised: 1 and want to identify these layers.

Original: By eye inspection, we find the threshold over which all these points are included in Fig.
Revised: By eye inspection, we find the threshold over which all these points are included in Fig.

Original: 1 is around 0.38 and so we select this value as a threshold.
Revised: 1 is around 0.38 and so we select this value as a threshold.

Original: Here, we show the raw result of the observation.
Revised: Here, we show the raw result of the observation.

Original: The layer name list is shown below:
Revised: The layer name list is shown below:

Original: 1 0.ln_1 , 0.attn.c_attn , 0.ln_2 , 0.mlp.c_fc , 0.mlp.c_proj , 0.mlp.dropout 2 1.ln_1 , 1.attn.c_attn , 1.ln_ 3 2.ln_1 , 2.ln_ 4 3.ln_1 , 3.ln_2 , 3.mlp.c_fc 5 4.ln_1 , 4.attn.c_attn , 4.ln_ 6 5.ln_1 ,
Revised: 1 0.ln_1 , 0.attn.c_attn , 0.ln_2 , 0.mlp.c_fc , 0.mlp.c_proj , 0.mlp.dropout 2 1.ln_1 , 1.attn.c_attn , 1.ln_ 3 2.ln_1 , 2.ln_ 4 3.ln_1 , 3.ln_2 , 3.mlp.c_fc 5 4.ln_1 , 4.attn.c_attn , 4.ln_ 6 5.ln_1 ,

Original: [Table]
Revised: [Table]

Original: .attn.c_attn ,
Revised: .attn.c_attn ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: _ 7 6.ln_1 , 6.attn.c_attn ,
Revised: _ 7 6.ln_1 , 6.attn.c_attn ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table]
Revised: [Table]

Original: .ln_1 ,
Revised: .ln_1 ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table]
Revised: [Table]

Original: .ln_1 ,
Revised: .ln_1 ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table] [Table]
Revised: [Table] [Table]

Original: .ln_1 ,
Revised: .ln_1 ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table]
Revised: [Table]

Original: .ln_1 ,
Revised: .ln_1 ,

Original: [Table]
Revised: [Table]

Original: .attn.c_attn ,
Revised: .attn.c_attn ,

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table]
Revised: [Table]

Original: .ln
Revised: .ln

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ln_
Revised: ln_

Original: [Table]
Revised: [Table]

Original: Listing
Revised: Listing

Original: [Table]
Revised: [Table]

Original: Note that, for example, 0.ln_1 is the first layer normalization layer ( ln_1 ) of the first transformer block ( 0 ).
Revised: Note that, for example, 0.ln_1 is the first layer normalization layer ( ln_1 ) of the first transformer block ( 0 ).

Original: We observe that 24 of 34 layers are layer norm modules ( ln_1 and ln_2 ).
Revised: We observe that 24 of 34 layers are layer norm modules ( ln_1 and ln_2 ).

Original: D.4 CKA Between Different Models
Revised: D.4 CKA Between Different Models

Original: To check if pre-trained models and the randomly initialized model converge to a similar representation or not, we compute the CKA between representations of fine-tuned these models.
Revised: To check if pre-trained models and the randomly initialized model converge to a similar representation or not, we compute the CKA between representations of fine-tuned these models.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: 12, where (a) is the CKA between GPT2 and iGPT, (b) is that between GPT2 and random initialization, and (c) is that between iGPT and random initialization.
Revised: 12, where (a) is the CKA between GPT2 and iGPT, (b) is that between GPT2 and random initialization, and (c) is that between iGPT and random initialization.

Original: We observe that CKA values in these heat maps are small, confirming that different models learn different representations.06121824303642485460667278849096102 GPT2 Layers R A ND O M I N I T
Revised: We observe that CKA values in these heat maps are small, confirming that different models learn different representations.

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]

Original: ) Random Init.

Original: vs iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: (c) GPT2 vs iGPT

Original: D.5 CKA Between Different Layers in a Model D.5.1 Post-Fine-Tuning
Revised: D.5 CKA Between Different Layers in a Model D.5.1 Post-Fine-Tuning

Original:  Appendix 5.1, we find that each layer of the pre-trained model and randomly initialized model learn representation differently.
Revised: In Appendix D.4, we find that each layer of the pre-trained model and randomly initialized model learn representation differently.

Original: In this section, we delve into what representation these models acquire.
Revised: In this section, we delve into what representation these models acquire.

Original: For that purpose, we compute CKA values of the different layers in the same model after fine-tuning.
Revised: For that purpose, we compute CKA values of the different layers in the same model after fine-tuning.

Original: The result for the state input token with the Hopper-medium dataset is shown in Fig.
Revised: The result for the state input token with the Hopper-medium dataset is shown in Fig.

Original: The results for other environments and input tokens are in Appendix J.1.
Revised: The results for other environments and input tokens are in Appendix J.1.

Original: 06121824303642485460667278849096102 GPT2 Layers 0714212835424956637077849198105 G P T 2 La y e r s 0. (a) GPT 0112233445566778899110121132143154165176187198209 IGPT Layers I G P T La y e r s 0. (b) iGPT 06121824303642485460667278849096102 RANDOM INIT Layers
Revised: [Figure]

Original: R A ND O M

Original: I N I T La y e r s 0. (c) Random Initialization Comparing the CKA matrix of pre-trained models (Fig. 13 (a) and Fig.
Revised: Comparing the CKA matrix of pre-trained models (Fig. 13 (a) and Fig.

Original: 13 (b)) with that of the randomly initialized model (Fig.
Revised: 13 (b)) with that of the randomly initialized model (Fig.

Original: 13 (c))
Revised: 13 (c))

Original: , we observe that the randomly initialized model has an almost equally divided lattice structure, while pre-trained models have some block-like structure.
Revised: , we observe that the randomly initialized model has an almost equally divided lattice structure, while pre-trained models have some block-like structure.

Original: In particular, the shallow blocks and the middle to top blocks of language-pre-trained model have a bit similar representation, respectively.
Revised: In particular, the shallow blocks and the middle to top blocks of language-pre-trained model have a bit similar representation, respectively.

Original: This result indicates that probably layers of random initialization process information separately, while some layers of language-pre-trained models coordinate to represent information.
Revised: This result indicates that probably layers of random initialization process information separately, while some layers of language-pre-trained models coordinate to represent information.

Original: D.5.2 Pre-Fine-Tuning
Revised: D.5.2 Pre-Fine-Tuning

Original: For just reference, we put the CKA matrix of pre-trained models before fine-tuning as well.
Revised: For just reference, we put the CKA matrix of pre-trained models before fine-tuning as well.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: We observe the grid structure at the initial state as well.
Revised: We observe the grid structure at the initial state as well.

Original: [-GPT2 Layers 0714212835424956637077849198105 G P T 
-]
Revised: [-GPT2 Layers 0714212835424956637077849198105 G P T 
-]

Original: La y e r s 0.

Original: (a) GPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: I G P T La y e r s 0.

Original: (b) iGPT 06121824303642485460667278849096102 RANDOM INIT Layers

Original: R A ND O M I N I T La y e r s 0. (c) Random Initialization
Revised: [Figure]

Original: E Mutual Information Between Hidden Representation and Data E.1 Details of Experiments
Revised: E Mutual Information Between Hidden Representation and Data E.1 Details of Experiments

Original: MINE, the definition of which is explained in Appendix E.2, estimates mutual information by training a neural network.
Revised: MINE, the definition of which is explained in Appendix E.2, estimates mutual information by training a neural network.

Original: The neural network we use is a feed-forward ReLU network with two hidden layers of width 400.
Revised: The neural network we use is a feed-forward ReLU network with two hidden layers of width 400.

Original: We train the neural network for 1000 iterations by Adam optimizer with a learning rate of 1 e − 4 .
Revised: We train the neural network for 1000 iterations by Adam optimizer with a learning rate of 1 e − 4 .

Original: Same as in Appendix D.1, we randomly sample 100 trajectories and use them to obtain the activation.
Revised: Same as in Appendix D.1, we randomly sample 100 trajectories and use them to obtain the activation.

Original: Thus, the dataset for mutual information calculation is the pair between theseactivation vectors and 100 trajectories for ˆ
Revised: Thus, the dataset for mutual information calculation is the pair between theseactivation vectors and 100 trajectories for ˆ

Original: I ( X ; T ) and 100 last action vectors for ˆ
Revised: I ( X ; T ) and 100 last action vectors for ˆ

Original: I
Revised: I

Original: ( Y ; T ) .
Revised: ( Y ; T ) .

Original: If the estimated value is NaN , we exclude the point from the figure.
Revised: If the estimated value is NaN , we exclude the point from the figure.

Original: The descriptive diagram that shows hidden activation and data we consider to compute estimated mutual information is Fig.
Revised: The descriptive diagram that shows hidden activation and data we consider to compute estimated mutual information is Fig.

Original: 15, where (a) is for ˆ I ( X ; T ) and (b) is for ˆ
Revised: 15, where (a) is for ˆ I ( X ; T ) and (b) is for ˆ

Original: I
Revised: I

Original: ( Y ; T ) .
Revised: ( Y ; T ) .

Original: This is an example of mutual information for the deep Transformer block.
Revised: This is an example of mutual information for the deep Transformer block.

Original: For the middleblock, we use block 6 for GPT and random initialization and block 12 for iGPT. Specifically, the outputs of the shallow, middle, and deep Transformer blocks are the outputs of 0.mlp.dropout , 6.mlp.dropout , and 11.mlp.dropout for GPT2 and randomly initialized models and 0.mlp.dropout , 12.mlp.dropout , and 23.mlp.dropout for
Revised: For the middle

Original: [Figure]
Revised: [Figure]

Original: E.2 Mutual Information Neural Estimation (MINE)
Revised: E.2 Mutual Information Neural Estimation (MINE)

Original: Given random variables X and T , whose joint probability is P XT and marginal distributions are P
Revised: Given random variables X and T , whose joint probability is P XT and marginal distributions are P

Original: X = (cid:82) T d P XT and P T = (cid:82) X d P XT , the mutual information of X and T are as follows:
Revised: X = (cid:82) T d P XT and P T = (cid:82) X d P XT , the mutual information of X and T are as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Denoting neural network with parameters θ ∈ Θ by NN θ and the empirical distribution of P characterized by n i.i.d.
Revised: Denoting neural network with parameters θ ∈ Θ by NN θ and the empirical distribution of P characterized by n i.i.d.

Original: samples by ˆ P ( n ) , MINE is defined as follows [58]:
Revised: samples by ˆ P ( n ) , MINE is defined as follows [57]:

Original: [Equation]
Revised: [Equation]

Original: Please refer to the previous study [58] for the details.
Revised: Please refer to the previous study [57] for the details.

Original: E.3 Mutual Information Without Considering Context
Revised: E.3 Mutual Information Without Considering Context

Original: As a complementary analysis, we also calculate the estimated mutual information between hidden representation and s t and a t .
Revised: As a complementary analysis, we also calculate the estimated mutual information between hidden representation and s t and a t .

Original: In other words, we compute ˆ
Revised: In other words, we compute ˆ

Original: I ( s t ; T l ( s t )) and ˆ I ( a t ; T l ( s t ))
Revised: I ( s t ; T l ( s t )) and ˆ I ( a t ; T l ( s t ))

Original: as ˆ I ( X ; T ) and ˆ I
Revised: as ˆ I ( X ; T ) and ˆ I

Original: ( Y ; T )
Revised: ( Y ; T )

Original: for all time steps in the context t = 1 , ..., K .
Revised: for all time steps in the context t = 1 , ..., K .

Original: Then, we take an average of them over the context.
Revised: Then, we take an average of them over the context.

Original: The descriptive diagram is shown in Fig.
Revised: The descriptive diagram is shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The result for Hopper-medium is shown in Fig.
Revised: The result for Hopper-medium is shown in Fig.

Original: Each point indicates the estimated mean mutual information and the shaded area is the standard deviation.
Revised: Each point indicates the estimated mean mutual information and the shaded area is the standard deviation.

Original: Just as we did in Section 5.1, we take an average over two adjacent elements for iGPT for comparison.
Revised: Just as we did in Section 5.1, we take an average over two adjacent elements for iGPT for comparison.

Original: We again observe that the randomly initialized model generally has more information on input and label, and iGPT has almost nothing about label-related information.
Revised: We again observe that the randomly initialized model generally has more information on input and label, and iGPT has almost nothing about label-related information.

Original: F Parameter Similarity F.1 Details of Experiments
Revised: F Parameter Similarity F.1 Details of Experiments

Original: The parameters considered in the analysis are those in Transformer blocks ( DecisionTransformer. transformer.h ).
Revised: The parameters considered in the analysis are those in Transformer blocks ( DecisionTransformer. transformer.h ).

Original: We concatenate all these parameters into a single vector and compute l 2 distance and cosine similarity for this vector.
Revised: We concatenate all these parameters into a single vector and compute l 2 distance and cosine similarity for this vector.

Original: For the post-fine-tuning model in Section 5.3, we use the model trained with the Hopper-medium dataset.
Revised: For the post-fine-tuning model in Section 5.3, we use the model trained with the Hopper-medium dataset.

Original: The results for other environments are in Appendix J.3.
Revised: The results for other environments are in Appendix J.3.

Original: F.2 Ticks’ Labels of Figures 3 and
Revised: F.2 Ticks’ Labels of Figures 3 and

Original: The labels of ticks of Figs.
Revised: The labels of ticks of Figs.

Original: 3 and 4 in Section 5.3 correspond to all parameter sets .
Revised: 3 and 4 in Section 5.3 correspond to all parameter sets .

Original: For example, one of the parameter sets is 0.ln_1.weight :
Revised: For example, one of the parameter sets is 0.ln_1.weight :

Original: this is the weight vector of the layer normalization in the first ( 0 ) transformer block ( h ).
Revised: this is the weight vector of the layer normalization in the first ( 0 ) transformer block ( h ).

Original: The labels for ticks of GPT2 and the randomly initialized model are summarized below:
Revised: The labels for ticks of GPT2 and the randomly initialized model are summarized below:

Original:  ’0.ln_1.weight’, ’0.ln_1.bias’, ’0.attn.c_attn.weight’, ’0.attn.c_attn.bias’, ’0.attn.c_proj.weight’, ’
Revised:  ’0.ln_1.weight’, ’0.ln_1.bias’, ’0.attn.c_attn.weight’, ’0.attn.c_attn.bias’, ’0.attn.c_proj.weight’, ’

Original: .attn.c_proj.bias’, ’0.ln_2.weight’, ’0.ln_2.bias’, ’0.mlp.c_fc.weight’, ’0.mlp.c_fc.bias’, ’0.mlp.c_proj.weight’, ’0.mlp.c_proj.bias’, ’1.ln_1.weight’, ’1.ln_1.bias’, ’1.attn.c_attn.weight’, ’1.attn.c_attn.bias’, ’1.attn.c_proj.weight’, ’1.attn.c_proj.bias’, ’1.ln_2.weight’, ’1.ln_2.bias’, ’1.mlp.c_fc.weight’, ’1.mlp.c_fc.bias’, ’1.mlp.c_proj.weight’, ’1.mlp.c_proj.bias’,
Revised: .attn.c_proj.bias’, ’0.ln_2.weight’, ’0.ln_2.bias’, ’0.mlp.c_fc.weight’, ’0.mlp.c_fc.bias’, ’0.mlp.c_proj.weight’, ’0.mlp.c_proj.bias’, ’1.ln_1.weight’, ’1.ln_1.bias’, ’1.attn.c_attn.weight’, ’1.attn.c_attn.bias’, ’1.attn.c_proj.weight’, ’1.attn.c_proj.bias’, ’1.ln_2.weight’, ’1.ln_2.bias’, ’1.mlp.c_fc.weight’, ’1.mlp.c_fc.bias’, ’1.mlp.c_proj.weight’, ’1.mlp.c_proj.bias’,

Original: ’2.ln_1.weight’, ’2.ln_1.bias’, ’2.attn.c_attn.weight’, ’2.attn.c_attn.bias’, ’2.attn.c_proj.weight’, ’2.attn.c_proj.bias’, ’2.ln_2.weight’, ’2.ln_2.bias’, ’2.mlp.c_fc.weight’, ’2.mlp.c_fc.bias’, ’2.mlp.c_proj.weight’, ’2.mlp.c_proj.bias’, ’3.ln_1.weight’, ’3.ln_1.bias’, ’3.attn.c_attn.weight’, ’3.attn.c_attn.bias’, ’3.attn.c_proj.weight’, ’3.attn.c_proj.bias’, ’3.ln_2.weight’, ’3.ln_2.bias’, ’3.mlp.c_fc.weight’, ’3.mlp.c_fc.bias’, ’3.mlp.c_proj.weight’, ’3.mlp.c_proj.bias’, ’4.ln_1.weight’, ’4.ln_1.bias’, ’4.attn.c_attn.weight’, ’4.attn.c_attn.bias’, ’4.attn.c_proj.weight’, ’4.attn.c_proj.bias’, ’4.ln_2.weight’, ’4.ln_2.bias’, ’4.mlp.c_fc.weight’, ’4.mlp.c_fc.bias’, ’4.mlp.c_proj.weight’, ’4.mlp.c_proj.bias’, ’5.ln_1.weight’, ’5.ln_1.bias’, ’5.attn.c_attn.weight’, ’5.attn.c_attn.bias’, ’5.attn.c_proj.weight’, ’5.attn.c_proj.bias’, ’5.ln_2.weight’, ’5.ln_2.bias’, ’5.mlp.c_fc.weight’, ’5.mlp.c_fc.bias’, ’5.mlp.c_proj.weight’, ’5.mlp.c_proj.bias’, ’6.ln_1.weight’, ’6.ln_1.bias’, ’6.attn.c_attn.weight’, ’6.attn.c_attn.bias’, ’6.attn.c_proj.weight’, ’6.attn.c_proj.bias’, ’6.ln_2.weight’, ’6.ln_2.bias’, ’6.mlp.c_fc.weight’, ’6.mlp.c_fc.bias’, ’6.mlp.c_proj.weight’, ’6.mlp.c_proj.bias’, ’7.ln_1.weight’, ’7.ln_1.bias’, ’7.attn.c_attn
Revised: ’2.ln_1.weight’, ’2.ln_1.bias’, ’2.attn.c_attn.weight’, ’2.attn.c_attn.bias’, ’2.attn.c_proj.weight’, ’2.attn.c_proj.bias’, ’2.ln_2.weight’, ’2.ln_2.bias’, ’2.mlp.c_fc.weight’, ’2.mlp.c_fc.bias’, ’2.mlp.c_proj.weight’, ’2.mlp.c_proj.bias’, ’3.ln_1.weight’, ’3.ln_1.bias’, ’3.attn.c_attn.weight’, ’3.attn.c_attn.bias’, ’3.attn.c_proj.weight’, ’3.attn.c_proj.bias’, ’3.ln_2.weight’, ’3.ln_2.bias’, ’3.mlp.c_fc.weight’, ’3.mlp.c_fc.bias’, ’3.mlp.c_proj.weight’, ’3.mlp.c_proj.bias’, ’4.ln_1.weight’, ’4.ln_1.bias’, ’4.attn.c_attn.weight’, ’4.attn.c_attn.bias’, ’4.attn.c_proj.weight’, ’4.attn.c_proj.bias’, ’4.ln_2.weight’, ’4.ln_2.bias’, ’4.mlp.c_fc.weight’, ’4.mlp.c_fc.bias’, ’4.mlp.c_proj.weight’, ’4.mlp.c_proj.bias’, ’5.ln_1.weight’, ’5.ln_1.bias’, ’5.attn.c_attn.weight’, ’5.attn.c_attn.bias’, ’5.attn.c_proj.weight’, ’5.attn.c_proj.bias’, ’5.ln_2.weight’, ’5.ln_2.bias’, ’5.mlp.c_fc.weight’, ’5.mlp.c_fc.bias’, ’5.mlp.c_proj.weight’, ’5.mlp.c_proj.bias’, ’6.ln_1.weight’, ’6.ln_1.bias’, ’6.attn.c_attn.weight’, ’6.attn.c_attn.bias’, ’6.attn.c_proj.weight’, ’6.attn.c_proj.bias’, ’6.ln_2.weight’, ’6.ln_2.bias’, ’6.mlp.c_fc.weight’, ’6.mlp.c_fc.bias’, ’6.mlp.c_proj.weight’, ’6.mlp.c_proj.bias’, ’7.ln_1.weight’, ’7.ln_1.bias’, ’7.attn.c_attn

Original: ’7.attn.c_proj.bias’, ’7.ln_2.weight’, ’7.ln_2.bias’, ’7.mlp.c_fc.weight’, ’7.mlp.c_fc.bias’, ’7.mlp.c_proj.weight’, ’7.mlp.c_proj.bias’, ’8.ln_1.weight’, ’8.ln_1.bias’, ’8.attn.c_attn.weight’, ’8.attn.c_attn.bias’, ’8.attn.c_proj.weight’, ’8.attn.c_proj.bias’, ’8.ln_2.weight’, ’8.ln_2.bias’, ’8.mlp.c_fc.weight’, ’8.mlp.c_fc.bias’, ’8.mlp.c_proj.weight’, ’8.mlp.c_proj.bias’, ’9.ln_1.weight’, ’9.ln_1.bias’, ’9.attn.c_attn.weight’, ’9.attn.c_attn.bias’, ’9.attn.c_proj.weight’, ’9.attn.c_proj.bias’, ’9.ln_2.weight’, ’9.ln_2.bias’, ’9.mlp.c_fc.weight’, ’9.mlp.c_fc.bias’, ’9.mlp.c_proj.weight’, ’9.mlp.c_proj.bias’, ’10.ln_1.weight’, ’10.ln_1.bias’, ’10.attn.c_attn.weight’, ’10.attn.c_attn.bias’, ’10.attn.c_proj.weight’, ’10.attn.c_proj.bias’, ’10.ln_2.weight’, ’10.ln_2.bias’, ’10.mlp.c_fc.weight’, ’10.mlp.c_fc.bias’, ’10.mlp.c_proj.weight’, ’10.mlp.c_proj.bias’, ’11.ln_1.weight’, ’11.ln_1.bias’, ’11.attn.c_attn.weight’, ’11.attn.c_attn.bias’, ’11.attn.c_proj.weight’, ’11.attn.c_proj.bias’, ’11.ln_2.weight’, ’11.ln_2.bias’, ’11.mlp.c_fc.weight’, ’11.mlp.c_fc.bias’, ’11.mlp.c_proj.weight’, ’11.mlp.c_proj.bias’
Revised: ’7.attn.c_proj.bias’, ’7.ln_2.weight’, ’7.ln_2.bias’, ’7.mlp.c_fc.weight’, ’7.mlp.c_fc.bias’, ’7.mlp.c_proj.weight’, ’7.mlp.c_proj.bias’, ’8.ln_1.weight’, ’8.ln_1.bias’, ’8.attn.c_attn.weight’, ’8.attn.c_attn.bias’, ’8.attn.c_proj.weight’, ’8.attn.c_proj.bias’, ’8.ln_2.weight’, ’8.ln_2.bias’, ’8.mlp.c_fc.weight’, ’8.mlp.c_fc.bias’, ’8.mlp.c_proj.weight’, ’8.mlp.c_proj.bias’, ’9.ln_1.weight’, ’9.ln_1.bias’, ’9.attn.c_attn.weight’, ’9.attn.c_attn.bias’, ’9.attn.c_proj.weight’, ’9.attn.c_proj.bias’, ’9.ln_2.weight’, ’9.ln_2.bias’, ’9.mlp.c_fc.weight’, ’9.mlp.c_fc.bias’, ’9.mlp.c_proj.weight’, ’9.mlp.c_proj.bias’, ’10.ln_1.weight’, ’10.ln_1.bias’, ’10.attn.c_attn.weight’, ’10.attn.c_attn.bias’, ’10.attn.c_proj.weight’, ’10.attn.c_proj.bias’, ’10.ln_2.weight’, ’10.ln_2.bias’, ’10.mlp.c_fc.weight’, ’10.mlp.c_fc.bias’, ’10.mlp.c_proj.weight’, ’10.mlp.c_proj.bias’, ’11.ln_1.weight’, ’11.ln_1.bias’, ’11.attn.c_attn.weight’, ’11.attn.c_attn.bias’, ’11.attn.c_proj.weight’, ’11.attn.c_proj.bias’, ’11.ln_2.weight’, ’11.ln_2.bias’, ’11.mlp.c_fc.weight’, ’11.mlp.c_fc.bias’, ’11.mlp.c_proj.weight’, ’11.mlp.c_proj.bias’

Original: Label list of
Revised: Label list of

Original: For iGPT, the number of transformer block is twice (from 0-11 to 0-23) but the configuration is the same.
Revised: For iGPT, the number of transformer block is twice (from 0-11 to 0-23) but the configuration is the same.

Original: G Gradient Analysis G.1 Details of Experiments
Revised: G Gradient Analysis G.1 Details of Experiments

Original: We randomly sample 100 samples for gradient norm in Fig.
Revised: We randomly sample 100 samples for gradient norm in Fig.

Original: 6 and 50 samples for gradient confusion in Fig.
Revised: 6 and 50 samples for gradient confusion in Fig.

Original: 5 and compute the gradient of the loss on each of these samples for all parameters of the models fine-tuned after 1 epoch.
Revised: 5 and compute the gradient of the loss on each of these samples for all parameters of the models fine-tuned after 1 epoch.

Original: The bar plot of Fig.
Revised: The bar plot of Fig.

Original: 5 is the minimum of the 50 × 50 =cosine similarities.
Revised: 5 is the minimum of the 50 × 50 =cosine similarities.

Original: Each point of Fig.
Revised: Each point of Fig.

Original: 6 corresponds to the gradient norm of each data sample ingradient norms.
Revised: 6 corresponds to the gradient norm of each data sample ingradient norms.

Original: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.
Revised: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.

Original: The whiskers extend from the box by 1.5 times the inter-quartile range.
Revised: The whiskers extend from the box by 1.5 times the inter-quartile range.

Original: For gradient clipping, we use a method in Pytorch [68] and set the maximum norm to be 0 .
Revised: For gradient clipping, we use a method in Pytorch [67] and set the maximum norm to be 0 .

Original: 25 , following the previous work [9].
Revised: 25 , following the previous work [9].

Original: The process of creating Fig.
Revised: The process of creating Fig.

Original: 7 is as follows.
Revised: 7 is as follows.

Original: We first compute gradient norms per parameter set and examine the bar plot that we will explain in Appendix G.2.
Revised: We first compute gradient norms per parameter set and examine the bar plot that we will explain in Appendix G.2.

Original: Noticing that two peaks exist in the figure, we check the label of the parameter sets and find that they are 0.ln_1.weight and 0.ln_1.bias .
Revised: Noticing that two peaks exist in the figure, we check the label of the parameter sets and find that they are 0.ln_1.weight and 0.ln_1.bias .

Original: Thus, we lump the remaining as others and only show the ratio in the main body of the paper to tell readers the main finding from the observation in an easy-to-understand manner.
Revised: Thus, we lump the remaining as others and only show the ratio in the main body of the paper to tell readers the main finding from the observation in an easy-to-understand manner.

Original: The parameter sets considered to compute the parameter set-wise gradient norms are the same as those in Appendix F.2.
Revised: The parameter sets considered to compute the parameter set-wise gradient norms are the same as those in Appendix F.2.

Original: G.2 Gradient Norm for Each Parameter
Revised: G.2 Gradient Norm for Each Parameter

Original: For visibility, we show only the ratio in Section 5.4 as Fig.
Revised: For visibility, we show only the ratio in Section 5.4 as Fig.

Original: The full results for gradient norms of each parameter are shown in Fig.
Revised: The full results for gradient norms of each parameter are shown in Fig.

Original: The labels of the ticks on the x-axis are the same as that in
Revised: The labels of the ticks on the x-axis are the same as that in

Original: Appendix F.2.
Revised: Appendix F.2.

Original: We immediately notice that the first two parameters ( 0.ln_1.weight and 0.ln_1.bias ) are much larger than the remains.
Revised: We immediately notice that the first two parameters ( 0.ln_1.weight and 0.ln_1.bias ) are much larger than the remains.

Original: [Figure]
Revised: [Figure]

Original: G.3 Analysis of the Effect of Gradient Clipping
Revised: G.3 Analysis of the Effect of Gradient Clipping

Original: In Section 5.4, we noted that large gradient and gradient clipping could be a cause of the bad performance of iGPT.
Revised: In Section 5.4, we noted that large gradient and gradient clipping could be a cause of the bad performance of iGPT.

Original: To further explore this point, we conducted a simple experiment.
Revised: To further explore this point, we conducted a simple experiment.

Original: The experiments of the previous study [9] and our experiments both used a Pytorch function called torch.nn.utils.clip_grad_norm_ 9 for gradient clipping.
Revised: The experiments of the previous study [9] and our experiments both used a Pytorch function called torch.nn.utils.clip_grad_norm_ 9 for gradient clipping.

Original: This function divides each gradient by the total norm of all gradients and multiplies the clipping constant (gradient norm clipping).
Revised: This function divides each gradient by the total norm of all gradients and multiplies the clipping constant (gradient norm clipping).

Original: However, as we pointed out in Section 5.4, the gradient norm values are dominated by only a few parameters (the layer normalization layer in the first block).
Revised: However, as we pointed out in Section 5.4, the gradient norm values are dominated by only a few parameters (the layer normalization layer in the first block).

Original: This large norm affects the normalization of all gradients, decreasing the informational value of most gradients.
Revised: This large norm affects the normalization of all gradients, decreasing the informational value of most gradients.

Original: Therefore, we hypothesized that one of the possible reasons why iGPT is difficult to learn can be the use of torch.nn
Revised: Therefore, we hypothesized that one of the possible reasons why iGPT is difficult to learn can be the use of torch.nn

Original: .utils.clip_grad_norm_
Revised: .utils.clip_grad_norm_

Original: function .
Revised: function .

Original: Thus we trained the image-pre-trained model without using gradient clipping.
Revised: Thus we trained the image-pre-trained model without using gradient clipping.

Original: The basic experimental setting is the same as that of Section 5.4 except that we trained the model for 10 epochs, instead of 40 epochs.
Revised: The basic experimental setting is the same as that of Section 5.4 except that we trained the model for 10 epochs, instead of 40 epochs.

Original: The results are shown in Figs 19 and 20 for action error and mean return, respectively.
Revised: The results are shown in Figs 19 and 20 for action error and mean return, respectively.

Original: The x-axis is the epoch.
Revised: The x-axis is the epoch.

Original: These figures show that eliminating gradient clipping does not immediately solve the catastrophic performance at least up to 10 epochs.
Revised: These figures show that eliminating gradient clipping does not immediately solve the catastrophic performance at least up to 10 epochs.

Original: However, we did find that action error is much smaller for no gradient clipping condition, indicating that the learning process of it seems to be more stable and efficient than the clipping was applied.
Revised: However, we did find that action error is much smaller for no gradient clipping condition, indicating that the learning process of it seems to be more stable and efficient than the clipping was applied.

Original: We also found that the return means seem to improve, albeit very slightly.
Revised: We also found that the return means seem to improve, albeit very slightly.

Original: Thus, we can conclude that gradient clipping is a cause of the bad performance of iGPT, though it does not seem to be critical.
Revised: Thus, we can conclude that gradient clipping is a cause of the bad performance of iGPT, though it does not seem to be critical.

Original: Exploring the critical factor that makes iGPT performance catastrophic is left a future work.
Revised: Exploring the critical factor that makes iGPT performance catastrophic is left a future work.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H Fine-Tuning with No Context Information H.1 Details of Experiments
Revised: H Fine-Tuning with No Context Information H.1 Details of Experiments

Original: The configuration of the training is the same as that described in Appendix B except that the context is 1 (no context).
Revised: The configuration of the training is the same as that described in Appendix B except that the context is 1 (no context).

Original: We evaluate the model per epoch and the normalized score in Table 4 is the result of the best checkpoint in the 40 checkpoints.
Revised: We evaluate the model per epoch and the normalized score in Table 4 is the result of the best checkpoint in the 40 checkpoints.

Original: The mean return reported in Fig.
Revised: The mean return reported in Fig.

Original: 8 and Table 4 is the average of the returns over the trajectory.
Revised: 8 and Table 4 is the average of the returns over the trajectory.

Original: Table 4 is the result of one random seed (seed = 666).
Revised: Table 4 is the result of one random seed (seed = 666).

Original: The mean and standard deviation of the normalized score for the two seeds (seed = 666 & 42) and the mean return of another seed (seed = 42) are reported in Appendix J.5.
Revised: The mean and standard deviation of the normalized score for the two seeds (seed = 666 & 42) and the mean return of another seed (seed = 42) are reported in Appendix J.5.

Original: In this paper, we mean context by the number of accessible time steps for a model to predict, following the previous study [36].
Revised: In this paper, we mean context by the number of accessible time steps for a model to predict, following the previous study [35].

Original: If the context length is K , the model can use inputs in past K steps to predict the action at the current step, while the context length is 1 (no context), the model has to predict only from the current time step.
Revised: If the context length is K , the model can use inputs in past K steps to predict the action at the current step, while the context length is 1 (no context), the model has to predict only from the current time step.

Original: [Figure]
Revised: [Figure]

Original: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context
Revised: H.2 Analysis of Why Randomly Initialized Model Fails for Hopper with No Context

Original: In Section 5.5, we observe that the performance of randomly initialized models for the Hopper environment is particularly worse than that for other environments (Table 4).
Revised: In Section 5.5, we observe that the performance of randomly initialized models for the Hopper environment is particularly worse than that for other environments (Table 4).

Original: In this Section, we further explore a possible cause of this observation.
Revised: In this Section, we further explore a possible cause of this observation.

Original: We speculate that this may be because how much of the range of context that needs to be looked at changes depending on the data set.
Revised: We speculate that this may be because how much of the range of context that needs to be looked at changes depending on the data set.

Original: For example, prior studies on decision transformer reported that the effect of context varied depending on the task [36].
Revised: For example, prior studies on decision transformer reported that the effect of context varied depending on the task [35].

Original: Improvement by context means that the context is important information for solving the task.
Revised: Improvement by context means that the context is important information for solving the task.

Original: Thus, it could be possible that Hopper is the dataset that requires more information from context than the other two data sets.
Revised: Thus, it could be possible that Hopper is the dataset that requires more information from context than the other two data sets.

Original: As a test of the hypothesis, we randomly sampled a batch sample and calculated the mutual information between action and state or return-to-go at the same time step and compare them between different environments; note that this is the mutual information between the data, not between the data and representation.
Revised: As a test of the hypothesis, we randomly sampled a batch sample and calculated the mutual information between action and state or return-to-go at the same time step and compare them between different environments; note that this is the mutual information between the data, not between the data and representation.

Original: The higher the value of the mutual information, the higher the mutual dependence between state or return-to-go and action at the same time step.
Revised: The higher the value of the mutual information, the higher the mutual dependence between state or return-to-go and action at the same time step.

Original: Hence, higher mutual information suggests that the model could predict action better only from the information at the current time step.
Revised: Hence, higher mutual information suggests that the model could predict action better only from the information at the current time step.

Original: In particular, we sample 100 samples of context length K = 20 .
Revised: In particular, we sample 100 samples of context length K = 20 .

Original: Then, for all time step t , we compute the estimated mutual information ˆ I ( s t ; a t ) between state s t and action a t and that ˆ
Revised: Then, for all time step t , we compute the estimated mutual information ˆ I ( s t ; a t ) between state s t and action a t and that ˆ

Original: I ( ˆ R t ; a t ) between return-to-go ˆ R t and action.
Revised: I ( ˆ R t ; a t ) between return-to-go ˆ R t and action.

Original: Other setup for mutual information estimation is the same as that of
Revised: Other setup for mutual information estimation is the same as that of

Original: Section 5.2.
Revised: Section 5.2.

Original: The result is shown in Fig 22, where (a) is the box plot for the state and action and (b) is that for the return-to-go and action.
Revised: The result is shown in Fig 22, where (a) is the box plot for the state and action and (b) is that for the return-to-go and action.

Original: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.
Revised: The top of the box is the 1st quartile and the bottom of the box is the 3rd quartile of the points.

Original: The whiskers extend from the box by 1.5 times the inter-quartile range.
Revised: The whiskers extend from the box by 1.5 times the inter-quartile range.

Original: Each point corresponds to different time step in the context.
Revised: Each point corresponds to different time step in the context.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We observe that the estimated mutual information between return-to-go and action is smaller on average for Hopper than for other environments (Fig.
Revised: We observe that the estimated mutual information between return-to-go and action is smaller on average for Hopper than for other environments (Fig.

Original: 22 (b)), though that between action and state does not differ that much (Fig. 

Revised: 22 (b)), though that between action and state does not differ that much (Fig.

Original: (a)).
Revised: (a)).

Original: Prior research has indicated that return-to-go information seems to be important for prediction [9].
Revised: Prior research has indicated that return-to-go information seems to be important for prediction [9].

Original: Hence, we can say that models have to use more information from the other steps in the context to solve the Hopper task than models do for other environments.
Revised: Hence, we can say that models have to use more information from the other steps in the context to solve the Hopper task than models do for other environments.

Original: This result supports our hypothesis above.
Revised: This result supports our hypothesis above.

Original: I More In-Depth Analysis of Context Dependence I.1 Replacement by the Pre-Trained Block I.1.1 Details of Experiments
Revised: I More In-Depth Analysis of Context Dependence I.1 Replacement by the Pre-Trained Block I.1.1 Details of Experiments

Original: The training configuration is the same as that of Section 5.5.
Revised: The training configuration is the same as that of Section 5.5.

Original: The smoothing factor of the exponential moving average for Fig.
Revised: The smoothing factor of the exponential moving average for Fig.

Original: 9 is 0.8.
Revised: 9 is 0.8.

Original: The action error is the mean square loss between the true action a t and the predicted action ˆ a t .
Revised: The action error is the mean square loss between the true action a t and the predicted action ˆ a t .

Original: Considering that the goal of this experiment is to highlight the effect of pre-training, we train the model for 10 epochs for Hopper and 5 epochs for HalfCheetah and
Revised: Considering that the goal of this experiment is to highlight the effect of pre-training, we train the model for 10 epochs for Hopper and 5 epochs for HalfCheetah and

Original: Walker2D because Fig.
Revised: Walker2D because Fig.

Original: 8 shows that the difference between the randomly initialized and pre-trained model is evident during these epochs.
Revised: 8 shows that the difference between the randomly initialized and pre-trained model is evident during these epochs.

Original: I.2 Attention Distance Analysis I.2.1 Details of Experiments
Revised: I.2 Attention Distance Analysis I.2.1 Details of Experiments

Original: In the main body of this paper, we mean attention distance by average/mean attention distance in the previous work [60, 44].
Revised: In the main body of this paper, we mean attention distance by average/mean attention distance in the previous work [59, 43].

Original: Because we use only one attention head in this study, there is a single point per sample in Fig.
Revised: Because we use only one attention head in this study, there is a single point per sample in Fig.

Original: We randomly sample 100 trajectory samples and compute attention for these trajectories.
Revised: We randomly sample 100 trajectory samples and compute attention for these trajectories.

Original: When calculating attention distance, we allow the model to access the context of
Revised: When calculating attention distance, we allow the model to access the context of

Original: K = 20 length and compute the attention distance up to the length.
Revised: K = 20 length and compute the attention distance up to the length.

Original: Each point of the box plot (Fig.
Revised: Each point of the box plot (Fig.

Original: 10) corresponds to the attention distance of each sample.
Revised: 10) corresponds to the attention distance of each sample.

Original: The configuration for the box plot is the same as that in Appendix G.
Revised: The configuration for the box plot is the same as that in Appendix G.

Original: J Results for Other Conditions J.1 Activation Similarity J.1.1 CKA Between Pre and Post-Fine-Tuning
Revised: J Results for Other Conditions J.1 Activation Similarity J.1.1 CKA Between Pre and Post-Fine-Tuning

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J
Revised: J

Original: CKA
Revised: CKA

Original: Between Pre and Post-Fine-Tuning (Seed = 42)
Revised: Between Pre and Post-Fine-Tuning (Seed = 42)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J.1.3 CKA Between Different Models

Original: [Figure]
Revised: [Figure]

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]

Original: [Figure]

Original: Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]

Original: Init

Original: [Figure]

Original: iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: [-06121824303642485460667278849096102 GPT2 Layers 
-]
Revised: [-06121824303642485460667278849096102 GPT2 Layers 
-]

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (b) Random Init.

Original: vs iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: (c) GPT2 vs iGPT 06121824303642485460667278849096102 GPT2 Layers R A ND O M I N I T
Revised: J.1.3 CKA Between Different Models

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (a) Random Init.

Original: vs GPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (b) Random Init.

Original: vs iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: [-06121824303642485460667278849096102 GPT2 Layers 
-]
Revised: [-06121824303642485460667278849096102 GPT2 Layers 
-]

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (a) Random Init.

Original: vs GPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (b) Random Init.

Original: vs iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: (c) GPT2 vs iGPT

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (a) Random Init

Original: [Figure]
Revised: [Figure]

Original: GPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]

Original: iGPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]

Original: J.1.4 CKA Between Different Layers in a Model
Revised: J.1.4 CKA Between Different Layers in a Model

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: I G P T La y e r s 0.

Original: [Figure][Figure] [Figure] R A ND O M
Revised: [Figure]

Original: I N I T

Original: La y e r s 0.

Original: [Figure]

Original: [Figure]

Original: 0714212835424956637077849198105 G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT

Original: [Figure]
Revised: [Figure]

Original: I G P T La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: iGPT

Original: [Figure]
Revised: [Figure]

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 06121824303642485460667278849096102 GPT2 Layers 0714212835424956637077849198105 G P T 2 La y e r s 0. (a [Figure] GPT 0112233445566778899110121132143154165176187198209 IGPT Layers
Revised: J.2 Mutual Information Between Hidden Representation and Data

Original: I G P T La y e r s 0.

Original: (b

Original: [Figure]
Revised: [Figure]

Original: 06121824303642485460667278849096102 RANDOM INIT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: ) Random Initialization

Original: 06121824303642485460667278849096102 GPT2 Layers

Original: G P T 2 La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: I G P T La y e r s 0.

Original: [Figure]
Revised: [Figure]

Original: iGPT

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 06121824303642485460667278849096102 GPT2 Layers 0714212835424956637077849198105 G P T 2 La y e r s 0.

Original: (a) GPT 0112233445566778899110121132143154165176187198209 IGPT Layers

Original: I G P T La y e r s 0.

Original: (b) iGPT 06121824303642485460667278849096102 RANDOM INIT Layers

Original: R A ND O M

Original: I N I T

Original: La y e r s 0.

Original: (c) Random Initialization

Original: J.2 Mutual Information Between Hidden Representation and Data

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: I ( Y ;
Revised: I ( Y ;

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Init GPT
Revised: Init GPT

Original: [Figure]
Revised: [Figure]

Original: ˆ
Revised: ˆ

Original: I ( Y ; T )
Revised: I ( Y ; T )

Original: J.3 Parameter Similarity J.3.1 Parameter Similarity (Other Environment)
Revised: J.3 Parameter Similarity J.3.1 Parameter Similarity (Other Environment)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: J.3.2 Parameter Similarity (Seed = 42)
Revised: J.3.2 Parameter Similarity (Seed = 42)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ||
Revised: ||

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: J.4 Gradient Analysis J.4.1 Gradient Analysis (Other Environments)
Revised: J.4 Gradient Analysis J.4.1 Gradient Analysis (Other Environments)

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: M i n o f c o ss i m ( 1 ,
Revised: M i n o f c o ss i m ( 1 ,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: iGPT
Revised: iGPT

Original: Random Init 0.
Revised: Random Init 0.

Original: M i n o f c o ss i m ( 1 , 2 )
Revised: M i n o f c o ss i m ( 1 , 2 )

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: li pped
Revised: li pped

Original: G r ad i en t N
Revised: G r ad i en t N

Original: [Figure]
Revised: [Figure]

Original: r m
Revised: r m

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C li pped
Revised: C li pped

Original: G r ad i en t N o r m
Revised: G r ad i en t N o r m

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: r ad i en t N o r m
Revised: r ad i en t N o r m

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J
Revised: J

Original: -Tuning with No Context Information
Revised: -Tuning with No Context Information

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: J.6.2 Attention Distance Analysis
Revised: J.6.2 Attention Distance Analysis

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: K Link to the Anonymized Repository
Revised: K Link to the Anonymized Repository

Original: The URL for our anonymized repository is https://anonymous.4open.science/r/pre-training-differentmodality-offline-rl-35A0/README.md
Revised: The URL for our anonymized repository is https://anonymous.4open.science/r/pre-training-differentmodality-offline-rl-35A0/README.md

Original: L
Revised: L

Original: Details
Revised: Details

Original: for Computation
Revised: for Computation

Original: The computational environment for our experiments is as follows:
Revised: The computational environment for our experiments is as follows:

Original: • GPU: NVIDIA GeForce GTX TITAN
Revised: • GPU: NVIDIA GeForce GTX TITAN

Original: X × 2GPU.
Revised: X × 2GPU.

Original: • CPU: Intel Xeon E5-2620 v3 2.4 GHz, 6 Cores × 2CPU.
Revised: • CPU: Intel Xeon E5-2620 v3 2.4 GHz, 6 Cores × 2CPU.

Original: • software: Ubuntu 20.04.4, GCC 9.4.0, CUDA 11.6.
Revised: • software: Ubuntu 20.04.4, GCC 9.4.0, CUDA 11.6.

Original: The computational time of fine-tuning for 40 epochs is around from 24 to 72 hours for context length
Revised: The computational time of fine-tuning for 40 epochs is around from 24 to 72 hours for context length

Original: K = 20 and around from 7 to 48 hours for context length K = 1 per run.
Revised: K = 20 and around from 7 to 48 hours for context length K = 1 per run.

Original: M Licences of Assets Used for Our Experiments M.1 Dataset
Revised: M Licences of Assets Used for Our Experiments M.1 Dataset

Original: We use the offline RL dataset of Mujoco in D4RL [39].
Revised: We use the offline RL dataset of Mujoco in D4RL [38].

Original: The license of the dataset in D4RL is a
Revised: The license of the dataset in D4RL is a

Original: Creative Commons Attribution 4.0 License (CC BY).
Revised: Creative Commons Attribution 4.0 License (CC BY).

Original: Thus, we can use this dataset with no consent as long as we follow the term of use.
Revised: Thus, we can use this dataset with no consent as long as we follow the term of use.

Original: M.2 Code
Revised: M.2 Code

Original: The code we used does not require special consent from authors as long as we follow the terms of use.
Revised: The code we used does not require special consent from authors as long as we follow the terms of use.

Original: Their licenses are as follows:
Revised: Their licenses are as follows:

Original: • https://github.com/machelreid/can-wikipedia-help-offline-rl: MIT Licence.
Revised: • https://github.com/machelreid/can-wikipedia-help-offline-rl: MIT Licence.

Original: • https://github.com/rail-berkeley/d4rl:
Revised: • https://github.com/rail-berkeley/d4rl:

Original: Apache License 2.0.
Revised: Apache License 2.0.

Original: • https://github.com/google-research/google-research/tree/master/representation_similarity
Revised: • https://github.com/google-research/google-research/tree/master/representation_similarity

Original: ApacheLicense2.0.
Revised: ApacheLicense2.0.

Original: • https://github.com/gtegner/mine-pytorch: MIT License.
Revised: • https://github.com/gtegner/mine-pytorch: MIT License.

Original: N Note on The Role of Each Sub-Sections in Section
Revised: N Note on The Role of Each Sub-Sections in Section

Original: Sections 5.1 and 5.2 show what may not be a cause of the good performance.
Revised: Sections 5.1 and 5.2 show what may not be a cause of the good performance.

Original: In other words, these sections eliminate some seemingly possible causes of good performance of language-pre-trained models, respectively.
Revised: In other words, these sections eliminate some seemingly possible causes of good performance of language-pre-trained models, respectively.

Original: These analyses further highlight the importance of attention distance for performance.
Revised: These analyses further highlight the importance of attention distance for performance.

Original: Without the analysis of Section 5.1, we cannot exclude the possibility that good performance comes from re-using pre-trained representation as well , which is common in the unimodal case as we explained in Section 5.1.
Revised: Without the analysis of Section 5.1, we cannot exclude the possibility that good performance comes from re-using pre-trained representation as well , which is common in the unimodal case as we explained in Section 5.1.

Original: Similarly, without the analysis of Section 5.2, the performance can benefit from fitting better into training data as well .
Revised: Similarly, without the analysis of Section 5.2, the performance can benefit from fitting better into training data as well .

Original: In that case, it would be less clear whether utilizing some prior knowledge is solely important.
Revised: In that case, it would be less clear whether utilizing some prior knowledge is solely important.

Original: Sections 5.4, 5.5, 5.6.1, and 5.6.2 discuss what may be a cause of the good performance, as summarized at the end of Section 5.
Revised: Sections 5.4, 5.5, 5.6.1, and 5.6.2 discuss what may be a cause of the good performance, as summarized at the end of Section 5.

Original: Section 5.3 provides complementary findings for these sections.
Revised: Section 5.3 provides complementary findings for these sections.

Original: For example, the existence of some unchanged parameters implies the possibility that some information is re-used, while the relatively larger change in only shallower layers of pre-trained models partially supports that these unchanged parameters do not seem to contradict with changed representation observed in Section 5.1, as explained in Appendix O.
Revised: For example, the existence of some unchanged parameters implies the possibility that some information is re-used, while the relatively larger change in only shallower layers of pre-trained models partially supports that these unchanged parameters do not seem to contradict with changed representation observed in Section 5.1, as explained in Appendix O.

Original: O Note on Relationship Between Parameter Change and Representation Change
Revised: O Note on Relationship Between Parameter Change and Representation Change

Original: In Section 5.1 we say that representation in some layers of the pre-trained model changes, while in
Revised: In Section 5.1 we say that representation in some layers of the pre-trained model changes, while in

Original: Section 5.3 we say that parameters of them in some layers do not change that much, or vice versa for the randomly initialized model.
Revised: Section 5.3 we say that parameters of them in some layers do not change that much, or vice versa for the randomly initialized model.

Original: These observations of Sections 5.1 and 5.3 are not necessarily contradictory because the change of parameters in layer ℓ does not correspond one-to-one with that in representation in the layer.
Revised: These observations of Sections 5.1 and 5.3 are not necessarily contradictory because the change of parameters in layer ℓ does not correspond one-to-one with that in representation in the layer.

Original: First, the representation of layer ℓ is affected by all parameters and input data prior to layer ℓ .
Revised: First, the representation of layer ℓ is affected by all parameters and input data prior to layer ℓ .

Original: So, even if the parameters of layer ℓ have not changed, if the parameters before layer ℓ have changed, the representation of layer ℓ may change.
Revised: So, even if the parameters of layer ℓ have not changed, if the parameters before layer ℓ have changed, the representation of layer ℓ may change.

Original: Second, since not all of the parameters of a neural network necessarily contribute to the output, the output of a layer may not change even if some parameters of that layer change.
Revised: Second, since not all of the parameters of a neural network necessarily contribute to the output, the output of a layer may not change even if some parameters of that layer change.

Original: For example, if the input value to ReLU is negative, the output will remain 0 no matter how much the parameters involved change.
Revised: For example, if the input value to ReLU is negative, the output will remain 0 no matter how much the parameters involved change.

Original: So, for example, it is possible that even if the parameters of the ℓ -layer change, the representation of the ℓ -layer remains the same.
Revised: So, for example, it is possible that even if the parameters of the ℓ -layer change, the representation of the ℓ -layer remains the same.

Original: Another example is when the values of two weights w ℓ 1 and w ℓ 2 in layer ℓ do not change the output of the vanilla feed-forward network because of its symmetry.
Revised: Another example is when the values of two weights w ℓ 1 and w ℓ 2 in layer ℓ do not change the output of the vanilla feed-forward network because of its symmetry.

Original: Another possible cause of this observation unique to the current analysis is that we use CKA to measure representational similarity.
Revised: Another possible cause of this observation unique to the current analysis is that we use CKA to measure representational similarity.

Original: CKA is designed to be invariant to some transformations on the representation matrix [40].
Revised: CKA is designed to be invariant to some transformations on the representation matrix [39].

Original: Thus, different representation matrix is regarded as the same under these transformations even when parameters are changed.
Revised: Thus, different representation matrix is regarded as the same under these transformations even when parameters are changed.

Original: P Note on Why Better Action Prediction Not Always Result in Better Return
Revised: P Note on Why Better Action Prediction Not Always Result in Better Return

Original: In Section 5.2, we explained that randomly initialized models seem to predict action better, while in
Revised: In Section 5.2, we explained that randomly initialized models seem to predict action better, while in

Original: Table 1, they perform worse than GPT2.
Revised: Table 1, they perform worse than GPT2.

Original: We will add a possible explanation of why better action prediction not necessarily comes to a better return.
Revised: We will add a possible explanation of why better action prediction not necessarily comes to a better return.

Original: The most likely cause of this observation is the current problem setup.
Revised: The most likely cause of this observation is the current problem setup.

Original: As mentioned in Appendix B, the medium dataset is early stopped and thus does not necessarily converge to an optimal policy.
Revised: As mentioned in Appendix B, the medium dataset is early stopped and thus does not necessarily converge to an optimal policy.

Original: This means that if the model accurately learns trajectory and predicts the next action, it does not always mean that it is the best action.
Revised: This means that if the model accurately learns trajectory and predicts the next action, it does not always mean that it is the best action.

Original: Hence, we can see that a low action error does not necessarily mean a large mean return.
Revised: Hence, we can see that a low action error does not necessarily mean a large mean return.

Original: However, We do not believe that the use of medium will hurt the validity of the analysis using this data since this dataset is not that pathological.
Revised: However, We do not believe that the use of medium will hurt the validity of the analysis using this data since this dataset is not that pathological.

Original: Another cause might be related to mutual information.
Revised: Another cause might be related to mutual information.

Original: The current analysis of mutual information shows that the representation of the hidden layer of the randomly initialized model holds more information as well of the input as well as the labels.
Revised: The current analysis of mutual information shows that the representation of the hidden layer of the randomly initialized model holds more information as well of the input as well as the labels.

Original: Since the neural net representation is a vector representation if a single vector contains both types of information, these types of information might have been mixed.
Revised: Since the neural net representation is a vector representation if a single vector contains both types of information, these types of information might have been mixed.

Original: This may make it difficult to properly utilize only the label information, making it harder to accurately predict the action.
Revised: This may make it difficult to properly utilize only the label information, making it harder to accurately predict the action.

Original: Another possibility is that the pre-trained model and the randomly initialized model differ in terms of which input token type (return-to-go, state, or action) information they encode.
Revised: Another possibility is that the pre-trained model and the randomly initialized model differ in terms of which input token type (return-to-go, state, or action) information they encode.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 2 (b), the language pre-trained model encodes information uniformly from the same token type: the hidden representation with high mutual information is the representation corresponding to the input from the triangles, i.e., the part corresponding to the return-to-go.
Revised: 2 (b), the language pre-trained model encodes information uniformly from the same token type: the hidden representation with high mutual information is the representation corresponding to the input from the triangles, i.e., the part corresponding to the return-to-go.

Original: On the other hand, in the randomly initialized models, there are variations (triangle, circle, and cross).
Revised: On the other hand, in the randomly initialized models, there are variations (triangle, circle, and cross).

Original: It was noted in a previous study that attention weight was strong between the same token types [9].
Revised: It was noted in a previous study that attention weight was strong between the same token types [9].

Original: Therefore, perhaps the pre-trained model might encode less total information, but the amount of information effectively utilized is not that different.
Revised: Therefore, perhaps the pre-trained model might encode less total information, but the amount of information effectively utilized is not that different.

Original: Finally, as pointed out in the explanation about the limitation, the limitations of mutual information as a metric might be a cause.
Revised: Finally, as pointed out in the explanation about the limitation, the limitations of mutual information as a metric might be a cause.