Original: There are two situations where the agent receives a reward: one is uponing arriving the good states at the last timestep, the agent receives a reward of 1.
Revised: There are two situations where the agent receives a reward: one is uponing arriving the good states at the last timestep, the agent receives a reward of 1.

Original: The other is upon the first ever transition into the bad state, the agent receives an “anti-shaped” reward of 0.1 with probability 0.5.
Revised: The other is upon the first ever transition into the bad state, the agent receives an “anti-shaped” reward of 0.1 with probability 0.5.

Original: Such design makes greedy algorithms without strategic exploration such as policy optimization methods easily fail.
Revised: Such design makes greedy algorithms without strategic exploration such as policy optimization methods easily fail.

Original: For the initial state distribution, the environment starts in s 0;0 or s 1;0 with equal probability.
Revised: For the initial state distribution, the environment starts in s 0;0 or s 1;0 with equal probability.

Original: The dimension of the observation is 2 ⌈ log( H + |S| +1) ⌉ .
Revised: The dimension of the observation is 2 ⌈ log( H + |S| +1) ⌉ .

Original: For the emission distribution, given a latent state s i ; h , the observation is generated by first concatenate the one hot vectors of the state and the horizon, adding i.i.d.
Revised: For the emission distribution, given a latent state s i ; h , the observation is generated by first concatenate the one hot vectors of the state and the horizon, adding i.i.d.

Original: N (0,0.1) noise at each entry, appending 0 at the end if necessary.
Revised: N (0,0.1) noise at each entry, appending 0 at the end if necessary.

Original: Then finally we apply a linear transfermation on the observation with a Hadamard matrix.
Revised: Then finally we apply a linear transfermation on the observation with a Hadamard matrix.

Original: Note that without a good feature or strategic exploration, it takes 10 H actions to reach the final goal with random actions.
Revised: Note that without a good feature or strategic exploration, it takes 10 H actions to reach the final goal with random actions.

Original: K. C ONSTRUCTION OF TRANSFER SETUP IN S ECTION . J. 
Revised: I.2 Construction of transfer setup in Section. 6.

Original: In this section we introduce the detailed construction of the experiment in Section.
Revised: In this section we introduce the detailed construction of the experiment in Section.

Original: J.1.

Original: For the source environment, we simply generate 5 random vanilla comblock environment described in Section.
Revised: For the source environment, we simply generate 5 random vanilla comblock environment described in Section.

Original: K.1.
Revised: I.1.

Original: Note that in this way we ensure that the emission distribution shares across the sources, but the latent dynamics are different because the optimal actions are independently randomly selected.
Revised: Note that in this way we ensure that the emission distribution shares across the sources, but the latent dynamics are different because the optimal actions are independently randomly selected.

Original: For the target environment, for each timestep h , we randomly acquire the optimal actions at h from one of the sources and set it to be the optimal action of the target environment at timestep h , if the selected optimal actions are different for the two good states.
Revised: For the target environment, for each timestep h , we randomly acquire the optimal actions at h from one of the sources and set it to be the optimal action of the target environment at timestep h , if the selected optimal actions are different for the two good states.

Original: Otherwise we keep sampling until they are different.
Revised: Otherwise we keep sampling until they are different.

Original: Note that under such construction, since we fix the emission distribution, Assumption 2. is satisfied if we set α = 1 for the source environment where we select the optimal action and α =for the other sources, at each timestep.
Revised: Note that under such construction, since we fix the emission distribution, Assumption 3. is satisfied if we set α = 1 for the source environment where we select the optimal action and α =for the other sources, at each timestep.

Original: To see how Assumption ?? is satisfied, recall that comblock environment naturally satisfies Assumption 2.3, and identical emission implies that the conditional ratio of all observations between source and target is 1.
Revised: To see how Assumption 5.1 is satisfied, recall that comblock environment naturally satisfies Assumption 3.3, and identical emission implies that the conditional ratio of all observations between source and target is 1.

Original: K. C ONSTRUCTION OF TRANSFER SETUP IN S ECTION . J. 
Revised: I.3 Construction of transfer setup in Section. 6.

Original: Now we introduce the construction of the Comblock with Partitioned Observation (ComblockPO) environment.
Revised: Now we introduce the construction of the Comblock with Partitioned Observation (ComblockPO) environment.

Original: Comparing with the vanilla comblock environment, the major difference is in the observation space.
Revised: Comparing with the vanilla comblock environment, the major difference is in the observation space.

Original: In this setting, the size of the observation depends on the number of source environments K .
Revised: In this setting, the size of the observation depends on the number of source environments K .

Original: Let the size of the original observation space be O = |O| , the size of the observation for comblock-PO is KO .
Revised: Let the size of the original observation space be O = |O| , the size of the observation for comblock-PO is KO .

Original: For the k -th source environment, where k ∈ [ K ] , the environment first generates the O -dimensional observation vector as in the original comblock, and then embed it to the ( k − 1) O -th to kO -th entries of the KO -dimensional observation vector, where it is 0 everywhere else.
Revised: For the k -th source environment, where k ∈ [ K ] , the environment first generates the O -dimensional observation vector as in the original comblock, and then embed it to the ( k − 1) O -th to kO -th entries of the KO -dimensional observation vector, where it is 0 everywhere else.

Original: Thus we can see that the observation space for each source environment is disjoint (and thus the name partitioned observations).
Revised: Thus we can see that the observation space for each source environment is disjoint (and thus the name partitioned observations).

Original: For the target enviornment, since the latent dynamcis are the same, we only need to design the emission distribution: for each latent state s i ; h , we assign the emission distribution uniformly at random from one of the sources.
Revised: For the target enviornment, since the latent dynamcis are the same, we only need to design the emission distribution: for each latent state s i ; h , we assign the emission distribution uniformly at random from one of the sources.

Original: K. I MPLEMENTATION DETAILS
Revised: I.4 Implementation details

Original: Our implementation builds on B RIEE (Zhang et al., 2022).
Revised: Our implementation builds on B RIEE (Zhang et al., 2022).

Original: In the Multi-task R EP L EARN stage, we requires our learned feature to predict the Bellman backup of all the sources simultaneously.
Revised: In the Multi-task R EP L EARN stage, we requires our learned feature to predict the Bellman backup of all the sources simultaneously.

Original: Therefore, in each iteration we have k discriminators and k sets of linear weights (instead of 1 in
Revised: Therefore, in each iteration we have k discriminators and k sets of linear weights (instead of 1 in

Original: B RIEE ), where k is the number of source environments.
Revised: B RIEE ), where k is the number of source environments.

Original: For the deployment stage we implement
Revised: For the deployment stage we implement

Original: LSVI following Algorithm 6.
Revised: LSVI following Algorithm 6.

Original: To create the training dataset for Multi-task R EP L EARN , for each ( i, j ) environment pairs where i ̸
Revised: To create the training dataset for Multi-task R EP L EARN , for each ( i, j ) environment pairs where i ̸

Original: = j , we collect 500 samples for each timestep h .
Revised: = j , we collect 500 samples for each timestep h .

Original: For each ( i, i ) environment pairs, we collect 500 × ( k − 1)
Revised: For each ( i, i ) environment pairs, we collect 500 × ( k − 1)

Original: × k samples for each timestep h , where k denotes the number of sources.
Revised: × k samples for each timestep h , where k denotes the number of sources.

Original: Thus we ensure that the number of samples from cross transition of different environments is the same as the number of samples from cross transition of the same environment.
Revised: Thus we ensure that the number of samples from cross transition of different environments is the same as the number of samples from cross transition of the same environment.

Original: For the online setting, we simply sample 1000 × ( k − 1) × k samples for each ( i, i ) cross transition to ensure that the total number of samples is the same for G-R EP T RANSFER and O-R EP T RANSFER .
Revised: For the online setting, we simply sample 1000 × ( k − 1) × k samples for each ( i, i ) cross transition to ensure that the total number of samples is the same for G-R EP T RANSFER and O-R EP T RANSFER .

Original: To sample the initial state action pair (i.e., (˜ s, ˜ a ) pair as in (1)), for 90% of the samples, we follow the final policy from each source environment trained using B RIEE .
Revised: To sample the initial state action pair (i.e., (˜ s, ˜ a ) pair as in (1)), for 90% of the samples, we follow the final policy from each source environment trained using B RIEE .

Original: For the remaining 10% , we follow the same policy to state ˜ s , and then take a uniform random policy to get ˜ a .
Revised: For the remaining 10% , we follow the same policy to state ˜ s , and then take a uniform random policy to get ˜ a .

Original: With this sampling scheme we ensure that Assumption 2.3 is satisfied.
Revised: With this sampling scheme we ensure that Assumption 3.3 is satisfied.

Original: In the setting of Section.
Revised: In the setting of Section.

Original: J.2, we follow a more simple procedure to ensure that the samples are more balanced among the three states: we skip the first sampling step from environment
Revised: 6.2, we follow a more simple procedure to ensure that the samples are more balanced among the three states: we skip the first sampling step from environment

Original: i (i.e., sample s given (˜ s, ˜ a ) ), and simply reset environment i to s , where s is one of the three states with equal probability, and generate the observation accordingly.
Revised: i (i.e., sample s given (˜ s, ˜ a ) ), and simply reset environment i to s , where s is one of the three states with equal probability, and generate the observation accordingly.

Original: Note that such visitation distribution is also possible in the online setting with a more nuanced sampling procedure, and in the experiment we use the same sampling procedure for both G-R EP T RANSFER and O-R EP T RANSFER for a fair comparison.
Revised: Note that such visitation distribution is also possible in the online setting with a more nuanced sampling procedure, and in the experiment we use the same sampling procedure for both G-R EP T RANSFER and O-R EP T RANSFER for a fair comparison.

Original: K. H YPERPARAMETERS
Revised: I.5 Hyperparameters

Original: In this section, we record the hyperparameters we try and the final hyperparameter we use for each baselines.
Revised: In this section, we record the hyperparameters we try and the final hyperparameter we use for each baselines.

Original: The hyperparameters for R EP T RANSFER in Section.
Revised: The hyperparameters for R EP T RANSFER in Section.

Original: J.1 is in Table.
Revised: 6.1 is in Table.

Original: The hyperparameters for R EP T RANSFER in Section.
Revised: The hyperparameters for R EP T RANSFER in Section.

Original: J.2 is in Table.
Revised: 6.2 is in Table.

Original: The hyperparameters for B RIEE is in Table.
Revised: The hyperparameters for B RIEE is in Table.

Original: We use the same set of hyperparameters for G-R EP T RANSFER and O-R EP T RANSFER . 
Revised: We use the same set of hyperparameters for G-R EP T RANSFER and O-R EP T RANSFER . [Table] [Table] [Table]

Revised: Decoder

Original: [Table]
Revised: [Table]

Revised: rate {1e-2} 1eDiscriminator f learning rate {1e-2} 1eDiscriminator f hidden layer size {256,512}

Revised: Discriminator f hidden layer number {2,3}

Revised: RepLearn Iteration T {30,40,50,100,150}

Revised: Decoder ϕ number of gradient steps {64,80,128,256}

Revised: Discriminator f number of gradient steps {64,80,128,256}

Revised: Decoder ϕ batch size {256,512}

Revised: Discriminator f batch size {256,512}

Revised: RepLearn regularization coefficient λ {0.01} 0.

Revised: Decoder ϕ softmax temperature {1}

Revised: Decoder ϕ 0 softmax temperature {0.1,1}

Original: [Table]
Revised: [Table]

Revised: bonus coefficient β {1, H 5 }

Revised: [Table]

Revised: regularization coefficient λ {1}

Revised: [Table]

Revised: size {1e5} 1e

Revised: [Table]

Revised: frequency {50}

Revised: Optimizer {SGD, Adam}

Revised: Adam

Revised: [Table]

Revised: {1e-2} 1e

Revised: [Table]

Revised: [Table]

Revised: learning rate {1e-2} 1e

Revised: [Table]

Revised: Discriminator f hidden layer size {256}

Revised: RepLearn Iteration T {30}

Revised: Decoder ϕ number of gradient steps {64}

Revised: Discriminator f number of gradient steps {64}

Revised: Decoder ϕ batch size {512}

Revised: Discriminator f batch size {512}

Revised: RepLearn regularization coefficient λ {0.01} 0.

Revised: Decoder ϕ softmax temperature {1}

Revised: Decoder ϕ 0 softmax temperature {0.1} 0.

Original: K. V ISUALIZATIONS
Revised: LSVI bonus coefficient β { H 5 } H LSVI [Table]

Revised: coefficient λ {1}

Revised: [Table]

Revised: size {1e5} 1e

Revised: [Table]

Revised: frequency {50}

Revised: I.6 Visualizations

Original: In this section we provide a comprehensive visualization of the decoders for all baselines in the target environment.
Revised: In this section we provide a comprehensive visualization of the decoders for all baselines in the target environment.

Original: We observe that the behaviors of all baselines are similar across the 5 random seeds.
Revised: We observe that the behaviors of all baselines are similar across the 5 random seeds.

Original: Thus to avoid redundancy, we only show the visualization from 1 random seed.
Revised: Thus to avoid redundancy, we only show the visualization from 1 random seed.

Original: K.6. V ISUALIZATIONS FROM S ECTION . J. 
Revised: I.6.1 Visualizations from Section. 6.

Original: We record the visualization of the 5 sources from Fig.
Revised: We record the visualization of the 5 sources from Fig.

Original: 3 to Fig.
Revised: 3 to Fig.

Original: 7; O-R EP T RANSFER in Fig.8;
Revised: 7; O-R EP T RANSFER in Fig.8;

Original: G-R EP T RANSFER in Fig.
Revised: G-R EP T RANSFER in Fig.

Original: 9; running B RIEE on target in Fig.
Revised: 9; running B RIEE on target in Fig.

Original: K.6. V ISUALIZATIONS FROM S ECTION . J.
Revised: I.6.2 Visualizations from

Revised: Section

Original: We record the visualization of the 2 sources from Fig.
Revised: We record the visualization of the 2 sources from Fig.

Original: 11 and Fig.
Revised: 11 and Fig.

Original: 12; O-R EP T RANSFER in Fig.13;
Revised: 12; O-R EP T RANSFER in Fig.13;

Original: G-R EP T RANSFER in Fig.
Revised: G-R EP T RANSFER in Fig.

Original: 14; running B RIEE on target in Fig.
Revised: 14; running B RIEE on target in Fig.

Original: Note that the features collapse[Table] [Figure] at some timesteps in Fig.
Revised: Note that the features collapse at some timesteps in Fig.

Original: 14 and Fig. 15, but this is acceptable because the optimal actions at those timesteps are the same for the collapsed states.
Revised: 14 and Fig. 15, but this is acceptable because the optimal actions at those timesteps are the same for the collapsed states.

Original: [Table]
Revised: [Figure]

Original: [Figure][Figure] [Table] [Figure]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Table][Figure] [Figure] [Table]
Revised: [Table]

Original: [Figure][Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: state
Revised: state

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Figure]
Revised: [Figure] [Figure]