Original: We design the following experimental procedure to demonstrate this  ability: (1) train the model on dataset D 1 which contains two contexts ( C 0 and C 1 ); (2) reset free parameters in the variational distribution q ( ν | ˆ ν ) and q ( µ | ˆ µ ) while preserving q ( θ | ˆ θ ) ; (3) re-train the model on dataset D 2 which contains the two original contexts and two new contexts ( C 2 and
Revised: We design the following experimental procedure to demonstrate this a ability: (1) train the model on dataset D 1 which contains two contexts ( C 0 and C 1 ); (2) reset free parameters in the variational distribution q ( ν | ˆ ν ) and q ( µ | ˆ µ ) while preserving q ( θ | ˆ θ ) ; (3) re-train the model on dataset D 2 which contains the two original contexts and two new contexts ( C 2 and

Original: C 3 ).
Revised: C 3 ).

Original: We present the training results on the sets D 1 and D 2 in Figure A16 and A17, respectively.
Revised: We present the training results on the sets D 1 and D 2 in Figure A16 and A17, respectively.

Original: a

Original: These results suggest that our model is able to learn new contexts ( C 2 and C 3 in Figure A17), while preserving the original contexts ( C 0 and C 1 in both Figures A16 and A17). 
Revised: These results suggest that our model is able to learn new contexts ( C 2 and C 3 in Figure A17), while preserving the original contexts ( C 0 and C 1 in both Figures A16 and A17). [Figure] [Figure]

Original: Z
Revised: Z

Original: ZZ Z
Revised: Z

Original: ZZ Z Z
Revised: Z

Original: Z
Revised: Z

Original: Zp ( z 0 )87.9 0.
Revised: Zp ( z 0 )

Original: P r o b a b ili t y ( % ) (a) Learned context model on D    80 Time ( t ) C C
Revised: [Figure]

Original: Z Z Z
Revised: [Figure] [Figure]

Original: Z Z(b)
Revised: [Figure]

Original: Learned context transitions on D

Original: [Figure]
Revised: [Figure]

Original: E.
Revised: E.

Original: C OMPARING TO POMDP AND C-MDP METHODS
Revised: C OMPARING TO POMDP AND C -MDP METHODS

Original: [-algo ↓ failure → hard soft α = 
-]
Revised: [-algo ↓ failure → hard soft α = 
-]

Original: .soft α

Original: [-= 
-]
Revised: [-= 
-]

Original: .soft α = 0 .

Original: FI-SAC 84 .

Original: 50 ± 1 .76 .

Original: 63 ± 8 .84 .

Original: 75 ± 3 .86 .

Original: 92 ± 1 .

Original: C-SAC 85 .

Original: 38 ± 1 .76 .

Original: 80 ± 8 .86 .

Original: 76 ± 2 .88 .

Original: 35 ± 1 .

Original: C-CEM 87 .

Original: 63 ± 0 . [Table] a We chosen the context set as C = {− 1 , χ } for χ ∈ {− 1 , 0 . 1 , 0 . 3 , 0 . 5 } .
Revised: We chosen the context set as C = {− 1 , χ } for χ ∈ {− 1 , 0 . 1 , 0 . 3 , 0 . 5 } .

Original: However, for χ > 0 , we again increased the force magnitude by two-fold.
Revised: However, for χ > 0 , we again increased the force magnitude by two-fold.

Original: For SAC-based algorithms we present the statistics for 50 episodes and 3 seeds.
Revised: For SAC-based algorithms we present the statistics for 50 episodes and 3 seeds.

Original: To avoid unfair comparison to POMDP and continual RL methods, we pick their best performance and explain why these methods are not well-suited for our problem.
Revised: To avoid unfair comparison to POMDP and continual RL methods, we pick their best performance and explain why these methods are not well-suited for our problem.

Original: For
Revised: For

Original: GPMM we run ten separate experiments with 15 episode each and picked three best runs and the best episode performance.
Revised: GPMM we run ten separate experiments with 15 episode each and picked three best runs and the best episode performance.

Original: For RNN-PPO we run three experiments, but picked the best learned policy over time and over the runs.
Revised: For RNN-PPO we run three experiments, but picked the best learned policy over time and over the runs.

Revised: Furthermore, we use the RNN in a more favorable setting as we assume that the number of contexts is known and can be hard coded in the RNN architecture.

Original: We present our experimental results in Table A6.
Revised: We present our experimental results in Table A6.

Original: While it seems that RNN-PPO learns to swing-up correctly for soft failures and GPMM almost learns to swing-up for χ = 0 .
Revised: While it seems that RNN-PPO learns to swing-up correctly for soft failures and GPMM almost learns to swing-up for χ = 0 .

Original: 5 , the learned belief models indicate  that this not so.
Revised: 5 , the learned belief models indicate a [Table] that this not so.

Original: In fact, plotting the evolution of the belief for RNN-PPO and the context evolution for GPMM illustrates that the algorithms do not learn the context model (see Figure 4 in the main text).
Revised: In fact, plotting the evolution of the belief for RNN-PPO and the context evolution for GPMM illustrates that the algorithms do not learn the context model (see Figure 4 in the main text).

Revised: In order to illustrate that it is not Gaussian Processes that cause failure in GPMM, we replace the Gaussian Process mixture with Deep Neural Network and Attentive Neural Process (Qin et al., 2019;

Revised: Kim et al., 2019) mixtures (DNNMM and ANPMM, respectively) using the code from Xu et al.

Revised: The results in this case are similar to the GPMM case, i.e., we manage to get reasonable rewards for α = 0 .

Revised: 5 , but fail for other cases.

Original: a
Revised: a