Original: M ETA -W ORLD C ONDITIONAL N EURAL P ROCESSES
Revised: M ETA -W ORLD C ONDITIONAL N EURAL P ROCESSES

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Meta-World Conditional Neural Processes (MW-CNP), a conditional world model generator that leverages sample efficiency and scalability of Conditional Neural Processes to allow an agent to sample from the generated world model.
Revised: We propose Meta-World Conditional Neural Processes (MW-CNP), a conditional world model generator that leverages sample efficiency and scalability of Conditional Neural Processes to allow an agent to sample from the generated world model.

Original: We intend to reduce the agent’s interaction with the target environment as much as possible.
Revised: We intend to reduce the agent’s interaction with the target environment as much as possible.

Original: Thus, MW-CNP meta-learns world models that use prior experience.
Revised: Thus, MW-CNP meta-learns world models that use prior experience.

Original: Using the world model generated from MW-CNP the RL agent can be conditioned on significantly fewer samples collected from the target environment to imagine the unseen environment.
Revised: Using the world model generated from MW-CNP the RL agent can be conditioned on significantly fewer samples collected from the target environment to imagine the unseen environment.

Original: We emphasize that the agent does not have access to the task parameters throughout training and testing.
Revised: We emphasize that the agent does not have access to the task parameters throughout training and testing.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A diverse set of skills can be acquired through incremental learning in environments with emergent complexity.
Revised: A diverse set of skills can be acquired through incremental learning in environments with emergent complexity.

Original: In line with this, world model generation has been an integral part of open-ended learning (Wang et al.
Revised: In line with this, world model generation has been an integral part of open-ended learning (Wang et al.

Original: Although creating a variety of challenging simulation environments is a promising direction, it is bounded by the limitations of the physics engine and the predetermined set of environment parameters.
Revised: Although creating a variety of challenging simulation environments is a promising direction, it is bounded by the limitations of the physics engine and the predetermined set of environment parameters.

Original: We propose generating world models from the agent’s own experience to overcome this problem.
Revised: We propose generating world models from the agent’s own experience to overcome this problem.

Original: Research on lifelong learning focus on learning incrementally and adapting rapidly to unknown tasks from incoming streams of data (Parisi et al.
Revised: Research on lifelong learning focus on learning incrementally and adapting rapidly to unknown tasks from incoming streams of data (Parisi et al.

Original: Challenges associated with lifelong learning include catastrophic forgetting, negative transfer, and resource limitations.
Revised: Challenges associated with lifelong learning include catastrophic forgetting, negative transfer, and resource limitations.

Original: In addition, data to be processed after deployment and optimal model hyperparameters are not available a priori.
Revised: In addition, data to be processed after deployment and optimal model hyperparameters are not available a priori.

Original: Meta reinforcement learning is an enduring area of interest that enables fast adaptation in test tasks.
Revised: Meta reinforcement learning is an enduring area of interest that enables fast adaptation in test tasks.

Original: Recent works have integrated meta-learning to offline RL (Mitchell et al.
Revised: Recent works have integrated meta-learning to offline RL (Mitchell et al.

Original: and lifelong learning (Finn et al.
Revised: and lifelong learning (Finn et al.

Original: ; Nagabandi et al.
Revised: ; Nagabandi et al.

Original: ; Berseth et al.
Revised: ; Berseth et al.

Original: In open-ended learning, a policy with high representational capacity can be utilized for increasingly challenging environments.
Revised: In open-ended learning, a policy with high representational capacity can be utilized for increasingly challenging environments.

Original: In this work, we are interested in few-shot learning in settings where the transition dynamics of the environment change across tasks.
Revised: In this work, we are interested in few-shot learning in settings where the transition dynamics of the environment change across tasks.

Original: More specifically, the agent is trained a the distribution of tasks with varying transition dynamics and expected to adapt to an unseen task with few samples from the target environment.
Revised: More specifically, the agent is trained a the distribution of tasks with varying transition dynamics and expected to adapt to an unseen task with few samples from the target environment.

Original: Sampling in the real world is expensive; hence reducing the number of samples used for fast adaptation is an important research direction for sim-to-real RL.
Revised: Sampling in the real world is expensive; hence reducing the number of samples used for fast adaptation is an important research direction for sim-to-real RL.

Revised: In this work we are interested in meta-reinforcement learning settings where the reward function stays the same but the transition function changes across multiple tasks.

Revised: In particular, each transition function depends on the environment parameters that are hidden during training and testing.

Revised: More concretely, imagine a setting where an agent moves to a goal location in different environments parametrized by different force fields.

Original:  We propose Meta-World CNP to generate world models from a few samples collected from the target environment.
Revised: Different force fields push the agent in different directions. Through trial and error, the agent learns to move robustly in environments with different transition dynamics. The agent is required to adapt quickly to a new target environment with hidden environment parameters and reward signals at test time. We propose Meta-World CNP to generate world models from a few samples collected from the target environment.

Original: Our contributions include (1) generating and learning world models with no access to target environment parameters, (2) sample efficient, fast adaptation to the unseen test environment.
Revised: Our contributions include (1) generating and learning world models with no access to target environment parameters, (2) sample efficient, fast adaptation to the unseen test environment.

Original: Requires 1/25 rollouts from the unseen target environment at test time compared to NoReward
Revised: Requires significantly less rollouts from the unseen target environment at test time compared to No

Original:  Meta-Learning (NORML) Yang et al.
Revised: Reward Meta-Learning (NORML) Yang et al.

Original: and (3) utilizing offline datasets of MarkovDecision
Revised: and (3) utilizing offline datasets of Markov

Original:  Process tuples for training.
Revised: Decision Process tuples for training.

Original:  P RELIMINARIES 2. C ONDITIONAL N EURAL P ROCESSES
Revised:  P RELIMINARIES 2. C ONDITIONAL N EURAL P ROCESSES

Original: Conditional Neural Processes are proposed as a novel neural architecture by (Garnelo et al.
Revised: Conditional Neural Processes are proposed as a novel neural architecture by (Garnelo et al.

Original: (2018) to predict the parameters of a probability distribution conditioned on a permutation invariant prior data and target input.
Revised: (2018) to predict the parameters of a probability distribution conditioned on a permutation invariant prior data and target input.

Original: CNPs attempt to represent a family of functions using Bayesian Inference and the high representational capacity of neural networks.
Revised: CNPs attempt to represent a family of functions using Bayesian Inference and the high representational capacity of neural networks.

Original: The architecture of CNPs consists of parameter sharing encoders and a decoder named the query network.
Revised: The architecture of CNPs consists of parameter sharing encoders and a decoder named the query network.

Original: In particular, first a random number of input and true output pairs ( x f i , y truef i ) are sampled from the function f i ∈ F .
Revised: In particular, first a random number of input and true output pairs ( x f i , y truef i ) are sampled from the function f i ∈ F .

Original: Each pair is encoded into a latent representation via the encoder networks.
Revised: Each pair is encoded into a latent representation via the encoder networks.

Original: Then, these representations are averaged along the dimensions to account for the invariance to permutation and the number of inputs.
Revised: Then, these representations are averaged along the dimensions to account for the invariance to permutation and the number of inputs.

Original: The resulting representation is concatenated with the target input query and fed to the query network.
Revised: The resulting representation is concatenated with the target input query and fed to the query network.

Original: The query network outputs the predicted mean and standard deviation for the queried input ( x qf i ) .
Revised: The query network outputs the predicted mean and standard deviation for the queried input ( x qf i ) .

Original: 2. N O -R EWARD M ETA L EARNING
Revised: 2. N O -R EWARD M ETA L EARNING

Original: NORML, proposed by (Yang et al.
Revised: NORML, proposed by (Yang et al.

Original: (2019)), is an extension of MAML-RL framework (Finn et al.
Revised: (2019)), is an extension of MAML-RL framework (Finn et al.

Original: (2017)) for settings where the environment dynamics change across tasks instead of the reward function.
Revised: (2017)) for settings where the environment dynamics change across tasks instead of the reward function.

Original: The goal of NORML is to utilize past experience to quickly adapt to tasks with unknown parameters from a few samples with missing reward signals.
Revised: The goal of NORML is to utilize past experience to quickly adapt to tasks with unknown parameters from a few samples with missing reward signals.

Original: Provided that the change in dynamics can be represented in ( s t , a t , s t +1 ) , NORML learns an pseudo-advantage function A ψ ( s t , a t , s t +1 ) .
Revised: Provided that the change in dynamics can be represented in ( s t , a t , s t +1 ) , NORML learns an pseudo-advantage function A ψ ( s t , a t , s t +1 ) .

Original: It is important to note that the aim of A ψ is to guide the meta-policy adaptation instead of fitting to the advantage function.
Revised: It is important to note that the aim of A ψ is to guide the meta-policy adaptation instead of fitting to the advantage function.

Original: A ψ is used to compute task specific parameters in the MAML inner loop from a set of state transitions of task i D train i that does not contain reward signal.
Revised: A ψ is used to compute task specific parameters in the MAML inner loop from a set of state transitions of task i D train i that does not contain reward signal.

Original: The learned advantage function is optimized in the MAML outer loop using the reward information present in rollouts D test i obtained from updated task-specific policy parameters.
Revised: The learned advantage function is optimized in the MAML outer loop using the reward information present in rollouts D test i obtained from updated task-specific policy parameters.

Original:  P ROPOSED M ETHOD : M ETA -W ORLD C ONDITIONAL N EURAL P ROCESS (MW-CNP)
Revised:  P ROPOSED M ETHOD : M ETA -W ORLD C ONDITIONAL N EURAL P ROCESS (MW-CNP)

Original: In this section, we present our method, named Meta-World Conditional Neural Processes (MWCNP), for learning world models using prior experience.
Revised: In this section, we present our method, named Meta-World Conditional Neural Processes (MWCNP), for learning world models using prior experience.

Original: We first describe the problem setup and then explain MW-CNP’s structure in detail.
Revised: We first describe the problem setup and then explain MW-CNP’s structure in detail.

Original: In few-shot learning, the goal is to quickly adapt to an unseen target task using a few labeled data in the target environment.
Revised: In few-shot learning, the goal is to quickly adapt to an unseen target task using a few labeled data in the target environment.

Original: Meta-World Conditional Neural Processes (MW-CNP) can reduce the number of samples required from the target environment by generating world models from fewer samples from the target environment with no access to the target environment parameter.
Revised: Meta-World Conditional Neural Processes (MW-CNP) can reduce the number of samples required from the target environment by generating world models from fewer samples from the target environment with no access to the target environment parameter.

Original: These models can then be used to obtain inexpensive rollouts for finetuning at test time.
Revised: These models can then be used to obtain inexpensive rollouts for finetuning at test time.

Original: Online Meta-Learning We denote the initial state distribution as ρ ( s 0 ) :
Revised: Online Meta-Learning We denote the initial state distribution as ρ ( s 0 ) :

Original: S → R , state transition distribution of task i as ρ i ( s t +1 | s t , a t ) :
Revised: S → R , state transition distribution of task i as ρ i ( s t +1 | s t , a t ) :

Original: S ×
Revised: S ×

Original: A × S → R , and the reward function as r t : S ×
Revised: A × S → R , and the reward function as r t : S ×

Original: A × S → R .
Revised: A × S → R .

Original: During meta-training we store transitions for each task i as a set of observations
Revised: During meta-training we store transitions for each task i as a set of observations

Original: B i = { ( s t , a t , s t +1 ) } nt =

Revised: B i = { ( s t , a t , s t +1 ) } nt =

Original: ⊂ S ×
Revised: ⊂ S ×

Original: A × S without the task parameter.
Revised: A × S without the task parameter.

Original: Hence during both training and testing the parameters of the state transition distribution are hidden.
Revised: Hence during both training and testing the parameters of the state transition distribution are hidden.

Original: After we obtain the meta-policy and learn the pseudo-advantage function A ψ ( s t , a t , s t +1 ) using
Revised: After we obtain the meta-policy and learn the pseudo-advantage function A ψ ( s t , a t , s t +1 ) using

Original: NORML, we train our MW-CNP model using the replay buffer of transitions B = { B i } ni =1 where n denotes the number of environments.
Revised: NORML, we train our MW-CNP model using the replay buffer of transitions B = { B i } n i =1 where n denotes the number of environments.

Original: Meta-World Conditional Neural Processes (MW-CNP)
Revised: Meta-World Conditional Neural Processes (MW-CNP)

Original: MW-CNP is trained in an offline fashion using unlabeled batches of Markov Decision Process (MDP) tuples collected during online metalearning.
Revised: MW-CNP is trained in an offline fashion using unlabeled batches of Markov Decision Process (MDP) tuples collected during online metalearning.

Original: Figure 1 illustrates the training procedure for MW-CNP.
Revised: Figure 1 illustrates the training procedure for MW-CNP.

Original: It is worth noting that the environment parameter is hidden during training and testing.
Revised: It is worth noting that the environment parameter is hidden during training and testing.

Original: In each training iteration, an unlabeled batch
Revised: In each training iteration, an unlabeled batch

Original: B i = { ( s t , a t , s t +1 ) } nt =0 is randomly sampled from the offline dataset.
Revised: B i = { ( s t , a t , s t +1 ) } nt =0 is randomly sampled from the offline dataset.

Original: Then, a set 
Revised: Then, a set of task-specific MDP tuples { ( s k , a k , s k +1 ) } k and a single MDP ( s q , a q , s ′ q ) are randomly sampled from the chosen batch B i . ( s q , a q , s ′ q ) is used for target state-action query [ s q , a q ] and true target next state label [ s ′ q ] .

Original: Each MDP tuple ( s k , a k , s k +1 ) is encoded into a fixed size representation using a parameter sharing encoder network.
Revised: Each MDP tuple ( s k , a k , s k +1 ) is encoded into a fixed size representation using a parameter sharing encoder network.

Original: These representations are then passed through an averaging module A to obtain a latent representation r of the hidden environment transition function used in batch B i .
Revised: These representations are then passed through an averaging module A to obtain a latent representation r of the hidden environment transition function used in batch B i .

Original: The resulting latent representation r is concatenated with the [ s q , a q ] to predict the distribution parameters µ q , σ q of the next state s ′ q given the latent representation r and the target query
Revised: The resulting latent representation r is concatenated with the [ s q , a q ] to predict the distribution parameters µ q , σ q of the next state s ′ q given the latent representation r and the target query

Original: [ s q , a q ] .
Revised: [ s q , a q ] .

Original: [Figure]
Revised: [Figure]

Original: The loss function of MW-CNP can be expressed as 
Revised: The loss function of MW-CNP can be expressed as µ

Original: [Equation]
Revised: [Equation]

Original: where f θ D , g θ E are the decoder the encoder networks, [ s q , a q ] is the target state action query, ( s k , a k , s ′ k ) are the randomly sampled transitions from the set of observations B i .
Revised: where f θ D , g θ E are the decoder the encoder networks, [ s q , a q ] is the target state action query, ( s k , a k , s ′ k ) are the randomly sampled transitions from the set of observations B i .

Original: Figure 2 illustrates the test procedure of MW-CNP.
Revised: Figure 2 illustrates the test procedure of MW-CNP.

Original: At test-time, the agent is allowed to collect a few samples from the unseen target environment.
Revised: At test-time, the agent is allowed to collect a few samples from the unseen target environment.

Original: These samples are encoded into representations of fixed size by an encoder network with shared weights.
Revised: These samples are encoded into representations of fixed size by an encoder network with shared weights.

Original: A shared representation of the target environment is obtained using an averaging module shown in Figure 2(I).
Revised: A shared representation of the target environment is obtained using an averaging module shown in Figure 2(I).

Original: Once the latent representation is obtained for the target environment with a hidden task parameter rollouts can be generated inexpensively.
Revised: Once the latent representation is obtained for the target environment with a hidden task parameter rollouts can be generated inexpensively.

Original: This representation is used to predict the parameters of the next-state distribution for the state-action query [ s q , a q ] .
Revised: This representation is used to predict the parameters of the next-state distribution for the state-action query [ s q , a q ] .

Original: For each MW-CNP generated rollout, same true initial state, sampled from the real target environment, is used.
Revised: For each MW-CNP generated rollout, same true initial state, sampled from the real target environment, is used.

Original: The inital state is fed to the stochastic meta-policy to obtain the action query a q Figure 2(II).
Revised: The inital state is fed to the stochastic meta-policy to obtain the action query a q Figure 2(II).

Original: Then, the predicted next-state is used to sample the next action query until the episode is terminated.
Revised: Then, the predicted next-state is used to sample the next action query until the episode is terminated.

Original: The rollouts ”hallucinated” from the MW-CNP are combined with the true rollout sampled from the real target environment.
Revised: The rollouts ”hallucinated” from the MW-CNP are combined with the true rollout sampled from the real target environment.

Original: The resulting set of rollouts are fed to the pseudo-advantage network.
Revised: The resulting set of rollouts are fed to the pseudo-advantage network.

Original: The learned pseudo-advantage network learned during online meta-learning uses ( s, a, s ′ ) tuple as input and outputs an advantage estimation value.
Revised: The learned pseudo-advantage network learned during online meta-learning uses ( s, a, s ′ ) tuple as input and outputs an advantage estimation value.

Original: Hence, once experience from the generated world model is collected, these experiences are fed to the learned advantage function in the form of ( s, a, s ′ ) .
Revised: Hence, once experience from the generated world model is collected, these experiences are fed to the learned advantage function in the form of ( s, a, s ′ ) .

Original: Finally, meta policy is finetuned for fast adaptation to the target task using the estimated advantage values and combined set of MW-CNP generated rollouts and a single target environment rollout.
Revised: Finally, meta policy is finetuned for fast adaptation to the target task using the as s’ ..

Original: as s’ .. s’s q a q π ( [Figure]
Revised: s’

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we analyze the performance of our method and compare it with NORML and the oracle in 2D point agent environment.
Revised: In this section, we analyze the performance of our method and compare it with NORML and the oracle in 2D point agent environment.

Original: The goal of the point agent, initialized at [x=0,y=0], is to move to the position [x=0,y=1], where x,y are the positions on the 2D plane.
Revised: The goal of the point agent, initialized at [x=0,y=0], is to move to the position [x=0,y=1], where x,y are the positions on the 2D plane.

Original: We are interested in a meta-RL setting where the reward function is identical across multiple tasks.
Revised: We are interested in a meta-RL setting where the reward function is identical across multiple tasks.

Original: Different tasks are created by generating different artificial force fields that push the agent in different directions ( ϕ ).
Revised: Different tasks are created by generating different artificial force fields that push the agent in different directions ( ϕ ).

Original: We use the same reward function, the negative Euclidean distance from the goal position, and hyperparameters used in NORML for comparative analysis.
Revised: We use the same reward function, the negative Euclidean distance from the goal position, and hyperparameters used in NORML for comparative analysis.

Original: In the Point Agent environmenttasks are defined over the [ − π, π ] interval.
Revised: In the Point Agent environmenttasks are defined over the [ − π, π ] interval.

Original: The agent is initially trained across a distribution of 5000 tasks, i.e. in environment withdifferent force fields.
Revised: The agent is initially trained across a distribution of 5000 tasks, i.e. in environment withdifferent force fields.

Original: Then, it is tested in an unseen target task.
Revised: Then, it is tested in an unseen target task.

Original: The rollouts obtained from the target environment will be referred to as actual rollouts, whereas the rollouts generated from the
Revised: The rollouts obtained from the target environment will be referred to as actual rollouts, whereas the rollouts generated from the

Original: MW-CNP will be named imagined rollouts.
Revised: MW-CNP will be named imagined rollouts.

Original: At test time, Oracle agent uses 25 actual rollouts from the unseen target environment.
Revised: At test time, Oracle agent uses 25 actual rollouts from the unseen target environment.

Original: 25 rollouts were used in the original NORML experiment hence we use the same number of rollouts for comparison.
Revised: 25 rollouts were used in the original NORML experiment hence we use the same number of rollouts for comparison.

Original: The NORML agent and the MW-CNP agent are allowed to use only a single rollout for finetuning the meta-policy.
Revised: The NORML agent and the MW-CNP agent are allowed to use only a single rollout for finetuning the meta-policy.

Original: By limiting the number of actual rollouts that can be used for finetuning we aim to compare sample efficiency of MW-CNP to the baseline NORML.
Revised: By limiting the number of actual rollouts that can be used for finetuning we aim to compare sample efficiency of MW-CNP to the baseline NORML.

Original: The average return obtained from the target environment after finetuning the meta policy of NORML with 1 rollout are shown in Fig.
Revised: The average return obtained from the target environment after finetuning the meta policy of NORML with 1 rollout are shown in Fig.

Original: 3 with green bars.
Revised: 3 with green bars.

Original: As shown, when
Revised: As shown, when

Original: MW-CNP and NORML used the same amount of actual rollouts (1 actual rollout) from the target environment, MW-CNP outperforms NORML dramatically.
Revised: MW-CNP and NORML used the same amount of actual rollouts (1 actual rollout) from the target environment, MW-CNP outperforms NORML dramatically.

Original: Even though the oracle agent trained with NORML used a significantly higher number of actual rollouts than MW-CNP (25 to 1), their performances are similar, as shown in illustrations in Figures 4, 5, 6 and the bar plots in 3.
Revised: Even though the oracle agent trained with NORML used a significantly higher number of actual rollouts than MW-CNP (25 to 1), their performances are similar, as shown in illustrations in Figures 4, 5, 6 and the bar plots in 3.

Original: Figure 3 shows the average return obtained in the target environment over tasks.
Revised: Figure 3 shows the average return obtained in the target environment over tasks.

Original: We compare results for NORML finetuning over 1 rollout, NORML finetuning over 25 rollouts (Oracle) and finetuning over a combination of 24 imagined rollouts from the MW-CNP model, and 1 actual rollout from the target environment.
Revised: We compare results for NORML finetuning over 1 rollout, NORML finetuning over 25 rollouts (Oracle) and finetuning over a combination of 24 imagined rollouts from the MW-CNP model, and 1 actual rollout from the target environment.

Original: At test-time MW-CNP uses a total of 25 combined rollouts similar to the oracle
Revised: At test-time MW-CNP uses a total of 25 combined rollouts similar to the oracle

Original: NORML that uses 25 actual rollouts.
Revised: NORML that uses 25 actual rollouts.

Original: [Figure]
Revised: [Figure]

Original: L O r a c l e M W - C N P ( o u r s ) We condition the MW-CNP model on the same rollout used for 1 rollout-finetunig.
Revised: We condition the MW-CNP model on the same rollout used for 1 rollout-finetunig.

Original: The results show that the samples generated from the agent’s imagination created by the MW-CNP generated sophisticated samples that can be used for finetuning the meta-policy for fast domain adaptation.
Revised: The results show that the samples generated from the agent’s imagination created by the MW-CNP generated sophisticated samples that can be used for finetuning the meta-policy for fast domain adaptation.

Original: All in all, MW-CNP requires significantly less interaction with the target environment compared to
Revised: All in all, MW-CNP requires significantly less interaction with the target environment compared to

Original: NORML, for fast adaptation to the unseen task.
Revised: NORML, for fast adaptation to the unseen task.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We showed that meaningful experience could be collected using the MW-CNP framework to guide the meta-policy adaptation.
Revised: We showed that meaningful hallucinated rollouts can be collected using the MW-CNP framework to guide the meta-policy adaptation.

Original: We compared the expected return obtained from finetuning  over generated data with MW-CNP and  ground truth data.
Revised: We compared the expected return obtained from finetuning [Figure] [Figure] over generated data with MW-CNP and the ground truth data.

Original: While our initial experiments are conducted in lowdimensional input space, CNPs were shown to perform well with high dimensional inputs like image[Figure] [Figure] data (Seker et al.
Revised: While our initial experiments are conducted in low-dimensional state and actions spaces such as the point agent, CNPs were shown to perform well with high dimensional inputs like image data (Seker et al.

Original: Therefore, we plan to extend our work and apply it to high-dimensional sensorimotor spaces, such as manipulator robots with RGB-D cameras.
Revised: Therefore, we plan to extend our work and apply it to high-dimensional sensorimotor spaces, such as manipulator robots with RGB-D cameras