Original: R EINFORCEMENT L EARNING WITH B AYESIAN C LASSIFIERS : E FFICIENT S KILL L EARNING FROM O UTCOME
Revised: R EINFORCEMENT L EARNING WITH B AYESIAN C LASSIFIERS : E FFICIENT S KILL L EARNING FROM O UTCOME

Original: E
Revised: E

Original: XAMPLES
Revised: XAMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Exploration in reinforcement learning is, in general, a challenging problem.
Revised: Exploration in reinforcement learning is, in general, a challenging problem.

Original: In this work, we study a more tractable class of reinforcement learning problems deﬁned by data that provides examples of successful outcome states.
Revised: In this work, we study a more tractable class of reinforcement learning problems deﬁned by data that provides examples of successful outcome states.

Original: In this case, the reward function can be obtained automatically by training a classiﬁer to classify states as successful or not.
Revised: In this case, the reward function can be obtained automatically by training a classiﬁer to classify states as successful or not.

Original: We argue that, with appropriate representation and regularization, such a classiﬁer can guide a reinforcement learning algorithm to an effective solution.
Revised: We argue that, with appropriate representation and regularization, such a classiﬁer can guide a reinforcement learning algorithm to an effective solution.

Original: However, as we will show, this requires the classiﬁer to make uncertaintyaware predictions that are very difﬁcult with standard deep networks.
Revised: However, as we will show, this requires the classiﬁer to make uncertaintyaware predictions that are very difﬁcult with standard deep networks.

Original: To address this, we propose a novel mechanism for obtaining calibrated uncertainty based on an amortized technique for computing the normalized maximum likelihood distribution.
Revised: To address this, we propose a novel mechanism for obtaining calibrated uncertainty based on an amortized technique for computing the normalized maximum likelihood distribution.

Original: We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions from data, while being able to guide algorithms towards the speciﬁed goal more effectively.
Revised: We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions from data, while being able to guide algorithms towards the speciﬁed goal more effectively.

Original: We show how using amortized normalized maximum likelihood for reward inference is able to provide effective reward guidance for solving a number of challenging navigation and robotic manipulation tasks which prove difﬁcult for other algorithms.
Revised: We show how using amortized normalized maximum likelihood for reward inference is able to provide effective reward guidance for solving a number of challenging navigation and robotic manipulation tasks which prove difﬁcult for other algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While reinforcement learning (RL) has been shown to successfully solve problems with careful reward design (Rajeswaran et al., 2018; OpenAI et al., 2019), RL in its most general form, with no assumptions on the dynamics or reward function, requires solving a challenging uninformed search problem in which rewards are sparsely observed.
Revised: While reinforcement learning (RL) has been shown to successfully solve problems with careful reward design (Rajeswaran et al., 2018; OpenAI et al., 2019), RL in its most general form, with no assumptions on the dynamics or reward function, requires solving a challenging uninformed search problem in which rewards are sparsely observed.

Original: Techniques which explicitly provide “rewardshaping” (Ng et al., 1999), or modify the reward function to guide learning, can help take some of the burden off of exploration, but shaped rewards can be difﬁcult to obtain without domain knowledge.
Revised: Techniques which explicitly provide “rewardshaping” (Ng et al., 1999), or modify the reward function to guide learning, can help take some of the burden off of exploration, but shaped rewards can be difﬁcult to obtain without domain knowledge.

Original: In this paper, we aim to reformulate the reinforcement learning problem to make it easier for the user to specify the task and to provide a tractable reinforcement learning objective.
Revised: In this paper, we aim to reformulate the reinforcement learning problem to make it easier for the user to specify the task and to provide a tractable reinforcement learning objective.

Original: Instead of requiring a reward function designed for an objective, our method instead assumes a user-provided set of successful outcome examples : states in which the desired task has been accomplished successfully.
Revised: Instead of requiring a reward function designed for an objective, our method instead assumes a user-provided set of successful outcome examples : states in which the desired task has been accomplished successfully.

Original: The algorithm aims to estimate the distribution over these states and maximize the probability of reaching states that are likely under the distribution.
Revised: The algorithm aims to estimate the distribution over these states and maximize the probability of reaching states that are likely under the distribution.

Original: Prior work on learning from success examples (Fu et al., 2018b; Zhu et al., 2020) focused primarily on alleviating the need for manual reward design.
Revised: Prior work on learning from success examples (Fu et al., 2018b; Zhu et al., 2020) focused primarily on alleviating the need for manual reward design.

Original: In our work, we focus on the potential for this mode of task speciﬁcation to produce more tractable
Revised: In our work, we focus on the potential for this mode of task speciﬁcation to produce more tractable

Original: RL problems and solve more challenging classes of tasks.
Revised: RL problems and solve more challenging classes of tasks.

Original: Intuitively, when provided with explicit examples of successful states, the RL algorithm should be able to direct its exploration, rather than simply hope to randomly chance upon potentially high reward states.
Revised: Intuitively, when provided with explicit examples of successful states, the RL algorithm should be able to direct its exploration, rather than simply hope to randomly chance upon potentially high reward states.

Original: The main challenge in instantiating this idea into a practical algorithm is performing appropriate uncertainty quantiﬁcation in estimating whether a given state corresponds to a successful outcome.
Revised: The main challenge in instantiating this idea into a practical algorithm is performing appropriate uncertainty quantiﬁcation in estimating whether a given state corresponds to a successful outcome.

Original: Our approach trains a classiﬁer to distinguish successful states, provided by the user, from those generated by the current policy, analogously to generative adversarial networks (Goodfellow et al., 2014) and previously proposed methods for inverse reinforcement learning (Fu et al., 2018a).
Revised: Our approach trains a classiﬁer to distinguish successful states, provided by the user, from those generated by the current policy, analogously to generative adversarial networks (Goodfellow et al., 2014) and previously proposed methods for inverse reinforcement learning (Fu et al., 2018a).

Original: In general, such a classiﬁer is not guaranteed to provide a good optimization landscape for learning the policy.
Revised: In general, such a classiﬁer is not guaranteed to provide a good optimization landscape for learning the policy.

Original: We discuss how a particular form of uncertainty quantiﬁcation based on the normalized maximum likelihood (NML) distribution produces better reward guidance for learning.
Revised: We discuss how a particular form of uncertainty quantiﬁcation based on the normalized maximum likelihood (NML) distribution produces better reward guidance for learning.

Original: We also connect our approach to count-based exploration methods, showing that a classiﬁer with suitable uncertainty estimates reduces to a count-based exploration method in the absence of any generalization across states, while also discussing how it improves over count-based exploration in the presence of good generalization.
Revised: We also connect our approach to count-based exploration methods, showing that a classiﬁer with suitable uncertainty estimates reduces to a count-based exploration method in the absence of any generalization across states, while also discussing how it improves over count-based exploration in the presence of good generalization.

Original: We then propose a practical algorithm to train success classiﬁers in a computationally efﬁcient way with NML, and show how this form of reward inference allows us to solve difﬁcult problems more efﬁciently, providing experimental results which outperform existing algorithms on a number of navigation and robotic manipulation domains.
Revised: We then propose a practical algorithm to train success classiﬁers in a computationally efﬁcient way with NML, and show how this form of reward inference allows us to solve difﬁcult problems more efﬁciently, providing experimental results which outperform existing algorithms on a number of navigation and robotic manipulation domains.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A number of techniques have been proposed to improve exploration.
Revised: A number of techniques have been proposed to improve exploration.

Original: These techniques either add reward bonuses that encourage a policy to visit novel states in a task-agnostic manner (Wiering and
Revised: These techniques either add reward bonuses that encourage a policy to visit novel states in a task-agnostic manner (Wiering and

Original: Schmidhuber, 1998; Auer et al., 2002; Schaul et al., 2011; Houthooft et al., 2016; Pathak et al., 2017; Tang et al., 2017; Stadie et al., 2015; Bellemare et al., 2016; Burda et al., 2018a; O’Donoghue, 2018) or perform Thompson sampling or approximate Thompson sampling based on a prior over value functions (Strens, 2000; Osband et al., 2013; 2016).
Revised: Schmidhuber, 1998; Auer et al., 2002; Schaul et al., 2011; Houthooft et al., 2016; Pathak et al., 2017; Tang et al., 2017; Stadie et al., 2015; Bellemare et al., 2016; Burda et al., 2018a; O’Donoghue, 2018) or perform Thompson sampling or approximate Thompson sampling based on a prior over value functions (Strens, 2000; Osband et al., 2013; 2016).

Original: While these techniques are uninformed about the actual task, we consider a constrained set of problems where examples of successes can allow for more task directed exploration.
Revised: While these techniques are uninformed about the actual task, we consider a constrained set of problems where examples of successes can allow for more task directed exploration.

Original: In real world problems, designing well-shaped reward functions makes exploration easier but often requires signiﬁcant domain knowledge (Andrychowicz et al., 2020), access to privileged information about the environment (Levine et al., 2016) and/or a human in the loop providing rewards (Knox and Stone, 2009; Singh et al., 2019b).
Revised: In real world problems, designing well-shaped reward functions makes exploration easier but often requires signiﬁcant domain knowledge (Andrychowicz et al., 2020), access to privileged information about the environment (Levine et al., 2016) and/or a human in the loop providing rewards (Knox and Stone, 2009; Singh et al., 2019b).

Original: Prior work has considered specifying rewards by providing example demonstrations, and inferring rewards with inverse RL (Abbeel and Ng, 2004; Ziebart et al., 2008; Ho and Ermon, 2016;
Revised: Prior work has considered specifying rewards by providing example demonstrations, and inferring rewards with inverse RL (Abbeel and Ng, 2004; Ziebart et al., 2008; Ho and Ermon, 2016;

Original: Fu et al., 2018a).
Revised: Fu et al., 2018a).

Original: This requires expensive expert demonstrations to be provided to the agent.
Revised: This requires expensive expert demonstrations to be provided to the agent.

Original: In contrast, our work has the minimal requirement of simply providing successful outcome states, which can be done cheaply.
Revised: In contrast, our work has the minimal requirement of simply providing successful outcome states, which can be done cheaply.

Original: This subclass of problems is also related to goal conditioned RL (Kaelbling, 1993; Schaul et al., 2015;
Revised: This subclass of problems is also related to goal conditioned RL (Kaelbling, 1993; Schaul et al., 2015;

Original: Zhu et al., 2017; Andrychowicz et al., 2017; Nair et al., 2018; Veeriah et al., 2018; Rauber et al., 2018; Warde-Farley et al., 2018; Colas et al., 2019; Ghosh et al., 2019; Pong et al., 2020) but is more general, since it allows for a more abstract notion of task success.
Revised: Zhu et al., 2017; Andrychowicz et al., 2017; Nair et al., 2018; Veeriah et al., 2018; Rauber et al., 2018; Warde-Farley et al., 2018; Colas et al., 2019; Ghosh et al., 2019; Pong et al., 2020) but is more general, since it allows for a more abstract notion of task success.

Original: A core idea behind our work is using a Bayesian classiﬁer to learn a suitable reward function.
Revised: A core idea behind our work is using a Bayesian classiﬁer to learn a suitable reward function.

Original: Bayesian inference with expressive models and high dimensional data can often be intractable, requiring assumptions on the form of the posterior (Hoffman et al., 2013; Blundell et al., 2015; Maddox et al., 2019).
Revised: Bayesian inference with expressive models and high dimensional data can often be intractable, requiring assumptions on the form of the posterior (Hoffman et al., 2013; Blundell et al., 2015; Maddox et al., 2019).

Original: In this work, we build on the concept of normalized maximum likelihood (Rissanen, 1996;
Revised: In this work, we build on the concept of normalized maximum likelihood (Rissanen, 1996;

Original: Shtar’kov, 1987), or NML, to learn Bayesian classiﬁers.
Revised: Shtar’kov, 1987), or NML, to learn Bayesian classiﬁers.

Original: Although NML is typically considered from the perspective of optimal coding (Grünwald, 2007; Fogel and Feder, 2018), we show how it can be used to learn success classiﬁers, and discuss its connections to exploration and reward shaping.
Revised: Although NML is typically considered from the perspective of optimal coding (Grünwald, 2007; Fogel and Feder, 2018), we show how it can be used to learn success classiﬁers, and discuss its connections to exploration and reward shaping.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this paper, we study a modiﬁed reinforcement learning problem, where instead of the standard reward function, the agent is provided with successful outcome examples .
Revised: In this paper, we study a modiﬁed reinforcement learning problem, where instead of the standard reward function, the agent is provided with successful outcome examples .

Original: This reformulation not only provides a modality for task speciﬁcation that may be more natural for users to provide in some settings (Fu et al., 2018b; Zhu et al., 2020; Singh et al., 2019a) but, as we will show, can also make learning easier.
Revised: This reformulation not only provides a modality for task speciﬁcation that may be more natural for users to provide in some settings (Fu et al., 2018b; Zhu et al., 2020; Singh et al., 2019a), but, as we will show, can also make learning easier.

Original: We also derive a meta-learned variant of the conditional normalized maximum likelihood (CNML) distribution for representing our reward function, in order to make evaluation tractable.
Revised: We also derive a meta-learned variant of the conditional normalized maximum likelihood (CNML) distribution for representing our reward function, in order to make evaluation tractable.

Original: We discuss background on successful outcome examples and CNML in this section.
Revised: We discuss background on successful outcome examples and CNML in this section.

Original: 3. R EINFORCEMENT L EARNING WITH E XAMPLES OF S UCCESSFUL O UTCOMES
Revised: 3. R EINFORCEMENT L EARNING WITH E XAMPLES OF S UCCESSFUL O UTCOMES

Original: Formally, we follow the framework proposed by Fu et al.
Revised: We follow the framework proposed by Fu et al.

Original: (2018b)
Revised: (2018b)

Original: and assume that we are provided with a Markov decision process (MDP) without a reward function, given by M , where M = ( S , A , T , γ, µ 0 ) , as well as successful outcome examples S
Revised: and assume that we are provided with a Markov decision process (MDP) without a reward function, given by M , where M = ( S , A , T , γ, µ 0 ) , as well as successful outcome examples S

Original: + =
Revised: + =

Original: { s k + } Kk =1 , which is a set of states in which the desired task has been accomplished.
Revised: { s k + } Kk =1 , which is a set of states in which the desired task has been accomplished.

Original: This formalism is easiest to describe in terms of the control as inference framework (Levine, 2018).
Revised: This formalism is easiest to describe in terms of the control as inference framework (Levine, 2018).

Original: The relevant graphical model in Figure 7 consists of states and actions, as well as binary success variables e t which represent the occurrence of a particular event.
Revised: The relevant graphical model in Figure 9 consists of states and actions, as well as binary success variables e t which represent the occurrence of a particular event.

Original: The agent’s goal is to cause this event to occur (e.g., a robot that is cleaning the ﬂoor must cause the “ﬂoor is clean” event to occur).
Revised: The agent’s objective is to cause this event to occur (e.g., a robot that is cleaning the ﬂoor must cause the “ﬂoor is clean” event to occur).

Original: Formally, we assume that the states in S + are sampled from the distribution p ( s t | e t
Revised: Formally, we assume that the states in S + are sampled from the distribution p ( s t | e t

Original: =
Revised: =

Original: True ) – that is, states where the desired event has taken place.
Revised: True ) – that is, states where the desired event has taken place.

Original: In this work, we focus on efﬁcient methods for solving this reformulation of the RL problem, by utilizing a novel uncertainty quantiﬁcation method to represent the distribution p ( e t | s t ) .
Revised: In this work, we focus on efﬁcient methods for solving this reformulation of the RL problem, by utilizing a novel uncertainty quantiﬁcation method to represent the distribution p ( e t | s t ) .

Original: In practice, prior methods that build on this and similar reformulations of the RL problem (Fu et al., 2018b) derive an algorithm where the reward function in RL is produced by a classiﬁer that estimates p ( e t =
Revised: In practice, prior methods that build on this and similar reformulations of the RL problem (Fu et al., 2018b) derive an algorithm where the reward function in RL is produced by a classiﬁer that estimates p ( e t =

Original: True | s t ) .
Revised: True | s t ) .

Original: Following the adversarial inverse reinforcement learning (AIRL) derivation (Fu et al., 2018a; Finn et al., 2016), it is possible to show that the correct source of negative examples for training this classiﬁer is the state distribution of the policy itself, π ( s ) .
Revised: Following the adversarial inverse reinforcement learning (AIRL) derivation (Fu et al., 2018a; Finn et al., 2016), it is possible to show that the correct source of negative examples for training this classiﬁer is the state distribution of the policy itself, π ( s ) .

Original: This insight results in a simple algorithm: at each iteration of the algorithm, the policy is updated to maximize the current reward, given by log p ( e t =
Revised: This insight results in a simple algorithm: at each iteration of the algorithm, the policy is updated to maximize the current reward, given by log p ( e t =

Original: True | s t ) , then samples from the policy are added to the set of negative examples
Revised: True | s t ) , then samples from the policy are added to the set of negative examples

Original: S − , and the classiﬁer is retrained on the original positive set S
Revised: S − , and the classiﬁer is retrained on the original positive set S

Original: + and the updated negative set S − .
Revised: + and the updated negative set S − .

Original: 3. C ONDITIONAL N ORMALIZED M AXIMUM L IKELIHOOD
Revised: 3. C ONDITIONAL N ORMALIZED M AXIMUM L IKELIHOOD

Original: Our method builds on the principle of conditional normalized maximum likelihood (NML) (Rissanen and Roos, 2007; Grünwald, 2007; Fogel and Feder, 2018), which we review brieﬂy.
Revised: Our method builds on the principle of conditional normalized maximum likelihood (NML) (Rissanen and Roos, 2007; Grünwald, 2007; Fogel and Feder, 2018), which we review brieﬂy.

Original: CNML is a method for performing k -way classiﬁcation, given a model class Θ and a dataset
Revised: CNML is a method for performing k -way classiﬁcation, given a model class Θ and a dataset

Original: D = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , ..., ( x n , y n ) } , and has been shown to provide better calibrated predictions and uncertainty estimates with minimax regret guarantees (Bibas et al., 2019).
Revised: D = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , ..., ( x n , y n ) } , and has been shown to provide better calibrated predictions and uncertainty estimates with minimax regret guarantees (Bibas et al., 2019).

Original: To predict the class of a query point x q , CNML constructs k augmented datasets by adding x q with a different label in each datasets, which we write as D ∪ ( x q , y = i ) , i ∈ (1 , 2 , ..., k ) .
Revised: To predict the class of a query point x q , CNML constructs k augmented datasets by adding x q with a different label in each datasets, which we write as D ∪ ( x q , y = i ) , i ∈ (1 , 2 , ..., k ) .

Original: CNML then deﬁnes the class distribution by solving the maximum likelihood estimation problem at query time for each of these augmented datasets to convergence, and normalize the likelihoods as follows:
Revised: CNML then deﬁnes the class distribution by solving the maximum likelihood estimation problem at query time for each of these augmented datasets to convergence, and normalize the likelihoods as follows:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, if x q is close to other datapoints in D , then the model will struggle to assign a high likelihood to labels that differ substantially from other nearby points.
Revised: Intuitively, if x q is close to other datapoints in D , then the model will struggle to assign a high likelihood to labels that differ substantially from other nearby points.

Original: However, if x q is far from all datapoints in D , then the different augmented MLE problems can easily classify x q as an arbitrary class, providing us with a likelihood closer to uniform.
Revised: However, if x q is far from all datapoints in D , then the different augmented MLE problems can easily classify x q as an arbitrary class, providing us with a likelihood closer to uniform.

Original: More formally, CNML is an application of the
Revised: More formally, CNML is an application of the

Original: NML principle for obtaining minimum description length models, and we refer readers to Grünwald (2007) for an in-depth discussion.
Revised: NML principle for obtaining minimum description length models, and we refer readers to Grünwald (2007) for an in-depth discussion.

Original: A major limitation of CNML is that it requires training an entire neural network to convergence on the entire augmented dataset every time we want to evaluate a test point’s class probabilities.
Revised: A major limitation of CNML is that it requires training an entire neural network to convergence on the entire augmented dataset every time we want to evaluate a test point’s class probabilities.

Original: We will address this issue in Section 5.
Revised: We will address this issue in Section 5.

Original:  B AYESIAN S UCCESS C LASSIFIERS FOR R EWARD I NFERENCE
Revised:  B AYESIAN S UCCESS C LASSIFIERS FOR R EWARD I NFERENCE

Original: Ideally, training a classiﬁer with the policy samples as negative examples as described in Section 3. should yield a smooth decision boundary between the well-separated negative and positive examples.
Revised: Ideally, training a classiﬁer with the policy samples as negative examples as described in Section 3. should yield a smooth decision boundary between the well-separated negative and positive examples.

Original: For example, Figure 1 depicts a simple 1-D example, where the agent starts at the left ( s 0 ) and the positive outcomes are at the right ( s + ) side of the environment.
Revised: For example, Figure 2 depicts a simple 1-D example, where the agent starts at the left ( s 0 ) and the positive outcomes are at the right ( s + ) side of the environment.

Original: Since the positives are on the right and the negatives are on the left, one might expect a classiﬁer to gradually increase its prediction of a success as we move to the right (Figure 1a), which would provide a dense reward signal for the policy to move to the right.
Revised: Since the positives are on the right and the negatives are on the left, one might expect a classiﬁer to gradually increase its prediction of a success as we move to the right (Figure 2a), which would provide a dense reward signal for the policy to move to the right.

Original: However, this idealized scenario rarely happens in practice.
Revised: However, this idealized scenario rarely happens in practice.

Original: Without suitable regularization, the decision boundary between the positive and negative examples may not be smooth.
Revised: Without suitable regularization, the decision boundary between the positive and negative examples may not be smooth.

Original: In fact, the decision boundary of an optimal classiﬁer may take on the form of a sharp boundary anywhere between the positive and negative examples in the early stages of training (Figure 1b).
Revised: In fact, the decision boundary of an optimal classiﬁer may take on the form of a sharp boundary anywhere between the positive and negative examples in the early stages of training (Figure 2b).

Original: As a result, the classiﬁer might provide little to no reward signal for the policy, since it can assign arbitrarily small probabilities to the states sampled from the policy.
Revised: As a result, the classiﬁer might provide little to no reward signal for the policy, since it can assign arbitrarily small probabilities to the states sampled from the policy.

Original: We note that this issue is not pathological: our experiments in Section 6 show that this poor reward signal issue happens in practice and can greatly hinder learning.
Revised: We note that this issue is not pathological: our experiments in Section 6 show that this poor reward signal issue happens in practice and can greatly hinder learning.

Original: In this section, we will discuss how an appropriate classiﬁer training method can avoid these uninformative rewards.
Revised: In this section, we will discuss how an appropriate classiﬁer training method can avoid these uninformative rewards.

Original: [Figure] 
Revised: [Figure] 4. R EGULARIZED S UCCESS C LASSIFIERS VIA N ORMALIZED M AXIMUM L IKELIHOOD

Original: To create effective shaping, we need to impose a prior on our classiﬁer so that it provides a more informative reward when evaluated at rarely visited states that lie on the path to successful outcomes.
Revised: To create effective shaping, we need to impose a prior on our classiﬁer so that it provides a more informative reward when evaluated at rarely visited states that lie on the path to successful outcomes.

Original: A more informative reward function is one that assigns higher rewards to the fringe of the states visited by the policy, because this reward function will encourage the policy to explore and move towards the desired states.
Revised: A more informative reward function is one that assigns higher rewards to the fringe of the states visited by the policy, because this reward function will encourage the policy to explore and move towards the desired states.

Original: We can construct such a reward function by imposing the prior that novel states have a non-negligible chance of being a success state.
Revised: We can construct such a reward function by imposing the prior that novel states have a non-negligible chance of being a success state.

Original: To do so, we train a Bayesian classiﬁer using conditional normalized maximum likelihood (CNML) (Shtar’kov, 1987), as we described in
Revised: To do so, we train a Bayesian classiﬁer using conditional normalized maximum likelihood (CNML) (Shtar’kov, 1987), as we described in

Original: Section 3, which is essentially imposing a uniform prior on the space of outcomes.
Revised: Section 3, which is essentially imposing a uniform prior on the space of outcomes.

Original: To use CNML for reward inference, the procedure is similar to the one described in Section 3.
Revised: To use CNML for reward inference, the procedure is similar to the one described in Section 3.

Original: We construct a dataset using the provided successful outcomes as positives and the on-policy samples as negatives.
Revised: We construct a dataset using the provided successful outcomes as positives and the on-policy samples as negatives.

Original: However, the label probabilities for RL are then produced by the CNML procedure described in Equation 1 to obtain the rewards r ( s ) =
Revised: However, the label probabilities for RL are then produced by the CNML procedure described in Equation 1 to obtain the rewards r ( s ) =

Original: p CNML ( e = 1 | s ) .
Revised: p CNML ( e = 1 | s ) .

Original: How does this affect reward assignment during learning?
Revised: How does this affect reward assignment during learning?

Original: To illustrate the impact of using CNML, wecan visualize a potential assignment of rewards with a CNML-based classiﬁer on the problem described earlier.
Revised: To illustrate the impact of using CNML, we visualize a potential assignment of rewards with a CNML-based classiﬁer on the problem described earlier.

Original: When the success classiﬁer is trained with CNML instead of standard maximum likelihood, intermediate unseen states would receive non-zero rewards rather than simply having vanishing likelihoods like inFigure 1b.
Revised: When the success classiﬁer is trained with CNML instead of standard maximum likelihood, intermediate unseen states would receive non-zero rewards rather than simply having vanishing likelihoods like in

Original:  The didactic illustration in Fig 1c and Fig 1d shows how the rewards obtained via NML might incentivize exploration.
Revised: Figure 2b. The didactic illustrations in Fig 2c and Fig 2d show how the rewards obtained via NML might incentivize exploration.

Original: When there is no generalization across states, the CNML likelihood will correspond directly to count based exploration (as we show below).
Revised: When there is no generalization across states, the CNML likelihood will correspond directly to count based exploration (as we show below).

Original: However, when there is actually some degree of generalization across states, the classiﬁer can provide more directed shaping towards the goal.
Revised: However, when there is actually some degree of generalization across states, the classiﬁer can provide more directed shaping towards the goal.

Original: 4. R ELATIONSHIP TO C OUNT -B ASED E XPLORATION
Revised: 4. R ELATIONSHIP TO C OUNT -B ASED E XPLORATION

Original: In this section weformally relate the success likelihoods obtained via CNML to commonly used exploration methods based on counts.
Revised: In this section we relate the success likelihoods obtained via CNML to commonly used exploration methods based on counts.

Original: Formally, we prove that the success classiﬁer trained with CNML is equivalent to a version of count-based exploration in the absence of any generalization across states (i.e., a fully tabular setting).
Revised: Formally, we prove that the success classiﬁer trained with CNML is equivalent to a version of count-based exploration in the absence of any generalization across states (i.e., a fully tabular setting).

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Suppose we are estimating success probabilities p ( e = 1 | s ) in the tabular setting, where we have a separate parameter independently for each state.
Revised: Suppose we are estimating success probabilities p ( e = 1 | s ) in the tabular setting, where we have a separate parameter independently for each state.

Original: Let N ( s ) denote the number of times state s has been visited by the policy, and let G ( s ) be the number of occurrences of state s in the successful outcomes.
Revised: Let N ( s ) denote the number of times state s has been visited by the policy, and let G ( s ) be the number of occurrences of state s in the successful outcome examples.

Original: Then the CNML probability p CNML ( e = 1 | s ) is equal to
Revised: Then the CNML probability p CNML ( e = 1 | s ) is equal to

Original: G ( s )+1 N ( s )+ G ( s )+2 .
Revised: G ( s )+1 N ( s )+ G ( s )+2 .

Original: For states that are never observed to be successful, we then recover inverse counts 1 N ( s )+2 .
Revised: For states that are never observed to be successful, we then recover inverse counts 1 N ( s )+2 .

Original: Refer to Appendix Appendix A.5 for a full proof.
Revised: Refer to Appendix A.6 for a full proof.

Original: Additionally, in practice, the learned classiﬁer attains non-negligible generalization.

Original: As long as this generalization is correlated with the dynamics of the environment, we expect this generalization to improve shaping, as we discuss next.

Original: 4. R EWARD S HAPING WITH B AYESIAN S UCCESS C LASSIFIERS
Revised: 4. R EWARD S HAPING WITH B AYESIAN S UCCESS C LASSIFIERS

Original: While the analysis above suggests that a CNML classiﬁer would give us something akin to an exploration bonus, the structure of the problem and the state space actually provides us more information to guide us towards the goal.
Revised: While the analysis above suggests that a CNML classiﬁer would give us something akin to an exploration bonus, the structure of the problem and the state space actually provides us more information to guide us towards the goal.

Original: In most environments (Brockman et al., 2016;
Revised: In most environments (Brockman et al., 2016;

Original: Yu et al., 2019)
Revised: Yu et al., 2019)

Original: the state space does not consist of independent and uncorrelated categorical variables, but is provided in a representation that relates at least roughly to the dynamics structure in the environment.
Revised: the state space does not consist of independent and uncorrelated categorical variables, but is provided in a representation that relates at least roughly to the dynamics structure in the environment.

Original: For instance, states close to the goal dynamically are also typically close to the goal in the metric space deﬁned by the states.
Revised: For instance, states close to the goal dynamically are also typically close to the goal in the metric space deﬁned by the states.

Original: Indeed, this observation is the basis of many commonly used heuristic reward shaping methods, such as rewards given by Euclidean distance to target states.
Revised: Indeed, this observation is the basis of many commonly used heuristic reward shaping methods, such as rewards given by Euclidean distance to target states.

Original: In this case, the task speciﬁcation can actually provide more information than simply performing uninformed count-based exploration.
Revised: In this case, the task speciﬁcation can actually provide more information than simply performing uninformed count-based exploration.

Original: Since the uncertainty aware classiﬁer described in Section 4. is built on top of features that are correlated with environment dynamics, it is able to incentivize task-aware directed exploration.
Revised: Since the uncertainty aware classiﬁer described in Section 4. is built on top of features that are correlated with environment dynamics, it is able to incentivize task-aware directed exploration.

Original: A depiction of this phenomenon is illustrated in Fig 1d.
Revised: A depiction of this phenomenon is illustrated in Fig 2d, and is demonstrated empirically in our experimental analysis in Section 6.

Original: As compared to CNML without generalization in Fig 1c, we expect the intermediate rewards to provide more shaping towards the goal.
Revised: As compared to CNML without generalization in Fig 2c, we expect the intermediate rewards to provide more shaping towards the goal.

Original: This is evident from our empirical results in Section 6 where BayCRL is able to signiﬁcantly outperform methods for task agnostic exploration. 
Revised: This is evident from our empirical results in Section 6 where BayCRL is able to signiﬁcantly outperform methods for task agnostic exploration. 4. O VERVIEW

Revised: In this section, we introduced the idea of Bayesian classiﬁers trained via CNML as a means to provide rewards for RL problems speciﬁed via examples of successful outcomes.

Revised: Standard MLE classiﬁers can provide exploration and reward shaping, whereas properly regularized classiﬁers can directly provide beneﬁts for exploration (as shown in Theorem 4.1) and reward shaping, as discussed in Section 4.3.

Revised: We describe an efﬁcient technique to train these types of Bayesian classiﬁers via meta-learning in the following section.

Original:  B AY CRL: T RAINING B AYESIAN S UCCESS C LASSIFIERS FOR O UTCOME D RIVEN RL VIA M ETA -L EARNING AND CNML
Revised:  B AY CRL: T RAINING B AYESIAN S UCCESS C LASSIFIERS FOR O UTCOME D RIVEN RL VIA M ETA -L EARNING AND CNML

Original: In Section 4, we discussed how Bayesian success classiﬁers can incentivize exploration and provide reward shaping to guide RL.
Revised: In Section 4, we discussed how Bayesian success classiﬁers can incentivize exploration and provide reward shaping to guide RL.

Original: However, the reward inference technique via CNML described in
Revised: However, the reward inference technique via CNML described in

Original: Section 4.1 is computationally intractable, as it requires optimizing maximum likelihood estimation problems to convergence on every data point we want to query.
Revised: Section 4.1 is computationally intractable, as it requires optimizing maximum likelihood estimation problems to convergence on every data point we want to query.

Original: In this section, we describe a novel approximation that allows us to instantiate this method in practice.
Revised: In this section, we describe a novel approximation that allows us to instantiate this method in practice.

Original: 5. M ETA -L EARNING FOR CNML
Revised: 5. M ETA -L EARNING FOR CNML

Original: We adopt ideas from meta-learning to amortize the cost of obtaining the CNML distribution.
Revised: We adopt ideas from meta-learning to amortize the cost of obtaining the CNML distribution.

Original: As noted in Section 4.1, the computation of the CNML distribution involves repeatedly solving maximum likelihood problems that share a signiﬁcant amount of common structure.
Revised: As noted in Section 4.1, the computation of the CNML distribution involves repeatedly solving maximum likelihood problems.

Original: We should use this commonality between problems  to quickly obtain CNML estimates without redoing the slow training process repeatedly.
Revised: While computationally daunting, these problems share a signiﬁcant amount of common structure, which we should exploit to quickly obtain CNML estimates without redoing the slow training process repeatedly.

Original: One set of techniques that are directly applicable is meta-learning for few shot classiﬁcation.
Revised: One set of techniques that are directly applicable is meta-learning for few shot classiﬁcation.

Original: Meta-learning uses a distribution of training problems to explicitly learn models that can quickly adapt to new problems.
Revised: Meta-learning uses a distribution of training problems to explicitly learn models that can quickly adapt to new problems.

Original: To apply meta-learning to the CNML problem, we can formulate each of the maximum likelihood problems described in Equation 1 as a separate task for meta-learning, and apply any standard meta-learning technique to obtain a model capable of few-shot adaptation to the MLE problems required for CNML.
Revised: To apply meta-learning to the CNML problem, we can formulate each of the maximum likelihood problems described in Equation 1 as a separate task for meta-learning, and apply any standard meta-learning technique to obtain a model capable of few-shot adaptation to the MLE problems required for CNML.

Original: While any meta-learning algorithm is applicable, we found model agnostic meta-learning (MAML) to be an effective choice of algorithm.
Revised: While any meta-learning algorithm is applicable, we found model agnostic meta-learning (MAML) to be an effective choice of algorithm.

Original: In short, MAML tries to learn a model that can quickly adapt to new tasks via a few steps of gradient descent.
Revised: In short, MAML tries to learn a model that can quickly adapt to new tasks via a few steps of gradient descent.

Original: We refer readers to Finn et al.
Revised: We refer readers to Finn et al.

Original: (2017) for a detailed overview.
Revised: (2017) for a detailed overview.

Original: This procedure is illustrated in Fig 8, and can be described as follows: given a dataset D
Revised: This procedure is illustrated in Fig 10, and can be described as follows: given a dataset D

Original: = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , ..., ( x n , y n ) } , 2 n different tasks τ
Revised: = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , ..., ( x n , y n ) } , 2 n different tasks τ

Original: i can be constructed, each corresponding to performing maximum likelihood estimation on the dataset with a certain proposed label for x i : max θ E ( x,y )
Revised: i can be constructed, each corresponding to performing maximum likelihood estimation on the dataset with a certain proposed label for x i : max θ E ( x,y )

Original: ∼D∪ ( x i ,y =0) log p ( y | x, θ ) or max θ E ( x,y ) ∼D∪ ( x i ,y =1) log p ( y | x, θ ) .
Revised: ∼D∪ ( x i ,y =0) log p ( y | x, θ ) or max θ E ( x,y ) ∼D∪ ( x i ,y =1) log p ( y | x, θ ) .

Original: Given these constructed tasks S ( τ ) , meta-training can be performed by performing the bi-level optimization described in Finn et al.
Revised: Given these constructed tasks S ( τ ) , meta-training can be performed by performing the bi-level optimization described in Finn et al.

Original: [Equation]
Revised: [Equation]

Original: This training procedure gives us parameters θ that can then be quickly adapted to provide the CNML distribution simply by performing a step of gradient descent.
Revised: This training procedure gives us parameters θ that can then be quickly adapted to provide the CNML distribution simply by performing a step of gradient descent.

Original: The model can be queried for the CNML distribution by starting from θ and taking one step of gradient descent for the query point augmented dataset, each with a different potential label.
Revised: The model can be queried for the CNML distribution by starting from θ and taking one step of gradient descent for the query point augmented dataset, each with a different potential label.

Original: These likelihoods are then normalized to provide the
Revised: These likelihoods are then normalized to provide the

Original: CNML distribution as follows:
Revised: CNML distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: This algorithm, which we call meta-NML , allows us to obtain normalized likelihood estimates without having to retrain maximum likelihood to convergence at every single query point, since the model can now solve maximum likelihood problems of this form very quickly.
Revised: This algorithm, which we call meta-NML , allows us to obtain normalized likelihood estimates without having to retrain maximum likelihood to convergence at every single query point, since the model can now solve maximum likelihood problems of this form very quickly.

Original: A complete detailed description and pseudocode of this algorithm are provided in Appendix A.2.
Revised: A complete detailed description and pseudocode of this algorithm are provided in Appendix A.2.

Original: 5. A PPLYING M ETA -NML TO S UCCESS C LASSIFICATION
Revised: Feedforward Inference Meta-NML Naive CNML

Original:  RL 1: User provides success examples S + 2: Initialize dataset D with S + as positives with label 1 .
Revised: Single input point 0.0004s 0.0090s 15.19s Epoch of RL 23.50s [Table]

Original: 3: Initialize policy π randomly. 4: for iteration i = 1 , 2 , ... do 5: Collect on-policy examples S − by executing π in the environment.
Revised: 05s 4hr 13min 34s

Original: Add on-policy examples S − to D as negatives with [-label 0 . 7: if iteration i mod k == 
-]
Revised: Crucially, we ﬁnd that metaNML is able to approximate the true NML outputs with [-label 0 . 7: if iteration i mod k == 
-]just one or a few gradient steps.

Original: then 8:
Revised: This makes it several orders of magnitude faster than naive CNML, which would normally require multiple passes through the entire dataset on each input point in order to train to convergence.

Original: Meta-train via meta-NML on tasks sampled from D using Equation9: Assign state rewards via Equation10:
Revised: Table 2 shows a comparison of runtimes for naive CNML and meta-NML on a simple two-dimensional dataset.

Original: Train π with RL algorithm
Revised: 5. A PPLYING M ETA -NML TO S UCCESS C LASSIFICATION

Original: We apply the meta-NML algorithm described above to learning Bayesian success classiﬁers for providing rewards for reinforcement learning, in our proposed algorithm, which we term BayCRL— Bayesian classiﬁers for reinforcement learning.
Revised: We apply the meta-NML algorithm described above to learning Bayesian success classiﬁers for providing rewards for reinforcement learning, in our proposed algorithm, which we term BayCRL— Bayesian classiﬁers for reinforcement learning.

Original: Similarly to Fu et al.
Revised: Similarly to Fu et al.

Original: (2018b), we can train our Bayesian classiﬁer by ﬁrst constructing a dataset D for binary classiﬁcation.
Revised: (2018b), we can train our Bayesian classiﬁer by ﬁrst constructing a dataset D for binary classiﬁcation.

Original: This is done by using the provided examples of successful outcomes as positives, and on-policy examples collected by the policy as negatives.
Revised: This is done by using the provided examples of successful outcomes as positives, and on-policy examples collected by the policy as negatives, balancing the number of sampled positives and negatives in the dataset.

Original: Given this dataset, the Bayesian classiﬁer parameters θ meta-NML can be trained via meta-NML as described in Equation 2.
Revised: Given this dataset, the Bayesian classiﬁer parameters θ R can be trained via meta-NML as described in Equation 2.

Original: The classiﬁer can then be used to directly and quickly assign rewards to a state s according to its probabilities r ( s ) =
Revised: The classiﬁer can then be used to directly and quickly assign rewards to a state s according to its probabilities r ( s ) =

Original: p meta-NML ( e = 1 | s ) (via a step of gradient descent, as described in Equation 3), and perform standard reinforcement learning.
Revised: p meta-NML ( e = 1 | s ) (via a step of gradient descent, as described in Equation 4), and perform standard reinforcement learning.

Original: [Equation]
Revised: [Equation]

Original: An overview of this algorithm is provided in Algorithm 1, and full details are in Appendix A.2.
Revised: An overview of this algorithm is provided in Algorithm 1, and full details are in Appendix A.2.

Revised: The rewards start off at an uninformative value of 0 .

Original: E XPERIMENTAL E VALUATION
Revised: 5 for all unvisited states at the beginning, and close to 1 for successful outcomes. As training progresses, more states are visited, added to the buffer and BayCRL starts to assign them progressively lower reward as they get visited more and more, thereby encouraging visiting of under-visited states. At convergence, all the non successful states will have a reward of close to 0 and states at the goal will have a reward of 0 .

Revised: 5 , since the numbers of positive and negative labels for successful outcomes will be balanced as described above.

Original: In our experimental evaluation we aim to answer the following questions: (1) Do the learning dynamics of prior classiﬁer-based reward learning methods provide informative rewards for RL?
Revised: In our experimental evaluation we aim to answer the following questions: (1) Do the learning dynamics of prior classiﬁer-based reward learning methods provide informative rewards for RL?

Original: (2) Does using BayCRL help address the exploration challenge when solving RL problems speciﬁed by successful outcomes?
Revised: (2) Does using BayCRL help address the exploration challenge when solving RL problems speciﬁed by successful outcomes?

Original: (3) Does using BayCRL help provide better reward shaping than simply performing naïvely uninformed exploration?
Revised: (3) Does using BayCRL help provide better reward shaping than simply performing naïvely uninformed exploration?

Original: To evaluate these questions, we evaluate our proposed algorithm BayCRL with the following setup.
Revised: To evaluate these questions, we evaluate our proposed algorithm BayCRL with the following setup.

Original: Further details and videos can be found at https://sites.google com/view
Revised: Further details and videos can be found at https://sites.google.com/view/baycrl/home

Original: /baycrl/home 6. E XPERIMENTAL S ETUP
Revised: 6. E XPERIMENTAL S ETUP

Original: We start off by understanding the algorithm behavior by evaluating it on maze navigation problems, which require avoiding several local optima before truly reaching the goal.
Revised: We start off by understanding the algorithm behavior by evaluating it on maze navigation problems, which require avoiding several local optima before truly reaching the goal.

Original: Then, to evaluate our method in more complex domains, we consider three robotic manipulation tasks that were considered in Fu et al. (2018b) with a Sawyer robot arm: door opening, tabletop object pushing, and 3D object picking.
Revised: Then, to evaluate our method in more complex domains, we consider three robotic manipulation tasks that were previously covered in Singh et al. (2019a) with a Sawyer robot arm: door opening, tabletop object pushing, and 3D object picking.

Original: As we show in our results, exploration in these environments is challenging and using naively chosen reward shaping often does not solve the problem at hand.
Revised: As we show in our results, exploration in these environments is challenging and using naively chosen reward shaping often does not solve the problem at hand.

Revised: More details on each environment and their associated challenges are available in Appendix A.4.1.

Original: [Figure]
Revised: [Figure]

Original: We compare with a number of prior algorithms and ablations.
Revised: We compare with a number of prior algorithms and ablations.

Original: To provide a comparison with a standard previous method based on success classiﬁers trained with an IRL-based adversarial method, we include the VICE algorithm Fu et al.
Revised: To provide a comparison with a standard previous method based on success classiﬁers trained with an IRL-based adversarial method, we include the VICE algorithm Fu et al.

Original: (2018b).
Revised: (2018b).

Revised: Note that this algorithm is quite related to BayCRL, but it uses a standard maximum likelihood classiﬁer rather than a Bayesian classiﬁer trained with normalized maximum likelihood and metalearning.

Original: We also include a comparison with a recently proposed technique for learning dynamical distances Hartikainen et al.
Revised: We also include a comparison with a recently proposed technique for learning dynamical distances (Hartikainen et al., 2019).

Revised: Both VICE and DDL have access to user-provided goal examples in the same format as BayCRL.

Original: We additionally include comparisons to algorithms for uninformed exploration to show that BayCRL does a more directed form of exploration and reward shaping.
Revised: We additionally include comparisons to algorithms for uninformed exploration to show that BayCRL does a more directed form of exploration and reward shaping.

Original: To provide an apples-toapples comparison, we use the same VICE method for training classiﬁers, but combine it with novelty-based exploration based on random network distillation (Burda et al., 2018b).
Revised: To provide an apples-to-apples comparison, we use the same VICE method for training classiﬁers, but combine it with novelty-based exploration based on random network distillation (Burda et al., 2018b) for the robotic manipulation tasks, and oracle inverse count bonuses for the maze navigation tasks.

Revised: Finally, we compare to two naive reward functions to demonstrate the importance of well-shaped rewards.

Revised: These are a sparse reward, which is only nonzero when the agent is within the threshold of success, and a heuristically shaped reward which simply uses L2 distance to the goal state.

Original: 6. C OMPARISONS WITH P RIOR A LGORITHMS
Revised: 6. C OMPARISONS WITH P RIOR A LGORITHMS

Original: We compare with prior algorithms on the domains described above.
Revised: We compare with prior algorithms on the domains described above.

Original: As we can see in Fig 4, BayCRL is able to very quickly learn how to solve these challenging exploration tasks, often reaching signiﬁcantly better asymptotic performance than most prior methods, and doing so signiﬁcantly more efﬁciently than VICE (Fu et al., 2018b) or DDL (Hartikainen et al., 2019).
Revised: As we can see in Fig 5, BayCRL is able to very quickly learn how to solve these challenging exploration tasks, often reaching signiﬁcantly better asymptotic performance than most prior methods, and doing so signiﬁcantly more efﬁciently than VICE (Fu et al., 2018b) or DDL (Hartikainen et al., 2019).

Original: This suggests that BayCRL is able to provide directed reward shaping and exploration that is substantially better than standard classiﬁer-based methods (e.g., VICE).
Revised: This suggests that BayCRL is able to provide directed reward shaping and exploration that is substantially better than standard classiﬁer-based methods (e.g., VICE).

Original: To isolate whether the beneﬁts purely come from exploration or also from task-aware reward shaping, we compare with methods that only perform uninformed, task-agnostic exploration.
Revised: To isolate whether the beneﬁts purely come from exploration or also from task-aware reward shaping, we compare with methods that only perform uninformed, task-agnostic exploration.

Original: On the maze environments, where we can discretize the state space, we compute ground truth count-based bonuses for exploration.
Revised: On the maze environments, where we can discretize the state space, we compute ground truth countbased bonuses for exploration.

Original: For the higher dimensional robotics tasks, we use RND (Burda et al., 2018b).
Revised: For the higher dimensional robotics tasks, we use RND (Burda et al., 2018b).

Original: From these comparisons, shown in Fig 4, it is clear that BayCRL signiﬁcantly outperforms methods that use novelty-seeking exploration, but do not otherwise provide effective reward shaping.
Revised: From these comparisons, shown in Fig 5, it is clear that BayCRL signiﬁcantly outperforms methods that use noveltyseeking exploration, but do not otherwise provide effective reward shaping.

Original: In combination with our visualizations in Section 6.4, this suggests that BayCRL is providing useful task-aware reward shaping more effectively than uniformed exploration methods.
Revised: In combination with our visualizations in Section 6.4, this suggests that BayCRL is providing useful task-aware reward shaping more effectively than uniformed exploration methods.

Original: We also compare BayCRL to a manually heuristically-designed shaped reward function, based on Euclidean distance.
Revised: We also compare BayCRL to a manually heuristically-designed shaped reward function, based on Euclidean distance.

Original: As shown in 
Revised: As shown in Fig 5,

Original: Fig 4, BayCRL generally outperforms simple manual shaping in terms of sample complexity and asymptotic performance, indicating that the learned shaping is non-trivial and adapted to the task.
Revised: BayCRL generally outperforms simple manual shaping in terms of sample complexity and asymptotic performance, indicating that the learned shaping is non-trivial and adapted to the task.

Original: 6. A BLATIONS
Revised: 6. A BLATIONS

Original: We ﬁrst evaluate the importance of meta-learning for estimating the NML distribution.
Revised: We ﬁrst evaluate the importance of meta-learning for estimating the NML distribution.

Original: In Figure 5, we see that naïvely estimating the NML distribution with a small number of gradient steps, without  meta-training, results in much worse performance.
Revised: In Figure 6, we see that naively estimating the NML distribution by taking a single gradient step and following the same process as evaluating meta-NML, but without any meta-training, results in much worse performance.

Original: Second, we analyze the importance of making the BayCRL classiﬁer aware of the task being solved, to understand whether BayCRL is informed by the success examples, or simply approximates count-based exploration.
Revised: Second, we analyze the importance of making the BayCRL classiﬁer aware of the task being solved, to understand whether BayCRL is informed by the success examples or simply approximates count-based exploration.

Original: To that end, we modify the training procedure so that the dataset D consists of only the on-policy negatives, and add the inferred reward from the Bayesian classiﬁer to the reward obtained by a standard MLE classiﬁer (similarly to the VICE+RND baseline).
Revised: To that end, we modify the training procedure so that the dataset D consists of only the on-policy negatives, and add the inferred reward from the Bayesian classiﬁer to the reward obtained by a standard MLE classiﬁer (similarly to the VICE+RND baseline).

Original: We see that this performs poorly, showing that the BayCRL classiﬁer is doing more than just performing count-based exploration.
Revised: We see that this performs poorly, showing that the BayCRL classiﬁer is doing more than just performing count-based exploration, and beneﬁts from better reward shaping due to the provided goal examples.

Original: 6. A NALYSIS OF B AY CRL
Revised: 6. A NALYSIS OF B AY CRL

Original:  To further analyze BayCRL, weﬁrst visualize theshaped rewards  on the Sawyer Pick task.
Revised: BayCRL and Reward Shaping. To better understand how BayCRL provides reward shaping, we visualize the rewards for various slices along the z axis on the Sawyer Pick task, which presents a signiﬁcant exploration challenge.

Original: In Fig 6a we see that the BayCRL rewards clearly correlate with the distance to the object’s goal position, whereas the MLE classiﬁer has a sharp, poorly-shaped decision boundary.
Revised: In Fig 7 we see that the BayCRL rewards clearly correlate with the distance to the object’s goal position, shown as a white star, thus guiding the robot to raise the ball to the desired location even if it has never reached the goal before. In contrast, the MLE classiﬁer has a sharp, poorly-shaped decision boundary.

Original:  Next, to understand the connection between BayCRL andcount based exploration, we visualize the reward landscape in an early iteration of the Zigzag Maze task with BayCRL  in Fig 6b.
Revised: BayCRL and Exploration. Next, to illustrate the connection between BayCRL and exploration, we compare the states visited by BayCRL (which uses a meta-NML classiﬁer) and by VICE (which uses a standard L2-regularized classiﬁer) in Figure 8.

Original: We see that  the algorithm maintains a fringe of least visited states and rewards those states more, explicitly encouraging exploration.
Revised: We see that BayCRL naturally incentivizes the agent to visit novel states, allowing it to navigate around local minima and reach the true goal.

Original: Furthermore, meta-NML is able to reasonably approximate the idealized
Revised: In contrast,

Original: NML rewards with just one gradient step, while also beneﬁting from additional shaping due to generalization of the classiﬁer.
Revised: VICE learns a misleading reward function which simply prioritizes closeness to the goal in xy space, causing the agent to stay on the wrong side of the wall and not explore further.

Original: See Appendix A.3 for a more detailed visualization.
Revised: Interestingly, despite incentivizing exploration, BayCRL does not simply visit all possible states; at convergence, it has only covered around 70% of the state space. In fact, we see in the scatterplots below that BayCRL prioritizes states that bring it closer to the goal and ignores ones that don’t, thus making use of the goal examples provided to it.

Original: (a) Visualization of reward shaping for Picking at various z values. BayCRL learns rewards that provide a smooth slope toward the goal, adapting to the policy and guiding it to learn the task, while [Figure]
Revised: This suggests that BayCRL beneﬁts from a combination of novelty-seeking behavior and effective reward shaping, allowing it to choose new states strategically rather than resorting to naive exploration.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we consider a subclass of reinforcement learning problems where a task is speciﬁed by examples of successful outcomes.
Revised: In this work, we consider a subclass of reinforcement learning problems where a task is speciﬁed by examples of successful outcomes.

Original: We analyze how solutions via standard success classiﬁers suffer from shortcomings, and training Bayesian classiﬁers allows for better exploration to solve challenging problems.
Revised: We analyze how solutions via standard success classiﬁers suffer from shortcomings, and training Bayesian classiﬁers allows for better exploration to solve challenging problems.

Original: We discuss how the NML distribution can provide us a way to train such
Revised: We discuss how the NML distribution can provide us a way to train such

Original: Bayesian classiﬁers, providing beneﬁts of exploration and reward shaping.
Revised: Bayesian classiﬁers, providing beneﬁts of exploration and reward shaping.

Original: To make learning tractable, we propose a  meta-learning approach to amortize the NML process.
Revised: To make learning tractable, we propose a novel meta-learning approach to amortize the NML process.

Original: While this work has shown the effectiveness of Bayesian classiﬁers for reward inference for tasks in simulation, it would be interesting to scale this solution to real world problems.
Revised: While this work has shown the effectiveness of Bayesian classiﬁers for reward inference for tasks in simulation, it would be interesting to scale this solution to real world problems.

Original: Additionally, obtaining a theoretical understanding of how reward shaping interacts with learning dynamics would be illuminating in designing reward schemes.
Revised: Additionally, obtaining a theoretical understanding of how reward shaping interacts with learning dynamics would be illuminating in designing reward schemes

Original: A. D ETAILED D ESCRIPTION OF M ETA -NML
Revised: A. D ETAILED D ESCRIPTION OF M ETA -NML

Original: We provide a detailed description of the meta-NML algorithm described in Section 5, and the details of the practical algorithm.
Revised: We provide a detailed description of the meta-NML algorithm described in Section 5, and the details of the practical algorithm.

Original: Given a dataset D = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , .., ( x n , y n ) } , the meta-NML procedure proceeds by ﬁrst constructing k ∗ n tasks from these data points, for a k shot classiﬁcation problem.
Revised: Given a dataset D = { ( x 0 , y 0 ) , ( x 1 , y 1 ) , .., ( x n , y n ) } , the meta-NML procedure proceeds by ﬁrst constructing k ∗ n tasks from these data points, for a k shot classiﬁcation problem.

Original: We will keep k = 2 for simplicity in this description, in accordance with the setup of binary success classiﬁers in RL.
Revised: We will keep k = 2 for simplicity in this description, in accordance with the setup of binary success classiﬁers in RL.

Original: Each task τ
Revised: Each task τ

Original: i is constructed by augmenting the dataset with a negative label D ∪ ( x i , y = 0) or a positive label D ∪ ( x i , y = 1) .
Revised: i is constructed by augmenting the dataset with a negative label D ∪ ( x i , y = 0) or a positive label D ∪ ( x i , y = 1) .

Original: Now that each task consists of solving the maximum likelihood problem for it’s augmented dataset, we can directly apply standard meta-learning algorithms to this setting.
Revised: Now that each task consists of solving the maximum likelihood problem for it’s augmented dataset, we can directly apply standard meta-learning algorithms to this setting.

Original: Building off the ideas in MAML (Finn et al., 2017), we can then train a set of model parameters θ such that after a single step of gradient descent it can quickly adapt to the optimal solution for the MLE problem on any of the augmented datasets.
Revised: Building off the ideas in MAML (Finn et al., 2017), we can then train a set of model parameters θ such that after a single step of gradient descent it can quickly adapt to the optimal solution for the MLE problem on any of the augmented datasets.

Original: This is more formally written as
Revised: This is more formally written as

Original: [Equation]
Revised: [Equation]

Original:  where L represents a standard classiﬁcation loss function, α is the learning rate, and the distribution of tasks p ( τ ) is constructed as described above.
Revised: [Equation] ) where L represents a standard classiﬁcation loss function, α is the learning rate, and the distribution of tasks p ( τ ) is constructed as described above.

Original: For a new query point x , these initial parameters can then quickly be adapted to provide the CNML distribution by taking a gradient step on each augmented dataset to obtain the approximatley optimal MLE solution, and normalizing these as follows:
Revised: For a new query point x , these initial parameters can then quickly be adapted to provide the CNML distribution by taking a gradient step on each augmented dataset to obtain the approximately optimal MLE solution, and normalizing these as follows:

Original: [Equation]
Revised: [Equation]

Original: This algorithm in principle can be optimized used any standard stochastic optimization method such as SGD, as described in Finn et al.
Revised: This algorithm in principle can be optimized using any standard stochastic optimization method such as SGD, as described in Finn et al.

Original: (2017), backpropagating through the inner loop gradient update.
Revised: (2017), backpropagating through the inner loop gradient update.

Original: For the speciﬁc problem setting that we consider, we have to employ some optimization tricks in order to enable learning:
Revised: For the speciﬁc problem setting that we consider, we have to employ some optimization tricks in order to enable learning:

Original: A.2. I MPORTANCE W EIGHTING ON Q UERY P OINT
Revised: A.2. I MPORTANCE W EIGHTING ON Q UERY P OINT

Original: Since only one datapoint is augmented to the training set, it can get challenging for stochastic optimization to pay attention to this datapoint with increasing dataset sizes.
Revised: Since only one datapoint is augmented to the training set, it can get challenging for stochastic optimization to pay attention to this datapoint with increasing dataset sizes.

Original: In order to make sure the optimization considers the query point, we include the query point and proposed label ( x q , y ) in every minibatch that is sampled, but downweight the loss computed on that point such that the overall objective remains unbiased.
Revised: In order to make sure the optimization considers the query point, we include the query point and proposed label ( x q , y ) in every minibatch that is sampled, but downweight the loss computed on that point such that the overall objective remains unbiased.

Original: This is simply doing importance weighting, with the query point downweighted by a factor of bN where b is batchsize, N is the total number of points in the dataset
Revised: This is simply doing importance weighting, with the query point downweighted by a factor of bN where b is batchsize, N is the total number of points in the dataset.

Original: . A.2. K ERNEL W EIGHTED T RAINING L OSS
Revised: A.2. K ERNEL W EIGHTED T RAINING L OSS

Original: The augmented dataset consists of points from the original dataset D and one augmented point ( x q , y ) .
Revised: The augmented dataset consists of points from the original dataset D and one augmented point ( x q , y ) .

Original: Given that we mostly care about having the proper likelihood on the query point, with an imperfect optimization process, the meta-training can yield solutions that are not very accurately representing true likelihoods on the query point.
Revised: Given that we mostly care about having the proper likelihood on the query point, with an imperfect optimization process, the meta-training can yield solutions that are not very accurately representing true likelihoods on the query point.

Original: To counter this, we introduce a kernel weighting into the loss function in Equation 6 during meta-training and subsequently meta-testing.
Revised: To counter this, we introduce a kernel weighting into the loss function in Equation 6 during meta-training and subsequently meta-testing.

Original: The kernel weighting modiﬁes the training loss function as:
Revised: The kernel weighting modiﬁes the training loss function as:

Original: [Equation]
Revised: [Equation]

Original: where x τ is the query point for task τ and K is a choice of kernel.
Revised: where x τ is the query point for task τ and K is a choice of kernel.

Original: We typically choose exponential kernels centered around x τ .
Revised: We typically choose exponential kernels centered around x τ .

Original: Intuitively, this allows the meta-optimization to mainly consider the datapoints that are copies of the query point in the dataset, or are similar to the query point, and ensures that they have the correct likelihoods, instead of receiving interfering gradient signals from the many other points in the dataset.
Revised: Intuitively, this allows the meta-optimization to mainly consider the datapoints that are copies of the query point in the dataset, or are similar to the query point, and ensures that they have the correct likelihoods, instead of receiving interfering gradient signals from the many other points in the dataset.

Original: To make hyperparameter selection intuitive, we designate the strength of the exponential kernel by a parameter λ dist , which is the Euclidean distance away from the query point at which the weight becomes 0.1.
Revised: To make hyperparameter selection intuitive, we designate the strength of the exponential kernel by a parameter λ dist , which is the Euclidean distance away from the query point at which the weight becomes 0.1.

Original: Formally, the weight of a point x in the loss function for query point x τ is computed as:
Revised: Formally, the weight of a point x in the loss function for query point x τ is computed as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: E XTRA G RADIENT S TEPS FOR M ETA -T ESTING
Revised: A.2. M ETA -T RAINING AT F IXED I NTERVALS

Original: Given that we are using MAML for meta-training, we can continue training at meta-test time using standard gradient descent.
Revised: While in principle meta-NML would retrain with every new datapoint, in practice we retrain metaNML once every k epochs.

Original: Taking a few more gradient steps on the meta-test time MLE objective allows the algorithm to be no worse than gradient descent in the worst case. When the meta-learner does not generalize as well, these extra gradient steps ensure that the estimated likelihoods are still well calibrated.
Revised: (In all of our experiments we set k = 1 , but we could optionally increase k if we do not expect the meta-task distribution to change much between epochs.) We warm-start the meta-learner parameters from the previous iteration of meta-learning, so every instance of meta-training only requires a few steps.

Original: While this gives us more accurate NML estimates, in practice we found that taking one gradient step was sufﬁcient to achieve good performance on our RL tasks.
Revised: We ﬁnd that this periodic training is a reasonable enough approximation, as evidenced by the strong performance of BayCRL in our experimental results in Section 6.

Original: A. M ETA -NML V ISUALIZATION W ITH A DDITIONAL G RADIENT S TEPS
Revised: A. M ETA -NML V ISUALIZATIONS A.3. M ETA -NML WITH A DDITIONAL G RADIENT S TEPS

Original: Below, we show a more detailed visualization of meta-NML outputs on data from the Zigzag Maze task.
Revised: Below, we show a more detailed visualization of meta-NML outputs on data from the Zigzag Maze task, and how these outputs change with additional gradient steps.

Original: For comparison, we also include therewards given by an ordinary classiﬁer trained using MLE (which gives a poorly shaped reward due to the sharp decision boundary), and the idealized NML rewards (which come from a discrete count-based classiﬁer).
Revised: For comparison, we also include the idealized NML rewards, which come from a discrete count-based classiﬁer. Meta-NML is able to resemble the ideal NML rewards fairly well with just 1 gradient step, providing both an approximation of a count-based exploration bonus and better shaping towards the goal due to generalization. By taking additional gradient steps, meta-NML can get arbitrarily close to the true NML outputs, which themselves correspond to inverse counts of 1 n +2 as explained in Theorem 4.1.

Original: Meta-NML is able to approximate the ideal NML rewards fairly well with just 1 gradient step, providing a much better shaped reward than the MLE classiﬁer.
Revised: While this would give us more accurate NML estimates, in practice we found that taking one gradient step was sufﬁcient to achieve good performance on our RL tasks. [Figure]

Original: By taking additional gradient steps, meta-NML can get arbitrarily close to the true NML outputs, which themselves correspond to inverse counts.
Revised: Below provide the runtimes for feedforward inference, naive CNML, and meta-NML on each of our evaluation domains.

Original: A. H YPERPARAMETER AND I MPLEMENTATION D ETAILS
Revised: We list both the runtimes for evaluating a single input, and for completing a full epoch of training during RL. These benchmarks were performed on an NVIDIA Titan X Pascal GPU. Per-input runtimes are averaged across 100 samples, and per-epoch runtimes are averaged across 20 epochs. [Table]

Revised: [Table]

Revised: Mazes (zigzag, spiral)

Revised: [Table]

Revised: [Table]

Revised: A. E XPERIMENTAL D ETAILS A.4. E NVIRONMENTS

Revised: Zigzag Maze and Spiral Maze: These two navigation tasks require moving through long corridors and avoiding several local optima in order to reach the goal.

Revised: For example, on Spiral Maze, the agent must not get stuck on the other side of the inner wall, even though that position would be close in L  distance to the desired goal.

Revised: On these tasks, a sparse reward is not informative enough for learning, while ordinary classiﬁer methods get stuck in local optima due to poor shaping near the goal.

Revised: Both of these environments have a continuous state space consisting of the ( x, y ) coordinates of the agent, ranging from ( − 4 , − 4) to (4 , 4) inclusive.

Revised: The action space is the desired velocity in the x and y directions, each ranging from − 1 to 1 inclusive.

Revised: Sawyer 2D Pusher:

Revised: This task involves using a Sawyer arm, constrained to move only in the xy plane, to push a randomly initialized puck to a ﬁxed location on a table.

Revised: The state space consists of the ( x, y, z ) coordinates of the robot end effector and the ( x, y ) coordinates of the puck.

Revised: The action space is the desired x and y velocities of the arm.

Revised: Sawyer Door Opening:

Revised: In this task, the Sawyer arm is attached to a hook, which it must use to open a door to a desired angle of 45 degrees.

Revised: The door is randomly initialized each time to be at a starting angle of between 0 and 15 degrees.

Revised: The state space consists of the ( x, y, z ) coordinates of the end effector and the door angle (in radians); the action space consists of ( x, y, z ) velocities.

Revised: Sawyer 3D Pick and Place: The Sawyer robot must pick up a ball, which is randomly placed somewhere on the table each time, and raise it to a ﬁxed ( x, y, z ) location high above the table.

Revised: This represents the biggest exploration challenge out of all the manipulation tasks, as the state space is large and the agent would normally not receive any learning signal unless it happened to pick up the ball and raise it, which is unlikely without careful reward shaping.

Revised: The state space consists of the ( x, y, z ) coordinates of the end effector, the ( x, y, z ) coordinates of the ball, and the tightness of the gripper (a continuous value between 0 and 1).

Revised: The robot can control its ( x, y, z ) arm velocity as well as the gripper value.

Revised: A.4. G ROUND T RUTH D ISTANCE M ETRICS

Revised: In addition to the success rate plots in Figure 5, we provide plots of each algorithm’s distance to the goal over time according to environment-speciﬁc distance metrics.

Revised: The distance metrics and success thresholds, which were used to compute the success rates in Figure 5, are listed in the table below.

Revised: Environment

Revised: Distance Metric Used

Revised: Success Threshold

Revised: Zigzag Maze

Revised: Manhattan distance to goal 0.

Revised: Spiral Maze

Revised: Manhattan distance to goal 0.

Revised: Sawyer 2D Pusher

Revised: Puck L2 distance to goal 0.

Revised: Sawyer Door Opening

Revised: Angle difference to goal (radians) 0.

Revised: Sawyer 3D Pick-and-Place

Revised: Ball L2 distance to goal 0.

Revised: A. H YPERPARAMETER AND I MPLEMENTATION D ETAILS

Original: We describe the hyperparmeter choices for our experiments here.
Revised: We describe the hyperparameter choices for our experiments here.

Original: We ﬁrst list the general hyperparameters that were shared across runs, then provide tables of additional hyperparameters we tuned over for each domain and algorithm.
Revised: We ﬁrst list the general hyperparameters that were shared across runs, then provide tables of additional hyperparameters we tuned over for each domain and algorithm.

Original: SAC
Revised: SAC

Original: Learning Rate 3 × 10 −
Revised: Learning Rate 3 × 10 −

Original: Discount Factor γ 0 .
Revised: Discount Factor γ 0 .

Original: Policy Type
Revised: Policy Type

Original: Gaussian
Revised: Gaussian

Original: Policy Hidden Sizes (512 , 512)
Revised: Policy Hidden Sizes (512 , 512)

Original: Policy Hidden Activation
Revised: Policy Hidden Activation

Original: ReLU
Revised: ReLU

Original: RL Batch Size
Revised: RL Batch Size

Original: Reward Scaling
Revised: Reward Scaling

Original: Replay Buffer Size 500 ,
Revised: Replay Buffer Size 500 ,

Original: Q Hidden Sizes (512 , 512)
Revised: Q Hidden Sizes (512 , 512)

Original: Q Hidden Activation
Revised: Q Hidden Activation

Original: ReLU
Revised: ReLU

Original: Q Weight Decay
Revised: Q Weight Decay

Original: Q Learning Rate 3 × 10 −
Revised: Q Learning Rate 3 × 10 −

Original: Target Network τ 5 × 10 −
Revised: Target Network τ 5 × 10 −

Original: BayCRL
Revised: BayCRL

Original: Adaptation batch size
Revised: Adaptation batch size

Original: Meta-training tasks per epoch
Revised: Meta-training tasks per epoch

Original: Meta-test set size
Revised: Meta-test set size

Original: VICE
Revised: VICE

Original: Classiﬁer Learning
Revised: Classiﬁer Learning

Original: Rate 1 × 10 −
Revised: Rate 1 × 10 −

Original: Classiﬁer Batch Size
Revised: Classiﬁer Batch Size

Original: Classiﬁer Optimizer
Revised: Classiﬁer Optimizer

Original: Adam
Revised: Adam

Original: RL Algorithm
Revised: RL Algorithm

Original: SAC
Revised: SAC

Original: RND
Revised: RND

Original: Hidden Layer Sizes (256 , 256)
Revised: Hidden Layer Sizes (256 , 256)

Original: [Table]
Revised: [Table]

Original: A.4. Z IGZAG M AZE H YPERPARAMETERS
Revised: A.5. Z IGZAG M AZE H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: A.4. S PIRAL M AZE H YPERPARAMETERS
Revised: A.5. S PIRAL M AZE H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: A.4. S AWYER P USH H YPERPARAMETERS
Revised: A.5. S AWYER P USH H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: A.4. S AWYER P ICK - AND -P LACE H YPERPARAMETERS
Revised: A.5. S AWYER P ICK - AND -P LACE H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: A.4. S AWYER D OOR O PENING H YPERPARAMETERS
Revised: A.5. S AWYER D OOR O PENING H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: A. P ROOF OF T HEOREM 1 CONNECTING NML AND INVERSE COUNTS
Revised: A. P ROOF OF T HEOREM 1 CONNECTING NML AND INVERSE COUNTS

Original: We provide the proof of Theorem 1 here for completeness
Revised: We provide the proof of Theorem 1 here for completeness.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: Suppose we are estimating success probabilities p ( e = 1 | s ) in the tabular setting, where we have a separate parameter independently for each state.
Revised: Suppose we are estimating success probabilities p ( e = 1 | s ) in the tabular setting, where we have a separate parameter independently for each state.

Original: Let N ( s ) denote the number of times state s has been visited by the policy, and let G ( s ) be the number of occurrences of state s in the successful outcomes.
Revised: Let N ( s ) denote the number of times state s has been visited by the policy, and let G ( s ) be the number of occurrences of state s in the successful outcomes.

Original: Then the CNML probability p CNML ( e = 1 | s ) is equal to
Revised: Then the CNML probability p CNML ( e = 1 | s ) is equal to

Original: G ( s )+1 N ( s )+ G ( s )+2 .
Revised: G ( s )+1 N ( s )+ G ( s )+2 .

Original: For states that are never observed to be successful, we then recover inverse counts [Equation] N [Equation]
Revised: For states that are never observed to be successful, we then recover inverse counts 1 N ( s )+2 .

Original: Proof.
Revised: Proof.

Original: In the fully tabular setting, our MLE estimates for p ( O | s ) are simply given by ﬁnding the best parameter p s for each state.
Revised: In the fully tabular setting, our MLE estimates for p ( O | s ) are simply given by ﬁnding the best parameter p s for each state.

Original: The proof then proceeds by simple calculation.
Revised: The proof then proceeds by simple calculation.

Original: For a state with n = N ( s ) negative occurrences and g = G ( s ) positive occurrences, the MLE estimate is simply given by gn + g .
Revised: For a state with n = N ( s ) negative occurrences and g = G ( s ) positive occurrences, the MLE estimate is simply given by gn + g .

Original: Now for evaluating CNML, we consider appending another instance for each class.
Revised: Now for evaluating CNML, we consider appending another instance for each class.

Original: The new parameter after appending a negative example is then g n + g +1 , which then assigns probability n +1 n + g + to the negative class.
Revised: The new parameter after appending a negative example is then g n + g +1 , which then assigns probability n +1 n + g + to the negative class.

Original: Similarly, after appending a positive example, the new parameter is g +1 n + g +1 , so we try to assign probability g +1 n + g +1 to the positive class.
Revised: Similarly, after appending a positive example, the new parameter is g +1 n + g +1 , so we try to assign probability g +1 n + g +1 to the positive class.

Original: Normalizing, we have
Revised: Normalizing, we have

Original: [Equation]
Revised: [Equation]

Original: When considering states that have only been visited on-policy, and are not included int the set of successful outcomes, then the likelihood reduces to
Revised: When considering states that have only been visited on-policy, and are not included in the set of successful outcomes, then the likelihood reduces to

Original: [Equation]
Revised: [Equation]