Original: I MITATION L EARNING FROM V ISUAL 
Revised: I MITATION L EARNING FROM V ISUAL D ATA

Original: D ATA

Original: WITH
Revised: WITH

Original: M ULTIPLE I NTENTIONS
Revised: M ULTIPLE I NTENTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs.
Revised: Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs.

Original: LfD algorithms generally assume learning from single task demonstrations.
Revised: LfD algorithms generally assume learning from single task demonstrations.

Original: In practice, however, it is more efﬁcient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering.
Revised: In practice, however, it is more efﬁcient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering.

Original: Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior.
Revised: Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data and often result in sub-optimal behavior.

Original: In this paper we present an LfD approach for learning multiple modes of behavior from visual data.
Revised: In this paper we present an LfD approach for learning multiple modes of behavior from visual data.

Original: Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network.
Revised: Our approach is based on a stochastic deep neural network (SNN) which represents the underlying intention in the demonstration as a stochastic activation in the network

Original: We present an efﬁcient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.
Revised: We present an efﬁcient algorithm for training SNNs , and for learning with vi sion inputs , we also propose an architecture that associates the intention with a stochastic attention module Furthermore

Original: We demonstrate our method on real robot visual object reaching tasks, and show that it can reliably learn the multiple behavior modes in the demonstration data.
Revised: , we demonstrate our method on real robot visual object reaching tasks and show that it can reliably learn the multiple behavior modes in the demonstration data.

Original: Video results are available at https://vimeo.com/240212286/fd401241b9 .
Revised: Video results are available at https://vimeo.com/240212286/fd401241b9 .

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: A key problem in robotic control is to simplify the problem of programming a complex behavior.
Revised: A key problem in robotic control is to simplify the problem of programming a complex behavior.

Original: Traditional control engineering approaches, which rely on accurate manual modeling of the system environment, are very challenging to apply in modern robotic applications where most sensory inputs come from images and other high-dimensional signals such as tactile feedback.
Revised: Traditional control engineering approaches, which rely on accurate manual modeling of the system environment, are very challenging to apply in modern robotic applications where most sensory inputs come from images and other high-dimensional signals such as tactile feedback.

Original: In contrast, imitation learning, or learning from demonstration (LfD) approaches (Schaal et al., 2003) aim to directly learn a control policy from mentor or expert demonstrations.
Revised: In contrast, imitation learning, or learning from demonstration (LfD) approaches (Schaal et al., 2003) aim to directly learn a control policy from mentor or expert demonstrations.

Original: The key advantages of LfD are simplicity and data-efﬁciency, and indeed, LfD has been successfully used for learning complex robot skills such as locomotion (Schaal et al., 2005), driving (Pomerleau, 1989;
Revised: The key advantages of LfD are simplicity and data-efﬁciency, and indeed, LfD has been successfully used for learning complex robot skills such as locomotion (Schaal et al., 2005), driving (Pomerleau, 1989;

Original: Ross et al., 2011), ﬂying (Abbeel & Ng, 2004), and manipulation (M ¨ ulling et al., 2013; Chebotar et al., 2016; Pastor et al., 2009).
Revised: Ross et al., 2011), ﬂying (Abbeel & Ng, 2004), and manipulation (M¨ulling et al., 2013; Chebotar et al., 2016; Pastor et al., 2009).

Original: Recently, advances in deep representation learning (Goodfellow et al., 2016) have facilitated LfD methods with high dimensional perception, such as mapping raw images directly to controls (Giusti et al., 2016).
Revised: Recently, advances in deep representation learning (Goodfellow et al., 2016) have facilitated LfD methods with high dimensional perception, such as mapping raw images directly to controls (Giusti et al., 2016).

Original: These advances are capable of learning generalizable skills (Levine et al., 2015), and offer a promising approach for modern industrial challenges such as pick and place tasks (Correll et al., 2016).
Revised: These advances are capable of learning generalizable skills (Levine et al., 2015), and offer a promising approach for modern industrial challenges such as pick and place tasks (Correll et al., 2016).

Original: One challenge in LfD, however, is learning different modes of the same task.
Revised: One challenge in LfD, however, is learning different modes of the same task.

Original: For example, consider learning to pick up an object from a pile.
Revised: For example, consider learning to pick up an object from a pile.

Original: The demonstrator can choose to pick up a different object each time, yet we expect LfD to understand that these are similar demonstrations of the same pickup skill, only with a different intention in mind .
Revised: The demonstrator can choose to pick up a different object each time, yet we expect LfD to understand that these are similar demonstrations of the same pickup skill, only with a different intention in mind .

Original: Moreover, we want the learned robot behavior to display a similar multi-modal 1 nature.
Revised: Moreover, we want the learned robot behavior to display a similar multi-modal 1 nature.

Original: Standard approaches for LfD with image inputs, such as learning with deep neural networks (NNs)
Revised: Standard approaches for LfD with image inputs, such as learning with deep neural networks (NNs)

Original: (Pomerleau, 1989; Giusti et al., 2016; Levine et al., 2015), are not suitable for learning multimodal behaviors.
Revised: (Pomerleau, 1989; Giusti et al., 2016; Levine et al., 2015), are not suitable for learning multimodal behaviors.

Original: In their essence, NNs learn a deterministic mapping from observation to control, which cannot represent the inherently multi-modal latent intention in the demonstrations.
Revised: In their essence, NNs learn a deterministic mapping from observation to control, which cannot represent the inherently multi-modal latent intention in the demonstrations.

Original: In prac- tice, this manifests as an ‘averaging’ of the different modes in the data (Bishop, 1994), leading to an undesirable policy.
Revised: In practice, this manifests as an ‘averaging’ of the different modes in the data (Bishop, 1994), leading to an undesirable policy.

Original: A straightforward approach for tackling the multi-modal problem in LfD is to add a label for each mode in the data.
Revised: A straightforward approach for tackling the multi-modal problem in LfD is to add a label for each mode in the data.

Original: Thus, in the pick-up task above, the demonstrator would also explicitly specify the object she intends to pick-up beforehand.
Revised: Thus, in the pick-up task above, the demonstrator would also explicitly specify the object she intends to pick-up beforehand.

Original: Such an approach has several practical shortcomings: it requires the demonstrator to record more data, and requires the possible intentions to be speciﬁed in advance, making it difﬁcult to use the same recorded data for different tasks.
Revised: Such an approach has several practical shortcomings: it requires the demonstrator to record more data, and requires the possible intentions to be speciﬁed in advance, making it difﬁcult to use the same recorded data for different tasks.

Original: More importantly, such a solution is conceptually ﬂawed – it solves an algorithmic challenge by placing additional burden on the client.
Revised: More importantly, such a solution is conceptually ﬂawed – it solves an algorithmic challenge by placing additional burden on the client.

Original: In this work, we propose an approach for LfD with multi-modal demonstrations that does not require any additional data labels.
Revised: In this work, we propose an approach for LfD with multi-modal demonstrations that does not require any additional data labels.

Original: Our method is based on a stochastic neural network model, which represents the latent intention as a random activation in the network.
Revised: Our method is based on a stochastic neural network model, which represents the latent intention as a random activation in the network.

Original: We propose a novel and efﬁcient learning algorithm for training stochastic networks, and present a network architecture suitable for
Revised: We propose a novel and efﬁcient learning algorithm for training stochastic networks, and present a network architecture suitable for

Original: LfD with raw image inputs, where the intention takes the form of a stochastic attention over features in the image.
Revised: LfD with raw image inputs, where the intention takes the form of a stochastic attention over features in the image.

Original: We show that our method can reliably reproduce behavior with multiple intentions in real-robot object reaching tasks.
Revised: We show that our method can reliably reproduce behavior with multiple intentions in real-robot object reaching tasks.

Original: Moreover, in scenarios where multiple intentions exist in the demonstration data, the stochastic neural networks perform better than their deterministic counterparts.
Revised: Moreover, in scenarios where multiple intentions exist in the demonstration data, the stochastic neural networks perform better than their deterministic counterparts.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: In this work we focus on a direct imitation learning approach, known as behavioral cloning (Pomerleau, 1989).
Revised: In this work we focus on a direct imitation learning approach, known as behavioral cloning (Pomerleau, 1989).

Original: This approach does not require any model of the task dynamics, and does not require additional queries of the mentor or robot execution rollouts beyond the collected demonstrations (though such can be used to improve performance (Ross et al., 2011)).
Revised: This approach does not require any model of the task dynamics, and does not require additional queries of the mentor or robot execution rollouts beyond the collected demonstrations (though such can be used to improve performance (Ross et al., 2011)).

Original: An alternative approach is inverse reinforcement learning (IRL) (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008), where a reward model that explains the demonstrated behavior is sought.
Revised: An alternative approach is inverse reinforcement learning (IRL) (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008), where a reward model that explains the demonstrated behavior is sought.

Original: Recently, model free IRL approaches that can learn complex behavior policies from high-dimensional data were proposed (Finn et al., 2016; Ho & Ermon, 2016).
Revised: Recently, modelfree IRL approaches that can learn complex behavior policies from high-dimensional data were proposed (Finn et al., 2016; Ho & Ermon, 2016).

Original: These approaches, however, rely on taking additional policy rollouts as a fundamental step of the method , which, in realistic robot applications, requires substantial resources.
Revised: These approaches, however, rely on taking additional policy rollouts as a fundamental step of the method , which, in realistic robot applications, requires substantial resources.

Original: Multi-task IRL learns from unlabeled demonstrations generated by varying intentions or objectives (Babes et al., 2011; Dimitrakakis & Rothkopf, 2012).
Revised: Multi-task IRL learns from unlabeled demonstrations generated by varying intentions or objectives (Babes et al., 2011; Dimitrakakis & Rothkopf, 2012).

Original: Dimitrakakis & Rothkopf (2012) propose a Bayesian approach for inferring the intention of an agent performing a series of tasks in a dynamic environment.
Revised: Dimitrakakis & Rothkopf (2012) propose a Bayesian approach for inferring the intention of an agent performing a series of tasks in a dynamic environment.

Original: Babes et al.
Revised: Babes et al.

Original: (2011) propose an EM for clustering the unlabeled demonstrations and then application of IRL for inferring the intention of a given cluster.
Revised: (2011) propose an EM for clustering the unlabeled demonstrations and then application of IRL for inferring the intention of a given cluster.

Original: Both approaches have been shown promising results on relatively simple low dimensional problems.
Revised: Both approaches have been shown promising results on relatively simple low dimensional problems.

Original: Several recent works on multi-task IRL (Hausman et al., 2017; Wang et al., 2017; Li et al., 2017) extended the generative adversarial imitation learning (GAIL) algorithm (Ho & Ermon, 2016) to high dimensional multi-modal demonstrations.
Revised: Several recent works on multi-task IRL (Hausman et al., 2017; Wang et al., 2017; Li et al., 2017) extended the generative adversarial imitation learning (GAIL) algorithm (Ho & Ermon, 2016) to high dimensional multi-modal demonstrations.

Original: Our approach, in comparison, does not require taking additional robot rollouts.
Revised: Our approach, in comparison, does not require taking additional robot rollouts.

Original: Recently, Rahmatizadeh et al.
Revised: Recently, Rahmatizadeh et al.

Original: (2017) proposed a method for learning multi-modal policies from raw visual inputs, using mixture density networks (Bishop, 1994) for generating outputs from a mixture of Gaussians distribution.
Revised: (2017) proposed a method for learning multi-modal policies from raw visual inputs, using mixture density networks (Bishop, 1994) for generating outputs from a mixture of Gaussians distribution.

Original: Their training method requires labeling each task with a speciﬁc signal.
Revised: Their training method requires labeling each task with a speciﬁc signal.

Original: While the modes, or intentions in our work can be seen as different tasks, we do not require any labeling of the intention in the demonstrations.
Revised: While the modes, or intentions in our work can be seen as different tasks, we do not require any labeling of the intention in the demonstrations.

Original: To our knowledge, this is the ﬁrst LfD approach that can handle multiple modes in the demonstrations and: (1) does not require additional robot rollouts, (2) does not require a label for the mode, and (3) can work with raw image inputs.
Revised: To our knowledge, this is the ﬁrst LfD approach that can handle multiple modes in the demonstrations and: (1) does not require additional robot rollouts, (2) does not require a label for the mode, and (3) can work with raw image inputs.

Original: The stochastic neural network model we use here is related to recently proposed generative models such as (conditional) variational autoencoders (VAEs) (Kingma & Welling, 2014;
Revised: The stochastic neural network model we use here is related to recently proposed generative models such as (conditional) variational autoencoders (VAEs) (Kingma & Welling, 2014;

Original: Sohn et al., 2015b), and generative adversarial nets (GANs) (Goodfellow et al., 2014; Mirza & Osindero, 2014).
Revised: Sohn et al., 2015), and generative adversarial nets (GANs) (Goodfellow et al., 2014; Mirza & Osindero, 2014).

Original: We opted for stochastic neural networks since GANs are known to have problems learning multimodal distributions (Arora & Zhang, 2017), and conditional VAEs (Pillai & Leonard, 2017) require training an additional encoding network, which proved to be difﬁcult in our experimental domain.
Revised: We opted for stochastic neural networks since GANs are known to have problems learning multi-modal distributions (Arora & Zhang, 2017), and conditional VAEs (Pillai & Leonard, 2017) require training an additional encoding network, which proved to be difﬁcult in our experimental domain.

Original:  PRELIMINARIES
Revised:  PRELIMINARIES

Original: We ﬁrst introduce some preliminary concepts for presenting our methods, and then present our problem formulation of imitation learning with multi-modal behaviors in expert demonstrations.
Revised: We ﬁrst introduce some preliminary concepts for presenting our methods, and then present our problem formulation of imitation learning with multi-modal behaviors in expert demonstrations.

Original: Learning from Demonstration To explicitly formulate the problem of imitation learning, let X
Revised: Learning from Demonstration To explicitly formulate the problem of imitation learning, let X

Original: and U denote the observation and action spaces of the robot, and let x t ∈ X and u t ∈ U denote an observation and a control command for the robot at time t .
Revised: and U denote the observation and action spaces of the robot, and let x t ∈ X and u t ∈ U denote an observation and a control command for the robot at time t .

Original: Given a data-set D of N trajectories T i with length T (for simplifying notations we drop the subscript i in T i ), where a demonstrated task is recorded in the form of sequential pairs of observations and actions T i = (cid:8) (cid:104) x
Revised: Given a data-set D of N trajectories T i with length T (for simplifying notations we drop the subscript i in T i ), where a demonstrated task is recorded in the form of sequential pairs of observations and actions T i = (cid:8) (cid:104) x

Original: i 1 , u i 1 (cid:105) , . .
Revised: i 1 , u i 1 (cid:105) , . .

Original: , (cid:104) x iT , u iT (cid:105) (cid:9) N i =1 , LfD aims to learn a policy P : X → P ( U ) that is parametrized by feature weight vector θ ∈ Θ , such that it reliably performs the task.
Revised: , (cid:104) x iT , u iT (cid:105) (cid:9) N i =1 , LfD aims to learn a policy P : X → P ( U ) that is parametrized by feature weight vector θ ∈ Θ , such that it reliably performs the task.

Original: Here P ( U ) represents the space of probability distributions deﬁned on the action space U .
Revised: Here P ( U ) represents the space of probability distributions deﬁned on the action space U .

Original: Since each observation is associated with an action label, the imitation learning policy can be found by solving the maximum-likelihood (ML) objective: θ ∗ ∈ arg max θ ∈ Θ 1 N (cid:80) Ni =1 log P ( u i 1: T | x i 1: T , θ ) , where we abbreviated the sequence of actions as u 1: T .
Revised: Since each observation is associated with an action label, the imitation learning policy can be found by solving the maximum-likelihood (ML) objective: θ ∗ ∈ arg max θ ∈ Θ 1 N (cid:80) Ni =1 log P ( u i 1: T | x i 1: T , θ ) , where we abbreviated the sequence of actions as u 1: T .

Original: = u 1 , . . .
Revised: = u 1 , . . .

Original: , u T , and the sequence of observations as x 1: T .
Revised: , u T , and the sequence of observations as x 1: T .

Original: = x 1 , . . .
Revised: = x 1 , . . .

Original: , x T .
Revised: , x T .

Original: This objective function is the empirical average of the conditional log likelihood, which is a consistent estimator of the expected conditional log likelihood: E [log P ( u 1: T | x 1: T , θ )] .
Revised: This objective function is the empirical average of the conditional log likelihood, which is a consistent estimator of the expected conditional log likelihood: E [log P ( u 1: T | x 1: T , θ )] .

Original: If, for example, the policy is Gaussian with parametrized mean vector f ( x ; θ ) and an identity co-variance matrix, then the above supervised learning problem is equivalent to an (cid:96) 2 regression problem with objective function 1 N (cid:80) Ni =1 (cid:80) 
Revised: If, for example, the policy is Gaussian with parametrized mean vector f ( x ; θ ) and an identity co-variance matrix, then the above supervised learning problem is equivalent to an (cid:96) 2 regression problem with objective function 1 N (cid:80) Ni =1 (cid:80) [Equation]

Original: Tt =1 (cid:107) u it − f ( x it ; θ ) (cid:107) 22 .
Revised: = [Equation]

Original: Stochastic Neural Networks Multilayer perceptrons (MLPs) are general purpose function approximators for nonlinear regression in feedforward neural networks (NNs)
Revised: Stochastic Neural Networks Multilayer perceptrons (MLPs) are general purpose function approximators for nonlinear regression in feedforward neural networks (NNs)

Original: (Goodfellow et al., 2016).
Revised: (Goodfellow et al., 2016).

Original: Parametrized by the NN weights θ , the output of the MLP, f ( x ; θ ) , is often interpreted as the sufﬁcient statistics of the conditional probability P ( u | x ; θ ) , if the conditional probability belongs to the exponential family (conditioned on the input x ).
Revised: Parametrized by the NN weights θ , the output of the MLP, f ( x ; θ ) , is often interpreted as the sufﬁcient statistics of the conditional probability P ( u | x ; θ ) , if the conditional probability belongs to the exponential family (conditioned on the input x ).

Original: For example, if P ( u | x ; θ ) is parametrized as an isotropic Gaussian distribution, it can be represented by N ( u | f ( x ; θ ) , I ) .
Revised: For example, if P ( u | x ; θ ) is parametrized as an isotropic Gaussian distribution, it can be represented by N ( u | f ( x ; θ ) , I ) .

Original: The parameters θ are typically learned by maximizing the expected log likelihood function.
Revised: The parameters θ are typically learned by maximizing the expected log likelihood function.

Original: However, since the MLP activation functions are all deterministic, by nature the model P ( u | x ; θ ) is a unimodal distribution.
Revised: However, since the MLP activation functions are all deterministic, by nature the model P ( u | x ; θ ) is a unimodal distribution.

Original: For many structured prediction problems, we are interested in a conditional distribution that is multimodal.
Revised: For many structured prediction problems, we are interested in a conditional distribution that is multimodal.

Original: To satisfy the multi-modality requirement, a common approach is to make the hidden variables in the NN stochastic.
Revised: To satisfy the multi-modality requirement, a common approach is to make the hidden variables in the NN stochastic.

Original: Sigmoid belief nets (SBNs) (Neal, 1992) are an early example of this idea, using binary stochastic hidden variables.
Revised: Sigmoid belief nets (SBNs) (Neal, 1992) are an early example of this idea, using binary stochastic hidden variables.

Original: However, inference in SBNs is generally intractable, and costly to compute in practice.
Revised: However, inference in SBNs is generally intractable, and costly to compute in practice.

Original: Recently, Tang & Salakhutdinov (2013) introduced the stochastic feedforward neural network (SNN) for modeling the multi-modal conditional distribution P ( u | x ; θ ) .
Revised: Recently, Tang & Salakhutdinov (2013) introduced the stochastic feedforward neural network (SNN) for modeling the multi-modal conditional distribution P ( u | x ; θ ) .

Original: Unlike SBNs, SNNs add to the deterministic NN latent features a stochastic latent variable z , and decompose the conditional distribution as: P ( u | x ; θ )
Revised: Unlike SBNs, SNNs add to the deterministic NN latent features a stochastic latent variable z , and decompose the conditional distribution as: P ( u | x ; θ )

Original: = (cid:80) z P ( u | x, z ; θ ) P ( z ) .
Revised: = (cid:80) z P ( u | x, z ; θ ) P ( z ) .

Original: It is also assumed that
Revised: It is also assumed that

Original: P ( u | x, z, θ ) and P ( z ) can be easily computed, for example, as in (Tang & Salakhutdinov, 2013), where z is represented by Bernoulli random variable nodes in the network.
Revised: P ( u | x, z, θ ) and P ( z ) can be easily computed, for example, as in (Tang & Salakhutdinov, 2013), where z is represented by Bernoulli random variable nodes in the network.

Original: For learning the parameters θ , Tang & Salakhutdinov (2013) proposed a generalized EM algorithm, where importance sampling is used in the E-step, and error back-propagation is used in the M-step.
Revised: For learning the parameters θ , Tang & Salakhutdinov (2013) proposed a generalized EM algorithm, where importance sampling is used in the E-step, and error back-propagation is used in the M-step.

Original: Problem Formulation
Revised: Problem Formulation

Original: In this work, we consider an imitation learning setting, where the mentor demonstrations of a particular task consist of multiple behaviors.
Revised: In this work, we consider an imitation learning setting, where the mentor demonstrations of a particular task consist of multiple behaviors.

Original: In particular, we assume that the demonstrator can perform the task in several different strategies, which we term as intentions .
Revised: In particular, we assume that the demonstrator can perform the task in several different strategies, which we term as intentions .

Original: As a concrete example, consider the task of picking up an object from a pile of different objects in the scene (see Figure 1(a) for an example).
Revised: As a concrete example, consider the task of picking up an object from a pile of different objects in the scene (see Figure 1(a) for an example).

Original: In this case, the data-set of mentor demonstrations consists of a list of trajectories, where the target object in each trajectory is inherently decided by the demonstrator, and not explicitly labeled.
Revised: In this case, the data-set of mentor demonstrations consists of a list of trajectories, where the target object in each trajectory is inherently decided by the demonstrator, and not explicitly labeled.

Original: Our goal is to learn a stochastic policy that accurately mimics the mentor’s policy, and accurately displays the multiple intentions demonstrated in the data.
Revised: Our goal is to learn a stochastic policy that accurately mimics the mentor’s policy, and accurately displays the multiple intentions demonstrated in the data.

Original:  MULTI MODAL L F D WITH SNN S
Revised:  MULTI MODAL L F D WITH SNN S

Original: In this section, we ﬁrst present our LfD formulation based on the SNN model for learning multiple intentions, and then propose a sampling based algorithm for learning the SNN parameters.
Revised: In this section, we ﬁrst present our LfD formulation based on the SNN model for learning multiple intentions, and then propose a sampling based algorithm for learning the SNN parameters.

Original: We then present a particular SNN architecture that is suitable for vision-based inputs, where the stochastic intention takes the form of an attention over image features.
Revised: We then present a particular SNN architecture that is suitable for vision-based inputs, where the stochastic intention takes the form of an attention over image features.

Original: 4. T HE SNN F ORMULATION
Revised: 4. T HE SNN F ORMULATION

Original: We model the intention of the demonstrator using a random vector z ∈ R M with probability distribution P ( z ) .
Revised: We model the intention of the demonstrator using a random vector z ∈ R M with probability distribution P ( z ) .

Original: For example, P ( z ) could be a unit normal distribution N (0 , I ) , or a vector of independent multinomial probabilities.
Revised: For example, P ( z ) could be a unit normal distribution N (0 , I ) , or a vector of independent multinomial probabilities.

Original: Here we assume that throughout a single trajectory, the intention does not change, and the intention is independent of the observations 2 .
Revised: Here we assume that throughout a single trajectory, the intention does not change, and the intention is independent of the observations 2 .

Original: Therefore, the conditional data likelihood is obtained by marginalizing out the random variable of intention:
Revised: Therefore, the conditional data likelihood is obtained by marginalizing out the random variable of intention:

Original: P ( u 1: T | x 1: T ; θ ) = (cid:80) z
Revised: P ( u 1: T | x 1: T ; θ ) = (cid:80) z

Original: P ( u 1: T | x 1: T , z ; θ ) P ( z ) .
Revised: P ( u 1: T | x 1: T , z ; θ ) P ( z ) .

Original: SNNs can be viewed as directed graphical models (see Figure 4 in the appendix for a diagram) where at each time t ∈ { 1 , . .
Revised: SNNs can be viewed as directed graphical models (see Figure 4 in the appendix for a diagram) where at each time t ∈ { 1 , . .

Original: , T } , the generative process starts from an observation x t , combines with a latent intention z , which is the same throughout the trajectory, and then generates an action u t .
Revised: , T } , the generative process starts from an observation x t , combines with a latent intention z , which is the same throughout the trajectory, and then generates an action u t .

Original: We also make the standard assumption that, given the intention z at each trajectory, the demonstrator policy is memory-less (a.k.a.
Revised: We also make the standard assumption that, given the intention z at each trajectory, the demonstrator policy is memory-less (a.k.a.

Original: Markov), which implies the following equality: P ( u 1: T | x 1: T , z ; θ ) = (cid:81)
Revised: Markov), which implies the following equality: P ( u 1: T | x 1: T , z ; θ ) = (cid:81)

Original: Tt =1 P ( u t | x t , z ; θ ) .
Revised: Tt =1 P ( u t | x t , z ; θ ) .

Original: Given an intention z , we model the action probability as log P ( u | z, x ; θ ) ∝ − d ( f ( x, z ; θ ) , u ) , where f is a deterministic NN that takes as input both the observation x and the intention z , and d is some distance metric.
Revised: Given an intention z , we model the action probability as log P ( u | z, x ; θ ) ∝ − d ( f ( x, z ; θ ) , u ) , where f is a deterministic NN that takes as input both the observation x and the intention z , and d is some distance metric.

Original: One immediate example is when d ( a, b ) = (cid:107) a − b (cid:107) 2 , one obtains P ( u | z, x ; θ ) as a normal distribution N ( u | f ( x, z ; θ ) , σ 2 ) for some MLP mean predictor f ( x, z ; θ ) and constant variance term σ 2 .
Revised: One immediate example is when d ( a, b ) = (cid:107) a − b (cid:107) 2 , one obtains P ( u | z, x ; θ ) as a normal distribution N ( u | f ( x, z ; θ ) , σ 2 ) for some MLP mean predictor f ( x, z ; θ ) and constant variance term σ 2 .

Original: In our experiments, we found the (cid:96) 1 distance function to work well.
Revised: In our experiments, we found the (cid:96) 1 distance function to work well.

Original: Note that when the intention variable z is ﬁxed, the output action follows a unimodal distribution.
Revised: Note that when the intention variable z is ﬁxed, the output action follows a unimodal distribution.

Original: However, since z is a random vector that is input to a nonlinear NN computation, the distribution of f ( x, z ; θ ) , and thereby the output distribution P ( u 1: T | x 1: T ; θ ) , can take a multi-modal form.
Revised: However, since z is a random vector that is input to a nonlinear NN computation, the distribution of f ( x, z ; θ ) , and thereby the output distribution P ( u 1: T | x 1: T ; θ ) , can take a multi-modal form.

Original: 4. T HE M ONTE C ARLO L EARNING A LGORITHM
Revised: 4. T HE M ONTE C ARLO L EARNING A LGORITHM

Original: We ﬁrst describe a basic Monte Carlo (MC) sampling algorithm for learning the parameters θ ∈ Θ of the SNN model in Section 4.1.
Revised: We ﬁrst describe a basic Monte Carlo (MC) sampling algorithm for learning the parameters θ ∈ Θ of the SNN model in Section 4.1.

Original: Let z 1 , . .
Revised: Let z 1 , . .

Original: , z N denote N samples of z , where z i ∼ P ( z ) .
Revised: , z N denote N samples of z , where z i ∼ P ( z ) .

Original: For each given parameter θ , sequence of observations x 1: T , and sequence of actions u 1: T , let r ( z ; x 1: T , u 1: T , θ ) =
Revised: For each given parameter θ , sequence of observations x 1: T , and sequence of actions u 1: T , let r ( z ; x 1: T , u 1: T , θ ) =

Original: P ( u 1: T | x 1: T , z ; θ ) be the reward function , which associates an intention with the data likelihood given it.
Revised: P ( u 1: T | x 1: T , z ; θ ) be the reward function , which associates an intention with the data likelihood given it.

Original: The reason we use this terminology is to later connect the likelihood maximization problem with risk-sensitive optimization concepts that will be key in our approach.
Revised: The reason we use this terminology is to later connect the likelihood maximization problem with risk-sensitive optimization concepts that will be key in our approach.

Original: A
Revised: A

Original: Monte Carlo approximation of the likelihood is given by,
Revised: Monte Carlo approximation of the likelihood is given by,

Original: [Equation]
Revised: [Equation]

Original: A direct approach for computing θ would be to directly maximize 1 N (cid:80) Ni 
Revised: A direct approach for computing θ would be to directly maximize 1 N (cid:80) Ni =1 log r ( z i

Original: [Equation]
Revised: [Equation]

Original: r
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: T , u

Original: [Equation]
Revised: [Equation]

Original: θ )
Revised: T

Original: [Equation]
Revised: [Equation]

Original:  which is a MC estimate of E z ∼ P [log r ( z ; x 1: T , u 1: T , θ )] , with gradient-based optimization.
Revised: [Equation] which is a MC estimate of E z ∼ P [log r ( z ; x 1: T , u 1: T , θ )] , with gradient-based optimization.

Original: By
Revised: By

Original: Jensen’s inequality, the above term is a lower bound of the data log likelihood log P ( u 1: T | x 1: T ; θ ) , corresponding to a maximum-likelihood approach.
Revised: Jensen’s inequality, the above term is a lower bound of the data log likelihood log P ( u 1: T | x 1: T ; θ ) , corresponding to a maximum-likelihood approach.

Original: While the estimator of the gradient is unbiased and consistent to E z ∼ P [ ∇ θ log r
Revised: While the estimator of the gradient is unbiased and consistent to E z ∼ P [ ∇ θ log r

Original: ( z ; x 1: T , u 1: T , θ )] , in practice, such an approach suffers from extremely high variance (with respect to intention z ).
Revised: ( z ; x 1: T , u 1: T , θ )] , in practice, such an approach suffers from extremely high variance (with respect to intention z ).

Original: To justify this observation, consider the sum of probabilities in (1).
Revised: To justify this observation, consider the sum of probabilities in (1).

Original: Since each sampled intention z i is given an equal weight in explaining the observed sequence of actions, even sampled intentions that are very different from the ones that generated the data (i.e., have high cost) are expected to produce a high likelihood.
Revised: Since each sampled intention z i is given an equal weight in explaining the observed sequence of actions, even sampled intentions that are very different from the ones that generated the data (i.e., have high cost) are expected to produce a high likelihood.

Original: 4. I NTENTION - DRIVEN L EARNING IN SNN S
Revised: 4. I NTENTION - DRIVEN L EARNING IN SNN S

Original: To reduce variance in training SNNs, in this section we introduce a sampling strategy whose gradient updates focuses only on the most correct underlying intentions – the intentions that have the lowest cost.
Revised: To reduce variance in training SNNs, in this section we introduce a sampling strategy whose gradient updates focuses only on the most correct underlying intentions – the intentions that have the highest reward.

Original: We analyze the bias and variance trade-offs of this new sampling gradient estimate, and show that the variance of our proposed approximation is lower than that of a naive MC approach.
Revised: We analyze the bias and variance trade-offs of this new sampling gradient estimate, and show that the variance of our proposed approximation is lower than that of a naive MC approach.

Original: For any given threshold α ∈ [0 , 1] , let q α ( θ ) denote the (upper)
Revised: For any given threshold α ∈ [0 , 1] , let q α ( θ ) denote the (upper)

Original: α − quantile of the reward function q α ( θ ) .
Revised: α − quantile of the reward function q α ( θ ) .

Original: = max w (cid:110)(cid:80) z : r ( z ; x 1: T ,u 1: T ,θ ) ≥ w P ( z ) ≤ α (cid:111) .
Revised: = max w (cid:110)(cid:80) z : r ( z ; x 1: T ,u 1: T ,θ ) ≥ w P ( z ) ≤ α (cid:111) .

Original: This associates with the α − quantile of the underlying intention with highest likelihood probability.
Revised: This associates with the α − quantile of the underlying intention with highest likelihood probability.

Original: We deﬁne an intention-driven probability distribution as follows:
Revised: We deﬁne an intention-driven probability distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: This quantity can be interpreted as a weighted distribution that only samples from the α % of the most correct underlying intentions (i.e., the intentions that best explain the mentor demonstrations).
Revised: This quantity can be interpreted as a weighted distribution that only samples from the α % of the most correct underlying intentions (i.e., the intentions that best explain the mentor demonstrations).

Original: The expected reward induced by the intention-driven distribution is given by:
Revised: The expected reward induced by the intention-driven distribution is given by:

Original: [Equation]
Revised: [Equation]

Original: which is equal to the conditional likelihood function of the α % most correct intentions.
Revised: which is equal to the conditional likelihood function of the α % most correct intentions.

Original: In the ﬁnancial risk literature, this metric is known as the expected shortfall (Rockafellar & Uryasev, 2000), and is typically used to evaluate the risky tail distribution of ﬁnancial assets.
Revised: In the ﬁnancial risk literature, this metric is known as the expected shortfall (Rockafellar & Uryasev, 2000), and is typically used to evaluate the risky tail distribution of ﬁnancial assets.

Original: While our setting is completely different, we will use tools developed for expected shortfall estimation in our approach.
Revised: While our setting is completely different, we will use tools developed for expected shortfall estimation in our approach.

Original: By deﬁnition of Q α , one has the following inequality.
Revised: By deﬁnition of Q α , one has the following inequality.

Original: [Equation]
Revised: [Equation]

Original: We propose to maximize E z ∼ Q
Revised: We propose to maximize E z ∼ Q

Original: α
Revised: α

Original: [ r ( z ; x 1: T , u 1: T , θ )] using Monte Carlo sampling techniques.
Revised: [ r ( z ; x 1: T , u 1: T , θ )] using Monte Carlo sampling techniques.

Original: In- tuitively, since the support of Q α is limited to the most likely z values, estimating it using sampling has lower variance than estimating the original likelihood.
Revised: In- tuitively, since the support of Q α is limited to the most likely z values, estimating it using sampling has lower variance than estimating the original likelihood.

Original: We will further elaborate on this point technically later in the section.
Revised: We will further elaborate on this point technically later in the section.

Original: However, this comes at the cost of adding a bias(cid:80) z [Equation] u [Equation]
Revised: However, this comes at the cost of adding a bias

Original: | x

Original: [Equation]
Revised: [Equation]

Original: θ ) P ( z ) .

Original: Empirically, we have found this procedure to work well.
Revised: Empirically, we have found this procedure to work well.

Original: To sample from Q α , we use empirical quantile estimation (Glynn, 1996).
Revised: To sample from Q α , we use empirical quantile estimation (Glynn, 1996).

Original: Let z ord 1 , . . .
Revised: Let z ord 1 , . . .

Original: , z ordN denote the MC samples z 1 , . . .
Revised: , z ordN denote the MC samples z 1 , . . .

Original: , z N sorted in descending order, according to the reward function r ( z ; x 1: T , u 1: T , θ ) .
Revised: , z N sorted in descending order, according to the reward function r ( z ; x 1: T , u 1: T , θ ) .

Original: Let N α = (cid:98) αN (cid:99) be the number of samples corresponding to the α − quantile.
Revised: Let N α = (cid:98) αN (cid:99) be the number of samples corresponding to the α − quantile.

Original: Then we have the following empirical estimate: E z ∼ Q
Revised: Then we have the following empirical estimate: E z ∼ Q

Original: α [ r ( z ; x 1: T , u 1: T , θ )]
Revised: α [ r ( z ; x 1: T , u 1: T , θ )]

Original: ≈
Revised: ≈

Original: N α (cid:80) N α
Revised: N α (cid:80) N α

Original: i =1 r ( z ordi ; x 1: T , u 1: T , θ ) .
Revised: i =1 r ( z ordi ; x 1: T , u 1: T , θ ) .

Original: It has been shown in Theorem 1 of Glynn (1996) that under standard assumptions, the above expression is a consistent estimator of E z ∼ Q
Revised: It has been shown in Theorem 1 of Glynn (1996) that under standard assumptions, the above expression is a consistent estimator of E z ∼ Q

Original: α [ r ( z ; x 1: T , u 1: T , θ )] with order O ( N − 1 / 2 ) , which we have shown above to be a lower bound to the likelihood function.
Revised: α [ r ( z ; x 1: T , u 1: T , θ )] with order O ( N − 1 / 2 ) , which we have shown above to be a lower bound to the likelihood function.

Original: For the special case of N α = 1 (when α = 1 /N ), we can replace the sorting operation with a simple min operation, yielding a simple and intuitive algorithm – we choose the sampled z with the lowest error for updating the parameters θ .
Revised: For the special case of N α = 1 (when α = 1 /N ), we can replace the sorting operation with a simple min operation, yielding a simple and intuitive algorithm – we choose the sampled z with the lowest error for updating the parameters θ .

Original: In practice, maximizing the log-likelihood of the data is known to work well (Goodfellow et al., 2016).
Revised: In practice, maximizing the log-likelihood of the data is known to work well (Goodfellow et al., 2016).

Original: In our case, we correspondingly maximize E z ∼ Q α [log r ( z ; x 1: T , u 1: T , θ )] , which, by the
Revised: In our case, we correspondingly maximize E z ∼ Q α [log r ( z ; x 1: T , u 1: T , θ )] , which, by the

Original: Jensen inequality 3 , is a lower bound on log E z ∼ Q
Revised: Jensen inequality 3 , is a lower bound on log E z ∼ Q

Original: α [ r ( z ; x 1: T , u 1: T , θ )] .
Revised: α [ r ( z ; x 1: T , u 1: T , θ )] .

Original: We therefore obtain the following gradient estimate G N,α ( θ ) :
Revised: We therefore obtain the following gradient estimate G N,α ( θ ) :

Original: = 1 N α (cid:80) N α
Revised: = 1 N α (cid:80) N α

Original: i =1 ∇ θ log r
Revised: i =1 ∇ θ log r

Original: ( z ordi ; x 1: T , u 1: T , θ ) .
Revised: ( z ordi ; x 1: T , u 1: T , θ ) .

Original: We term this sampling technique as intention-driven sampling (IDS).
Revised: We term this sampling technique as intention-driven sampling (IDS).

Original: Pseudocode is given in Algorithm 1.
Revised: Pseudocode is given in Algorithm 1.

Original: Optimistic Sampling:
Revised: Optimistic Sampling:

Original: To predict actions at test time , we ﬁrst sample z in the beginning of the episode and ﬁx it, and then use the NN to predict u t = f
Revised: To predict actions at test time , we ﬁrst sample z in the beginning of the episode and ﬁx it, and then use the NN to predict u t = f

Original: ( x t , z ) at every time step.
Revised: ( x t , z ) at every time step.

Original: While we could sample z from P ( z ) , this might result in a z value that has a low likelihood to reproduce the demonstrations, i.e., a z that has low reward.
Revised: While we could sample z from P ( z ) , this might result in a z value that has a low likelihood to reproduce the demonstrations, i.e., a z that has low reward.

Original: We observed this to be problematic in practice, and therefore devise an alternative approach, which we term optimistic sampling .
Revised: We observed this to be problematic in practice, and therefore devise an alternative approach, which we term optimistic sampling .

Original: We propose to store a set of the K most recent z values that obtained the highest reward during training, and at inference time sample a z uniformly from this set.
Revised: We propose to store a set of the K most recent z values that obtained the highest reward during training, and at inference time sample a z uniformly from this set.

Original: This corresponds to sampling z from Q α ( z ) , averaged over the training data.
Revised: This corresponds to sampling z from Q α ( z ) , averaged over the training data.

Original: Optimistic sampling dramatically improves the prediction performance in practice.
Revised: Optimistic sampling dramatically improves the prediction performance in practice.

Original: Analysis: By Theorem 4.2 of Hong & Liu (2009), G N,α ( θ ) is a consistent gradient estimator of the lower bound with asymptotic bias of O ( N − 1 / 2 ) .
Revised: Analysis: By Theorem 4.2 of Hong & Liu (2009), G N,α ( θ ) is a consistent gradient estimator of the lower bound with asymptotic bias of O ( N − 1 / 2 ) .

Original: In Appendix B, we deduce the following expression for the variance of G N,α ( θ ) :
Revised: In Appendix B, we deduce the following expression for the variance of G N,α ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: When N α = N , i.e., α = 1 , we obtain the variance of standard MC sampling , i.e., V
Revised: When N α = N , i.e., α = 1 , we obtain the variance of standard MC sampling , i.e., V

Original: MC = 1 N Var ( ∇ θ log r
Revised: MC = 1 N Var ( ∇ θ log r

Original: ( z ; x 1: T , u 1: T , θ )) .
Revised: ( z ; x 1: T , u 1: T , θ )) .

Original: On the other hand, when α → 1 /N the variance is bounded in O (1 /N 2 ) .
Revised: On the other hand, when α → 1 /N the variance is bounded in O (1 /N 2 ) .

Original: This result is due to the fact that (i) |∇ θ (log r ( z 1 ; x 1: T , u 1: T , θ ) −
Revised: This result is due to the fact that (i) |∇ θ (log r ( z 1 ; x 1: T , u 1: T , θ ) −

Original: q α ( θ )) | 1 { log r ( z 1 ; x 1: T , u 1: T , θ ) ≥ q α ( θ ) } is bounded by a constant that is O ( α ) , 5 (ii)
Revised: q α ( θ )) | 1 { log r ( z 1 ; x 1: T , u 1: T , θ ) ≥ q α ( θ ) } is bounded by a constant that is O ( α ) , 5 (ii)

Original: Var ( 1 { log r ( z 1 ; x 1: T , u 1: T , θ ) ≥
Revised: Var ( 1 { log r ( z 1 ; x 1: T , u 1: T , θ ) ≥

Original: q α ( θ ) } )
Revised: q α ( θ ) } )

Original: =
Revised: =

Original: α · (1 − α ) , and (iii) therefore V IDS
Revised: α · (1 − α ) , and (iii) therefore V IDS

Original: = O ( α · (1 − α ) / ( α 2 N ) · α 2 ) =
Revised: = O ( α · (1 − α ) / ( α 2 N ) · α 2 ) =

Original: O ( N − 2 ) .
Revised: O ( N − 2 ) .

Original: Therefore, one can treat α as a nob to trade-off bias (see (3)) and variance (see (4)) in IDS.
Revised: Therefore, one can treat α as a nob to trade-off bias (see (3)) and variance (see (4)) in IDS.

Original: Algorithm trajectory T i 1 Sample z1 , . . .
Revised: Algorithm 1: IDS z

Original:  , z N ∼ P (z ) 2 Set z ∗ =
Revised: . , z N ∼ P ( ) 2 Set z ∗ =

Original: arg max z i P ( u t : t + K | x t : t + K , z i ; θ ) 3 return ∇ θ log P ( u t : t + K| x t : t + K ,z ∗; θ ) , and z ∗
Revised: arg max z i P ( u t : t + K | x t : t + K , z i ; θ ) 3 return ∇ θ log P ( u t : t + K x t : t + K , ∗ ∗

Original: Comparison to SNNs: Broadly speaking, generalized EM algorithms (Tang & Salakhutdinov, 2013) can be seen as designing an importance sampling weight to reshape the sampling distribution, in order to lower the variance of the gradient estimate, by using entropy maximization or posterior distribution matching (more details can be found in Appendix A).
Revised: Comparison to SNNs: Broadly speaking, generalized EM algorithms (Tang & Salakhutdinov, 2013) can be seen as designing an importance sampling weight to reshape the sampling distribution, in order to lower the variance of the gradient estimate, by using entropy maximization or posterior distribution matching (more details can be found in Appendix A).

Original: For the speciﬁc
Revised: For the speciﬁc

Original: SNN algorithm of Tang & Salakhutdinov (2013) applied to our NN architecture, the importance weights correspond to a soft- max over the reward deﬁned above.
Revised: SNN algorithm of Tang & Salakhutdinov (2013) applied to our NN architecture, the importance weights correspond to a soft- max over the reward deﬁned above.

Original: In IDS the importance weight is w ( z ) = 

Revised: In IDS the importance weight is w ( z ) =

Original: α 1 { r ( z ; x 1: T , u 1: T , θ ) ≥
Revised: α 1 { r ( z ; x 1: T , u 1: T , θ ) ≥

Original: q α ( θ ) } , which for N α = 1 amounts to replacing the soft- max with a hard max .
Revised: q α ( θ ) } , which for N α = 1 amounts to replacing the soft- max with a hard max .

Original: Interestingly, these two similar algorithms were developed from very different ﬁrst principles.
Revised: Interestingly, these two similar algorithms were developed from very different ﬁrst principles.

Original: In practice, however, the IDS algorithm integrates naturally with optimistic sampling, which leads to signiﬁcantly better performance, as we show in our experiments.
Revised: In practice, however, the IDS algorithm integrates naturally with optimistic sampling, which leads to signiﬁcantly better performance, as we show in our experiments.

Original: 4. I NTENTION DRIVEN SNN A RCHITECTURE FOR S TOCHASTIC V ISUAL A TTENTION
Revised: 4. I NTENTION DRIVEN SNN A RCHITECTURE FOR S TOCHASTIC V ISUAL A TTENTION

Original: In this section we present Intention-SNN (henceforth I-SNN), an architecture that implements the stochastic intention as an attention over particular features in the image.
Revised: In this section we present Intention-SNN (henceforth I-SNN), an architecture that implements the stochastic intention as an attention over particular features in the image.

Original: This architecture is suitable for LfD domains where the visual observation contains information about multiple possible intentions, as in the object pick up task in Section 5.
Revised: This architecture is suitable for LfD domains where the visual observation contains information about multiple possible intentions, as in the object pick up task in Section 5.

Original: Our I-SNN architecture is presented in Figure 1, and is comprised of three modules.
Revised: Our I-SNN architecture is presented in Figure 1, and is comprised of three modules.

Original: The ﬁrst module is a standard multi-layer (fully) convolutional neural network (CNN) feature extractor (Goodfellow et al., 2016), followed by a spatial softmax layer.
Revised: The ﬁrst module is a standard multi-layer (fully) convolutional neural network (CNN) feature extractor (Goodfellow et al., 2016), followed by a spatial softmax layer.

Original: The CNN maps the input image onto C feature maps.
Revised: The CNN maps the input image onto C feature maps.

Original: The spatial softmax, introduced by Levine et al. (2015), calculates for each feature map in its input the corresponding ( x, y ) position in the image where this feature is most active.
Revised: The spatial softmax, introduced by Levine et al. (2015), calculates for each feature map in its input the corresponding ( x, y ) position in the image where this feature is most active.

Original: Let φ c,i,j denote the activation of feature map c at coordinate ( i, j ) .
Revised: Let φ c,i,j denote the activation of feature map c at coordinate ( i, j ) .

Original: The spatial softmax output for that feature is ( f c,x , f c,y ) , where f c,x = (cid:80) i,j exp( φ c,i,j ) · i/ (cid:80) i (cid:48) ,j (cid:48) exp( φ c, [Equation] (cid:48)  , [Equation]
Revised: The spatial softmax output for that feature is ( f c,x , f c,y ) , where f c,x = (cid:80) i,j exp( φ c,i,j ) · i/ (cid:80) i (cid:48) ,j (cid:48) exp( φ c,i (cid:48) ,j (cid:48) ) , and f c,y =

Original: (cid:48) ) , and f

Original: [Equation]

Original: ,y = (cid:80) i,j exp( φ c, [Equation] ,j ) · j/ (cid:80) i (cid:48) ,j (cid:48) exp( φ c,i (cid:48) ,j (cid:48) ) .
Revised: (cid:80) i,j exp( φ c,i,j ) · j/ (cid:80) i (cid:48) ,j (cid:48) exp( φ c,i (cid:48) ,j (cid:48) ) .

Original: Thus, the output of the spatial softmax is of dimensions C × 2 .
Revised: Thus, the output of the spatial softmax is of dimensions C × 2 .

Original: The second module applies a stochastic soft attention over the spatial softmax features.
Revised: The second module applies a stochastic soft attention over the spatial softmax features.

Original: We use a
Revised: We use a

Original: MLP to map the M -dimensional random intention vector z onto a C -dimensional vector w .
Revised: MLP to map the M -dimensional random intention vector z onto a C -dimensional vector w .

Original: We then apply a softmax to obtain the attention weight vector a ∈ R C , a c = exp( w c ) / (cid:80) c (cid:48)
Revised: We then apply a softmax to obtain the attention weight vector a ∈ R C , a c = exp( w c ) / (cid:80) c (cid:48)

Original: exp( w c (cid:48) )
Revised: exp( w c (cid:48) )

Original: (Xu et al., 2015).
Revised: (Xu et al., 2015).

Original: The attention weight is multiplied with the spatial softmax output to obtain the attention-modulated feature activations: f (cid:48)
Revised: The attention weight is multiplied with the spatial softmax output to obtain the attention-modulated feature activations: f (cid:48)

Original: c,x = f c,x · a c , and f (cid:48) c,y =
Revised: c,x = f c,x · a c , and f (cid:48) c,y =

Original: f c,y · a c , which are input to the control network along with the robot’s d -dimensional vector of current pose.
Revised: f c,y · a c , which are input to the control network along with the robot’s d -dimensional vector of current pose.

Original: The control network is a standard MLP mapping R C × 2+ d to P ( u ) .
Revised: The control network is a standard MLP mapping R C × 2+ d to P ( u ) .

Original: The intuition behind this architecture is that the stochastic intention can ‘select’ which features are relevant for making a control decision, by giving them higher weights in the attention.
Revised: The intuition behind this architecture is that the stochastic intention can ‘select’ which features are relevant for making a control decision, by giving them higher weights in the attention.

Original: When 
Revised: When 5 Intuitively, when the threshold α approaches 1 /N , the quantile q α ( θ ) increases (up to the maximum value of log reward) at rate O ( α ) , leaving less and less room for log rewards above the quantile. Concretely, the random variable | log r ( z 1 ; x 1: T , u 1: T , θ ) − q α ( θ ) | 1 { log r ( z 1 ; x 1: T , u 1: T , θ ) ≥ q α ( θ ) } converges to 0 at rate O ( α ) .

Original: , leaving the quantile.
Revised: Consequently, this random variable is bounded by O ( α ) .

Original: Concretely,
Revised: Analogous arguments can be applied for the case with gradients as well, more technical details can be found in Section 4 of Hong & Liu (2009).

Original: [Figure]
Revised: [Figure]

Original: multiple objects are in the scene, each object would naturally be represented by different features, therefore the intention in this architecture can correspond to attending to a particular object 6 .
Revised: multiple objects are in the scene, each object would naturally be represented by different features, therefore the intention in this architecture can correspond to attending to a particular object 6 .

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: We demonstrate the effectiveness of our approach on learning a reaching task with the IRB-120 robot (Figure 1b), where mentor trajectories are collected in the form of sequential image-action pairs.
Revised: We demonstrate the effectiveness of our approach on learning a reaching task with the IRB-120 robot (Figure 1b), where mentor trajectories are collected in the form of sequential image-action pairs.

Original: The main questions we seek to answer are: (1) Can our IDS learn to effectively reproduce multiple intentions in the demonstrations?
Revised: The main questions we seek to answer are: (1) Can our IDS learn to effectively reproduce multiple intentions in the demonstrations?

Original: (2) How does IDS approach compare to a standard deterministic
Revised: (2) How does IDS approach compare to a standard deterministic

Original: NN approach for LfD?
Revised: NN approach for LfD?

Original: (3) How does training an I-SNN using IDS compare with training it using the SNN algorithm 7 of Tang & Salakhutdinov (2013)?
Revised: (3) How does training an I-SNN using IDS compare with training it using the SNN algorithm 7 of Tang & Salakhutdinov (2013)?

Original: To maintain a fair comparison, we evaluated deterministic NN policies with identical structure as
Revised: To maintain a fair comparison, we evaluated deterministic NN policies with identical structure as

Original: I-SNN except for the stochastic intention module (cf.
Revised: I-SNN except for the stochastic intention module (cf.

Original: Section 4.4, and Figure 1).
Revised: Section 4.4, and Figure 1).

Original: 5. SNN I MPLEMENTATION D ETAILS
Revised: 5. SNN I MPLEMENTATION D ETAILS

Original: In all our experiments we used the following parameters for the SNN training, which we found to work well consistently across different domains:(1) Loss function : we represent the output distribution as log P ( u | h, x ; θ ) ∝ −(cid:107)
Revised: In all our experiments we used the following parameters for the SNN training, which we found to work well consistently across different domains:(1) Loss function : we represent the output distribution as log P ( u | h, x ; θ ) ∝ −(cid:107)

Original: f ( x, h ; θ ) − u (cid:107) 1 , where f ( x, h ; θ ) is the output of the control network, as described in Section 4.4.
Revised: f ( x, h ; θ ) − u (cid:107) 1 , where f ( x, h ; θ ) is the output of the control network, as described in Section 4.4.

Original: This corresponds to an L 1 regression loss, which we found to perform better than the popular L 2 loss in our experiments.
Revised: This corresponds to an L 1 regression loss, which we found to perform better than the popular L 2 loss in our experiments.

Original: (2) Monte Carlo samples : we chose N = 5 , which we found to work well.
Revised: (2) Monte Carlo samples : we chose N = 5 , which we found to work well.

Original: Higher values resulted in degraded performance at test time, due to the higher bias (see Section 4).
Revised: Higher values resulted in degraded performance at test time, due to the higher bias (see Section 4).

Original: (3) Intention variable dimension: we chose z ∈ R 5 for all experiments.
Revised: (3) Intention variable dimension: we chose z ∈ R 5 for all experiments.

Original: We did not observe this parameter to be sensitive, and dimensions from 2 to 10 performed similarly.
Revised: We did not observe this parameter to be sensitive, and dimensions from 2 to 10 performed similarly.

Original: (4) P ( z ) : a 5 -dimensional vector of independent uniform multinomials in { 0 : 4 } .
Revised: (4) P ( z ) : a 5 -dimensional vector of independent uniform multinomials in { 0 : 4 } .

Original: 5. R EAL W ORLD R OBOT R EACHING T ASK
Revised: 5. R EAL W ORLD R OBOT R EACHING T ASK

Original: In this task, depicted in Figure 3(a), the objective is to navigate the end-effector of robot to reach a point above one of 3 objects in the scene – a soap box, a blue electronic box and a measuring cup.
Revised: In this task, depicted in Figure 3(a), the objective is to navigate the end-effector of robot to reach a point above one of 3 objects in the scene – a soap box, a blue electronic box and a measuring cup.

Original: We used a 6-DOF IRB-120 robot where the control is a 3-dimensional Cartesian vector applied towhere twoobjects have exactly thesame appearance, and therefore thesame feature activation functionSuch cases can be handled by adding spatial information to the features or the attention module, which will be investigatedin future work.
Revised: We used a 6-DOF IRB-120 robot where the control is a 3-dimensional Cartesian vector applied to two exactly the appearance, and therefore the feature activation function spatial information to the features or the attention module, which will be investigated

Original: We also investigated using a conditional VAE (CVAE), however, despite extensive experimentation, we could not get the CVAE to work.
Revised: We also investigated using a conditional VAE (CVAE), however, despite extensive experimentation, we could not get the CVAE to work.

Original: We attribute this to the fact that the recognition network in a CVAE needs to map the image to a latent variable distribution that ‘explains’ the observed action.
Revised: We attribute this to the fact that the recognition network in a CVAE needs to map the image to a latent variable distribution that ‘explains’ the observed action.

Original: This mapping is complex, as it needs to understand from the image what goal the demonstration is aiming towards.
Revised: This mapping is complex, as it needs to understand from the image what goal the demonstration is aiming towards.

Original: Our approach, on the other hand, does not require a recognition module for reducing variance during training.
Revised: Our approach, on the other hand, does not require a recognition module for reducing variance during training.

Original: the the end effector u t = ( d(cid:126)x t , d(cid:126)y t , d(cid:126)z t ) .
Revised: the the end effector u t = ( d(cid:126)x t , d(cid:126)y t , d(cid:126)z t ) .

Original: The observations consists of (1) a 480 × 640 RGB image of the scene (further cropped and resized to 64 × 64 resolution) using a point-grey camera mounted at a ﬁxed angle; (2) the 3 dimensional end-effector Cartesian pose (see Figure 1 for more details).
Revised: The observations consists of (1) a 480 × 640 RGB image of the scene (further cropped and resized to 64 × 64 resolution) using a point-grey camera mounted at a ﬁxed angle; (2) the 3 dimensional end-effector Cartesian pose (see Figure 1 for more details).

Original: Data Collection: We denote a speciﬁc scene arrangement of object placements and initial endeffector pose as a task conﬁguration .
Revised: Data Collection: We denote a speciﬁc scene arrangement of object placements and initial endeffector pose as a task conﬁguration .

Original: Task conﬁgurations were randomly generated by arbitrarily placing the three objects on the work bench and randomly initializing the end-effector pose.
Revised: Task conﬁgurations were randomly generated by arbitrarily placing the three objects on the work bench and randomly initializing the end-effector pose.

Original: Once a task conﬁguration is generated, the position of the objects remain ﬁxed for the entire episode.
Revised: Once a task conﬁguration is generated, the position of the objects remain ﬁxed for the entire episode.

Original: For each task conﬁguration we collect 3 demonstrations, each generated by a human navigating the end-effector using a 3DConnexion space-mouse to reach one of the objects.
Revised: For each task conﬁguration we collect 3 demonstrations, each generated by a human navigating the end-effector using a 3DConnexion space-mouse to reach one of the objects.

Original: At each time step t , the observation o t together with the Cartesian control signal u t (see above) are recorded.
Revised: At each time step t , the observation o t together with the Cartesian control signal u t (see above) are recorded.

Original: We collected demonstrations from 468 different task conﬁgurations, for a total of 1404 demonstration trajectories.
Revised: We collected demonstrations from 468 different task conﬁgurations, for a total of 1404 demonstration trajectories.

Original: Training: We compare using IDS and the SNN algorithm of Tang & Salakhutdinov (2013) (henceforth SNN) for training the I-SNN architecture.
Revised: Training: We compare using IDS and the SNN algorithm of Tang & Salakhutdinov (2013) (henceforth SNN) for training the I-SNN architecture.

Original: To reduce the training time, we pre-trained the weights of the convolutional layers in the Feature Extraction module reusing the weights learned in the deterministic NN model.
Revised: To reduce the training time, we pre-trained the weights of the convolutional layers in the Feature Extraction module reusing the weights learned in the deterministic NN model.

Original: For optimization, we also used Adam (Kingma & Ba, 2014), with the default parameters using 90% of the data set for training and 10% for validation. .
Revised: For optimization, we also used Adam (Kingma & Ba, 2014), with the default parameters using 90% of the data set for training and 10% for validation. .

Original: Evaluation: We evaluate each model on 10 randomly generated task conﬁgurations, with 20 trials on each task conﬁguration, for a total of 200 trials.
Revised: Evaluation: We evaluate each model on 10 randomly generated task conﬁgurations, with 20 trials on each task conﬁguration, for a total of 200 trials.

Original: We run the model until it either succeeds to reach an object or fails.
Revised: We run the model until it either succeeds to reach an object or fails.

Original: For the IDS algorithm, we used optimistic sampling.
Revised: For the IDS algorithm, we used optimistic sampling.

Original: For SNN, we experimented with both optimistic and uniform sampling, however the latter resulted in a better overall performance.
Revised: For SNN, we experimented with both optimistic and uniform sampling, however the latter resulted in a better overall performance.

Original: Table 1 shows the overall success rate for every model across all 200 trials.
Revised: Table 1 shows the overall success rate for every model across all 200 trials.

Original: The deterministic NN model succeeded in reaching one of the objects only in 3 task conﬁgurations (and kept on reaching that same object for the 20 evaluations in each), and failed on the other 7 task conﬁgurations due to the averaging problem.
Revised: The deterministic NN model succeeded in reaching one of the objects only in 3 task conﬁgurations (and kept on reaching that same object for the 20 evaluations in each), and failed on the other 7 task conﬁgurations due to the averaging problem.

Original: The stochastic algorithms performed signiﬁcantly better by learning multiple modes of the problem.
Revised: The stochastic algorithms performed signiﬁcantly better by learning multiple modes of the problem.

Original: IDS signiﬁcantly out-performed the SNN algorithm, as we explain below.
Revised: IDS signiﬁcantly out-performed the SNN algorithm, as we explain below.

Original: We evaluate the mode learning ability by counting, for each task conﬁguration, how many different objects were reached, as depicted in Table 1 (last four columns).
Revised: We evaluate the mode learning ability by counting, for each task conﬁguration, how many different objects were reached, as depicted in Table 1 (last four columns).

Original: As expected, the deterministic
Revised: As expected, the deterministic

Original: NN could not reach more than one object.
Revised: NN could not reach more than one object.

Original: I-SNN trained by SNN algorithm, on the other hand, could reach two different (but ﬁxed) objects for 6 task conﬁgurations, and all three objects for the rest.
Revised: I-SNN trained by SNN algorithm, on the other hand, could reach two different (but ﬁxed) objects for 6 task conﬁgurations, and all three objects for the rest.

Original: The best performance is achieved by I-SNN trained by IDS, reaching all three objects in all the tasks, thereby demonstrating a strong ability to learn all the modes in the data.
Revised: The best performance is achieved by I-SNN trained by IDS, reaching all three objects in all the tasks, thereby demonstrating a strong ability to learn all the modes in the data.

Original: In the appendix (Figure 3), we show a histogram of reaching the different objects that demonstrates that IDS learned a near-uniform distribution over the modes.
Revised: In the appendix (Figure 3), we show a histogram of reaching the different objects that demonstrates that IDS learned a near-uniform distribution over the modes.

Original: Additionally, in the appendix (Figure 5) we visualize the features that I-SNN attends to, showing that in each episode the model consistently attends to the same object throughout the execution.
Revised: Additionally, in the appendix (Figure 5) we visualize the features that I-SNN attends to, showing that in each episode the model consistently attends to the same object throughout the execution.

Original: In Figure 2 we explain the superior performance of IDS over SNN.
Revised: In Figure 2 we explain the superior performance of IDS over SNN.

Original: Since IDS focuses only on the best samples (through the max ) compared to SNN which gives weight also to non best samples (through the soft- max ), IDS better ‘tunes’ the network for the best samples.
Revised: Since IDS focuses only on the best samples (through the max ) compared to SNN which gives weight also to non best samples (through the soft- max ), IDS better ‘tunes’ the network for the best samples.

Original: Combined with optimistic sampling, which draws these best samples during execution, this leads to better results.
Revised: Combined with optimistic sampling, which draws these best samples during execution, this leads to better results.

Original: [Table]
Revised: [Table]

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We presented an approach for learning from demonstrations that contain multiple modes of performing the same task.
Revised: We presented an approach for learning from demonstrations that contain multiple modes of performing the same task.

Original: Our method is based on stochastic neural networks, and represents the mode
Revised: Our method is based on stochastic neural networks, and represents the mode

Original: SNNIDS
Revised: SNNIDS

Original: Epoch
Revised: Epoch

Original: Loss (a)
Revised: Loss (a)

Original: Loss
Revised: Loss

Original: Epoch
Revised: Epoch

Original: SNNIDS (b)
Revised: SNNIDS (b)

Original: Loss
Revised: Loss

Original: Epoch
Revised: Epoch

Original: SNNIDS (c) of performing the task by a stochastic vector – the intention , which is given as input to a feedforward neural network.
Revised: SNNIDS (c) of performing the task by a stochastic vector – the intention , which is given as input to a feedforward neural network.

Original: We presented a simple and efﬁcient algorithm for training our models, and a particular implementation suitable for vision-based inputs.
Revised: We presented a simple and efﬁcient algorithm for training our models, and a particular implementation suitable for vision-based inputs.

Original: As we demonstrated in real-robot experiments, our method can reliably learn to reproduce the different modes in the demonstration data, and outperforms standard approaches in cases where such different modes exist.
Revised: As we demonstrated in real-robot experiments, our method can reliably learn to reproduce the different modes in the demonstration data, and outperforms standard approaches in cases where such different modes exist.

Original: In future work we intend to investigate the extension of this approach to more complex manipulation tasks such as grasping and assembly, and domains with a very large number of objects in the scene.
Revised: In future work we intend to investigate the extension of this approach to more complex manipulation tasks such as grasping and assembly, and domains with a very large number of objects in the scene.

Original: An interesting point in our model is tying the features to the intention by an attention mechanism, and we intend to further investigate recurrent attention mechanisms (Xu et al., 2015) that could offer better generalization at inference time.
Revised: An interesting point in our model is tying the features to the intention by an attention mechanism, and we intend to further investigate recurrent attention mechanisms (Xu et al., 2015) that could offer better generalization at inference time.

Original: A G ENERALIZED EM A LGORITHM FOR L EARNING SNN S VIA I MPORTANCE S AMPLING
Revised: A G ENERALIZED EM A LGORITHM FOR L EARNING SNN S VIA I MPORTANCE S AMPLING

Original: We review the work of Tang & Salakhutdinov (2013) as relevant for our setting.
Revised: We review the work of Tang & Salakhutdinov (2013) as relevant for our setting.

Original: Following their notation, we use h for the intention variable ( z in our formulation).

Original: For the importance sampling based approach, instead of maximizing the log likelihood function, consider maximizing the following evidence lower bound (ELBO):
Revised: For the importance sampling based approach, instead of maximizing the log likelihood function, consider maximizing the following evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original:  F isrelated to the Kullback Liebler divergence between P ( u 1: T | h, x 1: T ; θ ) and Q ( h | u 1: T , x 1: T )  as follows:
Revised: where F is the Kullback Liebler divergence between P ( u 1: T | z, x 1: T ; θ ) and Q ( z | u 1: T , x 1: T ) given as follows:

Original: [Equation]
Revised: [Equation]

Original: Most importantly
Revised: Most importantly

Original: [Equation]
Revised: [Equation]

Original: it has also been shown in Theorem 2 of Neal & Hinton (1998) that if Q and θ form a pair of local maximizer to F
Revised: it has also been shown in Theorem 2 of Neal & Hinton (1998) that if Q and θ form a pair of local maximizer to F

Original: [Equation]
Revised: [Equation]

Original: then θ is also a local maximum of the original likelihood maximization problem.
Revised: then θ is also a local maximum of the original likelihood maximization problem.

Original: To maximize F w.r.t Q , one has the closed form solution based on Bayes theorem:
Revised: To maximize F w.r.t Q [Equation] one has the closed form solution based on Bayes theorem:

Original: [Equation]
Revised: [Equation]

Original: Here, { h 1 , . . . , h N } is a sequence of latent random variables sampled i.i.d.
Revised: Here, { z 1 , . . . , z N } is a sequence of latent random variables sampled i.i.d.

Original: from the distribution
Revised: from the distribution

Original: P ( h ) .
Revised: P ( z ) .

Revised: Given parameter θ , denoted by θ old , immediately the posterior distribution Q that maximizes F is given by: Q ∗ ( z | x 1: T , u 1: T ) =

Revised: P ( z | x 1: T , u 1: T ; θ old ) .

Revised: In this case, the above loss function is equivalent to the complete data log-likelihood

Original:  Tang & Salakhutdinov (2013) present a generalized EM algorithm to train a SNN.
Revised: [Equation] which is a lower bound of the log likelihood. Furthermore, if θ = θ old , then clearly (cid:96) ∗ ( θ old , θ old ) is equal to the log-likelihood log P ( y 1: T | x 1: T ,θ old ) . Tang & Salakhutdinov (2013) present a generalized EM algorithm to train a SNN.

Original: In the E-step, the following approximate posterior distribution is used:
Revised: In the E-step, the following approximate posterior distribution is used:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is the the importance sampling weight.
Revised: is the the importance sampling weight.

Original: Recall that for our distribution model, r ( h ; x 1: T , u 1: T , θ old ) ∝ exp( − d ( f ( x, z ; θ ) , u )) , therefore we obtain that the importance weights correspond to a soft- max over the prediction error.
Revised: Recall that for our distribution model, r ( z ; x 1: T , u 1: T , θ old ) ∝ exp( − d ( f ( x, z ; θ ) , u )) , therefore we obtain that the importance weights correspond to a soft- max over the prediction error.

Original: In the M-step, the θ parameters are updated with the gradient vector with respect to the following optimization: θ ∈ arg max θ ∈ Θ ˆ (cid:96) ( θ, θ old ) , where
Revised: In the M-step, the θ parameters are updated with the gradient vector with respect to the following optimization: [Equation] θ, θ old ) , where

Original: [Equation]
Revised: [Equation]

Original: is the empirical expected log likelihood, and ˆ Q is the posterior distribution from the E-step.
Revised: is the empirical expected log likelihood, and ˆ Q is the posterior distribution from the E-step.

Original: Here we drop the last term in F because in our case Q that does not depend on θ .
Revised: Here we drop the last term in F because in our case Q that does not depend on θ .

Original: Correspondingly, the gradient estimate is given by:
Revised: Correspondingly, the gradient estimate is given by:

Original: [Equation]
Revised: [Equation]

Original: the equality is due to the facts that
Revised: the equality is due to the facts that

Original: [Equation]
Revised: [Equation]

Original: ) = log r ( h ; x 1: T ,y 1: T , θ ) + log P ( h ) and distribution P ( h ) is independent of θ . 
Revised: and distribution P ( z ) is independent of θ . To better understand this estimator, we will analyze the bias and variance of the gradient estimator. Based on the construction of importance sampling weight, immediately the gradient estimator is consistent. Furthermore, under certain regular assumptions, the bias is O ( N − 1 / 2 ) .

Revised: (This means the gradient estimator is asymptotically unbiased.)

Original: B T ECHNICAL P ROOF OF THE V ARIANCE IN I NTENTION - DRIVEN S AMPLING
Revised: Furthermore, the variance of this estimator is given by [Equation] where the integrand is given by v [Equation]

Revised: z ;

Revised: [Equation]

Revised: B T ECHNICAL P ROOF OF THE V ARIANCE IN I NTENTION - DRIVEN S AMPLING

Original: In this section, we study the variance of the gradient estimate of CVaR ( R ( z ; θ )) .
Revised: In this section, we study the variance of the gradient estimate of CVaR ( R ( z ; θ )) .

Original: This proof follows analogously from the analysis in Section 4.3 of Hong & Liu (2009) for the case of estimating asymptotic variance in gradient.
Revised: This proof follows analogously from the analysis in Section 4.3 of Hong & Liu (2009) for the case of estimating asymptotic variance in gradient.

Original: Here we use R ( z ; θ ) as the shorthand notation of the log reward function log r ( z ; x 1: T , y 1: T , θ ) .
Revised: Here we use R ( z ; θ ) as the shorthand notation of the log reward function log r ( z ; x 1: T , y 1: T , θ ) .

Original: For any given cut-off level α and sample size n , consider the following update formula of the CVaR gradient estimate
Revised: For any given cut-off level α and sample size n , consider the following update formula of the CVaR gradient estimate

Original: [Equation]
Revised: [Equation]

Original: where S ( z i ) =
Revised: where S ( z i ) =

Original: 1 { R ( z i ; θ ) ≥
Revised: 1 { R ( z i ; θ ) ≥

Original: q α ( θ ) } , ∀ i [Equation]
Revised: q α ( θ ) } , ∀ i , and { z 1 , . . .

Original: and { z 1 , . . . , z N } is sampled in an i.i.d.
Revised: , z N } is sampled in an i.i.d.

Original: fashion from
Revised: fashion from

Original: P ( h ) .
Revised: P ( h ) .

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: The correlation comes from the fact that quantile is deﬁned based on the order statistics of the reward.
Revised: The correlation comes from the fact that quantile is deﬁned based on the order statistics of the reward, see Section 4 of Hong & Liu (2009) for more details.

Original: Now notice the following property:
Revised: Now notice the following property:

Original: [Equation]
Revised: [Equation]

Original: where N i : N is the i − th order statistic from the N observations of reward { R ( z i ; θ ) } Ni =1 . 
Revised: where L i : N is the i − th order statistic from the N observations of reward { R ( z i ; θ ) } Ni =1 . Under the event that R ( z 1 ; θ ) >

Original: This implies that N  (cid:100) 
Revised: L (cid:100) Nα (cid:101)− 1: N − 2 and R ( z 2 ; θ ) > L (cid:100) [Equation]

Revised: : N − 2 , by the deﬁnition of the order statistic and by the i.i.d.

Revised: assumption of { z 1 , . .

Revised: . , z N } , one can deduce that L (cid:100)

Original: Nα (cid:101)− 1: N − 2 is independent of R ( z 1 ; θ ) and R ( z 2 ; θ ) .
Revised: Nα (cid:101)− 1: N − 2 is independent of R ( z 1 ; θ ) and R ( z 2 ; θ ) .

Original: Furthermore, the above condition implies,
Revised: Equipped with this condition, one has the following expression:

Original: [Equation]
Revised: [Equation]

Original: N − 2 )] 2 , where f ( N (cid:100)
Revised: where f ( L (cid:100)

Original: Nα (cid:101)− 1: N − 2 ) :
Revised: Nα (cid:101)− 1: N − 2 ) :

Original: = E [ ∇ θ R ( z 1 ; θ ) S ( z 1 )] . Following the same lines ofarguments, one can also show that E [ ∇ θ R ( z1 ; θ ) S ( z 1 )] 2 = E
Revised: = E [ ∇ θ R ( z 1 ; θ ) S ( z 1 ) | L (cid:100) Nα (cid:101)− 1: N − 2 ] is the conditional expectation of ∇ θ R ( z ; θ ) S ( z ) w.r.t.

Original: [ f ( N (cid:100) 
Revised: random variable L (cid:100) Nα (cid:101)− 1: N − 2 .

Original:  Nα 
Revised: The ﬁrst equality follows from the fact that z 1 , and z 2 are i.i.d. random variables that are independent of L (cid:100) Nα (cid:101)− 1: N − 2 . On the other hand, following the same lines of analysis, one can also show that

Original: [Equation]
Revised: [Equation]

Original: 1: N − 1 )] 2 .

Original: Combining all the above analysis together, one has the following expression:
Revised: Therefore, combining all the above analysis together, one has the following expression:

Original: [Equation]
Revised: [Equation]

Original: From the results in Proposition 4.3 to 4.4 of Hong & Liu (2009), with q (cid:48) α ( θ ) = ∇ θ q α ( θ ) , we have that
Revised: From the results in Proposition 4.3 to 4.4 of Hong & Liu (2009) [Equation] with q (cid:48) α ( θ ) = ∇ θ q α ( θ ) denote the gradient of the quantile

Original: Var

Original: [Equation]
Revised: [Equation]

Original: and
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the variance of G N can be expressed as:
Revised: the variance of G N can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: This provides the variance of intention-driven sampling gradient estimate
Revised: This provides the variance of intention-driven sampling gradient estimate

Original: [Equation]
Revised: [Equation]

Original: C A DDITIONAL M ATERIAL
Revised: C A DDITIONAL M ATERIAL

Revised: x t u t

Revised: T (a) x t u t z

Revised: T ⍵ (b)