Original: A TTENTION - DRIVEN R OBOTIC M ANIPULATION
Revised: A TTENTION - DRIVEN R OBOTIC M ANIPULATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks.
Revised: Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks.

Original: This is partly due to the fact that reinforcement learning algorithms are notoriously difﬁcult and time consuming to train, which is exacerbated when training from images rather than full-state inputs.
Revised: This is partly due to the fact that reinforcement learning algorithms are notoriously difﬁcult and time consuming to train, which is exacerbated when training from images rather than full-state inputs.

Original: As humans perform manipulation tasks, our eyes closely monitor every step of the process with our gaze focusing sequentially on the objects being manipulated.
Revised: As humans perform manipulation tasks, our eyes closely monitor every step of the process with our gaze focusing sequentially on the objects being manipulated.

Original: With this in mind, we present our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of real-world sparserewarded tasks without any prior task knowledge.
Revised: With this in mind, we present our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of real-world sparserewarded tasks without any prior task knowledge.

Original: ARM splits the complex task of manipulation into a 3 stage pipeline: (1) a Q-attention agent extracts interesting pixel locations from RGB and point cloud inputs, (2) a next-best pose agent that accepts crops from the Q-attention agent and outputs poses, and (3) a control agent that takes the goal pose and outputs joint actions.
Revised: ARM splits the complex task of manipulation into a 3 stage pipeline: (1) a Q-attention agent extracts interesting pixel locations from RGB and point cloud inputs, (2) a next-best pose agent that accepts crops from the Q-attention agent and outputs poses, and (3) a control agent that takes the goal pose and outputs joint actions.

Original: We show that current state-of-the-art reinforcement learning algorithms catastrophically fail on a range of RLBench tasks, whilst ARM is successful within a few hours.
Revised: We show that current state-of-the-art reinforcement learning algorithms catastrophically fail on a range of RLBench tasks, whilst ARM is successful within a few hours.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite their potential, continuous-control reinforcement learning (RL) algorithms have many ﬂaws: they are notoriously data hungry, often fail with sparse rewards, and struggle with long-horizon tasks.
Revised: Despite their potential, continuous-control reinforcement learning (RL) algorithms have many ﬂaws: they are notoriously data hungry, often fail with sparse rewards, and struggle with long-horizon tasks.

Original: The algorithms for both discrete and continuous RL are almost always evaluated on benchmarks that give shaped rewards (Brockman et al., 2016; Tassa et al., 2018), a privilege that is not feasible for training real-world robotic application across a broad range of tasks.
Revised: The algorithms for both discrete and continuous RL are almost always evaluated on benchmarks that give shaped rewards (Brockman et al., 2016; Tassa et al., 2018), a privilege that is not feasible for training real-world robotic application across a broad range of tasks.

Original: Motivated by the observation that humans focus their gaze close to objects being manipulated (Land et al., 1999), we propose an Attention-driven Robotic Manipulation (ARM) algorithm that consists of a series of algorithmagnostic components, that when combined, results in a method that is able to perform a range of challenging, sparsely-rewarded manipulation tasks.
Revised: Motivated by the observation that humans focus their gaze close to objects being manipulated (Land et al., 1999), we propose an Attention-driven Robotic Manipulation (ARM) algorithm that consists of a series of algorithmagnostic components, that when combined, results in a method that is able to perform a range of challenging, sparsely-rewarded manipulation tasks.

Original: Our algorithm operates through a pipeline of modules: our novel Q-attention module ﬁrst extracts interesting pixel locations from RGB and point cloud inputs by treating images as an environment, and pixel locations as actions.
Revised: Our algorithm operates through a pipeline of modules: our novel Q-attention module ﬁrst extracts interesting pixel locations from RGB and point cloud inputs by treating images as an environment, and pixel locations as actions.

Original: Using the pixel locations we crop the RGB-D images, signiﬁcantly reducing input size, and feed this to a next-best-pose continuous-control agent that outputs 6D poses, which is trained with our novel conﬁdence-aware critic .
Revised: Using the pixel locations we crop the RGB and point cloud inputs, signiﬁcantly reducing input size, and feed this to a next-best-pose continuous-control agent that outputs 6D poses, which is trained with our novel conﬁdence-aware critic .

Original: These goal poses are then used by a control algorithm that continuously outputs motor velocities.
Revised: These goal poses are then used by a control algorithm that continuously outputs motor velocities.

Original: As is common with sparsely-rewarded tasks, we improve initial exploration through the use of demonstrations.
Revised: As is common with sparsely-rewarded tasks, we improve initial exploration through the use of demonstrations.

Original: However, rather than simply inserting these directly into the replay buffer, we use a keyframe discovery strategy that chooses interesting keyframes along demonstration trajectories that is fundamental to training our Q-attention module.
Revised: However, rather than simply inserting these directly into the replay buffer, we use a keyframe discovery strategy that chooses interesting keyframes along demonstration trajectories that is fundamental to training our Q-attention module.

Original: Rather than storing the transition from an initial state to a keyframe state, we use our demo augmentation method which also stores the transition from intermediate points along a trajectories to the keyframe states; thus greatly increasing the proportion of initial demo transitions in the replay buffer.
Revised: Rather than storing the transition from an initial state to a keyframe state, we use our demo augmentation method which also stores the transition from intermediate points along a trajectories to the keyframe states; thus greatly increasing the proportion of initial demo transitions in the replay buffer.

Original: All of these improvements result in an algorithm that starkly outperforms other state-of-the-art methods when evaluated on 10 RLBench (James et al., 2020) tasks (Figure 1) that range in difﬁculty.
Revised: All of these improvements result in an algorithm that starkly outperforms other state-of-the-art methods when evaluated on 10 RLBench (James et al., 2020) tasks (Figure 1) that range in difﬁculty.

Original: To summarise, we propose the following contributions: (1) An  attention mechanism that is learnedexplicitly via Q-Learning, rather than the implicit attention that is commonly seen in the NLP and vision community; (2) A conﬁdence-aware Q function that predicts pixel-wise Q values and conﬁ- dence values, resulting in improved convergence times; (3) A keyframe discovery strategy and demo augmentation method that go hand-in-hand to improve the utilisation of demonstrations in RL.
Revised: To summarise, we propose the following contributions: (1) An off-policy hard attention mechanism that is learned via Q-Learning, rather than the on-policy hard attention and soft attention that is commonly seen in the NLP and vision community; (2) A conﬁdence-aware Q function that predicts pixel-wise Q values and conﬁdence values, resulting in improved convergence times; (3) A keyframe discovery strategy and demo augmentation method that go hand-in-hand to improve the utilisation of demonstrations in RL.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The use of reinforcement learning (RL) is prevalent in many areas of robotics, including legged robots (Kohl & Stone, 2004;
Revised: The use of reinforcement learning (RL) is prevalent in many areas of robotics, including legged robots (Kohl & Stone, 2004;

Original: Hwangbo et al., 2019), aerial vehicles (Sadeghi & Levine, 2017), and manipulation tasks, such as pushing (Finn & Levine, 2017), peg insertion (Levine et al., 2016;
Revised: Hwangbo et al., 2019), aerial vehicles (Sadeghi & Levine, 2017), and manipulation tasks, such as pushing (Finn & Levine, 2017), peg insertion (Levine et al., 2016;

Original: Zeng et al., 2018; Lee et al., 2019), throwing (Ghadirzadeh et al., 2017; Zeng et al., 2020), ballin-cup (Kober & Peters, 2009), cloth manipulation (Matas et al., 2018), and grasping (Kalashnikov et al., 2018; James et al., 2019b).
Revised: Zeng et al., 2018; Lee et al., 2019), throwing (Ghadirzadeh et al., 2017; Zeng et al., 2020), ballin-cup (Kober & Peters, 2009), cloth manipulation (Matas et al., 2018), and grasping (Kalashnikov et al., 2018; James et al., 2019b).

Original: Despite the abundance of work in this area, there has yet to be a general manipulation method that can tackle a range of challenging, sparsely-rewarded tasks without needing access to privileged simulation-only abilities (e.g. reset to demonstrations (Nair et al., 2018), asymmetric actor-critic (Pinto et al., 2018), reward shaping (Rajeswaran et al., 2018),  (James al
Revised: Despite the abundance of work in this area, there has yet to be a general manipulation method that can tackle a range of challenging, sparsely-rewarded tasks without needing access to privileged simulation-only abilities (e.g. reset to demonstrations (Nair et al., 2018), asymmetric actor-critic (Pinto et al., 2018), reward shaping (Rajeswaran et al., 2018), and auxiliary tasks (James et al., 2017)).

Original: Crucial to our method is the proposed Q-attention.
Revised: Crucial to our method is the proposed Q-attention.

Original: Soft and hard attention are prominent methods in both natural language processing (NLP)
Revised: Soft and hard attention are prominent methods in both natural language processing (NLP)

Original: (Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2018) and computer vision (Xu et al., 2015; Zhang et al., 2019).
Revised: (Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2018) and computer vision (Xu et al., 2015; Zhang et al., 2019).

Original: Soft-attention deterministically multiplies an attention map over the image feature map, whilst hard-attention uses the attention map stochastically to sample one or a few features on the feature map (which is optimised by maximising an approximate variational lower bound or equivalently via  REINFORCE (Williams, 1992)).
Revised: Soft attention deterministically multiplies an attention map over the image feature map, whilst hard attention uses the attention map stochastically to sample one or a few features on the feature map (which is optimised by maximising an approximate variational lower bound or equivalently via (on-policy) REINFORCE (Williams, 1992)).

Original: Despite sharing a similar human-inspired motivation and goal, ourproposed Q-attention varies in its formulation.
Revised: Given that we perform non-differentiable cropping, our Q-attention is closest to hard attention, but with the distinction that we learn this in an off-policy way.

Original: Rather than outputting a probability density function, we output the value for choosing each pixel, choose the highest, and then crop the observations at these pixel locations. We learn this via Q-learning, by treating the image as  an RL environment, where pixel locations act as actions.
Revised: This is key, as ‘traditional’ hard attention is unable to be used in an off-policy setting.

Original: We expand on the subtle differencesbetween Q-attention and attention in Appendix 7.
Revised: We therefore see Q-attention as an off-policy hard attention. We elaborate further on these differences in Section 4.1.

Original: Our proposed conﬁdence-aware critic (used to train the next-best pose agent) takes its inspiration from the pose estimation community (Wang et al., 2019; Wada et al., 2020).
Revised: Our proposed conﬁdence-aware critic (used to train the next-best pose agent) takes its inspiration from the pose estimation community (Wang et al., 2019; Wada et al., 2020).

Original: There exists a small amount of work in estimating uncertainty with Q-learning in discrete domains (Clements et al., 2019; Hoel et al., 2020); our work uses a continuous Q-function to predict both Q and conﬁdence values for each pixel, which lead to improved stability when training, and is not used during action selection.
Revised: There exists a small amount of work in estimating uncertainty with Q-learning in discrete domains (Clements et al., 2019; Hoel et al., 2020); our work uses a continuous Q-function to predict both Q and conﬁdence values for each pixel, which lead to improved stability when training, and is not used during action selection.

Original: Our approach makes use of demonstrations, which has been applied in a number of works (Vecerik et al., 2017; Matas et al., 2018; Kalashnikov et al., 2018; Nair et al., 2018), but while successful, they make limited use of the demonstrations and still can take many samples to converge.
Revised: Our approach makes use of demonstrations, which has been applied in a number of works (Vecerik et al., 2017; Matas et al., 2018; Kalashnikov et al., 2018; Nair et al., 2018), but while successful, they make limited use of the demonstrations and still can take many samples to converge.

Original: Rather than simply inserting these directly into the replay buffer, we instead make sure of our keyframe discovery and demo augmentation to maximise demonstration utility.
Revised: Rather than simply inserting these directly into the replay buffer, we instead make sure of our keyframe discovery and demo augmentation to maximise demonstration utility.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The reinforcement learning paradigm assumes an agent interacting with an environment consisting of states s ∈ S , actions a ∈ A , and a reward function R ( s t , a t ) , where s t and a t are the state and action at time step t respectively.
Revised: The reinforcement learning paradigm assumes an agent interacting with an environment consisting of states s ∈ S , actions a ∈ A , and a reward function R ( s t , a t ) , where s t and a t are the state and action at time step t respectively.

Original: The goal of the agent is then to discover a policy π that results in maximising the expectation of the sum of discounted rewards: E π [ (cid:80) t γ t R ( s t , a t )] , where future rewards are weighted with respect to the discount factor γ ∈ [0 , 1) .
Revised: The goal of the agent is then to discover a policy π that results in maximising the expectation of the sum of discounted rewards: E π [ (cid:80) t γ t R ( s t , a t )] , where future rewards are weighted with respect to the discount factor γ ∈ [0 , 1) .

Original: Each policy π has a corresponding value function Q ( s, a ) , which represents the expected return when following the 
Revised: Each policy π has a corresponding value function Q ( s, a ) , which represents the expected return when following the a

Original: Our Q-attention module builds from Deep Q-learning (Mnih et al., 2015), a method that approximated the value function Q θ , with a deep convolutional network, whose parameters θ are optimised by sampling mini-batches from a replay buffer D and using stochastic gradient descent to minimise the loss: E ( s t , a t , s t +1 ) 
Revised: Our Q-attention module builds from Deep Q-learning (Mnih et al., 2015), a method that approximated the value function Q θ , with a deep convolutional network, whose parameters θ are optimised by sampling mini-batches from a replay buffer D and using stochastic gradient descent to minimise the loss: E ( s t , a t , s t +1 ) ∼D [( r + γmax [Equation]

Original: ∼D [( r + γmax a (cid:48) Q θ (cid:48) ( s t +1 , a (cid:48) )
Revised: Q [Equation] (cid:48) ( s t +1 , a (cid:48) )

Original: − Q θ ( s t , a t )) 2 ] , where Q θ (cid:48) is a target network; a periodic copy of the online network Q θ which is not directly optimised.
Revised: − Q θ ( s t , a t )) 2 ] , where Q θ (cid:48) is a target network; a periodic copy of the online network Q θ which is not directly optimised.

Original: Our next-best pose agent builds upon SAC (Haarnoja et al., 2018), however, the agent is compatible with any off-policy, continuous-control RL algorithm.
Revised: Our next-best pose agent builds upon SAC (Haarnoja et al., 2018), however, the agent is compatible with any off-policy, continuous-control RL algorithm.

Original: SAC is a maximum entropy RL algorithm that, in addition to maximising the sum of rewards, also maximises the entropy of a policy: E
Revised: SAC is a maximum entropy RL algorithm that, in addition to maximising the sum of rewards, also maximises the entropy of a policy: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [ R
Revised: [ R

Original: [Equation]
Revised: [Equation]

Original: s t ,
Revised: s t ,

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: + [Equation] H ( π ( ·| s t ))]] , where α is a temperature parameter that determines the relative importance between the entropy and reward.
Revised: + α H ( π ( ·| s t ))]] , where α is a temperature parameter that determines the relative importance between the entropy and reward.

Original: The goal then becomes to maximise a soft Q-function by minimising the following Bellman residual:
Revised: The goal then becomes to maximise a soft Q-function by minimising the following Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: The policy is updated towards the Boltzmann policy with temperature α , with the Q-function taking the role of (negative) energy.
Revised: The policy is updated towards the Boltzmann policy with temperature α , with the Q-function taking the role of (negative) energy.

Original: Speciﬁcally, the goal is to minimise the Kullback-Leibler divergence between the policy and the Boltzman policy:
Revised: Speciﬁcally, the goal is to minimise the Kullback-Leibler divergence between the policy and the Boltzman policy:

Original: [Equation]
Revised: [Equation]

Original: Minimising the expected KL-divergence to learn the policy parameters was shown to be equivalent to maximising the expected value of the soft Qfunction:
Revised: Minimising the expected KL-divergence to learn the policy parameters was shown to be equivalent to maximising the expected value of the soft Qfunction:

Original: [Equation]
Revised: [Equation]

Original:  M ETHOD
Revised:  M ETHOD

Original: Our method can be split into a 3-phase pipeline.
Revised: Our method can be split into a 3-phase pipeline.

Original: Phase 1 (Section 4.1) consists of a high-level pixel agent that selects areas of interest using our novel Q-attention module.
Revised: Phase 1 (Section 4.1) consists of a high-level pixel agent that selects areas of interest using our novel Q-attention module.

Original: Phase 2 (Section 4.2) consists of a next-best pose prediction phase where the pixel location from the previous phase is used to crop the incoming observations and then predict a 6D pose.
Revised: Phase 2 (Section 4.2) consists of a next-best pose prediction phase where the pixel location from the previous phase is used to crop the incoming observations and then predict a 6D pose.

Original: Finally, phase 3 (Section 4.3) is a low-level control agent that accepts the predicted next-best pose and executes a series of actions to reach the given goal.
Revised: Finally, phase 3 (Section 4.3) is a low-level control agent that accepts the predicted next-best pose and executes a series of actions to reach the given goal.

Original: Before training, we ﬁll the replay buffer with demonstrations using our keyframe discoveryand demo augmentation strategy (Section 4.4) thatsigniﬁcantly improves training speed.
Revised: Before training, we ﬁll the replay buffer with demonstrations using our keyframe discovery (Section 4.4) that

Original: The full pipeline is summarised in Figure 2 and Algorithm 1.
Revised: The full pipeline is summarised in Figure 2 and Algorithm 1.

Original: All experiments are run in RLBench (James et al., 2020), a large-scale benchmark and learning environment for vision-guided manipulation built around CoppeliaSim (Rohmer et al., 2013) and PyRep (James et al., 2019a).
Revised: All experiments are run in RLBench (James et al., 2020), a large-scale benchmark and learning environment for vision-guided manipulation built around CoppeliaSim (Rohmer et al., 2013) and PyRep (James et al., 2019a).

Original: At each time step, we extract an observation from the front-facing camera that consists of an RGB image b and a depth image d , along with proprioceptive information z from the arm (consisting of end-effector pose and gripper open/close state).
Revised: At each time step, we extract an observation from the front-facing camera that consists of an RGB image b and a depth image d , along with proprioceptive information z from the arm (consisting of end-effector pose and gripper open/close state).

Original: Using known camera intrinsics and extrinsics, we process each depth image to produce a point cloud p (in world coordinates) projected from the view of the front-facing camera, producing a ( H × W × 3) ‘image’.
Revised: Using known camera intrinsics and extrinsics, we process each depth image to produce a point cloud p (in world coordinates) projected from the view of the front-facing camera, producing a ( H × W × 3) ‘image’.

Original: 4. Q- ATTENTION
Revised: 4. Q- ATTENTION

Original: Motivated by the role of vision and eye movement in the control of human activities (Land et al., 1999), we propose a Q-attention module that, given RGB and  point cloud inputs, outputs 2D pixel locations of the next area of interest.
Revised: Motivated by the role of vision and eye movement in the control of human activities (Land et al., 1999), we propose a Q-attention module that, given RGB and organised point cloud inputs, outputs 2D pixel locations of the next area of interest.

Original: With these pixel locations, we crop the RGB and  point cloud inputs and thus drastically reduce the input size to the next stage of the pipeline.
Revised: With these pixel locations, we crop the RGB and organised point cloud inputs and thus drastically reduce the input size to the next stage of the pipeline.

Original: The module shares similar human-inspired motivation to the attention seen in NLP (Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2018) and computer vision (Xu et al., 2015; Zhang et al., 2019), but differs in its formulation.

Original: NLP-based attention is implicitly learned (i.e. without an explicit loss), where as our Q-attention is explicitly learned via Q-learning, where images are treated as the ‘environment’, and pixel locations are treated as the ‘actions’.
Revised: Our Q-attention is explicitly learned via Q-learning, where images are treated as the ‘environment’, and pixel locations are treated as the ‘actions’.

Original: Given our Q-attention function QA θ , we extract the coordinates of pixels with the highest value:( x [Equation] [Equation] =
Revised: Given our Q-attention function QA θ , we extract the coordinates of pixels with the highest value:

Original: [Equation]
Revised: [Equation]

Original: 2D [Equation] [Equation] The parameters of the Q-attention are optimised by using stochastic gradient descent to minimise the loss:
Revised: The parameters of the Q-attention are optimised by using stochastic gradient descent to minimise the loss:

Original: [Equation]
Revised: [Equation]

Original: where s = ( b , p ) , QA θ (cid:48) is the target Q-function, and (cid:107) QA (cid:107) is an L 2 loss on the per-pixel output of the Q function; in practice, we found that this leads to increased robustness against the common problem of overestimation of Q values.
Revised: where s = ( b , p ) , QA θ (cid:48) is the target Q-function, and (cid:107) QA (cid:107) is an L 2 loss on the per-pixel output of the Q function (which we call Q regularisation ); in practice, we found that this leads to increased robustness against the common problem of overestimation of Q values.

Original: The Q-attention network follows a light-weight U-Net style architecture (Ronneberger et al., 2015), which is summarised in Figure 2.
Revised: The Q-attention network follows a light-weight U-Net style architecture (Ronneberger et al., 2015), which is summarised in Figure 2.

Original: Example per-pixel outputs of the Q-attention are shown in Figure 3.
Revised: Example per-pixel outputs of the Q-attention are shown in Figure 3.

Original: With the suggested coordinates from Q-attention, we perform a ( 16 × 16 ) crop on both the ( 128 × 128 ) RGB and  point cloud data: b (cid:48) , p (cid:48)
Revised: With the suggested coordinates from Q-attention, we perform a ( 16 × 16 ) crop on both the ( 128 × 128 ) RGB and organised point cloud data: b (cid:48) , p (cid:48)

Original: =
Revised: =

Original: crop ( b , p , ( x , y )) .
Revised: crop ( b , p , ( x , y )) .

Original: Notably, there is no explicit reward for choosing a pixel, but instead an implicit reward that comes from the output of the method pipeline as a whole (i.e. the same reward signal is used to train both the Q-attention and the next-best pose agent).
Revised: Notably, there is no explicit reward for choosing a pixel, but instead an implicit reward that comes from the output of the method pipeline as a whole (i.e. the same reward signal is used to train both the Q-attention and the next-best pose agent).

Original: This leads to a cyclic dependency between the two agents: the lower-level next-best pose agent relies on receiving good crops from the Q-attention agent, whilst the Q-attention agent needs the next-best pose agent to perform well in order to get its implicit reward.
Revised: This leads to a cyclic dependency between the two agents: the lower-level next-best pose agent relies on receiving good crops from the Q-attention agent, whilst the Q-attention agent needs the next-best pose agent to perform well in order to get its implicit reward.

Original: This is where delicate handling of demonstrations is key, which we discuss in
Revised: This is where delicate handling of demonstrations is key, which we discuss in

Original: Section 4.4.
Revised: Section 4.4.

Revised: The module shares similar human-inspired motivation to the attention seen in NLP (Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2018) and computer vision (Xu et al., 2015; Zhang et al., 2019), but differs in its formulation.

Revised: Soft attention multiplies an attention map over the image feature map, whilst hard attention uses the attention map to sample one or a few features on the feature maps or inputs.

Revised: Given that we perform non-differentiable cropping, we categorise our Q-attention as hard attention, but with 2 core differences: (1) most importantly, ‘traditional’ hard attention is optimised (on-policy) by maximising an approximate variational lower bound or equivalently via

Revised: REINFORCE (Williams, 1992), whereas our Q-attention is trained off-policy; this is crucial because our demonstration data, by deﬁnition, is off-policy, and therefore renders existing hard attention approaches unusable for demonstration-driven RL.

Revised: The output of ‘traditional’ hard attention carry different semantics: a score function in the case of REINFORCE hard attention, and Q-value (expected cumulative reward of choosing that crop) in the case of Q-attention.

Original: 4. N EXT - BEST P OSE A GENT
Revised: 4. N EXT - BEST P OSE A GENT

Original: Our next-best pose agent accepts cropped RGB b (cid:48) and  point cloud p (cid:48) inputs, and outputs a 6D pose.
Revised: Our next-best pose agent accepts cropped RGB b (cid:48) and organised point cloud p (cid:48) inputs, and outputs a 6D pose.

Revised: This next-best pose agent is run every time the robot reaches the previously selected pose.

Original: We represent the 6D pose via a translation e ∈ R 3 and a unit quaternion q ∈ R 4 , and restrict the w output of q to a positive number, therefore restricting the network to output unique unit quaternions.
Revised: We represent the 6D pose via a translation e ∈ R 3 and a unit quaternion q ∈ R 4 , and restrict the w output of q to a positive number, therefore restricting the network to output unique unit quaternions.

Original: The gripper action h ∈ R 1 lies between 0 and 1, which is then discretised to a binary open/close value.
Revised: The gripper action h ∈ R 1 lies between 0 and 1, which is then discretised to a binary open/close value.

Original: The combined action therefore is a = { e , q , h } .
Revised: The combined action therefore is a = { e , q , h } .

Original: To train this next-best pose agent, we use a modiﬁed version of SAC (Haarnoja et al., 2018) where we modify the soft Q-function (Equation 1) to be a conﬁdence-aware soft Q-function.
Revised: To train this next-best pose agent, we use a modiﬁed version of SAC (Haarnoja et al., 2018) where we modify the soft Q-function (Equation 1) to be a conﬁdence-aware soft Q-function.

Original: Recent work in 6D pose estimation (Wang et al., 2019; Wada et al., 2020) has seen the inclusion of a conﬁdence score c with the pose prediction output for each dense-pixel.
Revised: Recent work in 6D pose estimation (Wang et al., 2019; Wada et al., 2020) has seen the inclusion of a conﬁdence score c with the pose prediction output for each dense-pixel.

Original: Inspired by this, we augment our
Revised: Inspired by this, we augment our

Original: Q function with a per-pixel conﬁdence c ij , where we output a conﬁdence score for each Q-value prediction (resulting in a (16 × 16 × 2) output).
Revised: Q function with a per-pixel conﬁdence c ij , where we output a conﬁdence score for each Q-value prediction (resulting in a (16 × 16 × 2) output).

Original: To achieve this, we weight the per-pixel Bellman loss with the per-pixel conﬁdence, and add a conﬁdence regularisation term: 
Revised: To achieve this, we weight the per-pixel Bellman loss with the per-pixel conﬁdence, and add a conﬁdence regularisation term: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where s = ( b (cid:48) , p (cid:48) , z ) .
Revised: where s = ( b (cid:48) , p (cid:48) , z ) .

Original: With this, low conﬁdence will result in a low Bellman error but would incur a high penalty from the second term, and vice versa.
Revised: With this, low conﬁdence will result in a low Bellman error but would incur a high penalty from the second term, and vice versa.

Original: We use the Q value that has the highest conﬁdence when training the actor.
Revised: We use the Q value that has the highest conﬁdence when training the actor.

Original: As an aside, we also tried applying this conﬁdence-aware method to the policy, though empirically we found no improvement.
Revised: As an aside, we also tried applying this conﬁdence-aware method to the policy, though empirically we found no improvement.

Original: In practice we make use of the clipped double-Q trick (Fujimoto et al., 2018), which takes the minimum Q-value between two Q networks, but have omitted in the equations for brevity.
Revised: In practice we make use of the clipped double-Q trick (Fujimoto et al., 2018), which takes the minimum Q-value between two Q networks, but have omitted in the equations for brevity.

Original: Finally, the actor’s policy parameters can be optimised by minimising the loss as deﬁned in Equation 3.
Revised: Finally, the actor’s policy parameters can be optimised by minimising the loss as deﬁned in Equation 3.

Original: 4. C ONTROL A GENT
Revised: 4. C ONTROL A GENT

Original: Given the next-best pose suggestion from the previous stage, we give this to a goal-conditioned control function f ( s t , g t ) , which given state s t and goal g t , outputs motor velocities that drives theend-effector towards the goal.
Revised: Given the next-best pose suggestion from the previous stage, we give this to a goal-conditioned control function f ( s t , g t ) , which given state s t and goal g t , outputs motor velocities that drives the

Original: This function can take on many forms, but two noteworthy solutions would be either motion planning in combination with a feedback-control or a learnable policy trained with imitation/reinforcement learning.

Original: Given that the environmental dynamics are limited in the benchmark, we opted for the motion planning solution.

Original: Given the target pose, we perform path planning using the SBL (S´anchez & Latombe, 2003) planner within OMPL (S¸ucan et al., 2012), and use Reﬂexxes Motion Library for on-line trajectory generation.

Original: If the target pose is out of reach, we terminate the episode and supply a reward of − 1 .

Original: This path planning and trajectory generation is conveniently encapsulated by the ‘ABS EE POSE PLAN WORLD FRAME’ action mode in RLBench (James et al., 2020).

Original: 4. K EYFRAME S ELECTION & D EMO A UGMENTATION

Original: Initialise Q-attention networks QA θ 1 , QA θ 2 , critic networks Q πρ 1 , Q πρ 2 , and actor network π φ with random parameters θ 1 , θ 2 , ρ 1 , ρ 2 , φ
Revised: Initialise Q-attention networks QA θ 1 , QA θ 2 , critic networks Q πρ 1 , Q πρ 2 , and actor network π φ with random parameters θ 1 , θ 2 , ρ 1 , ρ 2 , φ

Original: Initialise target networks θ (cid:48) 1 ← θ 1 , θ (cid:48) 2 ← θ 2 , ρ (cid:48)
Revised: Initialise target networks θ (cid:48) 1 ← θ 1 , θ (cid:48) 2 ← θ 2 , ρ (cid:48)

Original: 1 ← ρ 1 , ρ (cid:48) 2 ← ρ
Revised: 1 ← ρ 1 , ρ (cid:48) 2 ← ρ

Original: Initialise replay buffer D with demos and apply keyframe selection and demo augmentation for each iteration do for each environment step do ( b t , p t , z t ) ← s t ( x t , y t ) ← argmax 2D a (cid:48)
Revised: Initialise replay buffer D with demos and apply keyframe selection and demo augmentation for each iteration do for each environment step do ( b t , p t , z t ) ← s t ( x t , y t ) ← argmax 2D a (cid:48)

Original: QA θ (( b t , p t ) , a (cid:48) )
Revised: QA θ (( b t , p t ) , a (cid:48) )

Original: (cid:46)
Revised: (cid:46)

Original: Use Q-attention to get pixel coords b (cid:48) t , p (cid:48)
Revised: Use Q-attention to get pixel coords b (cid:48) t , p (cid:48)

Original: t ← crop ( b t , p t , ( x t , y t )) a t ∼ π φ ( a t | ( b (cid:48) t , p (cid:48) t , z t ))
Revised: t ← crop ( b t , p t , ( x t , y t )) a t ∼ π φ ( a t | ( b (cid:48) t , p (cid:48) t , z t ))

Original: (cid:46)
Revised: (cid:46)

Original: Sample pose from the policy while target not reached do v ← f ( s , a t ) (cid:46) Get joint velocities from control agent s t +1 , r ← env.step ( v )
Revised: Sample pose from the policy while target not reached do v ← f ( s , a t ) (cid:46) Get joint velocities from control agent s t +1 , r ← env.step ( v )

Original: D ← D ∪ { ( s t , a t , r , s t +1 , ( x t , y t ))
Revised: D ← D ∪ { ( s t , a t , r , s t +1 , ( x t , y t ))

Original: } (cid:46) Store the transition in the replay pool for each gradient step do θ i ← θ i − λ QA ˆ ∇
Revised: } (cid:46) Store the transition in the replay pool for each gradient step do θ i ← θ i − λ QA ˆ ∇

Original: θ
Revised: θ

Original: i J QA ( θ i ) for i ∈ { 1 , 2 } (cid:46) Update Q-attention parameters
Revised: i J QA ( θ i ) for i ∈ { 1 , 2 } (cid:46) Update Q-attention parameters

Original: ρ i ← ρ
Revised: ρ i ← ρ

Original: i − λ Q π ˆ ∇ ρ i J Q π ( ρ i ) for i ∈ { 1 , 2 } (cid:46) Update critic parameters φ ← φ − λ π ˆ ∇ φ J π ( φ ) (cid:46) Update policy weights θ (cid:48)
Revised: i − λ Q π ˆ ∇ ρ i J Q π ( ρ i ) for i ∈ { 1 , 2 } (cid:46) Update critic parameters φ ← φ − λ π ˆ ∇ φ J π ( φ ) (cid:46) Update policy weights θ (cid:48)

Original: i ←
Revised: i ←

Original: τθ i + (1 − τ ) θ (cid:48)
Revised: τθ i + (1 − τ ) θ (cid:48)

Revised: i for i ∈ { 1 , 2 } (cid:46) Update Q-attention target network weights ρ (cid:48)

Original: i for i ∈ {
Revised: i ←

Original: [Figure] (cid:46) [Figure]
Revised: τρ i + (1 − τ ) ρ (cid:48)

Original: Q
Revised: i for i ∈ { 1 , 2 } (cid:46) Update critic target network weights end-effector towards the goal. This function can take on many forms, but two noteworthy solutions would be either motion planning in combination with a feedback-control or a learnable policy trained with imitation/reinforcement learning. Given that the environmental dynamics are limited in the benchmark, we opted for the motion planning solution.

Original: [Figure]
Revised: Given the target pose, we perform path planning using the SBL (S ´ anchez & Latombe, 2003) planner within OMPL (S¸ucan et al., 2012), and use Reﬂexxes Motion Library for on-line trajectory generation. If the target pose is out of reach, we terminate the episode and supply a reward of − 1 . This path planning and trajectory generation is conveniently encapsulated by the ‘ABS EE POSE PLAN WORLD FRAME’ action mode in RLBench (James et al., 2020). 4. K EYFRAME S ELECTION & D EMO A UGMENTATION

Original: In this section, we outline how we maximise the utility of given demonstrations in order to complete sparsely reward tasks.
Revised: In this section, we outline how we maximise the utility of given demonstrations in order to complete sparsely reward tasks.

Original: We assume to have a teacher policy π ∗ (e.g. motion planners or human teleoperatives) that can generate trajectories consisting of a series of states and actions: τ =
Revised: We assume to have a teacher policy π ∗ (e.g. motion planners or human teleoperatives) that can generate trajectories consisting of a series of states and actions: τ =

Original: [( s 1 , a 1 ) , . .
Revised: [( s 1 , a 1 ) , . .

Original: , ( s T , a T )] .
Revised: , ( s T , a T )] .

Original: In this case, we assume that the demonstrations come from RLBench (James et al., 2020).
Revised: In this case, we assume that the demonstrations come from RLBench (James et al., 2020).

Original: The keyframe selection process iterates over each of the demo trajectories τ and runs each of the stateaction pairs ( s , a ) through a function K : R D
Revised: The keyframe selection process iterates over each of the demo trajectories τ and runs each of the stateaction pairs ( s , a ) through a function K : R D

Original: → B which outputs a boolean deciding if the given trajectory point should be treated as a keyframe.
Revised: → B which outputs a boolean deciding if the given trajectory point should be treated as a keyframe.

Original: The keyframe function K could include a number of constraints.
Revised: The keyframe function K could include a number of constraints.

Original: In practice we found that performing a disjunction over two simple conditions worked well; these included (1) change in gripper state (a common occurrence when something is grasped or released), and (2) velocities approaching near zero (a common occurrence when entering pre-grasp poses or entering a new phase of a task).
Revised: In practice we found that performing a disjunction over two simple conditions worked well; these included (1) change in gripper state (a common occurrence when something is grasped or released), and (2) velocities approaching near zero (a common occurrence when entering pre-grasp poses or entering a new phase of a task).

Original: It is likely that as tasks get more complex, K will inevitably need to become more sophisticated via learning or simply through more conditions, e.g. sudden changes in direction or joint velocity, large changes in pixel values, etc.
Revised: It is likely that as tasks get more complex, K will inevitably need to become more sophisticated via learning or simply through more conditions, e.g. sudden changes in direction or joint velocity, large changes in pixel values, etc.

Revised: Figure 5 shows RGB observations from the keyframe selection process fromtasks.

Original: At each keyframe, we use the known camera intrinsics and extrinsics to project the end-effector pose at state s t +1 into the image plane of state s t , giving us pixel locations of the end-effector at the next keyframe.
Revised: At each keyframe, we use the known camera intrinsics and extrinsics to project the end-effector pose at state s t +1 into the image plane of state s t , giving us pixel locations of the end-effector at the next keyframe.

Original: This stage is crucial to breaking the cyclic dependency (mentioned in Section 4.1) between the Q-attention and next-best pose agent, as these projected pixel coordinates act as optimal actions for the Q-attention agent.
Revised: This stage is crucial to breaking the cyclic dependency (mentioned in Section 4.1) between the Q-attention and next-best pose agent, as these projected pixel coordinates act as optimal actions for the Q-attention agent.

Original: Using this keyframe selection method, each trajectory results in N = length ( keyframes ) transitions being stored into the replay buffer.
Revised: Using this keyframe selection method, each trajectory results in N = length ( keyframes ) transitions being stored into the replay buffer.

Original: To further increase the utility of demonstrations, we apply demo augmentation which stores the transition from an intermediate point along the trajectories to the keyframe states.
Revised: To further increase the utility of demonstrations, we apply demo augmentation which stores the transition from an intermediate point along the trajectories to the keyframe states.

Original: Formally, for each point ( s t , a t ) along the trajectory starting from keyframe k
Revised: Formally, for each point ( s t , a t ) along the trajectory starting from keyframe k

Original: i , we calculate the transformation of the end-effector pose (taken from s t ) at time step t to the end-effector pose at the time step associated with keyframe
Revised: i , we calculate the transformation of the end-effector pose (taken from s t ) at time step t to the end-effector pose at the time step associated with keyframe

Original: k
Revised: k

Original: i
Revised: i

Original: + i .
Revised: + i .

Original: This transformation can then be used as the action for the next-best pose agent.
Revised: This transformation can then be used as the action for the next-best pose agent.

Original: We repeat this process for every M th point along the trajectory (which we set to M = 5 ).
Revised: We repeat this process for every M th point along the trajectory (which we set to M = 5 ).

Original: The keyframe selection and demo augmentation is visualised in Figure 4.
Revised: The keyframe selection and demo augmentation is visualised in Figure 4.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we aim to answer the following questions: (1) Are we able to successfully learn across a range of sparsely-rewarded manipulation tasks?
Revised: In this section, we aim to answer the following questions: (1) Are we able to successfully learn across a range of sparsely-rewarded manipulation tasks?

Original: (2) Which of our proposed components contribute the most to our success?
Revised: (2) Which of our proposed components contribute the most to our success?

Original: (3) How sensitive is our method to the number of demonstrations?
Revised: (3) How sensitive is our method to the number of demonstrations and to the crop size?

Original: To answer these, we benchmark our approach using RLBench (James et al., 2020).
Revised: To answer these, we benchmark our approach using RLBench (James et al., 2020).

Original: Of the 100 tasks, we select 10 (shown in Figure 1) that we believe to be achievable from using only the front-facing camera.
Revised: Of the 100 tasks, we select 10 (shown in Figure 1) that we believe to be achievable from using only the front-facing camera.

Original: We leave tasks that require multiple cameras to future work.
Revised: We leave tasks that require multiple cameras to future work.

Original: RLBench was chosen due to its emphasis on vision-based manipulation benchmarking and because it gives access to a wide variety of tasks with demonstrations.
Revised: RLBench was chosen due to its emphasis on vision-based manipulation benchmarking and because it gives access to a wide variety of tasks with demonstrations.

Revised: Each task has a completely sparse reward of +1 which is given only on task completion, and 0 otherwise.

Original: The ﬁrst of our questions can be answered by attending to Figure 5.
Revised: The ﬁrst of our questions can be answered by attending to Figure 6.

Original: All baseline algorithms (SAC,
Revised: All baseline algorithms (SAC,

Original: TD3 and QT-Opt) are in their ‘vanilla’ form, and do not contain any of our proposed contributions:
Revised: TD3 and QT-Opt) are in their ‘vanilla’ form, and do not contain any of our proposed contributions:

Original: Q-attention, conﬁdence-aware critic, keyframe selection, and demo augmentation.
Revised: Q-attention, conﬁdence-aware critic, keyframe selection, and demo augmentation.

Original: All methods receive the exact same 200 demonstration sequences, which are loaded into the replay buffer prior to training.
Revised: All methods receive the exact same 200 demonstration sequences, which are loaded into the replay buffer prior to training.

Original: The baseline agents are architecturally similar to the next-best pose agent, but with a few differences to account for missing Q-attention (and so receives the full, uncropped RGB and  point cloud data) and missing conﬁdence-aware critic (and so outputs single Q-values rather than per-pixel values).
Revised: The baseline agents are architecturally similar to the next-best pose agent, but with a few differences to account for missing Q-attention (and so receives the full, uncropped RGB and organised point cloud data) and missing conﬁdence-aware critic (and so outputs single Q-values rather than per-pixel values).

Original: Speciﬁcally, the architecture uses the same RGB and point cloud fusion as shown in Figure 2.
Revised: Speciﬁcally, the architecture uses the same RGB and point cloud fusion as shown in Figure 2.

Original: Feature maps from the shared representation are concatenated with the reshaped proprioceptive input and fed to both the actor and critic.
Revised: Feature maps from the shared representation are concatenated with the reshaped proprioceptive input and fed to both the actor and critic.

Original: The baseline actor uses 3 convolution layers (64 channels, 3 × 3 ﬁlter size, 2 stride), who’s output feature maps are maxpooled and sent through 2 dense layers (64 nodes) and results in an action distribution output.
Revised: The baseline actor usesconvolution layers (64 channels, 3 × 3 ﬁlter size, 2 stride), who’s output feature maps are maxpooled and sent through 2 dense layers (64 nodes) and results in an action distribution output.

Original: The critic baseline uses 3 residual convolution blocks (128 channels, 3 × 3 ﬁlter size, 2 stride), who’s output feature maps are maxpooled and sent through 2 dense layers (64 nodes) and results in a single Q-value output.
Revised: The critic baseline uses 3 residual convolution blocks (128 channels, 3 × 3 ﬁlter size, 2 stride), who’s output feature maps are maxpooled and sent through 2 dense layers (64 nodes) and results in a single Qvalue output.

Original: All methods use the LeakyReLU activation, layer normalisation in the convolution layers, learning rate of 3 × 10 − [Figure] and τ = 5 − 4  .
Revised: All methods use the LeakyReLU activation, layer normalisation in the convolution layers, learning rate of 3 × 10 − 3 , soft target update of τ = 5 − 4 , and a reward scaling of 100 .

Original: Training and exploration were done asynchronously with a single agent (to emulate a real-world robot training scenario) that would continuously load checkpoints every 100 training steps.
Revised: Training and exploration were done asynchronously with a single agent (to emulate a real-world robot training scenario) that would continuously load checkpoints every 100 training steps.

Original: The results in Figure 5 show that baseline state-of-the-art methods are unable to accomplish any
Revised: The results in Figure 6 show that baseline state-of-the-art methods are unable to accomplish any

Original: RLBench tasks, whilst our method is able to accomplish the tasks in small number of environment steps; 5 , 000 environment steps equating to about an hour of robot interaction time (meaning ‘take lid off saucepan’ being solved in about two hours).
Revised: RLBench tasks, whilst our method is able to accomplish the tasks in small number of environment steps; 5 , 000 environment steps equating to about an hour of robot interaction time (meaning ‘take lid off saucepan’ being solved in about two hours).

Original: We suggest that the reason why our results starkly outperform other state-of-the-art methods is because of two key reasons that go hand-inhand: (1) Reducing the input dimensionality through Q-attention that immensely reduces the burden on the (often difﬁcult and unstable to train) continuous control algorithm; (2) Combining this with our keyframe selection method that enables the Q-attention network to quickly converge and suggest meaningful points of interest to the next-best pose agent.
Revised: We suggest that the reason why our results starkly outperform other state-of-the-art methods is because of two key reasons that go hand-inhand: (1) Reducing the input dimensionality through Q-attention that immensely reduces the burden on the (often difﬁcult and unstable to train) continuous control algorithm; (2) Combining this with our keyframe selection method that enables the Q-attention network to quickly converge and suggest meaningful points of interest to the next-best pose agent.

Original: We wish to stress that perhaps given enough training time some of these baseline methods may eventually start to succeed, however we found no evidence of this.
Revised: We wish to stress that perhaps given enough training time some of these baseline methods may eventually start to succeed, however we found no evidence of this.

Original: In Figure 6a, we perform an ablation study to evaluate which of the proposed components contribute the most to the success of our method.
Revised: In Figure 7a, we perform an ablation study to evaluate which of the proposed components contribute the most to the success of our method.

Original: To perform this ablation, we chose 2 tasks of varying difﬁculty: ‘take lid off saucepan’ and ‘put rubbish in bin’ .
Revised: To perform this ablation, we chose 2 tasks of varying difﬁculty: ‘take lid off saucepan’ and ‘put rubbish in bin’ .

Original: The ablation clearly shows that the Q-attention (combined with keyframe selection) is crucial to achieving the tasks, whilst the demo augmentation and conﬁdence-aware critic aid in overall stability and increase ﬁnal performance.
Revised: The ablation clearly shows that the Q-attention (combined with keyframe selection) is crucial to achieving the tasks, whilst the demo augmentation, conﬁdence-aware critic, and Q regularisation aid in overall stability and increase ﬁnal performance.

Revised: When swapping the Q-attention module with a soft attention Xu et al.

Revised: (2015) module, we found that performance was similar to that of the ‘vanilla’ baselines.

Revised: This result is unsurprising, as soft attention is implicitly learned (i.e. without an explicit loss), where as our Q-attention is explicitly learned via (off-policy) Q-learning, and so it can make greater use of the highly-informative keyframes given from the keyframe selection process.

Revised: Note that we cannot compare to ‘traditional’ hard attention because it requires on-policy training, as explained in Section 4.1.

Original: Our ﬁnal set of experiments in Figure 6b show how robust our method is when varying the number of demonstrations given.
Revised: Figure 7b shows how robust our method is when varying the number of demonstrations given.

Original: The results show that our method performs robustly, even when given 50% fewer demos, however as the task difﬁculty increase (from ‘take lid off saucepan’ to ‘put rubbish in bin’ ), the harmful effect of having less demonstrations is more severe. 
Revised: The results show that our method performs robustly, even when given 50% fewer demos, however as the task difﬁculty increase (from ‘take lid off saucepan’ to ‘put rubbish in bin’ ), the harmful effect of having less demonstrations is more severe. Our ﬁnal set of experiments in Figure 7c shows how our [Figure] method performs across varying crop sizes.

Original:   C ONCLUSION
Revised: As the task difﬁculty increases, the harmful effect of a larger crop size becomes more prominent; suggesting that one of the key beneﬁts of the Q-attention is to drastically reduce the input size to the next-best pose agent, making the RL optimisation much easier. It’s clear that a trade-off must be made between choosing smaller crops to increase training size, and choosing larger crops to incorporate more of the surrounding area. We found that setting the crops at 16 × 16 across all tasks performed well.  C ONCLUSION

Original: We have presented our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of real-world sparsely-rewarded tasks.
Revised: We have presented our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of real-world sparsely-rewarded tasks.

Original: We validate our method on 10 RLBench tasks of varying difﬁculty, and show that many commonly used state-of-the-art methods catastrophically fail.
Revised: We validate our method on 10 RLBench tasks of varying difﬁculty, and show that many commonly used state-of-the-art methods catastrophically fail.

Original: We show that Q-attention (along with the keyframe selection) is key to our success, whilst the conﬁdence-aware critic and demo augmentation contribute to achieving high ﬁnal performance.
Revised: We show that Q-attention (along with the keyframe selection) is key to our success, whilst the conﬁdence-aware critic and demo augmentation contribute to achieving high ﬁnal performance.

Original: Despite our strong experimental results, there are undoubtedly areas of weakness.
Revised: Despite our strong experimental results, there are undoubtedly areas of weakness.

Original: The control agent (ﬁnal agent in the pipeline) uses path planning and on-line trajectory generation, which for these tasks are adequate; however, this would need to be replaced with an alternative agent for tasks that have dynamic environments (e.g. moving target objects, moving obstacles, etc) or complex contact dynamics (e.g. peg-in-hole).
Revised: The control agent (ﬁnal agent in the pipeline) uses path planning and on-line trajectory generation, which for these tasks are adequate; however, this would need to be replaced with an alternative agent for tasks that have dynamic environments (e.g. moving target objects, moving obstacles, etc) or complex contact dynamics (e.g. peg-in-hole).

Original: We look to future work for swapping this with a goal-conditioned reinforcement learning policy, or similar.
Revised: We look to future work for swapping this with a goal-conditioned reinforcement learning policy, or similar.

Original: Another weakness is that we only evaluate on tasks that can be done with the front-facing camera; however we are keen to explore many of the other tasks RLBench has to offer by adapting the method to accommodate multiple camera inputs in future work.
Revised: Another weakness is that we only evaluate on tasks that can be done with the front-facing camera; however we are keen to explore many of the other tasks RLBench has to offer by adapting the method to accommodate multiple camera inputs in future work.

Original: Q- ATTENTION D ISCUSSION

Original: A key difference between typical attention and Q-attention, is that we crop the observations, and not the features.

Original: This is crucial, as in hard-attention style approaches, we are unable to propagate gradients back to the attention module (due to undifferentiable cropping); this leaves the next-best pose with the difﬁcult task of interpreting dynamically changing features that is has no inﬂuence on.

Original: By instead cropping the observation we alleviate this problem, as the next-best pose agent can now extract its own, bespoke features.

Original: Moreover it allows for greater separation of responsibility where the Q-attention agent can focus on high-level task understanding, while the next-best pose agent can focus on the lower-level manipulation/grasp understanding.

Original: When swapping the Q-attention module with a soft-attention Xu et al.

Original: (2015) module, we found that performance was similar to that of the ‘vanilla’ baselines shown in Figure 5.

Original: This result is unsurprising, as soft-attention has no way of making use of the highly-informative keyframe selection during the training process.

Original: To summarise, soft-attention is not ideal because it does not make use of keyframe selection, while hard-attention has to interpret dynamically changing features that is has no inﬂuence on.