Original: T OWARD E FFECTIVE D EEP R EINFORCEMENT L EARNING FOR 3D R OBOTIC M ANIPULATION : E ND - TO -E ND L EARNING FROM M ULTIMODAL R AW S ENSORY D ATA
Revised: Toward Effective Deep Reinforcement Learning for 3D Robotic Manipulation: Multimodal End-to-End Reinforcement Learning from Visual and Proprioceptive Feedback

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: Abstract

Original: Sample-efficient reinforcement learning (RL) methods capable of learning directly from raw sensory data without the use of human-crafted representations would open up real-world applications in robotics and control.
Revised: Sample-efficient reinforcement learning (RL) methods capable of learning directly from raw sensory data without the use of human-crafted representations would open up real-world applications in robotics and control.

Original: Recent advances in visual
Revised: Recent advances in visual

Original: RL have shown that learning a latent representation together with existing RL algorithms closes the gap between state-based and image-based training.
Revised: RL have shown that learning a latent representation together with existing RL algorithms closes the gap between state-based and image-based training.

Original: However, image-based training is still significantly sample-inefficient with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training.
Revised: However, image-based training is still significantly sample-inefficient with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training.

Original: In this study, we propose an effective model-free off-policy
Revised: In this study, we propose an effective model-free off-policy

Original: RL method for 3D robotic manipulation that can be trained in an end-to-end manner from multimodal raw sensory data obtained from a vision camera and a robot’s joint encoders, without the need for human-crafted representations.
Revised: RL method for 3D robotic manipulation that can be trained in an end-to-end manner from multimodal raw sensory data obtained from a vision camera and a robot’s joint encoders, without the need for human-crafted representations.

Original: Notably, our method is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.
Revised: Notably, our method is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.

Original: Our method, which we dub MERL :
Revised: Our method, which we dub MERL :

Original: M ultimodal E nd-to-end R einforcement L earning, results in a simple but effective approach capable of significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DeepMind Control.
Revised: M ultimodal E nd-to-end R einforcement L earning, results in a simple but effective approach capable of significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DeepMind Control.

Original:  I NTRODUCTION
Revised:  Introduction

Original: Deep reinforcement learning (deep RL), the effective combination of RL and deep learning, has allowed RL methods to attain remarkable results across a wide range of domains, including board and video games under discrete action space (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019) and robotics and control under continuous action space (Levine et al., 2016; Zhu et al., 2020;
Revised: Deep reinforcement learning (deep RL), the effective combination of RL and deep learning, has allowed RL methods to attain remarkable results across a wide range of domains, including board and video games under discrete action space (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019) and robotics and control under continuous action space (Levine et al., 2016; Zhu et al., 2020;

Original: Kalashnikov et al., 2021; Ibarz et al., 2021; Kroemer et al., 2021).
Revised: Kalashnikov et al., 2021; Ibarz et al., 2021; Kroemer et al., 2021).

Original: Many deep RL studies use human-crafted representations, for it is commonly known that state-based training operating on a coordinate state is significantly more sample-efficient than raw sensory data–based training (for example, image-based training).
Revised: Many deep RL studies use human-crafted representations, for it is commonly known that state-based training operating on a coordinate state is significantly more sample-efficient than raw sensory data–based training (for example, image-based training).

Original: However, the use of human-crafted representations poses several major limitations and issues for 3D robotic manipulation: (a) human-crafted representations cannot perfectly represent the robot environment; (b) in the case of real-world applications, a separate module for environmental perception is required to obtain the environment state; and (c) state-based training based on human-crafted representations is not capable of using a neural network architecture repeatedly for different tasks, even for small variations in the environment such as changes in the number, size, or shape of objects.
Revised: However, the use of human-crafted representations poses several major limitations and issues for 3D robotic manipulation: (a) human-crafted representations cannot perfectly represent the robot environment; (b) in the case of real-world applications, a separate module for environmental perception is required to obtain the environment state; and (c) state-based training based on human-crafted representations is not capable of using a neural network architecture repeatedly for different tasks, even for small variations in the environment such as changes in the number, size, or shape of objects.

Original: Over the last three years, the RL community has made significant headway on these limitations and issues by significantly improving sample efficiency in image-based training (Hafner et al., 2019a;b; 2020; 2022; Lee et al., 2020a; Srinivas et al., 2020; Yarats et al., 2020; 2021a;b;c).
Revised: Over the last three years, the RL community has made significant headway on these limitations and issues by significantly improving sample efficiency in image-based training (Hafner et al., 2019a,b, 2020, 2022; Lee et al., 2020a; Srinivas et al., 2020; Yarats et al., 2020, 2021a,b,c).

Original: A key insight of such studies is that the learning of better low-dimensional representations from image pixels is achieved through an autoencoder (Yarats et al., 2021c), variational inference (Hafner et al., 2019a;b; 2020; 2022; Lee et al., 2020a), contrastive learning (Srinivas et al., 2020; Yarats et al., 2021b), or data augmentation (Yarats et al., 2020; 2021a), which in turn has helped to improve sample efficiency significantly.
Revised: A key insight of such studies is that the learning of better low-dimensional representations from image pixels is achieved through an autoencoder (Yarats et al., 2021c), variational inference (Hafner et al., 2019a,b, 2020, 2022; Lee et al., 2020a), contrastive learning (Srinivas et al., 2020; Yarats et al., 2021b), or data augmentation (Yarats et al., 2020, 2021a), which in turn has helped to improve sample efficiency significantly.

Original: Recently, some visual RL studies have solved 3D continuous control problems such as quadruped and humanoid locomotion tasks from the DeepMind Control (DMC) suite, which in turn has helped to bridge the gap between state-based and image-based training (Hafner et al., 2020;
Revised: Recently, some visual RL studies have solved 3D continuous control problems such as quadruped and humanoid locomotion tasks from the DeepMind Control (DMC) suite, which in turn has helped to bridge the gap between state-based and image-based training (Hafner et al., 2020;

Original: Yarats et al., 2021a).
Revised: Yarats et al., 2021a).

Original: Despite such significant progress in visual RL, image-based training is still notably sample inefficient with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training.
Revised: Despite such significant progress in visual RL, image-based training is still notably sample inefficient with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training.

Original: In this paper, we propose an effective deep RL method for 3D robotic manipulation, which we dub MERL :
Revised: In this paper, we propose an effective deep RL method for 3D robotic manipulation, which we dub MERL :

Original: M ultimodal E nd-to-end R einforcement L earning.
Revised: M ultimodal E nd-to-end R einforcement L earning.

Original: MERL is a simple but effective model-free RL method that can be trained in an end-to-end manner from two different types of raw sensory data (RGB images and proprioception) having a multimodality that differs in dimensionality and value range, without the need for human-crafted representations.
Revised: MERL is a simple but effective model-free RL method that can be trained in an end-to-end manner from two different types of raw sensory data (RGB images and proprioception) having a multimodality that differs in dimensionality and value range, without the need for human-crafted representations.

Original: Notably, MERL is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.
Revised: Notably, MERL is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.

Original: Compared to current state-of-the-art visual RL and state-based RL methods, MERL provides significant improvements in sample efficiency, learning performance, and training stability in relation to three 3D robotic manipulation tasks from DMC ( jaco reach duplo , jaco move box , and jaco lift box ) (Tunyasuvunakool et al., 2020).
Revised: Compared to current state-of-the-art visual RL and state-based RL methods, MERL provides significant improvements in sample efficiency, learning performance, and training stability in relation to three 3D robotic manipulation tasks from DMC ( jaco reach duplo , jaco move box , and jaco lift box ) (Tunyasuvunakool et al., 2020).

Original: To the best of our knowledge, MERL is the first model-free off-policy method not only to learn a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, but also to show a new state-of-the-art performance by significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability.
Revised: To the best of our knowledge, MERL is the first model-free off-policy method not only to learn a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, but also to show a new state-of-the-art performance by significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability.

Original: The main contributions of the paper can be summarized as follows: (1) the introduction of an endto-end approach learning directly from multimodal raw sensory data for the efficient learning of a policy for use in the field of 3D robotic manipulation; (2) a demonstration of the fact that the approach significantly outperforms current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to three 3D robotic manipulation tasks from DMC; and (3) the provision of a deep RL method capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.
Revised: The main contributions of the paper can be summarized as follows: (1) the introduction of an endto-end approach learning directly from multimodal raw sensory data for the efficient learning of a policy for use in the field of 3D robotic manipulation; (2) a demonstration of the fact that the approach significantly outperforms current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to three 3D robotic manipulation tasks from DMC; and (3) the provision of a deep RL method capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.

Original:  R ELATED WORK 2. R EINFORCEMENT L EARNING FOR 3D R OBOTIC M ANIPULATION
Revised:  Related work 2.1 Reinforcement Learning for 3D Robotic Manipulation

Original: Deep RL has seen widespread success across a variety of domains, including board and video games and robotics and control (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019; Levine et al., 2016; Zhu et al., 2020;
Revised: Deep RL has seen widespread success across a variety of domains, including board and video games and robotics and control (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019; Levine et al., 2016; Zhu et al., 2020;

Original: Kalashnikov et al., 2021; Ibarz et al., 2021; Kroemer et al., 2021).
Revised: Kalashnikov et al., 2021; Ibarz et al., 2021; Kroemer et al., 2021).

Original: In recent years, a number of deep RL methods have been successfully applied to 3D robotic manipulation tasks ranging from ‘easy’ (for example, reach target ) to ‘hard’ (for example, assembly ) (Yamada et al., 2020; Lee et al., 2019b; 2021b;
Revised: In recent years, a number of deep RL methods have been successfully applied to 3D robotic manipulation tasks ranging from ‘easy’ (for example, reach target ) to ‘hard’ (for example, assembly ) (Yamada et al., 2020; Lee et al., 2019b, 2021b;

Original: Nam et al., 2022).
Revised: Nam et al., 2022).

Original: However, many such methods require hand-engineered components for perception, state estimation, and low-level control, for they learn from human-crafted representations.
Revised: However, many such methods require hand-engineered components for perception, state estimation, and low-level control, for they learn from human-crafted representations.

Original: Such methods are commonly referred to as state-based RL.
Revised: Such methods are commonly referred to as state-based RL.

Original: Over the last three years, visual RL that learns directly from image pixels has been greatly advanced and has seen significant improvements in sample efficiency in 3D continuous control problems, which in turn has helped to bridge the gap between state-based and image-based training (Hafner et al., 2019a;b; 2020; 2022; Srinivas et al., 2020; Yarats et al., 2020; 2021a;b;c; Wu et al., 2022).
Revised: Over the last three years, visual RL that learns directly from image pixels has been greatly advanced and has seen significant improvements in sample efficiency in 3D continuous control problems, which in turn has helped to bridge the gap between state-based and image-based training (Hafner et al., 2019a,b, 2020, 2022; Srinivas et al., 2020; Yarats et al., 2020, 2021a,b,c; Wu et al., 2022).

Original: A key insight of such studies is that the learning of better low-dimensional representations from image pixels is achieved through various techniques such as an autoencoder, variational inference, contrastive learning, or data augmentation.
Revised: A key insight of such studies is that the learning of better low-dimensional representations from image pixels is achieved through various techniques such as an autoencoder, variational inference, contrastive learning, or data augmentation.

Original: Currently, DrQ-v2 (Yarats et al., 2021a) shows a state-of-the-art performance in visual RL studies by solving complex 3D locomotion tasks from the DMC suite (Tassa et al., 2018) such as humanoid locomotion , previously unattained by model-free visual RL.
Revised: Currently, DrQ-v2 (Yarats et al., 2021a) shows a state-of-the-art performance in visual RL studies by solving complex 3D locomotion tasks from the DMC suite (Tassa et al., 2018) such as humanoid locomotion , previously unattained by model-free visual RL.

Original: In this study, we propose MERL, a deep RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, including RGB images and proprioception.
Revised: In this study, we propose MERL, a deep RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, including RGB images and proprioception.

Original: Here, the RGB images and proprioception are obtained from a fixed vision camera and a robot’s joint encoders, respectively.
Revised: Here, the RGB images and proprioception are obtained from a fixed vision camera and a robot’s joint encoders, respectively.

Original: Experimental results on 3D robotic manipulation tasks from DMC (Tunyasuvunakool et al., 2020)
Revised: Experimental results on 3D robotic manipulation tasks from DMC (Tunyasuvunakool et al., 2020)

Original: show that MERL significantly improves sample efficiency compared to state-based RL and DrQ-v (Yarats et al., 2021a), the current state-of-the-art visual RL.
Revised: show that MERL significantly improves sample efficiency compared to state-based RL and DrQ-v (Yarats et al., 2021a), the current state-of-the-art visual RL.

Original: 2. R EPRESENTATION L EARNING FOR M ULTIMODAL I NPUTS
Revised: 2.2 Representation Learning for Multimodal Inputs

Original: The complementary nature of heterogeneous sensor modalities, such as vision, audio, language, haptic, range, and proprioceptive data, has previously been explored with respect to perception and decision-making.
Revised: The complementary nature of heterogeneous sensor modalities, such as vision, audio, language, haptic, range, and proprioceptive data, has previously been explored with respect to perception and decision-making.

Original: For instance, many studies have explored the correlation between such different modalities, including the correlation between visual and auditory data in relation to speech recognition (Yang et al., 2017; Afouras et al., 2018) or sound source localization (Tian et al., 2018;
Revised: For instance, many studies have explored the correlation between such different modalities, including the correlation between visual and auditory data in relation to speech recognition (Yang et al., 2017; Afouras et al., 2018) or sound source localization (Tian et al., 2018;

Original: Owens &
Revised: Owens &

Original: Efros, 2018); the correlation between visual and haptic data for grasping (Calandra et al., 2018; Narita & Kroemer, 2021) or manipulation (Lee et al., 2019a; 2020b; 2021a); the correlation between visual and tactile data for object tracking (Yu & Rodriguez, 2018; Lambert et al., 2019) or shape completion (Wang et al., 2018); and the correlation between visual and ranging data for robot navigation (Liu et al., 2017; Cai et al., 2020).
Revised: Efros, 2018); the correlation between visual and haptic data for grasping (Calandra et al., 2018; Narita & Kroemer, 2021) or manipulation (Lee et al., 2019a, 2020b, 2021a); the correlation between visual and tactile data for object tracking (Yu & Rodriguez, 2018; Lambert et al., 2019) or shape completion (Wang et al., 2018); and the correlation between visual and ranging data for robot navigation (Liu et al., 2017; Cai et al., 2020).

Original: While many of these studies have contributed to improvements in perception performance through multimodal representation learning, in this study, we are interested in improving the performance of RL through multimodal representation learning; in particular, the efficient joint learning of a latent multimodal representation and a policy.
Revised: While many of these studies have contributed to improvements in perception performance through multimodal representation learning, in this study, we are interested in improving the performance of RL through multimodal representation learning; in particular, the efficient joint learning of a latent multimodal representation and a policy.

Original: In addition, some neuroscience studies have proved that the interdependence and concurrency of different sensory inputs aid perception and manipulation (Edelman, 1987; Lacey & Sathian, 2016; Bohg et al., 2017).
Revised: In addition, some neuroscience studies have proved that the interdependence and concurrency of different sensory inputs aid perception and manipulation (Edelman, 1987; Lacey & Sathian, 2016; Bohg et al., 2017).

Original: Accordingly, we use two heterogeneous sensory data: RGB images and proprioception (for example, joint angles and velocities) to efficiently learn a policy for 3D robotic manipulation.
Revised: Accordingly, we use two heterogeneous sensory data: RGB images and proprioception (for example, joint angles and velocities) to efficiently learn a policy for 3D robotic manipulation.

Original:  B ACKGROUND 3. E ND - TO -E ND R EINFORCEMENT L EARNING FROM M ULTIMODAL R AW S ENSORY D ATA
Revised:  Background 3.1 End-to-End Reinforcement Learning from Multimodal Raw Sensory Data

Original: We formulate continuous control problems on the basis of multimodal raw sensory data (that is,
Revised: We formulate continuous control problems on the basis of multimodal raw sensory data (that is,

Original: RGB image and proprioception) as an infinite-horizon Markov decision process (MDP) (Bellman, 1957).
Revised: RGB image and proprioception) as an infinite-horizon Markov decision process (MDP) (Bellman, 1957).

Original: In such a setting, we stack three consecutive prior RGB images to properly approximate the environment’s underlying state (Mnih et al., 2013).
Revised: In such a setting, we stack three consecutive prior RGB images to properly approximate the environment’s underlying state (Mnih et al., 2013).

Original: Accordingly, such an MDP can be described as a tuple ( O , A , P, R, γ, p 0 ) , where O = {O image , O prop } denotes the state space, including the three consecutive high-dimensional RGB images, O image , and the low-dimensional proprioception vector,
Revised: Accordingly, such an MDP can be described as a tuple ( O , A , P, R, γ, p 0 ) , where O = {O image , O prop } denotes the state space, including the three consecutive high-dimensional RGB images, O image , and the low-dimensional proprioception vector,

Original: O prop ; A denotes the action space; P : O × A → ∆( O ) denotes the transition dynamics that define a probability distribution over the next state given the current state and action; R : O × A → [0 , 1] denotes the reward function that maps the current state and action to a reward; γ ∈ [0 , 1) denotes a discount factor; and p 0 ∈ ∆( O ) denotes the probability distribution of the initial state o 0 .
Revised: O prop ; A denotes the action space; P : O × A → ∆( O ) denotes the transition dynamics that define a probability distribution over the next state given the current state and action; R : O × A → [0 , 1] denotes the reward function that maps the current state and action to a reward; γ ∈ [0 , 1) denotes a discount factor; and p 0 ∈ ∆( O ) denotes the probability distribution of the initial state o 0 .

Original: The goal is to find a policy, π : O → ∆( A ) , that maximizes the expected discounted sum of rewards, E π [ (cid:80) ∞ t =1 γ t r t ] , where o 0 ∼ p 0 , and ∀ t we have a t ∼ π ( · | o t ) , o t +1 ∼ p ( ·
Revised: The goal is to find a policy, π : O → ∆( A ) , that maximizes the expected discounted sum of rewards, E π [ (cid:80) ∞ t =1 γ t r t ] , where o 0 ∼ p 0 , and ∀ t we have a t ∼ π ( · | o t ) , o t +1 ∼ p ( ·

Original: | o t , a t ) , and r t =
Revised: | o t , a t ) , and r t =

Original: R ( o t , a t ) .
Revised: R ( o t , a t ) .

Original: 3. D EEP D ETERMINISTIC P OLICY G RADIENT
Revised: 3.2 Deep Deterministic Policy Gradient

Original: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) is an off-policy actor–critic RL algorithm for continuous control that concurrently learns a Q-function, Q ϕ , and a deterministic policy, π θ .
Revised: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) is an off-policy actor–critic RL algorithm for continuous control that concurrently learns a Q-function, Q ϕ , and a deterministic policy, π θ .

Original: For this, DDPG uses Q-learning (Watkins & Dayan, 1992) to learn Q ϕ by minimizing the one-step
Revised: For this, DDPG uses Q-learning (Watkins & Dayan, 1992) to learn Q ϕ by minimizing the one-step

Original: Bellman residual J ϕ ( D ) =
Revised: Bellman residual J ϕ ( D ) =

Original: E ( o t ,a t ,r t ,o t +1 )
Revised: E ( o t ,a t ,r t ,o t +1 )

Original: ∼D (cid:2) ( Q ϕ ( o t , a t ) − r t − γQ ¯ ϕ
Revised: ∼D (cid:2) ( Q ϕ ( o t , a t ) − r t − γQ ¯ ϕ

Original: ( o t +1 , π θ ( o t +1 )))
Revised: ( o t +1 , π θ ( o t +1 )))

Original: 2 (cid:3) .
Revised: 2 (cid:3) .

Original: The policy π θ is learned by employing Deterministic Policy Gradient (DPG) (Silver et al., 2014) and maximizing J θ ( D ) =
Revised: The policy π θ is learned by employing Deterministic Policy Gradient (DPG) (Silver et al., 2014) and maximizing J θ ( D ) =

Original: E o t ∼D [ Q ϕ ( o t , π θ ( o t ))] , where π θ ( o t ) approximates argmax a Q ϕ
Revised: E o t ∼D [ Q ϕ ( o t , π θ ( o t ))] , where π θ ( o t ) approximates argmax a Q ϕ

Original: ( o t , a ) .
Revised: ( o t , a ) .

Original: Here,
Revised: Here,

Original: D is a replay buffer of environment transitions and ¯ ϕ is an exponential moving average of the weights.
Revised: D is a replay buffer of environment transitions and ¯ ϕ is an exponential moving average of the weights.

Original: DDPG is amenable to incorporate n -step returns when estimating temporal difference (TD) error beyond a single step.
Revised: DDPG is amenable to incorporate n -step returns when estimating temporal difference (TD) error beyond a single step.

Original: In practice, n -step returns allow for faster reward propagation and have been previously used in policy gradient and Q-learning methods (Hessel et al., 2018; Barth-Maron et al., 2018; Li & Faisal, 2021).
Revised: In practice, n -step returns allow for faster reward propagation and have been previously used in policy gradient and Q-learning methods (Hessel et al., 2018; Barth-Maron et al., 2018; Li & Faisal, 2021).

Original: 3. S OFT A CTOR -C RITIC
Revised: 3.3 Soft Actor-Critic

Original: Soft actor-critic (SAC) (Haarnoja et al., 2018a;b) is an off-policy actor–critic RL algorithm for continuous control problems that concurrently learns a Q-function, Q ϕ , a stochastic policy, π θ , and a temperature,
Revised: Soft actor-critic (SAC) (Haarnoja et al., 2018a,b) is an off-policy actor–critic RL algorithm for continuous control problems that concurrently learns a Q-function, Q ϕ , a stochastic policy, π θ , and a temperature,

Original: α , on the basis of a maximum-entropy framework.
Revised: α , on the basis of a maximum-entropy framework.

Original: For this, SAC performs soft policy evaluation and improvement steps at each iteration, with the goal of maximizing a trade-off between expected return and entropy using the following γ -discounted maximum-entropy objective: E ( o t ,a t ) ∼D
Revised: For this, SAC performs soft policy evaluation and improvement steps at each iteration, with the goal of maximizing a trade-off between expected return and entropy using the following γ -discounted maximum-entropy objective: E ( o t ,a t ) ∼D

Original: [ R ( o t , a t )
Revised: [ R ( o t , a t )

Original: + αH ( π ( · | o t ))] , where α is the temperature that balances between optimizing for the reward and for the stochasticity of the policy.
Revised: + αH ( π ( · | o t ))] , where α is the temperature that balances between optimizing for the reward and for the stochasticity of the policy.

Original: 3. I MAGE A UGMENTATION IN R EINFORCEMENT L EARNING
Revised: 3.4 Image Augmentation in Reinforcement Learning

Original: Image augmentation techniques have been commonly used in computer vision research not only to avoid overfitting, but also to achieve state-of-the-art performance (Shorten & Khoshgoftaar, 2019).
Revised: Image augmentation techniques have been commonly used in computer vision research not only to avoid overfitting, but also to achieve state-of-the-art performance (Shorten & Khoshgoftaar, 2019).

Original: Recently, in visual RL, DrQ-v2 (Yarats et al., 2021a), which is an improvement on DrQ (Yarats et al., 2020), provides a new state-of-the-art performance in relation to 3D locomotion tasks from the DMC suite (Tassa et al., 2018) by adding image augmentation in the form of random shifts.
Revised: Recently, in visual RL, DrQ-v2 (Yarats et al., 2021a), which is an improvement on DrQ (Yarats et al., 2020), provides a new state-of-the-art performance in relation to 3D locomotion tasks from the DMC suite (Tassa et al., 2018) by adding image augmentation in the form of random shifts.

Original: The use of such image augmentation techniques is now an essential factor in achieving a new state-of-the-art performance in image-based training (Srinivas et al., 2020; Schwarzer et al., 2020;
Revised: The use of such image augmentation techniques is now an essential factor in achieving a new state-of-the-art performance in image-based training (Srinivas et al., 2020; Schwarzer et al., 2020;

Original: Yarats et al., 2020; 2021a; Hansen & Wang, 2021).
Revised: Yarats et al., 2020, 2021a; Hansen & Wang, 2021).

Original:  MERL: M ULTIMODAL E ND - TO -E ND R EINFORCEMENT L EARNING
Revised:  MERL: Multimodal End-to-End Reinforcement Learning

Original: In this section, we describe MERL, an effective deep RL method for 3D robotic manipulation; specifically, a model-free off-policy actor–critic RL algorithm that can be trained in an end-to-end manner from multimodal raw sensory data (high-dimensional RGB images and a low-dimensional proprioception vector).
Revised: In this section, we describe MERL, an effective deep RL method for 3D robotic manipulation; specifically, a model-free off-policy actor–critic RL algorithm that can be trained in an end-to-end manner from multimodal raw sensory data (high-dimensional RGB images and a low-dimensional proprioception vector).

Original: Figure 1 illustrates our method (MERL), which learns a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.
Revised: Figure 1 illustrates our method (MERL), which learns a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data.

Original: Here, MERL uses an image augmentation technique and a convolutional neural network (CNN) to encode the images; a multi-layer perception (MLP) to encode the proprioception; an MLP and layer normalization with scaling to learn the latent multimodal representation from the encoded visual and proprioceptive representations; and actor–critic networks to learn the policy.
Revised: Here, MERL uses an image augmentation technique and a convolutional neural network (CNN) to encode the images; a multi-layer perception (MLP) to encode the proprioception; an MLP and layer normalization with scaling to learn the latent multimodal representation from the encoded visual and proprioceptive representations; and actor–critic networks to learn the policy.

Original: 4. M ULTIMODAL R EPRESENTATION L EARNING FOR 3D R OBOTIC M ANIPULATION
Revised: 4.1 Multimodal Representation Learning for 3D Robotic Manipulation

Original: Replay Buffer Unlike in visual RL such as DrQ-v2 or state-based RL such as DDPG and SAC, our method stores transition ( o t , a t , r t , γ t , o t +1 ) , including multimodal observations, in the replay buffer, for we use two different types of raw sensory data (RGB images and proprioception).
Revised: Replay Buffer Unlike in visual RL such as DrQ-v2 or state-based RL such as DDPG and SAC, our method stores transition ( o t , a t , r t , γ t , o t +1 ) , including multimodal observations, in the replay buffer, for we use two different types of raw sensory data (RGB images and proprioception).

Original: Regarding the transition, o t denotes a multimodal observation (three consecutive prior RGB images, o image t , and a proprioception vector, o prop t ) at time t ; a t denotes the robot’s action (for example, joint torques) at time t ; r t denotes the reward at time t ; γ t denotes a discount factor at time t ; and o t +1 denotes a multimodal observation (three consecutive prior RGB images, o image t +1 , and a proprioception vector, o prop t +1 ) at time t + 1 .
Revised: Regarding the transition, o t denotes a multimodal observation (three consecutive prior RGB images, o image t , and a proprioception vector, o prop t ) at time t ; a t denotes the robot’s action (for example, joint torques) at time t ; r t denotes the reward at time t ; γ t denotes a discount factor at time t ; and o t +1 denotes a multimodal observation (three consecutive prior RGB images, o image t +1 , and a proprioception vector, o prop t +1 ) at time t + 1 .

Original: Regarding the proprioception vector, it can include the joint positions, velocities, or torques of the robotic arm and hand or the position or rotation of the robot’s end-effector, or all of the above.
Revised: Regarding the proprioception vector, it can include the joint positions, velocities, or torques of the robotic arm and hand or the position or rotation of the robot’s end-effector, or all of the above.

Original: Image Augmentation A deep neural network is a powerful tool with which to learn a latent representation from high-dimensional data; however, such a tool requires a substantial amount of training data.
Revised: Image Augmentation A deep neural network is a powerful tool with which to learn a latent representation from high-dimensional data; however, such a tool requires a substantial amount of training data.

Original: The computer vision community has made great headway regarding this challenge by using various image augmentation techniques (Shorten & Khoshgoftaar, 2019).
Revised: The computer vision community has made great headway regarding this challenge by using various image augmentation techniques (Shorten & Khoshgoftaar, 2019).

Original: In this study, we apply a random shift image augmentation technique to image observations of the environment, as in DrQ-v2.
Revised: In this study, we apply a random shift image augmentation technique to image observations of the environment, as in DrQ-v2.

Original: Specifically, we pad each side of an 84 × 84 image with four repeating boundary pixels.
Revised: Specifically, we pad each side of an 84 × 84 image with four repeating boundary pixels.

Original: Next, we replace each repeated pixel value with the average of the four nearest pixel values (that is, we apply bilinear interpolation to the shifted image).
Revised: Next, we replace each repeated pixel value with the average of the four nearest pixel values (that is, we apply bilinear interpolation to the shifted image).

Original: Finally, we select a random 84 × 84 crop, yielding the original image shifted by ± 4 pixels.
Revised: Finally, we select a random 84 × 84 crop, yielding the original image shifted by ± 4 pixels.

Original: Image Encoder
Revised: Image Encoder

Original: The augmented images, ˆ o image , obtained by means of the random shift augmentation technique, are embedded into a low-dimensional latent vector, r image , by means of a CNN encoder.
Revised: The augmented images, ˆ o image , obtained by means of the random shift augmentation technique, are embedded into a low-dimensional latent vector, r image , by means of a CNN encoder.

Original: Regarding the CNN encoder, we use the same architecture as in DrQ (Yarats et al., 2020), which was first introduced in SAC+AE (Yarats et al., 2021c).
Revised: Regarding the CNN encoder, we use the same architecture as in DrQ (Yarats et al., 2020), which was first introduced in SAC+AE (Yarats et al., 2021c).

Original: This is due to the fact that DrQ has successfully shown a range of encoders of differing architecture and capacity to perform equally well on continuous control problems in the case where a random shift augmentation technique is applied to original image observations.
Revised: This is due to the fact that DrQ has successfully shown a range of encoders of differing architecture and capacity to perform equally well on continuous control problems in the case where a random shift augmentation technique is applied to original image observations.

Original: The encoding process can be concisely formulated as r image =
Revised: The encoding process can be concisely formulated as r image =

Original: f ξ ( aug ( o image )) , where f ξ denotes the image encoder, aug denotes the random shift augmentation, and o image denotes the original image observation.
Revised: f ξ ( aug ( o image )) , where f ξ denotes the image encoder, aug denotes the random shift augmentation, and o image denotes the original image observation.

Original: Proprioception Encoder Regarding the proprioception encoder, we use a simple MLP.
Revised: Proprioception Encoder Regarding the proprioception encoder, we use a simple MLP.

Original: The encoding process can be concisely summarized as r prop =
Revised: The encoding process can be concisely summarized as r prop =

Original: g ζ ( o prop ) , where g ζ denotes the proprioception encoder and o prop denotes the original proprioception observation.
Revised: g ζ ( o prop ) , where g ζ denotes the proprioception encoder and o prop denotes the original proprioception observation.

Original: Note here that the best performance is achieved when the proprioception encoder is set as the identity encoder.
Revised: Note here that the best performance is achieved when the proprioception encoder is set as the identity encoder.

Original: This is due to the fact that the proprioception is already well-encoded by itself (see Appendix C.1 for more details).
Revised: This is due to the fact that the proprioception is already well-encoded by itself (see Appendix C.1 for more details).

Original: Multimodal Fusion MERL uses two different types of raw sensory data as input: the RGB images from a fixed vision camera and the proprioception from the joint encoders of the robot arm and hand.
Revised: Multimodal Fusion MERL uses two different types of raw sensory data as input: the RGB images from a fixed vision camera and the proprioception from the joint encoders of the robot arm and hand.

Original: The heterogeneous nature of this data requires us to use domain-specific encoders to obtain the unique characteristics of each modality, which we then fuse into a single latent multimodal representation vector, r mm , of dimension d .
Revised: The heterogeneous nature of this data requires us to use domain-specific encoders to obtain the unique characteristics of each modality, which we then fuse into a single latent multimodal representation vector, r mm , of dimension d .

Original: For visual feedback, we stack three consecutive prior RGB images, apply the random shift augmentation technique to the stacked images, normalize the resulting augmented images, and use a CNN to encode the normalized images into a latent representation vector, r image .
Revised: For visual feedback, we stack three consecutive prior RGB images, apply the random shift augmentation technique to the stacked images, normalize the resulting augmented images, and use a CNN to encode the normalized images into a latent representation vector, r image .

Original: For proprioception, we use an MLP to encode the proprioception into a latent representation vector, r prop .
Revised: For proprioception, we use an MLP to encode the proprioception into a latent representation vector, r prop .

Original: For multimodal fusion, we use an MLP and layer normalization with scaling to produce a low-dimensional scaled latent vector, of which the range is ( − 1 , 1) for each modality.
Revised: For multimodal fusion, we use an MLP and layer normalization with scaling to produce a low-dimensional scaled latent vector, of which the range is ( − 1 , 1) for each modality.

Original: Each scaled latent vector is concatenated to produce a single latent multimodal representation vector, r mm of dimension d .
Revised: Each scaled latent vector is concatenated to produce a single latent multimodal representation vector, r mm of dimension d .

Original: The multimodal fusion process can be succinctly summarized as r mm = h ψ ( r image , r prop ) , where r image =
Revised: The multimodal fusion process can be succinctly summarized as r mm = h ψ ( r image , r prop ) , where r image =

Original: f ξ ( aug ( o image )) denotes the encoded visual representation from the RGB images and r prop =
Revised: f ξ ( aug ( o image )) denotes the encoded visual representation from the RGB images and r prop =

Original: g ζ ( o prop ) denotes the encoded proprioceptive representation from the proprioception.
Revised: g ζ ( o prop ) denotes the encoded proprioceptive representation from the proprioception.

Original: 4. M ODEL -F REE A CTOR –C RITIC R EINFORCEMENT L EARNING
Revised: 4.2 Model-Free Actor–Critic Reinforcement Learning

Original: Although SAC has been the de facto off-policy RL algorithm for many RL methods over the past few years, it is prone to suffering from policy entropy collapse.
Revised: Although SAC has been the de facto off-policy RL algorithm for many RL methods over the past few years, it is prone to suffering from policy entropy collapse.

Original: Recently, DrQ-v2 showed that using
Revised: Recently, DrQ-v2 showed that using

Original: DDPG instead of SAC as a learning algorithm leads to better performance (that is, more robustness and more stability) in relation to the tasks from the DMC suite, including 3D continuous control problems.
Revised: DDPG instead of SAC as a learning algorithm leads to better performance (that is, more robustness and more stability) in relation to the tasks from the DMC suite, including 3D continuous control problems.

Original: For this reason, we opt for DDPG as a backbone actor–critic RL algorithm to learn a policy from a latent multimodal representation.
Revised: For this reason, we opt for DDPG as a backbone actor–critic RL algorithm to learn a policy from a latent multimodal representation.

Original: As in DrQ-v2, we also augment DDPG with n -step returns to estimate TD error: the augmentation leads to faster learning progress by accelerating reward propagation.
Revised: As in DrQ-v2, we also augment DDPG with n -step returns to estimate TD error: the augmentation leads to faster learning progress by accelerating reward propagation.

Original: Here, we opt for augmenting DDPG with 3-step returns.
Revised: Here, we opt for augmenting DDPG with 3-step returns.

Original: For the augmentation, we sample a mini-batch of transitions, τ = ( o t , a t , r t : t + n − 1 , o t + n ) , from the replay buffer, D .
Revised: For the augmentation, we sample a mini-batch of transitions, τ = ( o t , a t , r t : t + n − 1 , o t + n ) , from the replay buffer, D .

Original: In addition, we use clipped double Q-learning to reduce over-estimation bias in the target value.
Revised: In addition, we use clipped double Q-learning to reduce over-estimation bias in the target value.

Original: For the learning, we train two Q-functions, Q ϕ 1 and Q ϕ 2 , by minimizing the following two losses:
Revised: For the learning, we train two Q-functions, Q ϕ 1 and Q ϕ 2 , by minimizing the following two losses:

Original: [Equation]
Revised: [Equation]

Original: where TD target, y , is defined as follows:
Revised: where TD target, y , is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where r mm_c t = h ψ critic ( f ξ ( aug ( o image t )) , g ζ ( o

Original: [Equation]

Original:  r mm_c 
Revised: where r mm_c t

Original: [Equation]
Revised: [Equation]

Original: + n = h ψ critic (

Original: [Equation] aug [Equation] )) , g ζ (
Revised: a

Original: [Equation]
Revised: [Equation]

Original: )) , a t = π θ ( r mm_a t ) + ϵ , and a t + n = π θ ( r mm_a t + n ) + ϵ .
Revised: θ ( r mm_a t ) + ϵ , and a t + n = π θ ( r mm_a t + n ) + ϵ .

Original: Here, ϵ represents exploration noise sampled from clip ( N (0 , σ 2 ) , − c, c ) , which is similar to TD3 (Fujimoto et al., 2018); ¯ ϕ 1 and ¯ ϕ 2 represent the exponential moving averages of the weights for the Q target networks; and ψ critic represents the multimodal fusion weights for the critic.
Revised: Here, ϵ represents exploration noise sampled from clip ( N (0 , σ 2 ) , − c, c ) , which is similar to TD3 (Fujimoto et al., 2018); ¯ ϕ 1 and ¯ ϕ 2 represent the exponential moving averages of the weights for the Q target networks; and ψ critic represents the multimodal fusion weights for the critic.

Original: The final version of MERL uses decoupled multimodal fusion networks for the actor and critic, h ψ actor and h ψ critic , respectively.
Revised: The final version of MERL uses decoupled multimodal fusion networks for the actor and critic, h ψ actor and h ψ critic , respectively.

Original: We note here that we use the most recent weights of ξ for the image encoder, ζ for the proprioception encoder, and ψ for the multimodal fusion, to encode o t and o t + n .
Revised: We note here that we use the most recent weights of ξ for the image encoder, ζ for the proprioception encoder, and ψ for the multimodal fusion, to encode o t and o t + n .

Original: Finally, we train the deterministic actor, π θ , using DPG by maximizing the expected returns, as follows:
Revised: Finally, we train the deterministic actor, π θ , using DPG by maximizing the expected returns, as follows:

Original: [Equation]
Revised: [Equation]

Original: where r mm_a t = h ψ actor ( f ξ ( aug ( o image t )) , g ζ ( o prop t )) and a t = π θ ( r mm_a t ) + ϵ .
Revised: where r mm_a t = h ψ actor ( f ξ ( aug ( o image t )) , g ζ ( o prop t )) and a t = π θ ( r mm_a t ) + ϵ .

Original: Here, ψ actor represents the multimodal fusion weights for the actor.
Revised: Here, ψ actor represents the multimodal fusion weights for the actor.

Original: We note here that we prevent the actor’s gradients from updating the image and proprioception encoders, for SAC+AE found that this in fact hinders the agent’s performance.
Revised: We note here that we prevent the actor’s gradients from updating the image and proprioception encoders, for SAC+AE found that this in fact hinders the agent’s performance.

Original:  E XPERIMENTS
Revised:  Experiments

Original: In this section, we provide an empirical evaluation of MERL in relation to three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) (Tunyasuvunakool et al., 2020).
Revised: In this section, we provide an empirical evaluation of MERL in relation to three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) (Tunyasuvunakool et al., 2020).

Original: We first present a comparison to prior methods, including current state-of-the-art visual RL and state-based RL methods, with respect to sample efficiency, learning performance, and training stability.
Revised: We first present a comparison to prior methods, including current state-of-the-art visual RL and state-based RL methods, with respect to sample efficiency, learning performance, and training stability.

Original: We then present an ablation study that guided the final version of MERL.
Revised: We then present an ablation study that guided the final version of MERL.

Original: We believe that
Revised: We believe that

Original: MERL is the first deep RL method capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data in relation to 3D robotic manipulation tasks from DMC.
Revised: MERL is the first deep RL method capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data in relation to 3D robotic manipulation tasks from DMC.

Original: 5. S ETUP
Revised: 5.1 Setup

Original: Environment Setup We consider an environment for 3D robotic manipulation that provides multimodal raw sensory data as observation.
Revised: Environment Setup We consider an environment for 3D robotic manipulation that provides multimodal raw sensory data as observation.

Original: In this context, we consider learning directly from multimodal observations, including RGB images and proprioception.
Revised: In this context, we consider learning directly from multimodal observations, including RGB images and proprioception.

Original: Here, the RGB images represent stacks of three consecutive RGB images of size 84 × 84, stacked along the channel dimension to enable inference of dynamic information such as velocity and acceleration, and the proprioception represents joint angles and velocities for the robotic arm and hand.
Revised: Here, the RGB images represent stacks of three consecutive RGB images of size 84 × 84, stacked along the channel dimension to enable inference of dynamic information such as velocity and acceleration, and the proprioception represents joint angles and velocities for the robotic arm and hand.

Original: The action space is bounded by ( − 1 , 1) .
Revised: The action space is bounded by ( − 1 , 1) .

Original: Task Setup We design a set of three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) to include two different types of raw sensory data (that is, RGB images obtained from a fixed vision camera and proprioception obtained from a robot’s joint encoders) as observation, where task success requires joint reasoning over visual and proprioceptive feedback.
Revised: Task Setup We design a set of three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) to include two different types of raw sensory data (that is, RGB images obtained from a fixed vision camera and proprioception obtained from a robot’s joint encoders) as observation, where task success requires joint reasoning over visual and proprioceptive feedback.

Original: In addition, for each task, we randomize the configuration of the initial positions of the robot and the box at the beginning of each episode, during both training and testing, to enhance the robustness and generalization of the model.
Revised: In addition, for each task, we randomize the configuration of the initial positions of the robot and the box at the beginning of each episode, during both training and testing, to enhance the robustness and generalization of the model.

Original: Reward Design We adopt a staged, structured, and multi-component reward function to guide the
Revised: Reward Design We adopt a staged, structured, and multi-component reward function to guide the

Original: RL algorithm, which simplifies the challenge of exploration and leads to effective policy learning (Lee et al., 2020b; Yu et al., 2020).
Revised: RL algorithm, which simplifies the challenge of exploration and leads to effective policy learning (Lee et al., 2020b; Yu et al., 2020).

Original: The reward function, R, is a combination of a reaching reward, pushing reward, vertical reward, floating reward, and lifting reward, or subsets thereof for simpler tasks that only include reaching or pushing.
Revised: The reward function, R, is a combination of a reaching reward, pushing reward, vertical reward, floating reward, and lifting reward, or subsets thereof for simpler tasks that only include reaching or pushing.

Original: With this design, the reward is bounded by [0, 1] per timestep.
Revised: With this design, the reward is bounded by [0, 1] per timestep.

Original: 5. C OMPARISON TO P RIOR M ETHODS
Revised: 5.2 Comparison to Prior Methods

Original: Baselines We compare our method to DrQ-v2 (Yarats et al., 2021a), which currently provides the best performance for 3D continuous control problems within model-free visual RL methods, with respect to sample efficiency, learning performance, and training stability.
Revised: Baselines We compare our method to DrQ-v2 (Yarats et al., 2021a), which currently provides the best performance for 3D continuous control problems within model-free visual RL methods, with respect to sample efficiency, learning performance, and training stability.

Original: We also compare it to the state-based RL algorithms DDPG (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018b), which provide upper-bound performance with respect to sample efficiency for many RL studies.
Revised: We also compare it to the state-based RL algorithms DDPG (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018b), which provide upper-bound performance with respect to sample efficiency for many RL studies.

Original: Evaluation We present our experimental results for three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ), in Figure 2.
Revised: Evaluation We present our experimental results for three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ), in Figure 2.

Original: Our empirical study in Figurereveals that MERL considerably outperforms both the state-based RL algorithms (DDPG and SAC) and the state-of-the-art visual RL algorithm (DrQ-v2) with respect to sample efficiency, learning performance, and training stability.
Revised: Our empirical study in Figurereveals that MERL considerably outperforms both the state-based RL algorithms (DDPG and SAC) and the state-of-the-art visual RL algorithm (DrQ-v2) with respect to sample efficiency, learning performance, and training stability.

Original: This suggests that a coordinate state used in state-based RL is insufficient to represent the robot environment and that visual RL requires significantly more data to learn a latent representation, both of which in turn suggest that learning from multimodal raw sensory data is required to achieve the best performance with respect to sample efficiency and learning performance in relation to 3D robotic manipulation.
Revised: This suggests that a coordinate state used in state-based RL is insufficient to represent the robot environment and that visual RL requires significantly more data to learn a latent representation, both of which in turn suggest that learning from multimodal raw sensory data is required to achieve the best performance with respect to sample efficiency and learning performance in relation to 3D robotic manipulation.

Original: Notably, MERL’s advantage is more pronounced in the case of more difficult 3D robotic manipulation tasks (for example, jaco move box and jaco lift box ), where exploration is more challenging.
Revised: Notably, MERL’s advantage is more pronounced in the case of more difficult 3D robotic manipulation tasks (for example, jaco move box and jaco lift box ), where exploration is more challenging.

Original: Most notably, to the best of our knowledge, MERL is the first model-free RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data while showing a new state-of-the-art performance.
Revised: Most notably, to the best of our knowledge, MERL is the first model-free RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data while showing a new state-of-the-art performance.

Original: 5. A BLATION S TUDY
Revised: 5.3 Ablation Study

Original: We conduct an ablation study that leads to the final version of MERL.
Revised: We conduct an ablation study that leads to the final version of MERL.

Original: Our findings are summarized in Figure 3 and detailed below. 
Revised: Our findings are summarized in Figure 3 and detailed below. [Figure]

Original: Decoupled Multimodal Fusion We examine our decision to use a decoupled architecture for multimodal fusion compared to using a conventional shared architecture primarily used in multimodal representation learning (Lee et al., 2019a; 2020b).
Revised: Decoupled Multimodal Fusion We examine our decision to use a decoupled architecture for multimodal fusion compared to using a conventional shared architecture primarily used in multimodal representation learning (Lee et al., 2019a, 2020b).

Original: We conduct some experiments on two different architectures for multimodal fusion: decoupled and shared.
Revised: We conduct some experiments on two different architectures for multimodal fusion: decoupled and shared.

Original: As shown in Figure 3a, the decoupled architecture provides better performance in relation to the three 3D robotic manipulation tasks from
Revised: As shown in Figure 3a, the decoupled architecture provides better performance in relation to the three 3D robotic manipulation tasks from

Original: DMC ( jaco reach duplo, jaco move box , and jaco lift box ) compared to the conventional shared architecture.
Revised: DMC ( jaco reach duplo, jaco move box , and jaco lift box ) compared to the conventional shared architecture.

Original: This suggests that the decoupled architecture for multimodal fusion has a greater impact on actor–critic learning compared to the conventional shared architecture.
Revised: This suggests that the decoupled architecture for multimodal fusion has a greater impact on actor–critic learning compared to the conventional shared architecture.

Original: The greater impact may be[Figure] due to the fact that in the decoupled architecture latent multimodal representations fed into the actor are separately learned to maximize the learning performance of the actor, and likewise in the case of the critic.
Revised: The greater impact may be due to the fact that in the decoupled architecture latent multimodal representations fed into the actor are separately learned to maximize the learning performance of the actor, and likewise in the case of the critic.

Original: We also found that, in the case of the shared architecture for multimodal fusion, updating the multimodal fusion with the actor’s gradients greatly hinders the agent’s performance.
Revised: We also found that, in the case of the shared architecture for multimodal fusion, updating the multimodal fusion with the actor’s gradients greatly hinders the agent’s performance.

Original: Multimodal Representation Dimensions We investigate the required level of compactness of the latent multimodal representation for 3D robotic manipulation by changing the dimensionality of the representation.
Revised: Multimodal Representation Dimensions We investigate the required level of compactness of the latent multimodal representation for 3D robotic manipulation by changing the dimensionality of the representation.

Original: We hypothesize that a more compact representation may increase the tractability of
Revised: We hypothesize that a more compact representation may increase the tractability of

Original: RL yet capture less information for 3D robotic manipulation tasks.
Revised: RL yet capture less information for 3D robotic manipulation tasks.

Original: We perform several experiments at different representation dimensions: dim = 64, 128, and 256.
Revised: We perform several experiments at different representation dimensions: dim = 64, 128, and 256.

Original: As shown in Figure 3b, dim =provides the best performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ).
Revised: As shown in Figure 3b, dim =provides the best performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ).

Original: In the case of dim = 64, less information for the tasks was captured, and in the case of dim = 256, the learning performance was degraded owing to an increase in the latent state space used in policy learning.
Revised: In the case of dim = 64, less information for the tasks was captured, and in the case of dim = 256, the learning performance was degraded owing to an increase in the latent state space used in policy learning.

Original: Layer Normalization We examine the efficacy of layer normalization (Ba et al., 2016) commonly used in transformer architectures (Vaswani et al., 2017; Wang et al., 2019).
Revised: Layer Normalization We examine the efficacy of layer normalization (Ba et al., 2016) commonly used in transformer architectures (Vaswani et al., 2017; Wang et al., 2019).

Original: We hypothesize that the application of layer normalization to our multimodal fusion will play a crucial role in controlling the gradient scales, as in a transformer architecture, and will lead to favorable sample efficiency.
Revised: We hypothesize that the application of layer normalization to our multimodal fusion will play a crucial role in controlling the gradient scales, as in a transformer architecture, and will lead to favorable sample efficiency.

Original: We conduct some experiments for two different cases: multimodal fusion with and without layer normalization applied.
Revised: We conduct some experiments for two different cases: multimodal fusion with and without layer normalization applied.

Original: As shown in Figure 3c, the former case produces significantly better performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) than the latter case.
Revised: As shown in Figure 3c, the former case produces significantly better performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) than the latter case.

Original: This suggests that any increase in speed or stability of the
Revised: This suggests that any increase in speed or stability of the

Original: RL process is due to the use of normalized multimodal representations (via layer normalization) in actor–critic networks.
Revised: RL process is due to the use of normalized multimodal representations (via layer normalization) in actor–critic networks.

Original: In addition, we perform scaling via tanh before feeding the representation into the critic network in order to match the latent multimodal representation and action to the same scale.
Revised: In addition, we perform scaling via tanh before feeding the representation into the critic network in order to match the latent multimodal representation and action to the same scale.

Original: Exploration Noise We investigate the efficacy of the scheduled exploration noise used in DrQ-v2, where the scheduled exploration noise has different levels of exploration at different stages of learning.
Revised: Exploration Noise We investigate the efficacy of the scheduled exploration noise used in DrQ-v2, where the scheduled exploration noise has different levels of exploration at different stages of learning.

Original: We hypothesize that a scheduled exploration noise will be more helpful to improve sample efficiency by having a more stochastic exploration at the beginning of training compared to a fixed exploration noise.
Revised: We hypothesize that a scheduled exploration noise will be more helpful to improve sample efficiency by having a more stochastic exploration at the beginning of training compared to a fixed exploration noise.

Original: We perform some experiments at two different levels of exploration noise: decaying and fixed.
Revised: We perform some experiments at two different levels of exploration noise: decaying and fixed.

Original: As shown in Figure 3d, contrary to the results of the DrQ-v2 study (Yarats et al., 2021a), having a fixed level of exploration noise provides better learning performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) than having a scheduled level of exploration noise.
Revised: As shown in Figure 3d, contrary to the results of the DrQ-v2 study (Yarats et al., 2021a), having a fixed level of exploration noise provides better learning performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) than having a scheduled level of exploration noise.

Original: This suggests that the data accumulated through having a stochastic exploration at the beginning of training degrades the learning performance, especially in the case of the task jaco lift box .
Revised: This suggests that the data accumulated through having a stochastic exploration at the beginning of training degrades the learning performance, especially in the case of the task jaco lift box .

Original: Accordingly, we need to set different levels of exploration noise depending on the characteristics of a given task.
Revised: Accordingly, we need to set different levels of exploration noise depending on the characteristics of a given task.

Original: For example, for a task that requires a lot of stochastic exploration at the beginning of training, such as humanoid locomotion , a scheduled exploration noise is more effective, whereas, in the case of 3D robotic manipulation tasks, a fixed stochastic exploration is more effective.
Revised: For example, for a task that requires a lot of stochastic exploration at the beginning of training, such as humanoid locomotion , a scheduled exploration noise is more effective, whereas, in the case of 3D robotic manipulation tasks, a fixed stochastic exploration is more effective.

Original:  C ONCLUSION
Revised:  Conclusion

Original: We have proposed MERL, a conceptually simple but effective model-free RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data such as RGB images and proprioception.
Revised: We have proposed MERL, a conceptually simple but effective model-free RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data such as RGB images and proprioception.

Original: Our experimental results show that MERL significantly outperforms both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DMC.
Revised: Our experimental results show that MERL significantly outperforms both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DMC.

Original: To the best of our knowledge, MERL is the first effective RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data while showing a new state-of-the-art performance.
Revised: To the best of our knowledge, MERL is the first effective RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data while showing a new state-of-the-art performance.

Original: We hope this work will serve as a guideline for future multimodal end-to-end RL research.
Revised: We hope this work will serve as a guideline for future multimodal end-to-end RL research.

Original: In future work, we will seek to use more sensors (for example, depth and haptic sensors) of differing modalities, which will lead to more dexterous manipulation, and to test and validate MERL in a real-world robotic environment through sim-to-real.
Revised: In future work, we will seek to use more sensors (for example, depth and haptic sensors) of differing modalities, which will lead to more dexterous manipulation, and to test and validate MERL in a real-world robotic environment through sim-to-real.

Original: In addition, we hope to explore the use of transformer architectures and object-centric learning to obtain a better representation. 
Revised: In addition, we hope to explore the use of transformer architectures and object-centric learning to obtain a better representation. Acknowledgments

Revised: This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government.

Revised: [22ZR1100, A Study of Hyper-Connected Thinking Internet

Revised: Technology by autonomous connecting, controlling and evolving ways]

Revised: A Algorithm Details

Revised: standard deviation σ value c

Revised: Image augmentations: Random shift aug for t ← 1 · · · T do a t ← π θ ( h ψ actor ( f ξ ( o image t ) , g ζ ( o prop t )))

Revised: + ϵ and ϵ ∼ N (0 , σ 2 )

Revised: o t +1 ∼ P ( ·

Revised: | o t , a t )

Revised: D ← ( o t , a t , R ( o t , a t ) , o t +1 ) ∪ D

Revised: U PDATE C RITIC ( D )

Revised: U PDATE A CTOR ( D ) end for procedure

Revised: U PDATE C RITIC

Revised: ( D ) { ( o t , a t , r t : t + n − 1 , o t + n ) } ∼ D r mm_c t , r mm_c t + n ← h ψ critic ( f ξ ( aug ( o image t )) , g ζ ( o prop t )) , h ψ critic ( f ξ ( aug ( o image t + n )) , g ζ ( o prop t + n ))

Revised: r mm_a t , r mm_a t + n ← h ψ actor ( f ξ ( aug ( o image t )) , g ζ ( o prop t )) , h ψ actor ( f ξ ( aug ( o image t + n )) , g ζ ( o prop t + n )) a t + n ← π θ ( r mm_a t + n ) + ϵ and ϵ ∼ clip ( N (0 , σ 2 ) , − c, c )

Revised: Compute L ϕ 1 ,ψ critic ,ξ,ζ and L ϕ 2 ,ψ critic ,ξ,ζ ξ ← ξ − α ∇ ξ ( L ϕ 1 ,ψ critic ,ξ,ζ + L ϕ 2 ,ψ critic ,ξ,ζ ) ζ ← ζ − α ∇ ζ

Revised: ( L ϕ 1 ,ψ critic ,ξ,ζ + L ϕ 2 ,ψ critic ,ξ,ζ ) ψ critic ← ψ critic − α ∇ ψ critic ( L ϕ 1 ,ψ critic ,ξ,ζ + L ϕ 2 ,ψ critic ,ξ,ζ ) ϕ

Revised: k ← ϕ k − α ∇ ϕ k L ϕ k ,ψ critic ,ξ,ζ ∀ k ∈ { 1 , 2 } ¯ ϕ k ← (1 − τ )¯

Revised: ϕ k + τϕ k ∀ k ∈ { 1 , 2 } end procedure procedure U PDATE

Revised: A CTOR ( D ) { ( o t ) } ∼ D r mm_c t ← h ψ critic ( f ξ ( aug ( o image t )) , g ζ ( o prop t ))

Revised: r mm_a t ← h ψ actor ( f ξ ( aug ( o image t )) , g ζ ( o prop t )) a t ← π θ ( r mm_a t ) + ϵ and ϵ ∼ clip ( N (0 , σ 2 ) , − c, c )

Revised: Compute L θ,ψ actor ψ actor ← ψ actor − α ∇ ψ actor L θ,ψ actor θ ← θ − α ∇ θ L θ,ψ actor end procedure

Revised: B Task Details

Revised: In this section, we describe details of the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) with respect to task descriptions, reward design, and task visualizations.

Revised: B.1 Task Descriptions

Revised: We design a set of three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) (Tunyasuvunakool et al., 2020) to include two different types of raw sensory data (that is, RGB images obtained from a fixed vision camera and proprioception obtained from a robot’s joint encoders) as observation, where task success requires joint reasoning over visual and proprioceptive feedback.

Revised: For each task, we randomize the configuration of the initial positions of the robot and the box at the beginning of each episode, during both training and testing, to enhance the robustness and generalization of the model.

Revised: In the case of the task jaco reach duplo , the robot (more precisely, the center point of the robot’s end-effector) is required to reach a duplo randomly placed on a workspace.

Revised: The task is considered successful when the robot reaches within 5 cm of the duplo.

Revised: In the case of the task jaco move box , the robot is required to move a box randomly placed on a workspace to a specific target position.

Revised: The task is considered successful when the box reaches within 1 cm of the target position.

Revised: In the case of the task jaco lift box , the robot is required to lift a box randomly placed on a workspace to a specific target position.

Revised: The task is considered successful when the box reaches within 1 cm of the target position.

Revised: B.2 Reward Design

Revised: We adopt a staged, structured, and multi-component reward function to guide the RL algorithm, which simplifies the challenge of exploration and leads to effective policy learning (Lee et al., 2020b;

Revised: Yu et al., 2020).

Revised: The reward function, R , is a combination of a reaching reward, pushing reward, vertical reward, floating reward, and lifting reward, or subsets thereof for simpler tasks that only include reaching or pushing.

Revised: With this design, the reward is bounded by [0 , 1] per timestep.

Revised: [Table]

Revised: Duplo

Revised: The reward function for task jaco reach duplo is defined as follows:

Revised: [Equation]

Revised: Jaco Move Box The reward function for task jaco move box is defined as follows:

Revised: [Equation]

Revised: Jaco Lift Box The reward function for task jaco lift box is defined as follows:

Revised: [Equation]

Revised: where r reaching = 1 N tolerance (cid:0) dist ( p tcp , p obj ) , ε reaching (cid:1) , r pushing = 1 N tolerance (cid:0) dist ( p target , p obj ) , ε pushing (cid:1) , r vertical = 1 N tolerance (cid:0) cosdist ( v hand , u -z ) , 1 − ε vertical (cid:1) , r floating =

Revised: 1 N z obj z target , and r lifting = 1 N tolerance (cid:0) dist ( p obj , p target ) , ε lifting (cid:1) .

Revised: p tcp represents the position of the end-effector’s center point, p obj represents the position of the object (here, the box’s center point), p target represents the target position, z obj represents the z -axis value of the object’s center point, z target represents the z -axis value of the target position, v hand represents a unit vector vertical to the robot hand, and u -z represents a unit vector [0 , 0 , − 1] .

Revised: The number of stages is represented by N .

Revised: Here, we adopted the same function tolerance as in DMC (Tunyasuvunakool et al., 2020).

Revised: B.3 Task Visualizations

Revised: Figure 4 provides visualizations for behaviors generated by MERL in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ).

Revised: The visualization of behaviors learned by MERL during the training procedure is shown in Figure 4a.

Revised: We note here that MERL solves all three of the 3D robotic manipulation tasks in less than 1M environment steps.

Revised: Figure 4b illustrates the visualization of successful trajectories generated by MERL.

Revised: [Figure]

Revised: C Implementation Details

Revised: In this section, we describe our implementation details for MERL.

Revised: We use PyTorch as a deep learning tool and DMC and MuJoCo for our simulation.

Revised: We conduct our experiments on a workstation with an Intel i9-9980XE CPU and Nvidia Quadro RTX 8000 GPU.

Revised: C.1 Network Architectures

Revised: For all networks, we initialize the weight matrix of the convolutional and fully-connected layers with an orthogonal initialization (Saxe et al., 2013) and set the bias to be zero.

Revised: Image Encoder Network The image encoder network is modeled as four convolutional layers with 3 × 3 kernels and 32 channels, as in SAC+AE (Yarats et al., 2021c).

Revised: An ReLU activation is applied after each convolutional layer.

Revised: We use stride 1 everywhere, except for the first convolutional layer, which has stride 2.

Revised: Here, we note that only the critic optimizer is allowed to update the image encoder network weights (that is, we prevent the actor’s gradients from updating the image encoder network weights).

Revised: Proprioception Encoder Network The proprioception encoder network is modeled as a 2-layer

Revised: MLP with ReLU activations after each layer.

Revised: Here, the final version of our method uses the proprioception encoder as the identity encoder.

Revised: This is because as shown in Figure 5, contrary to the results in (Lee et al., 2019a, 2020b), the case where the proprioception encoder is set as the identity encoder provides better performance in relation to the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ), compared to the case where the proprioception encoder is set as an MLP encoder.

Revised: This suggests that the proprioception itself is enough to learn a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner, without the need for additional encodings; that is, it is already well-encoded.

Revised: Note that only the critic optimizer is allowed to update the proprioception encoder network weights.

Revised: Multimodal Fusion Network The encoded visual and proprioceptive representations are fed into multimodal fusion to learn a latent multimodal representation.

Revised: The multimodal fusion network is modeled as a single fully-connected layer normalized by LayerNorm (Ba et al., 2016).

Revised: After

Revised: LayerNorm , the tanh nonlinearity is applied to the output of the fully-connected layer, which serves to match the latent multimodal representation and action to the same scale.

Revised: Finally, each output is concatenated to produce a single latent multimodal representation vector of dimension d .

Revised: The weights of each multimodal fusion network for actor and critic are updated by the gradients of actor and critic, respectively.

Revised: Actor and Critic Networks

Revised: As in TD3 (Fujimoto et al., 2018), we use clipped double Q-learning for the critic network, where each Q-function is parametrized as a 3-layer (in the case of the task jaco reach box ) or 6-layer (in the case of the tasks jaco move box and jaco lift box ) MLP with LayerNorm and ReLU activations after each layer except the last.

Revised: The actor network is also modeled as a 3-layer (in the case of the task jaco reach box ) or 6-layer (in the case of the tasks jaco move box and jaco lift box ) MLP with LayerNorm and ReLU activations after each layer except the last, which outputs the mean and covariance for the diagonal Gaussian that represents the policy.

Revised: The hidden dimension is set to 1024 for both the critic and the actor.

Revised: C.2 Hyper-Parameters

Revised: We provide a comprehensive overview of the hyper-parameters used in the case of the three 3D robotic manipulation tasks from DMC ( jaco reach duplo, jaco move box , and jaco lift box ) in Table

Revised: [Table]

Revised: [Table]