Original: Reproducing Softmax Deep Double Deterministic Policy Gradients
Revised: Replicating Softmax Deep Double Deterministic Policy Gradients

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: SummaryScope of Reproducibility
Revised: Reproducibility Summary Scope of Reproducibility

Original: We attempt to reproduce Pan et al.
Revised: We attempt to reproduce Pan et al.

Original: [19] claim that Softmax Deep Double Deterministic Policy Gradient (SD3) achievessuperior performance over Twin Delayed Deep Double Deterministic Policy Gradient (TD3)
Revised: [25] claim that Softmax Deep Double Deterministic Policy Gradient (SD3) achievessuperior performance over Twin Delayed Deep Double Deterministic Policy Gradient (TD3)

Original: [9] on continuous controlreinforcement learning tasks.
Revised: [11] on continuous control reinforcement learning tasks.

Original: We utilize both environments that were used by the paper and expand to include somenot present.
Revised: We utilize both environments that were used by the paper and expand to includesome not present.

Original: Methodology
Revised: Methodology

Original: We compare the performance of TD3 and SD3 on a variety of continuous control tasks.
Revised: We compare the performance of TD3 and SD3 on a variety of continuous control tasks.

Original: We use the author’s PyTorchcode but also provide Tensorﬂow implementations of SD3 and TD3 (which we did not use for optimization reasons).
Revised: We use the authors’ PyTorchcode but also provide Tensorﬂow implementations of SD3 and TD3 (which we did not use for optimization reasons).

Original: For the control tasks we utilize OpenAI Gym environments with PyBullet implementations, as opposed to MuJoCo, inan effort to bolster claims of generalization and to avoid exclusionary research practices.
Revised: For the control tasks we utilize OpenAI Gym environments with PyBullet implementations, as opposed to MuJoCo, inan effort to bolster claims of generalization and to avoid exclusionary research practices.

Original: Experiments are conductedboth on similar environments in the original paper and those that were not mentioned.
Revised: Experiments are conductedboth on similar environments in the original paper and those that were not mentioned.

Original: Results
Revised: Results

Original: Overall we reach similar, albeit much milder, conclusions as the paper, speciﬁcally, that SD3 outperforms TD3 onsome of continuous control tasks.
Revised: Overall we reach similar, albeit much milder, conclusions as the paper, speciﬁcally, that SD3 outperforms TD3 onsome of continuous control tasks.

Original: However, the advantage is not always as readily apparent as in the original work.
Revised: However, the advantage is not always as readily apparent as in the original work.

Original: Algorithmic performance was comparable on most environments, with SD3 providing limited evidence of deﬁnitivesuperiority.
Revised: Algorithmic performance was comparable on most environments, with SD3 providing limited evidence of deﬁnitivesuperiority.

Original: Further investigation and improvements are warranted.
Revised: Further investigation and improvements are warranted.

Original: The results are not directly comparable to theoriginal paper due to differences in physics simulators.
Revised: The results are not directly comparable to theoriginal paper due to differences in physics simulators.

Original: Additionally, we did not perform hyperparameter optimization,which could potentially bolster returns on some environments.
Revised: Additionally, we did not perform hyperparameter optimization,which could potentially bolster returns on some environments.

Original: What was easy
Revised: What was easy

Original: The author’s made their code extremely easy to use, run, modify and rewrite in a different package.
Revised: The authors’ made their code extremely easy to use, run, modify and rewrite in a different package.

Original: Because everythingwas available on their github and required only common reinforcement learning packages it was quick and painless torun.
Revised: Because everythingwas available on their github and required only common reinforcement learning packages it was quick and painless torun.

Original: It was trivial to use the algorithms on different environments from different packages and collect their results foranalysis.
Revised: It was trivial to use the algorithms on different environments from different packages and collect their results foranalysis.

Original: What was difﬁcult
Revised: What was difﬁcult

Original: One of the biggest difﬁculties was the time and resource consumption’s of the experiments.
Revised: One of the biggest difﬁculties was the time and resource consumption of the experiments.

Original: Running each algorithmon each environment with a sufﬁcient number of random seeds took the vast majority of the time.
Revised: Running each algorithmon each environment with a sufﬁcient number of random seeds took the vast majority of the time.

Original: We had a total runtime of around 310 GPU hours (or 13 days).
Revised: We had a total runtime of around 310 GPU hours (or 13 days).

Original: Time was our primary constraint and was the primary reason we did no investigate other environments.
Revised: Time was our primary constraint and was the primary reason we did no investigate other environments.

Original: Simulator differences also proved to be somewhat challenging
Revised: Simulator differences also proved to be somewhat challenging

Original: Communication with original authors
Revised: Communication with original authors

Original: Our contact with the authors was limited to a discussion we had at their poster presentation at NeurIPS
Revised: Our contact with the authors was limited to a discussion we had at their poster presentation at NeurIPS

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (RL) has achieved a great deal in the past decade.
Revised: Deep reinforcement learning (RL) has achieved a great deal in the past decade.

Original: From mastering games such as Go[21], Dota 2 [5], StarCraft II [25], and Atari [4], to precision robotic control [2] and robotic movements [11].
Revised: From mastering games such as Go[27], Dota 2 [5], StarCraft II [32], and Atari [4], to precision robotic control [2] and robotic movements [14].

Original: However,none of these are solved environments and we are always looking to ﬁnd better and faster algorithms.
Revised: However,there is still substantial room to grow, and RL suffers from a number of problems. Problems such as brittleness tohyperparameters [17] and small code level changes [9], inferior performance to far simpler methods [12], [23], andweak generalization [21] that ultimately make RL difﬁcult to use in any real world applications [8]. This problem ofgeneralization is key to the investigation presented here.

Original: Here we strive toevaluate a novel approach presented in Pan et al.
Revised: In this work, we attempt to evaluate the generalizability of thenovel continuous control reinforcement algorithm presented in Pan et al.

Original: [19], the Softmax Deep Double Deterministic Policy Gradient (SD3).
Revised: [25], the Softmax Deep Double Deterministic Policy Gradient (SD3).

Original: SD3 presents an empirical and theoretical argument for the usage of the softmax operator in continuous control reinforcement learning tasks.
Revised: Pan et al. [25] presents an empirical and theoretical argument for the usage of the softmax operator in continuouscontrol reinforcement learning tasks.

Original: It is standard practice to use the softmax operator for stochastic algorithms in discreteenvironments.
Revised: While the softmax operator is standard practice in discrete policy gradientalgorithms, it usage in continuous environments is rare.

Original: However, with the recent successes of entropy maximizing RL [10],[11], [12], [13],  [7], [26], there hasbeen an increase of interest in the softmax operator for RL [3], [22].
Revised: However, with the recent successes of entropy maximizing RL [10], [13], [14], [15], [16], [7], [33], there has been an increase of interest in the softmax operator for RL [3],[28] which is functionally similar to entropy maximization.

Original: SD3 continues this trend, utilizing the softmaxoperator to expand upon and gain improvements over TD3 [9] on the MuJoCo benchmark.
Revised: SD3 continues this trend, utilizing the softmax operatorto expand upon and claim improvements over TD3 [11] on the MuJoCo benchmark.

Original: In this work, we attempt to replicate the results of Pan et al.
Revised: In this work, we attempt to evaluate the generalization of Pan et al.

Original: [19] utilizing a variety of environments based in the opensource PyBullet physics simulator.
Revised: [25] utilizing a variety of environments based in theopen source PyBullet physics simulator.

Original: These environments include those from the original paper and in addition to somethat were not present, to test SD3’s ability to generalize to other problems.
Revised: These environments include PyBullet reimplementations of the environmentsfrom the original paper, in addition to some that were not present. These environments are chosen to test SD3’s abilityto generalize to other environments.

Original: These test environments utilize the PyBulletphysics simulator and are adapted for OpenAI Gym [6] via PyBullet-Gym and includes many similar environmentsfrom the MuJoCo benchmark.
Revised: These test environments utilize the PyBullet physics simulator and are adapted for OpenAI Gym [6] via PyBullet-Gym and includes some similar environments from the MuJoCo benchmark and manyunique ones.

Original: We use PyBullet over MuJoco to support efforts to make reinforcement learning moreequitable [18], and we reject exclusionary MuJoCo usage and conduct all of our experiments exclusively on free andopen source software.
Revised: We use PyBullet over MuJoco to support efforts to make reinforcement learning more equitable [24], andwe reject exclusionary MuJoCo usage, conducting all of our experiments exclusively on free and open source software.

Original: Note that all code and results will be available for a ﬁnal copy (but are not presented here topreserve anonymity).
Revised: Note that all code and results will be available for a ﬁnal copy (but are not presented here to preserve anonymity).

Original:  Preliminaries2.1 Reinforcement Learning Background
Revised:  Preliminaries2.1 Reinforcement Learning Background

Original: Prior to getting evaluating the claims of the paper, a proper background is essential.

Original: Reinforcement learning is a ﬁeldof machine learning in which an agent seeks to maximize a numerical reward signal from an environment [23].
Revised: Reinforcement learning is a ﬁeld of machine learning in which an agent seeks to maximize a numerical reward signalfrom an environment [29].

Original: RL isoften formalized as a Markov Decision Process (MDP), deﬁned by the tuple hS , A , R, γ i .
Revised: RL environments are often formalized as a Markov Decision Process (MDP), deﬁned bythe tuple hS , A , R, γ i .

Original: Here S represents the set ofstates, A the set of actions, R the reward and γ the reward discount between 0 and 1.
Revised: Here S represents the set of states, A the set of actions, R the reward and γ the reward discount,γ ∈ [0 , 1] .

Original: It is common to also see a P inthis tuple representing the probability of state transitions; however, our environments are not stochastic and therefore P = 1 .
Revised: It is common to also see a P in this tuple representing the probability of state transitions; however, ourenvironments are not stochastic and therefore P = 1 .

Original: The goal of an RL algorithm is to design (or learn) a policy π such that it maximizes the expected return (alsocalled objective):J = E
Revised: The goal of an RL algorithm is to design (or learn) a policy, π ,such that it maximizes the expected return (also called objective):

Original:  hP Tt =

Revised: J = E hP Tt =

Original: γ t r ( s t , a t ) |
Revised: γ t r ( s t , a t ) |

Original: π i .
Revised: π i .

Original: There are a number of techniques to design the policy π , butthe dominant trend in contemporary RL  is  to use nonlinear function approximators (i.e. deep neural networks) to learn value and policy functions.
Revised: There are a number of techniques to design the policy π , but in contemporary RL it is standard practice to use nonlinear function approximators (i.e. deep neural networks) to learn value and policy functions.

Original: The focus of this workare actor-critic algorithms.
Revised: While there are a numberof classes of algorithms, the focus of this work is on model-free, off-policy, actor-critic algorithms.

Original: In these systems an actor (or policy) network outputs the actions and is updated witha critic network that learns the value function.
Revised: In these systemsan actor (or policy) network outputs the actions and is updated with a critic network that learns the value function.

Original: These policy functions can either be stochastic or deterministic, i.e.they can either output the mean and standard deviation of the policy distribution for an action to be chosen from, oroutput a single value for the action.
Revised: These policy functions can either be stochastic or deterministic, i.e. they can either output the mean and standarddeviation of the policy distribution for an action to be chosen from, or output a single value for the action.

Original: We will largely be considering deterministic policy algorithms.
Revised: Both SD3and TD3 are deterministic policy algorithms.

Original: The policy isnetwork, parameterized by θ , is denoted as π θ .
Revised: The policy is network, parameterized by θ , is denoted as π θ .

Original: This allows us to represent the objective function J , as an expectation, J ( π θ ) =
Revised: This allowsus to represent the objective function J , as an expectation, J ( π θ ) =

Original: R S r ( s, π θ ( s )) ds = E
Revised: R S r ( s, π θ ( s )) ds = E

Original: [ R ( s, π θ ( s ))] , take the gradient to yield ∇ J ( π θ ) 
Revised: [ R ( s, π θ ( s ))] , taking thegradient of the objective function yields ∇ J ( π θ ) =

Original: = R S ∇ π θ ( s ) ∇ Q ( s, π θ ( s )) ds = 
Revised: R S ∇ π θ ( s ) ∇ Q ( s, π θ ( s )) ds = E [ ∇ π θ ( s ) ∇ Q ( s, π θ ( s ))] , where

Original: E [ ∇ π θ ( s ) ∇ Q ( s, π θ ( s ))] , where Q ( s, a ) indicates the expected return of taking action a in state s [20].
Revised: Q ( s, a ) indicates the expected return of taking action a in state s [26].

Original: This is knownas the deterministic policy gradient.
Revised: This is known as the deterministic policygradient (DPG).

Original: There are a variety of techniquesto optimize this function and we will outlinehow these techniques lead to SD3.
Revised: There are a variety of techniques and improvements to this formulation building upon each other toachieve better results.

Original: For a visual outline of this see Figure 1.
Revised: For a visual outline of this see Figure 1.

Revised: The x-axis is time, and reading this ﬁgure left to rightshows how the algorithms have built upon each other over time.

Revised: Each box is an algorithm and the arrow from one boxindicates that the pointed to algorithm took ideas and techniques from the previous algorithm.

Original: DPG  is essentially what is presented above, the policy is updated via the gradient above and the Q network is updated with the standard Bellman error:
Revised: DPG algorithm is essentially the same as described above, the policy is updated via the gradient above and the Qnetwork is updated with the standard Bellman error:

Original: L ( Q θ )
Revised: L ( Q θ )

Original: =
Revised: =

Original: r t + γQ θ ( s t +1 , π θ ( s t +1 ))
Revised: r t + γQ θ ( s t +1 , π θ ( s t +1 ))

Original: − Q θ ( s t , a t ) [20].
Revised: − Q θ ( s t , a t ) [ 26]. However,

Revised: DPG is very hard to train suffering from severe hyperparameter brittleness and lack of exploration.

Original: DDPG improvedupon DPG by adding noise to the policies to increase exploration, and by adding target networks for the policyand value networks [9].
Revised: DDPG improvedupon DPG by adding noise to the policies to increase exploration, and by adding target networks for the policy andvalue networks to improve stability [11].

Original: TD3 attempts to address the well know overestimation problem in Q learning. Because E [ max a Q ( s t , a )] ≥ max a E [ Q ( s t , a )] , Q function approximators often overestimate the Q value leading to convergence problems [16].
Revised: DDPG still suffers from over-estimations of the Q value, which happensbecause overestimation is selected for when updating the Q value, i.e. E [ max a Q ( s t , a )] ≥ max a E [ Q ( s t , a )] , oftenleading to convergence problems [20]. TD3 attempts to address the well know overestimation problem in Q learning.

Original: To address these overestimation errors, TD3 borrows from Double DQN [24] and usestwo(hence the name ’twin’)
Revised: To address these overestimation errors, TD3 borrows from Double DQN [31] and uses the idea of using two (hence thename ’twin’)

Original: Q function approximators  in addition to updating the policy network less frequently [9].
Revised: Q function approximators to prevent overestimations, in addition TD3 updates the policy network lessfrequently [ 11].

Original: While TD3 does successfully address the overestimation problem, it introduces the new underestimation problem, something SD3 tries to combat [19].
Revised: While TD3 does successfully address the overestimation problem, it introduces the new underestimation problem, something SD3 tries to combat [25].

Original: Parallel to this are the entropy based methods: soft Q Learning [10] and soft actor-critic[12].
Revised: Parallel to these developments are the improvements in entropymaximization methods: soft Q Learning [13] and soft actor-critic [15].

Original: These methods maximize a different objective than presented above due to the addition of an entropy term: J ( π ) =
Revised: These methods maximize a different objectivethan presented above due to the addition of an entropy term: J ( π ) =

Original: P Tt =

Revised: P Tt =

Original: E [ r ( s t , a t ) + α H ( π ( s t ))] .
Revised: E [ r ( s t , a t ) + α H ( π ( s t ))] .

Original: This entropy objective (under optimal conditions) is functionally the sameas the softmax operator.
Revised: This entropyobjective (under optimal conditions) is functionally the same as the softmax operator.

Original: [Figure]
Revised: 2.2 SD3

Original: SD3 is essentially the same as TD3, with  the main difference being the use of the softmax operator in the valuefunction Bellman error.
Revised: SD3 has a number of similarities with TD3, the main difference being the use of the softmax operator in the value function Bellman error.

Original: Hence, key to understanding SD3 is understanding the softmax operator.
Revised: Hence, key to understanding SD3 is understanding the softmax operator.

Original: The softmax operator is common in RL problems, but is typically seen in discrete action spaces where it is easy to calculate: σ ( z ) i =
Revised: The softmax operator iscommon in RL problems, but is typically seen in discrete action spaces where it is easy to calculate: σ ( z ) i =

Original: e zi
Revised: e zi

Original: P Kj ={+=
+}
Revised: P Kj {+=
+}

Original: [Equation]

Original:  zj .
Revised: e zj .

Original: In continuousspace action spaces, the form becomes computationally intractable:σ ( Q ( s, · )) = 
Revised: If we consider the softmax of the Q function, in continuous action spaces it becomes computationally intractable:σ ( Q ( s, · )) = R A

Original: [Equation]
Revised: [Equation]

Revised: Q

Original: [Equation]
Revised: [Equation]

Original: s,
Revised: s,

Original: [Equation]
Revised: [Equation]

Original: ′ Q ( s, a ) da .
Revised: ) da .

Original: However, Pan et al.
Revised: If we could utilize the softmax operator, Pan et al.

Original: [19] proves helpful bounds on the differencebetween max q Q ( s, a ) and σ ( Q ( s, a )) , showing that the softmax operator does not overestimate and worst case onlyslightly underestimate.
Revised: [25] proves helpfulbounds on the difference between max a Q ( s, a ) and σ ( Q ( s, a )) , showing that the softmax operator does not overestimate and worst case only slightly underestimate.

Original: In order to make the continuous softmax operator computationally feasible, SD3 utilizes asampling technique from Haarnoja et al. [13]: σ ( Q ( s, · )) =
Revised: In order to make the continuous softmax operator computationallyfeasible, SD3 utilizes a sampling technique from [16]: σ ( Q ( s, · )) =

Original: E [ exp ( βQ ( s, a )) Q ( s, a ))] / E [ exp ( βQ ( s, a ))] .
Revised: E [ exp ( βQ ( s, a )) Q ( s, a ))] / E [ exp ( βQ ( s, a ))] .

Revised: This sampling technique can also incorporate advancements in importance sampling, but that is unused in SD3.

Original: To illuminate the similarities and differences between the TD3 and SD3 we present them side by side highlighting a few keydifferences (blue are highlighted in both to show differences and red are only highlighted in one to show addition of anew feature). 
Revised: Toilluminate the similarities and differences between the TD3 and SD3 we present them side by side highlighting a fewkey differences (blue are highlighted in both to show differences and red are only highlighted in one to show additionof a new feature). See Algorithms 1 and 2.  Methodology 3.1 Target Questions

Revised: In order to provide a thorough assessment of the paper, the claims it makes, and the conclusions it draws, we presentthree central questions.

Revised: These questions serve to guide our analysis, and we evaluate them after presenting the results.

Original: Initialize value networks Q 1 , Q 2 with parameters θ 1 , θ 2 Initialize policy network π with parameters φ
Revised: • To what extent can we replicate the superior performance of SD3 over TD3 on PyBullet reimplementationsof the used MuJoCo environments? • To what extent does this performance generalize to other continuous control tasks? • What improvements can be made to the SD3 algorithm? 3.2 Experimental Setup

Revised: Although we provide TensorFlow [1] implementations of both TD3 and SD3, we run all experiments using the authors’provided PyTorch implementations.

Revised: Our reasoning is twofold.

Revised: First, our code is less optimized than the PyTorch codeand it is therefore more computationally feasible to use the PyTorch code.

Revised: Secondly, RL algorithms are notoriously difﬁcult to re-implement [30] and we wish to avoid any challenges to our implementations.

Revised: Even little differences,such as rounding vs. truncating ﬂoating points, can result is performance differences.

Revised: We wanted to make our claimsabout the algorithm as presented, and our claims are strengthened by utilization of their code.

Revised: The PyBullet gymadaptions and implementations can be found here.

Revised: We begin by collecting data for the 6 of the environments in theoriginal paper, speciﬁcally: Ant, Hopper, Lunar Lander, Walker2D, Humanoid, and Half Cheetah environments.

Revised: Wealso evaluated three additional environments: Pendulum, InvertedDoublePendulum and HumanoidFlagrun (one of thehardest environments).

Revised: Our experiments runs were setup using the same framework as the original paper: we collecteddata for 1 million iterations and repeated each experiment ﬁve times with a different random seed each time.

Revised: For theextended experiments we only collected three runs due to time constraints.

Revised: All experiments were conducted on twopersonal computers with CUDA enabled GPUs.

Revised: Depending on the environment each run would take between 2 - 8hours.

Original:  Initialize target networks Q ′ 1 , Q ′ 2 , π ′ with parameters θ ′ 1 ← θ 1 , θ ′ 2 ← θ 2 , φ ′ ← φ
Revised: Algorithm Initialize value networks Q 1 , Q 2 with parameters θ 1 , θ 2 Initialize policy network π with parameters φ Initialize target networks Q ′ 1 , Q ′ 2 , π ′ with parameters θ ′ 1 ← θ 1 , θ ′ 2 ← θ 2 , φ ′ ← φ

Original: Initialize replay buffer D for for t = 0 to T do
Revised: Initialize replay buffer D for for t = 0 to T do

Original: Select noisy action a ← π ( s ) + N and observe reward and new state s ′
Revised: Select noisy action a ← π ( s ) + N and observe reward and new state s ′

Original: Store h s, a, r, s ′ i in D
Revised: Store h s, a, r, s ′ i in D

Original: Randomly sample N tuples from D
Revised: Randomly sample N tuples from D

Original: y ← r + γmin i =1 , 2 Q θ ′
Revised: y ← r + γmin i =1 , 2 Q θ ′

Original: i ( s ′ , π φ ( s ′ )
Revised: i ( s ′ , π φ ( s ′ )

Original: + N )
Revised: + N )

Original: Update critics via the loss
Revised: Update critics via the loss

Original: L ← 1 N P ( y − Q
Revised: L ← 1 N P ( y − Q

Original: θ i ( s, a ))
Revised: θ i ( s, a ))

Original: if policy update then
Revised: if policy update then

Original: Update φ via gradient
Revised: Update φ via gradient

Original: N P ∇ π φ ( s ) ∇ Q θ 1 ( s, π φ ( s ) + N )if target update then θ ′
Revised: N P ∇ π φ ( s ) ∇ Q θ 1 ( s, π φ ( s ) + N ) θ ′

Original: i ←
Revised: i ←

Original: τθ i + (1 − τ ) θ ′ i
Revised: τθ i + (1 − τ ) θ ′ i

Original: φ ′ ← τφ
Revised: φ ′ ← τφ

Original: + (1 − τ ) φ ′
Revised: + (1 − τ ) φ ′

Original: Algorithm 2:

Original: Initialize value networks Q 1 , Q 2 with parameters θ 1 , θ
Revised: Initialize value networks Q 1 , Q 2 with parameters θ 1 , θ

Original: Initialize policy networks π 1 , π 2 with parameters φ 1 , φ
Revised: Initialize policy networks π 1 , π 2 with parameters φ 1 , φ

Original: Initialize target networks Q ′ 1 , Q ′ 2 , π ′ 1 , π ′ 2 with parameters θ ′ 1 ← θ 1 , θ ′ 2 ← θ 2 , φ ′ 1 ← φ 1 , φ ′ 2 ← φ
Revised: Initialize target networks Q ′ 1 , Q ′ 2 , π ′ 1 , π ′ 2 with parameters θ ′ 1 ← θ 1 , θ ′ 2 ← θ 2 , φ ′ 1 ← φ 1 , φ ′ 2 ← φ

Original: Initialize replay buffer D for for t = 0 to T do
Revised: Initialize replay buffer D for for t = 0 to T do

Original: Select action a ← π i ( s ) , i ← max i =1 , 2 Q i ( s, π 1 ( s )) and observe reward and new state s ′
Revised: Select action a ← π i ( s ) , i ← max i =1 , 2 Q i ( s, π 1 ( s )) and observe reward and new state s ′

Original: Store h s, a, r, s ′ i in D for i = 1, 2 do
Revised: Store h s, a, r, s ′ i in D for i = 1, 2 do

Original: Randomly sample N tuples from D
Revised: Randomly sample N tuples from D

Original: Sample K noises ǫ ˆ a ′ ← π θ ′ 1 ( s ) + ǫ ˆ Q ← min i =1 , 2 ( Q θ ′ i ( s ′ , ˆ a ′ ))
Revised: Sample K noises ǫ ˆ a ′ ← π θ ′ 1 ( s ) + ǫ ˆ Q ← min i =1 , 2 ( Q θ ′ i ( s ′ , ˆ a ′ ))

Original: σ ( ˆ Q ) ← exp ( β ˆ Q ( s ′ , ˆ a ′ ))
Revised: σ ( ˆ Q ) ← exp ( β ˆ Q ( s ′ , ˆ a ′ ))

Original: ˆ Q ( s ′ , ˆ a ′ ) /exp
Revised: ˆ Q ( s ′ , ˆ a ′ ) /exp

Original: ( β ˆ Q ( s ′ , ˆ a ′ ))
Revised: ( β ˆ Q ( s ′ , ˆ a ′ ))

Original: y ← r + γσ ( ˆ Q )
Revised: y ← r + γσ ( ˆ Q )

Original: Update Q θ i via the loss L = 1 N P ( [Equation]
Revised: Update Q θ i via the loss L = 1 N P ( y − Q θ i ( s, a ))

Original: Update φ i via gradient
Revised: Update φ i via gradient

Original: [Equation] if target update then
Revised: N P ∇ π φ i ( s ) ∇ Q θ i ( s, π φ i ( s ))

Original: [Equation]
Revised: θ ′

Original: In order to assess the validity of the paper and the conclusions it makes, we present three central questions. • To what extent can we replicate the superior performance of SD3 over TD3 on the given environments?• To what extent does this performance generalize to other continuous control tasks?• What improvements can be made to the SD3 algorithm?
Revised: i ←

Original: Although we provide TensorFlow [1] implementations of both TD3 and SD3, we run all experiments using the author’sprovided PyTorch implementations.
Revised: τθ i + (1 − τ ) θ ′ i

Original: This strengthens our results on generalizability, and is also more efﬁcient due tocode efﬁciency differences. As stated before, we opted not to use MuJoCo, favoring the open source pybullet whichhas the same environments as MuJoCo (in addition to many others). The PyBullet gym adaptions and implementationscan be found here. We begin by collecting data for the 6 of the 8 environments in the original paper, speciﬁcally: Ant,
Revised: φ ′

Original: Hopper, Lunar Lander, Walker2D, Humanoid, and Half Cheetah environments. We also evaluated three additionalenvironments: Pendulum, InvertedDoublePendulum and HumanoidFlagrun. Similar to the original paper we collecteddata for 1 million iterations and repeated each experiment ﬁve times with a different random seed each time.
Revised: i ← τφ i + (1 − τ ) φ ′

Original: Forthe extended experiments we only collected three runs due to time constraints. All experiments were conducted ontwo personal computers with CUDA enabled GPUs, speciﬁcally a GTX 1060 and a GTX 1080. Depending on theenvironment each run would take between 2 - 8 hours. 3.3 Hyperparameters
Revised: i

Original: We largely used the same hyperparameters as the original paper.
Revised: We use the same hyperparameters as the original paper.

Original: For all environments and algorithms refer to Table 1.[Table] and
Revised: For all environments and algorithms refer to Table 1.

Original: [Table]
Revised: [Table]

Original: , N

Original: N ( [Table] ) γ [Table]
Revised: [Table]

Original: τ

Original: [Table]
Revised: [Table]

Original: only)

Original: K

Original: The one notable difference is that Pan et al.
Revised: The one notable difference is that Pan et al.

Original: [19] uses a separate set of hyperparameters for the Humanoid environmentwhich we do not do.
Revised: [25] uses a separate set of hyperparameters for the Humanoid environmentwhich we do not do.

Original: The second important note on hyperparameters is the SD3 unique hyperparameter β .
Revised: The second important note on hyperparameters is the SD3 unique hyperparameter β , whichscales the softmax operation.

Original: In theoriginal paper this is determined to be a speciﬁc value for each environment, ranging from 0.001 to 500.
Revised: In the original paper this is determined to be a speciﬁc value for each environment,ranging from 0.001 to 500.

Original: On theenvironments utilized in the paper we use the same values of β .
Revised: On the environments utilized in the paper we use the same values of β .

Original: Given our time and computational constraints, we donot perform ablation studies on the extended environments to determine theoptimal β . merely adopting values fromsimilar environments.
Revised: For the extendedenvironments we adopt the values of β from similar environments.

Original: For all β value see Table 
 
Revised: For all β value see Table 
 [Table]

Original:  Results
Revised:  Results

Original: Our results are overall indicative that SD3 does provide an advantage on the some of the environments, although theseadvantages are relatively small.
Revised: Our results are overall indicative that SD3 does provide an advantage on the some of the environments, although theseadvantages are relatively small.

Revised: On all 9 environments, SD3 performers an average of 7.7% better than TD3; howeverthis comes at an increased computational cost and is not consistently superior.

Original: 4.1 Results on Paper Benchmarks
Revised: 4.1 Results on Paper Benchmarks

Original: Results for six of theeight original environments can be seen in Figure 2.
Revised: Results for six of the original environments can be seen in Figure 2.

Original: Blue represents SD3 and red TD3.
Revised: The blue represents SD3 and red TD3.

Original: Theshaded area represents a conﬁdence interval of one standard deviation.
Revised: The shadedarea represents a conﬁdence interval of one standard deviation.

Original: The exact numerical reward values of the [Table] PyBullet are not directly comparable  to the MuJoCo rewards; the environments are similar but not exactly the samewith PyBullet being supposedly harder.
Revised: While the exact numerical rewards of the PyBulletare not directly comparable 1 to the MuJoCo rewards (hence we cannot overlay the original papers results directlyon these graphs); the environments are evaluate the same goal and the same physics. The results are also presentedin Table 3.

Original:  The key takeaway from these graphs is the relative performance of the twoalgorithms.
Revised: This table shows the best average reward (over 5 runs) and the associated standard deviation. The betterperforming result is bolded. From these results we can see that SD3 outperforms TD3 on 2/6 of the environments. Thismay look as though these algorithms are effectively the same (as the humanoid performance difference in minuscule).

Original: SD3 presents slight improvements for Ant, comparable performance for Lunar Lander, Half Cheetah,and Hopper, and deﬁnitively superior performance on Walker2D.
Revised: [Table] However, further analysis gives a slight edge to SD3. The average performance of SD3 is 9% better than TD3 2 onthese environments. This is very minor improvements (and not a reliable one) and comes at the cost of increasedcomputation time.

Original: Walker2D is the only environment that one coulduniversally recommend SD3 over TD3 as in every other environment the standard deviation curves overlap.
Revised: Walker2D is the only environment that one could universally recommend SD3 over TD3 as inevery other environment the standard deviation curves overlap.

Original: Therewas a potential issue with the Humanoid PyBullet environment, which has seemed to have hardly any change for bothalgorithms.
Revised: There seems to be consistent failues in the Humanoid PyBullet environment, which both algorithms fail to learn. 4.2 Additional results not present in the original paper

Original: [Figure]
Revised: [Figure]

Original:  Standard benchmarks are often problematic and may not always generalize (a reinforcement learning of "teaching forthe test" so to speak).
Revised: [Table] Standard benchmarks are often problematic and may not always generalize (a reinforcement learning of "teaching forthe test" so to speak).

Original: While there are proposals for other metrics of evaluation [14], we choose to simply evaluate thealgorithm on environments that are not part of the "standard" benchmark.
Revised: While there are proposals for other metrics of evaluation [18], we choose to do a simple testfor generalization by expanding the testing on environments that are not part of the "standard" benchmark.

Original: These  three environments present similarlyto the environmental results.
Revised: These These three environments offer similar results to the paper environments, a positive indication for generalization.

Original: Speciﬁcally on Pendulum and Humanoid Flagrun, SD3 offers slight advantages but ondouble inverted pendulum SD3 is far too volatile and performs inferior to TD3.
Revised: SD3performs worse on the majority of the environments once again, while also performing (on average) 5% better 3 .

Original: We acknowledge that these may notbe the optimal results for SD3, that being said, the β values are well within the standard range.
Revised: Notethat this is without substantial hyperparameter ( β ) optimization. This is indicative that even without full knowledgeof hyperparameters, reasonable choices can lead to an advantage.

Original:  These results are lessabout the potential strength of SD3 compared to TD3, and more about the generalizability and out of the box usability.
Revised: The point of these extended results is less about adirect comparison These results are less about the potential strength of SD3 compared to TD3, and more about thegeneralizability and out of the box usability.

Original:  Discussion
Revised:  Discussion

Original: First, let us consider the target questions we set in the beginning of this work.
Revised: The results presented above are not entirely conclusive.

Original: To what extent can we replicate thesuperior performance of SD3 over TD3 on  the given environments?
Revised: Although they indicate that, on average, SD3 performs superiorto TD3, this is not the end all be all. SD3 only outperforms TD3 on 3 out of the 9 environments, but it outperforms TD3 substantially (hence why the average is in its favor).

Original: Somewhat.
Revised: Note too that SD3 is a more computationally expensivealgorithm (only by a small margin, comparable to its performance gains). However, the runtime of the algorithm doesnot include the necessary hyperparameter optimization for the β value that would be needed (which could require 5-10additional runs) for any real world applications. The advantages appear to be more nuanced than the original papersuggested. 5.2 Target Questions

Original:  We were able to replicate generallygood performance  of SD3.
Revised: Next, let us consider the target questions we set in the beginning of this work. To what extent can we replicate thesuperior performance of SD3 over TD3 on the given environments? In short, we we largely not able to. We were ableto replicate superior performance on some environments and on average, but the majority of environments we werenot able to.

Original: However, the size of the advantages present in the original paper do not appear to be aslarge here.
Revised: In addition, the size of the advantages present in the original paper do not appear to be as large here.

Original: Pan et al.
Revised: Panet al.

Original: [19] presents 3 environments that SD3 deﬁnitively (i.e. the standard deviation curves do notoverlap) performs better on: Half Cheetah, Ant, Walker2d, Hopper.
Revised: [25] presents 4 environments that SD3 deﬁnitively (i.e. the standard deviation curves do not overlap) performsbetter on: Half Cheetah, Ant, Walker2d, Hopper.

Original: However, we were only able to replicate this levelof superior performance on Walker2d.
Revised: However, we were only able to replicate this level of superiorperformance on Walker2d.

Original: On the other three environments, SD3’s advantage was minimal to nonexistent.
Revised: On the other three environments, SD3’s advantage was minimal to nonexistent.

Original: This is not to suggest that there is anything wrong with their results, rather, that the results may not generalize as well as hoped.
Revised: This is notto suggest that there is anything wrong with their results, rather, that the results may not generalize (even to extremelysimilar environments, with minorly different reward scaling).

Original: Because PyBullet is not the same as MuJoCo (although the tasks are comparable) the exact comparison isn’tnecessarily important; however, the lack of clear advantages is.
Revised: This is unfortunate, as decreasing the brittleness of RLalgorithms is necessary for real world applications.

Original: To what extent does this performance generalize to other continuous control tasks?
Revised: To what extent does this performance generalize to other continuous control tasks?

Original: As was mentioned above, thisgeneralization is weak.
Revised: As was mentioned above, thisgeneralization is weak.

Original: The generalization is minimal even to the same environments in a different physics simulatorand this is also true for the new environments.
Revised: The generalization is minimal even to the same environments in a different physics simulatorand this is also true for the new environments.

Original: We cannot say  SD3 is the inferior algorithm onPendulum and Double
Revised: We cannot say deﬁnitively that SD3 is the inferior algorithm on

Original:  Inverted Pendulum as we did not do the full ablation studies to determine the optimal β values.
Revised: Pendulum and Double Inverted Pendulum as we did not do the full ablation studies to determine the optimal β values.

Original: However, we can saythat if one wants SD3 to perform deﬁnitely better, speciﬁc values of β are needed, and even then performance maynot be superior to TD3.
Revised: However, we can say that if one wants SD3 to perform deﬁnitely better, speciﬁc values of β are needed, and even thenperformance may not be superior to TD3.

Original: This need for hyperparameter optimization is a weakness of the algorithm.
Revised: This need for hyperparameter optimization is a weakness of the algorithm.

Original: Given the alreadynumerous challenges of real world RL [8], requiring extensive trial and error to obtain the necessary parameters foralgorithmic superiority is a steep price.
Revised: Given the already numerous challenges of real world RL [8], requiring extensive trial and error to obtain the necessaryparameters for algorithmic superiority is a steep price.

Revised: Given the results on a new simulator and new environments,the SD3 does not appear to generalize particularly well.

Revised: Of course, this is the case for many algorithms (and is notnecessarily a unique ﬂaw of SD3).

Original: What improvement can be made to the SD3 algorithm?
Revised: What improvement can be made to the SD3 algorithm?

Revised: Although improving generalizability is an important problem,solutions are much more difﬁcult.

Original: The most obvious improvement is to make the β parameterautomatically adjustable.
Revised: However, one solution that would help would be to enable automatic adjustmentsof the β value.

Revised: This is not only prohibitive when using SD3 for environments (as hyperparameter optimization can be expensive) but also over the course of a single run, the optimal β might differ.

Original: This idea is very similar to the improvements made to Soft-Actor critic.
Revised: This idea is very similar to theimprovements made to Soft-Actor Critic.

Original: In the original paper,the entropy parameter, α , was determined via trial and error [12]; however, automating this parameter showed to bemore effective from both a computation expense and a maximum reward standpoint [13].
Revised: In the original paper, the entropy parameter, α , was determined via trial anderror [15]; however, automating this parameter showed to be more effective from both a computation expense and amaximum reward standpoint [16]. The exact technique may not be able to carry over, as the nature of the problemsare different, but it is certainly worth investigating.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we evaluate the reprehensibility of the paper Softmax Deep Double Deterministic Policy Gradients [ 19].
Revised: In this work, we evaluate the replicatability of the paper Softmax Deep Double Deterministic Policy Gradients [25].

Original: To promote inclusive research practices, we ran all code on the open source PyBullet physics engine.
Revised: Topromote inclusive research practices, we ran all code on the open source PyBullet physics engine.

Original: Our resultsgenerally align with the original paper’s but are not very compelling overall. Some environments saw somewhat betterperformance with SD3, but many others saw similar performance with both SD3 and TD3.
Revised: Our results generallyalign with the original paper’s claims about SD3’s superior performance over TD3 overall, are not very compelling. SD3 failed to offer an advantage on the majority of environments evaluated.

Original: This may warrant furtherinvestigation there isthe potential that hyperparameter optimization would bolster the performance of SD3.
Revised: However, the average performance boostwarrants further investigation and there is potential that hyperparameter optimization would bolster the performanceof SD3.

Original: It is worthnoting that this level of scrutiny is not applied to all algorithms, and we make no claims that other SotA continuouscontrol algorithms would generalize any better.
Revised: It is worth noting that this level of scrutiny is not applied to all algorithms, and we make no claims that other SotA continuous control algorithms would generalize any better.