Paper: Reinforcement Learning with Elastic Time Steps (ICLR-2024)

Reviewer comment (Writing):
The paragraph immediately above section 3.1 indicates that there are a lot of loose ends that are not being discussed in detail, and which may strongly affect results. The imprecision of this discussion (e.g., what is a “partial MPC” and what role does the PID serve if you already use MPC?) suggests that the work may be somewhat immature.

Author response:
- In response to Q2, as described in Section 3.1, our model only provides the
  execution time of each action, which is only one part of the control of an
  actual robot. We cite MPC as just one of many possible control algorithms. We
  agree that the description can cause confusion, and we will modify the
  text to "an appropriate control algorithm."

Reviewer comment (Theory):
The reward structure discussed section 3.1 is *not* what one would properly call a “multi-objective optimization problem.” A distinguishing trait of such problems is the concept of “Pareto optimality” which encodes all of the tradeoffs among optimal performance with respect to each separate objective. By assuming a fixed weighting, this paper effectively reduces the problem to a standard optimization problem (and picks a single point on the Pareto frontier). I recommend consulting [3] for further details.

Author response:
- In response to Q3, it is a pertinent suggestion. Our core idea is to free as
  many computing resources as possible, which can then be used for perception,
  security, etc. Our description here can be improved according to the Definition
  1, we scalarize this multi-objective problem into a single value through
  weighting. We will remove the description of "multiple-objective" to ensure
  there is no ambiguity.

Reviewer comment (Theory):
The construction in Definition 1 is not as clear as it could be. For instance: are the R terms intended to be functions of state (and action)? If so, why does it make sense to only accrue reward at the times when actions are changed? Doesn’t that lead to some obvious opportunities for reward hacking? For example, could an agent decide to plow straight through some region of low reward for a bunch of (unactuated) time steps? Also, can R_t and R_\epsilon be evaluated at every time, or only at the end of an episode? Evidently, at every time t, but then I am lost as to why the agent is incentivized to minimize n, the length (in steps) of an elastic time step. I am lost.

Author response:
- In response to Q4, yes, we expect R to be a function of state (and action).
  The reward R_t for the task can be arbitrarily defined, and it does not
  necessarily require an action change to obtain the reward value. We will
  improve our description of Definition 1 to underline that R is used to
  evaluate each step individually.

Reviewer comment (Experiments):
The test environment is not very clearly explained, or at the very least, suggests a very basic question: wouldn’t it make more sense for the policy to output a force, rather than a target position? This would remove the need for lower-level tracking control (MPC, PID) and also mitigate the “measure of distance” question above, I believe.

Author response:
- In response to Q6, it is a pertinent suggestion. Setting the output to force
  is indeed a better choice. We will add additional experiments to the paper.

Reviewer comment (Writing):
I do not follow the “six dimensions of the state in the environment” - in fact, I count 9: 2 each for agent/obstacle/goal position, 2 for agent velocity, and 1 for duration. What am I missing? In the same paragraph, the discussion of semi-Markov processes and recurrence is rather opaque. Use of words like “might” and “could” lead me to wonder how clearly this point is understood. I suggest clarifying the language here.

Author response:
- In response to Q7, yes, you are right. We initially thought that time and past
  action values should not belong to the state. But we agree they should also be
  correctly defined as part of the state. We will restate the relevant
  definitions.

Reviewer comment (Evaluation):
There is little to no interpretation of the behavior of the proposed policies. Results here indicate some differences in aggregate behavior (although the interpretation to that effect should really require some error bars as above), but it would really help to understand what is going on if the authors expanded upon Fig. 7 to illustrate what was going on in the environments in these situations and why it made sense to change the control rate as shown.

Author response:
- In response to Q9, Figure 7 records the time-related generation strategy of
  the SEAC model in this environment. We do not have additional drawings because
  SAC and PPO are both fixed times (such as 5 Hz mentioned in the article).
  Figure 7 needs to be examined together with Figure 8: the SEAC model strategy
  shows the use of fewer time steps in Figure 8. In Figure 6, SEAC also has a
  shorter completion time, showing an advantage of SEAC in terms of total energy
  consumption. We will add some additional explanation diagrams based on Figure
  7 to explain the working of SEAC, SAC, and PPO strategies to make our work
  more understandable and readable.

Reviewer comment (Experiments):
The main motivation here seems to be saving computational resources. Obviously, most control systems are pretty lightweight and so I imagine these savings really come in from the perception side, e.g., if you no longer have to process big images at high frame rate. Experimental results to illustrate these savings more directly than the abstraction of “number of repeated actions” would be highly motivating.

Author response:
- In response to Q10, thank you for recognizing our core concept of reducing the use of
  computing resources. Based on your feedback, we will focus more on this aspect
  in the text if modifications are needed. Unfortunately, we have not
  yet completed the hardware experiment.

Reviewer comment (Experiments):
Figures 5 and 6 could be more clear about indicating that the right hand sides are insets of the left. Also, why were the methods run for so long - it seems they all converged quite a bit earlier and then for some reason PPO destabilized. Something seems off here.

Author response:
- In response to Q13, because the environment and tasks we set are relatively
  simple, these algorithms have converged around 3 million steps. However, we
  noticed that PPO would be unstable in this final period. We are not fully able to
  explain the behavior of the neural network, and we are reporting it as is.

Reviewer comment (Presentation):
Fig. 7 says “epochs” instead of “configurations.”

Author response:
- In response to Q14, thanks for the feedback. It should not be "epoch" but
  "episode." We will fix it.

Reviewer comment (Experiments):
There are many existing studies with varying time (e.g. option framework, action repetitions…). Authors introduce some notions of options and semi-MDP in the appendix, but without clear definitions of each notation, which makes it harder to see the clear connections to the main work and the option framework. It was not clear how the authors validated Bellman-like equations for the elastic time case. Assuming the algorithm is properly derived from the option framework, it is necessary to compare to the existing work based on the framework. Or at least it should show significant practical results compared to the existing work; it seems the experiments are not for sufficiently complex tasks.

Author response:
- In response to Q1, our work differs from action repetition or the options
  framework (HRL for extended actions). These methods do not reduce the amount
  of steps and thus do not reduce computational resources. For example, the
  agent with SEAC policy can reduce the frequency at which it samples the
  environment. However, we will add relevant references to
  improve our literature review so that readers can more clearly understand the
  differences with our work.

  As for the issue in Appendix A, we will modify the relevant description to
  clarify it. Although we do not have complex experimental results, we are
  working in this direction. Our core idea can save computing resources compared
  with methods such as action repetition.

Reviewer comment (Novelty):
Existing environments such as OpenAI Gym can be easily adjusted to include time as information for states; it is unclear what the authors mean by “…additional input and output information that is not available within existing RL environments…”. Note that simulators anyway need to run with a small time interval to maintain accuracy, and action durations can be just a repetition of that.

Author response:
- In response to Q2, additional input refers to the historical value of the
  agent, the duration of the previous step, and the action value of the last
  step. The other output refers to the duration of the action to be performed.
  We have a maximum frequency limit to ensure accurate execution of the
  action, which is 100Hz. The above information can be found in Sections 3 and 5
  of the article.

Reviewer comment (Presentation):
Figure 5 is a bit hard to parse: why are times in seconds negative? This could be guessed, but it would be better to make them crystal clear.

Author response:
- In response to Q3, thanks for pointing it out. We will fix Figure 5.

Reviewer comment (Experiments):
It would be better to show a baseline with the 100 Hz (fixed) case, not 5.0 Hz, since the elastic one uses 1 to 100 Hz.

Author response:
- In response to Q4, it is a good proposal. We can do more groups, such as 1 Hz,
  10 Hz, 60 Hz, 100 Hz, etc. We will use your feedback to improve our
  experiments.

Reviewer comment (Writing):
What is the action space A? Is it the Cartesian product of “action” and “time”?

Author response:
- In response to Q7, the action set is (time for next step, move distance on X,
  move distance on Y). Yes, it is the Cartesian product of "time" and "action".
  The specific information can be viewed in Definition 10 of Appendix B.

Reviewer comment (Writing):
The paragraph after Definition 1 (“We validate our reward strategy…”) could be rephrased to highlight SEAC differences compared to SAC.

Author response:
In response to Q4, it is a pertinent suggestion. The loss function of SEAC is
  consistent with that of SAC. As you proposed next, we will modify the
  expression here so that people can clearly understand that we only implement
  Definition 1 through SAC. The same idea can be achieved with other types of
  reinforcement learning algorithms. It is worth noting that we didn't create a
  new reinforcement learning algorithm.

Reviewer comment (Writing):
What do you mean when you say “…giving a high probability that the agent can discover the optimal solution to complete the task”? Maybe this sentence can be rephrased to make the exploration strategy clearer.

Author response:
In response to Q5, we will rewrite it as "make the agent learn the best time and
  corresponding action policy".

Reviewer comment (Presentation):
In general, from the sentence starting “we assume the agent…” to the sentence ending with “…Bellman equation”, I find the flow of the text, which can be read while referring to the scheme on Fig 3, a little hard to follow, in the sense that it jumps from one block to another one (of the Fig.3) without a precise order. Incorporating more references to the visual scheme and aligning the text with the functional flow of the figure 3 (rather than simply listing the meaning of the symbols) could help the readability.

Author response:
In response to Q6, Q7, and Q8, they are pertinent suggestions. We can modify
  Figure 3 and its caption describe more clearly how some low-level
  parameters, such as time t, etc., participate in the training process.

Reviewer comment (Presentation):
You mention that one major contribution of the SEAC is to include the execution time of each action to the output, but this term is not explicitly indicated on Fig.3, together with the At.

Author response:
In response to Q6, Q7, and Q8, they are pertinent suggestions. We can modify
  Figure 3 and its caption describe more clearly how some low-level
  parameters, such as time t, etc., participate in the training process.

Reviewer comment (Presentation):
The meaning of the double arrows in Fig.3 is not very clear to me. Maybe an explanation could be included either on the caption or on the main text.

Author response:
In response to Q6, Q7, and Q8, they are pertinent suggestions. We can modify
  Figure 3 and its caption describe more clearly how some low-level
  parameters, such as time t, etc., participate in the training process.

Reviewer comment (Reproducibility):
The impact value of the execution is defined, based on the chosen environment, as the target movement distance. Do you have in mind some examples of different implementations for different problems?

Author response:
In response to Q9, yes, such as the classic robot arm problem: one assumes
  that the output of the strategy set is a force set. Suppose you want to
  quickly grab a fragile object, such as an egg, without considering other
  interferences, such as the material of the robot arm. In that case, the grip
  strength of the robot arm and direction are essential, and the duration of the
  force is even more critical. Any application of reinforcement learning
  algorithms in real-world environments of continuous control type can adopt our
  elastic time step idea.

Reviewer comment (Writing):
In the end of paragraph 3.1, when you say “our objective is for the agent to learn the optimal execution time”, is the execution time equivalent to the action time, and therefore represented by Tt?

Author response:
In response to Q11, yes, the execution time equals the action time.

Reviewer comment (Writing):
Typo: “but but” in the sentence starting with “it is worth noting…” in paragraph 3.2

Author response:
In response to Q12, sorry, we will fix it.

Reviewer comment (Presentation):
Since the SAEC loss functions are (if I understand well) equal to those of SAC, rather than simply reporting the definitions, I would suggest to reorganize Section 4 to better explain how your formulation of the reward function is included in the update steps of the RL algorithm.

Author response:
In response to Q14, it is a good suggestion. We can rephrase Section 4 to show
  how some low-level parameters work with the model training based on a modified
  Figure 3.

Reviewer comment (Writing):
Section 5: When you refer to the “three RL algorithms”, do you mean SEAC, SAC and PPO? In this case, you should first say that you are comparing SEAC results with SAC and PPO in the text, otherwise it is not clear to the reader.

Author response:
In response to Q15, it is a good suggestion. We will modify the description to
  explain what kind of algorithms we are comparing.

Reviewer comment (Presentation):
I think that Fig. 7, as it is, is not very informative. It shows that SEAC dynamically changes the control rate, but it doesn’t allow to evaluate whether it does it in a meaningful way. Showing the scenario and/or information about the corresponding actions would make the concept clearer.

Author response:
In response to Q16, our original intention with Figure 7 was to inform readers
  that SEAC takes fewer time steps to achieve the set goals. The agent will
  first take a long step to achieve the goal and then fine-tune it. We will add
  some model explanation diagrams based on Figure 7 to explain the workings of
  SEAC, SAC, and PPO strategies to make our work more understandable and
  readable.

Reviewer comment (Presentation):
I feel Fig.8 would be more readable by inverting x and y axes (evaluation metric on the y-axis). Furthermore, you mention the overall reward both in the section and in the figure caption, but is the overall reward shown somewhere?

Author response:
In response to Q17, it is a good suggestion. We can change it to a rain cloud
  image. Figure 5 is realistic about the overall reward. As defined in 1, the
  total reward includes the number of steps and time used.