Paper: Scaling Safe Learning-based Control to  Long-Horizon Temporal Tasks (ICLR-2024)

Reviewer comment (Writing):
nit: there are quite a few sporadic syntax and grammar issues, e.g. in the first paragraph there are two instances of improper spacing near punctuation marks. Similar issues around references: you may consider using the \citep{} option for parenthetical citations.

Author response:
We will carefully review the language and polish it, we will also use \citep{} for citations.

Reviewer comment (Theory):
In Section 3, the paper clearly states that a condition must be satisfied for all initial states, yet the training objective is an expectation measured at only a finite number of samples. The appendix contains an abrupt mention of this point which leads me to believe there is more going on here, but it is a fairly serious omission from the main text and is also not really clarified in the appendix.

Author response:
You are right. The problem that we state is the ultimate goal for our training algorithm, but in the main body of the paper, we do not address this problem. However, we added a footnote (see footnote 3 in the updated text document) to clarify this point for the reader. We need to add, there are prior techniques in the literature (such as [1]) which can give us provable statistical guarantees. We refer you to Appendix H, which applies the technique from [1] to give probabilistic verification guarantees. We note that we still use the relaxed training objective (expectation of robustness values over initial conditions rather than guaranteeing that the min robustness value over all initial conditions is positive). We choose the relaxed condition because ensuring the stricter condition is intractable for both classical control techniques (due to highly nonlinear environments) and neural network controllers (due to lack of guaranteed optimization algorithms).

Reviewer comment (Theory):
In “Challenge 1” and elsewhere, it is asserted that existing frameworks for stochastic optimization cannot handle non-smooth objectives. This is patently false: every ReLU network ever has been non-smooth, and yet SGD/Adam/… seem to do just fine. Obviously, there is also a rich theory in non-smooth optimization, sub-gradient methods, etc. My objection here is mainly that the paper just asserts that smoothness is critical without ever supporting that claim.

Author response:
You are absolutely right, this was poor writing on our part. What we meant to say is that in the context of the robustness of STL specifications, non-smoothness becomes a deterrent. For example, previous work in [3] compares smooth STL semantics with MILP based formulations that use non-smooth, min-max based STL semantics, and shows considerable advantages in using smooth semantics. Thus, we assume community knowledge of sub-optimality in using non-smooth STL semantics for controller synthesis to justify our use of smooth semantics that are guaranteed lower bounds on the actual robustness.

Reviewer comment (Theory):
Where is the proof of Lemma 1? It seems important, but is nowhere to be found. Same for the equation at the bottom of page 4.

Author response:
We omitted a formal proof due to lack of space, but hoped to give insight into the proof in the paragraphs preceding the Lemma in Section 3.1. Here’s a brief proof sketch:
STL2NN is syntactically equivalent to the robustness function in [2] as it is simply a reformulation of min/max operations in the robustness function as ReLU units. The main idea here is that:
\\[
	\\min(a_1 , a_2) =  a_1 - ReLU(a_1-a_2)    \\quad(i.a)  \\quad \\& \\quad \\ max(a_1 , a_2) = a_2 + ReLU(a_1-a_2)   \\quad(i.b)
\\]
Next, we can show that
\\[ 
\\forall x: \\quad  \\text{swish}(x) < ReLU(x)  \\quad(ii.a) \\quad \\& \\quad  \\forall x:  \\quad \\text{soft-plus}(x) > ReLU(x)  \\quad(ii.b)
\\]

This follows from standard nonlinear inequalities about log/exp functions. For example, here’s the proof for (ii.a) by doing case analysis:
Note that $ ReLU(x) = \\max(x,0)$ and $\\text{swish}(x) = x/(1+e^{-bx}), \\quad b>0$.

Case I: $ x > 0  \\quad \\implies \\ max(x,0) = x$. Since, $\\forall x \\in \\mathbb{R}:\quad  e^{-bx} > 0$, we conclude $ (1+e^{-bx}) > 1$, so $x/(1+e^{-bx}) < x$ 

Case II: $x \\leq 0 \\quad \\implies \\ max(x,0) = 0$, while $\\text{swish}(x)$ is a negative number. <completing the proof>

We can similarly show that $\\forall x \\in \\mathbb{R} \\quad  \\text{soft-plus}(x) > ReLU(x)$

We can use (ii.a) and (ii.b) to show that replacing ReLU in (i.a) by soft-plus and ReLU in (i.b) by swish guarantees that the RHS expressions are lower-bounds on min() and max() respectively.
 
To see this, let’s define the smooth functions  $\\mathsf{mins}()$ and $\\mathsf{maxs}()$  as follows:
\\[   
  \\mathsf{mins}(a_1 , a_2) =  a_1 - \\text{soft-plus}(a_1-a_2) , \\quad (iii.a) \\quad \\& \\quad   \\mathsf{maxs}(a_1, a_2) = a_2 + \\text{swish}(a_1-a_2) \\quad (iii.b)
\\] 
From (ii.a), (ii.b) we can see that 
\\[
\\min(a_1 , a_2) >  \\mathsf{mins}(a_1,a_2) \\quad (iv-a)  \\quad \\& \\quad   \\max(a_1 , a_2) > \\mathsf{maxs}(a_1,a_2) \\quad (iv-b).
\\]
Rest of the proof follows by structural induction on the syntax tree of a given STL formula. Essentially, the robustness computation of an STL formula requires the robustness values of all subformulas over all times in the relevant time-scope of the formula. We can show that for all STL operators (i.e. $\wedge, \vee$, $\mathbf{F}$, $\mathbf{G}$, $\mathbf{U}$, $\mathbf{R}$), the robustness computation is essentially a $\\min()$ or $\\max()$ over robustness values of sub-formulas. Since robustness computation using $\\mathsf{mins}()$ and $\\mathsf{maxs}()$ is a lower bound for the actual $\\min()/\\max()$ operation, the robustness computation operations using $\\mathsf{mins}()/\\mathsf{maxs}()$ semantics also lower bound the robustness of the actual formula.

We also added a proof sketch to the updated text document, please see Appendix J.

Reviewer comment (Theory):
The modified swish and soft plus functions below Ex. 1 appear to be non-smooth, and what smoothness there is derives only from numerical precision issues. This is less than satisfying. Surely if we are hitting numerical precision issues that points to something more subtle going on, right? For example, it is well-documented in the literature that unstable closed loop dynamics yield exploding gradients in these kinds of policy optimization problems.

Author response:
You are right that these functions are not smooth. However, please note that even a smooth function will face numerical precision issues due to fundamental limits on the smallest number that can be represented in any modern computer. Both Matlab and Python have a pre-specified tolerance for numerical precision, and they do not recognize numbers less than this precision. As we have shown in our response to reviewer 1, by picking a value (720) for the hyper-parameter $\tau$, we can make the difference in the derivative of the modified swish and soft-plus functions with their smooth form, as small as $10^{-310}$. Increasing $\tau$ will make the difference even smaller. The numerical precision for Python/Matlab for “double” precision numbers is of the order of $10^{-308}$ (we selected $\\tau=50$ in our experiments). Thus, we argue that from a computational perspective, the functions can be practically smooth. We also remark that the advantage offered by the modified swish/soft-plus functions outweighs any concerns about computational difficulties when compared to using log-exp-sum or the Boltzmann operator. The latter operators make it impossible to compare both large and very small numbers due to numerical issues and thus impossible to compute smooth versions of the min/max operations. For example, log-exp-sum can not find the maximum of 80 and 81 if b=10,
\\[
\\text{log-exp-sum}: \\quad \\widetilde{max}(80,81) = \\frac{1}{10} \\log (\\exp(10 \times 80)+ \\exp(10 \times 81))
\\]
as you see $\\exp(800)$ and $\\exp(810)$ are not computable in MATLAB.

Reviewer comment (Writing):
What is the parameter \bar\rho in Alg. 1? (nit: note also that “Algo” is not a common abbreviation for Algorithm. I believe the IEEE standard, at at least, is “Alg.”)

Author response:
We will revise the abbreviation for Algorithm. The parameter $\\bar{\\rho}$ serves as a lower threshold for the required robustness that the controller must attain. When $\\bar{\\rho}=0$, the controller is trained solely to satisfy the given temporal specification. Alternatively, setting $\\bar{\\rho} > 0$ allows us to enhance safety levels by achieving a robustness greater than zero.

Reviewer comment (Reproducibility):
Is the matrix S actually being approximated at specific rows, or are entire rows being left out?

Author response:
You are correct – this is a good catch. It is true that guarantees indeed come from evaluating the smoothed STL formula. What we wanted to say is that the re-smoothing process may decrease the value of STL2LB. The sentence as written was incorrect/confusing, and we have removed it from the updated text document.

Reviewer comment (Theory):
How is it more efficient to compute the gradient at a sampled point? Doesn’t this essentially require backpropagating through all time steps from the end of time to the beginning, regardless of whether or not you are going to then throw away some of the gradient information?

Author response:
We have fixed this.

Reviewer comment (Theory):
Please help me to understand the last sentence before Section 4. Didn’t the guarantees come from simply evaluating the (potentially smoothed) STL formula from every initial condition—i.e., the optimization objective? Why does it matter if we change the algorithm used to approximate gradients?

Author response:
There are two main messages that we want to convey in this paper: (1) it is important for the smooth robustness semantics of STL to be rigorous lower-bounds for the actual robustness, (2) synthesizing neural network based controllers for long horizon temporal tasks faces problems of vanishing and exploding gradients, and to make learning-based synthesis feasible, we need better algorithms to approximate the gradient.

The related work in this space is along two orthogonal directions: (1) the kind of smooth approximations to min/max used to give smooth semantics for STL robustness, (2)  the type of control policy obtained by the synthesis algorithm. Most of the related work along the second direction, focuses on generating {\\em open loop} control policies (i.e. directly synthesizing the actions). It is well-known that open loop policies are not robust to noise, hence we prefer closed-loop feedback controllers. The only work that synthesizes closed-loop feedback control is a baseline that can lead to an apples-to-apples comparison. The comparison to open-loop policy synthesis is unfair because the complexity of generating open-loop policies is much lesser.

The other dimension is about smooth approximations of min/max. Superficially, the computational burden using STL2LB and other smooth semantics may look similar. However, during training, if robustness values of the signal predicates become high or low, then other smooth min/max versions give incorrect answers, potentially giving nonsensical values for the gradient. This is a serious computational issue.  You can see example 1 in the paper how the gradient computation will get affected.

    1- Lars Lindemann, Lejun Jiang, Nikolai Matni, and George J. Pappas. Risk of stochastic systems for temporal logic specifications, 2022. URL https://arxiv.org/abs/2205.14523.

    2- Alexandre Donzé and Oded Maler. Robust satisfaction of temporal logic over real-valued signals. In International Conference on Formal Modeling and Analysis of Timed Systems, pp. 92–106. Springer, 2010.

    3- Pant, Yash Vardhan, Houssam Abbas, and Rahul Mangharam. "Smooth operator: Control using the smooth robustness of temporal logic." 2017 IEEE Conference on Control Technology and Applications (CCTA). IEEE, 2017.

    4 - Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.

Reviewer comment (Experiments):
Lack of comparisons made to the state-of-the-art methods [1,2,3] for control using STL. The presented algorithms are shown without reporting results on enough competing methods. This brings into question the relative benefits of the given approach.

Author response:
Weakness 1 :  All other papers using smooth semantics of STL do not use neural network based feedback controllers, and it is unclear how comparing against them can demonstrate the effectiveness of our approach. For example, in [3,7,1], the authors do not train a controller, but instead only obtain an open-loop control policy from a single initial state.  The only previous work that also trains closed-loop controllers is [5], and the summary of our comparison is in Section 4, under the heading, “Comparison.” Both our paper and [5] train controllers over a set of initial states. 

Reinforcement Learning based techniques such as [9, 10] are either tabular techniques that do not use NN controllers, or may not use smooth semantics [11],  may not be scalable on  long time horizons [5, 4, 12, 1 , 2, 3 ] (longest considered time horizon in these papers is 100 time steps). 

A separate study could be done where we compare only the function STL2LB with other notions of smooth semantics in the context of synthesizing open-loop control policies.  Superficially, the computational burden using STL2LB and other smooth semantics may look similar. However, during training, if robustness values of the signal predicates become high or low, then other smooth min/max versions give incorrect answers, potentially giving nonsensical values for the gradient. We added Appendix L to the updated text document that provides this comparison. Here we intentionally select a spacious environment to have large enough quantitative values for predicates which results in failure for [3],[4], and we also show that STL2LB successfully solves this problem.

Reviewer comment (Experiments):
The motivation for a smooth lower bound on the STL score is mentioned but not sufficiently justified (viz. empirically). It would be interesting to see how far the STL2NN method would work without the approximations provided by STL2LB in Algorithm 1. Another useful smoothing technique could be as introduced in [3].

Author response:
Weakness 2 :  Although we have already made a comparison between STL2LB and the smooth formula utilized in [5], we will also  include a comparison with the notion introduced in [3]. We were not aware of this paper at the time of submission, so we thank you for giving us a pointer to this very useful work.

The value computed by STL2NN is the same as the robustness function in [6]. However, it is well known in the STL community that the semantics in [6] do not lead to an efficient training process due to the function not being differentiable (see the abstract of [4] and their experimental results).

Reviewer comment (Theory):
Assumes differentiability of the simulator environment and knowledge of its transition functions to calculate the policy parameter gradients. This may be infeasible in many problems.

Author response:
Weakness 3 :  We agree that the assumption to access the transition function may be infeasible for many problems. However, we can combine our technique with methods for learning dynamical system models which are common in control theory and model based reinforcement learning.

Reviewer comment (Writing):
In which steps in Algorithm 2 is the critical predicate $h^*$ and $k^*$ used? This is not entirely clear to me.

Author response:
Question 1 :  The critical time is used in line 11 of Alg.2. Once the critical predicate h* is found and the critical time k* is located on the trajectory, we approximate the gradient of h* with respect to control parameters. To that end, we sample arbitrary time steps between k=0 and k=k* while we make it certain that k=0 and k=k* are both included in the set of sampled times. We then generate the sampled trajectory, and finally we utilize this sampled trajectory to compute the approximate gradient of h* with respect to the control parameters. We note that the use of critical predicates and critical times allows us to approximate the gradient. This idea of sampled trajectory and gradient sampling is similar in spirit to the idea of drop-out layers [8].

Reviewer comment (Experiments):
Is it possible to include comparisons to other algorithms in the same environment such as [1]? If not, why is that the case?

Author response:
Question 2 :  The work in [1] uses smooth semantics of STL but does not use neural network based feedback controllers, but instead, it focuses on obtaining an open-loop control policy from a single initial state.It is unclear how comparing against this work can demonstrate the effectiveness of our approach. Furthermore, there is no explicit consideration of long horizon temporal tasks in [1], and it is unlikely that the technique in [1] will cope with numerical issues introduced by long horizon specifications. We will include new results that experimentally demonstrate this and update the rebuttal with a summary.

Reviewer comment (Reproducibility):
The introduction mentions an RNN-based implementation, but this is not explained further in the text. Could there be a section in the appendix with more implementation details? Is there a clear benefit versus not using a fully connected network with the observation (and say the current time) as input being the policy?

Author response:
Question 3 :   

We agree that we could have explained this further.  Our technique does not use an RNN-based controller. What we want to highlight is that the composed system, consisting of the plant dynamics, the neural network-based controller unrolled over the task horizon creates a structure that is very similar to that of sequence-to-sequence RNNs. Thus, the challenges of training such RNN structures are also challenges that we face. We will make this clearer in the text to avoid confusion.

We added more details in the appendix K of the updated text document, with an illustration that shows the structure of the control system that we consider. In fact, both your suggestions already exist in the paper: the control action is a fully connected feedforward neural network and is only dependent on the state at a given time, and the value of the time step itself.

Reviewer comment (Presentation):
The paper was difficult to follow. In particular, section 3.3 detailing the sampling based approximation of the gradient was quite hard to understand. A figure to help illustrate the trajectory subsampling approach would significantly improve the clarity of the paper.

Author response:
Weakness 1 :  The gradient approximation proposed in this paper can be viewed as a form of drop-out [1]. In drop-out, while neurons are used for back-propagation with some probability, in our setting, we replace entire layers with an approximate value ( A simple numerical example is also provided for  Reviewer 4 ). We included a figure (See figure 2) in Appendix A  for an example of gradient approximation technique to add clarity to this discussion.

Let me explain our gradient approximation method from the perspective of drop-out layers [1]. Considering NN controllers rolled out over the trajectory, in drop-out layers, we remove randomly selected nodes from a neural network (in our case, this corresponds to removing random nodes from arbitrary time-points of the NN representing the controller rolled out over the trajectory horizon). Drop-out requires the node to be absent in both the forward pass and the backward pass in the backpropagation algorithm. To alleviate the problem of vanishing and exploding gradients, instead of picking arbitrary neurons in the controller (at arbitrary time-points), we sample certain time-steps and select all of the controller neurons in that time-step. 
Due to the reduced number of multiplications, this alleviates this problem of vanishing/ exploding gradients. However, we cannot simply remove all controller nodes at a specific time step, as the resulting composed NN can become disconnected. Thus, when we drop out selected neurons, we replace those neurons with a node representing a constant function whose value is equal to the evaluation of the controller unit in the forward pass. This argument is the main motivation behind the definition of sampled trajectory.

Reviewer comment (Experiments):
The impact of STL2LB as compared to other strategies to smooth STL formulae was not clearly demonstrated.

Author response:
Weakness 2 :  All other papers using smooth semantics of STL do not use neural network based feedback controllers, and it is unclear how comparing against them can demonstrate the effectiveness of our approach. For example, in [3,4,5,8], the authors do not train a controller, but instead only obtain an open-loop control policy from a single initial state.  The only previous work that also trains closed-loop controllers is [2], and the summary of our comparison is in Section 4, under the heading, “Comparison.” Both our paper and [2] train controllers over a set of initial states. Reinforcement Learning based techniques such as [9, 10] are either tabular techniques that do not use NN controllers, or may not use smooth semantics [11], and may  not scale to long time horizons [2, 3, 4, 5 , 6, 7, 8 ] (longest considered time horizon in these papers is 100 time steps). 

A separate study could be done where we compare only the function STL2LB with other notions of smooth semantics in the context of synthesizing open-loop control policies.  We expect that the computational efficiency to be similar as other approaches; however, we remark that only our approach will guarantee soundness, i.e., if given a single initial state a control policy is obtained that has positive value of the STL2LB function, then the system behavior under this policy is guaranteed to satisfy the STL formula. This is not true for other smooth semantics. We argue that this study is somewhat orthogonal to the objectives of our paper (scaling neural network controllers to satisfy long-horizon tasks, and pushing the training algorithm towards satisfying safety guarantees).

Reviewer comment (Experiments):
The experimental comparisons were very limited: quantitative comparisons were only presented against ablations of the proposed approach, and not against many of the other cited works on training NN policies to satisfy STL objectives. In addition, there was no comparison against an approach which used the critical predicate-based time sampling, but without the waypoint functions. Thus it was not clear what improvement the critical predicate-based sampling strategy had over random time sampling.

Author response:
Weakness 3 :  We omitted the ablation study that involved using only critical predicate-based time sampling but not waypoint functions due to space restrictions. Our study shows that critical predicate-based time sampling still outperforms random time-sampling (even without waypoint functions), and adding waypoints functions boosts the performance even further. We have included new experiments to demonstrate this aspect. Please see Appendix G in the updated text document.

Reviewer comment (Theory):
The paper would be strengthened with theoretical results detailing what factors impact the quality of the sampling-based gradient approximation.

Author response:
Weakness 4 :  This is indeed a very interesting suggestion that we will consider for future work. Critical predicate-based sampling is quite property and system dependent, it is unclear if we can make general statements about the training algorithm for all STL formulas and environments. However, the general idea of dropping layers in an unrolled recurrent structure can be explored theoretically. One idea is to characterize the worst-case error introduced by the drop-out-like approximations [1] (that we introduce), and show that our training algorithm converges to the same (local) optimum as a training algorithm that does not use any dropout. We will look at the literature from stochastic gradient optimization for future work.

Reviewer comment (Experiments):
Do you have empirical results investigating how well the sampling-based gradient strategy approximates the true gradient?

Author response:
Question 1 :     We remark that the true gradient involves backpropagation over the entire trajectory, which often leads to vanishing or exploding gradients in long horizon specifications. Our sampling-based gradient, on the other hand, does not have this problem. By definition, these two gradients are not expected to be close to each other in value; however, we expect their directions to be approximately similar. However, it is still difficult to check if the inner product of the unit vectors in each gradient’s direction is 1 (due to the exploding/vanishing gradient problem). Hence, our approach to show this is through the ablation study in Table 1. We can argue that drop-out based techniques [1] have efficacy due to similar reasons.

Reviewer comment (Theory):
The solution to the computational problems with the swish and softplus functions seems to break differentiability, especially in the case of the swish function. Would this lead to issues during optimization?

Author response:
Question 2:  You are right, the modified swish function is not differentiable. However, in practice, we observe that we can pick the hyperparameter $\tau$ to guarantee that the derivatives are numerically close when we approach the point of non-differentiability from either side. Recall that
\\begin{equation}
\\text{if $\\zeta > -\\tau/b$}  \\quad \\text{then} \\quad \\widetilde{swish}(\\zeta)=
swish(\\zeta),
\\end{equation}
\\begin{equation}
\\text{if $\\zeta < -\\tau/b$}  \\quad  \\text{then} \\quad \\widetilde{swish}(\\zeta)=0 
\\end{equation}
For example, we can set $ \\tau = 720$. For $b=10$, then the point of non-differentiability is when x = -72. At this point, derivative of $\\widetilde{swish}(x)$  is  $-1.46\times10^{-310}$ (approaching from the *right*) and 0 approach from the left, which is numerically an insignificant difference, and smaller than the least number that can be represented in double precision in MATLAB. Similarly, for modified softplus, the derivatives from either side at x = 72 are identical from the perspective of numerical precision. However in our experiments we set $ \\tau$ =50, and for b=10 the point of non-differentiability is when $x=-5$, At this point the derivative of $ \\widetilde{swish}(x)$ is  $-9.4509 \\times 10^{-21}$, while the derivative of $y=0$ at $x=-5$ is equal to 0. Thus, we  conclude a difference of $9.4509 \\times 10^{-21}$, in practice, may not be problematic regarding differentiability. In addition, a simple calculation for the derivative of $\\text{softplus}(x )$ at $x=5$, $b=10$ implies the slope of $y=softplus(x)$ at $x=5 $ is $(1 - 1.9287 \\times 10^{-22})$ which is very close to $1$ this is while the slope of $y=x$ is exactly equal to $1$. Thus, we conclude a difference of $1.9287 \\times 10^{-22}$, in practice, may not provide computational error related to differentiability.