Paper: Neural Contractive Dynamical Systems (ICLR-2024)

Reviewer comment (Experiments):
Limitation of baseline method choices: The baselines compared to are all focused on asymptotic stability guarantees. While these are the most relevant methods for comparison, it would be interesting to see how imitation learning methods without any stability guarantee works on the tasks, to demonstrate the necessity of stability guarantee.

Author response:
- “*Limitation of baseline method choices: The baselines compared to are all focused on asymptotic stability guarantees. While these are the most relevant methods for comparison, it would be interesting to see how imitation learning methods without any stability guarantee works on the tasks, to demonstrate the necessity of stability guarantee.*”  
    - **Answer:** As promised, we have performed new experiments that include imitation learning without stability guarantees. To do so, we have chosen two different baseline approaches, including MLP and Neural Ordinary Differential Equation (NeuralODE) networks. The MLP-based baseline uses a neural network with $2$ hidden layers each with $100$ neurons with $\tanh$ activation function. The Neural ODE-based baseline model is implemented based on [1], with 2 hidden layers each with 100 neurons. The model is then integrated into a NeuralODE framework. The NeuralODE is configured with the 'adjoint' sensitivity method and utilizes the 'dopri5' solver for both the forward and adjoint passes. Additionally, adjustable tolerances for the adjoint solver are set with `atol_adjoint=1e-8` and `rtol_adjoint=1e-8`. It is important to mention, that these baselines directly produce the velocity and model $\dot{x}= f(x)$. These models were trained with the same cost function as NCDS’s Jacobian network defined in Equation 14.  Both networks were trained for 1000 epochs with an Adam optimizer.  Figure 15-c and -d show that both models effectively model the observed dynamics (vector field); however, as anticipated, stability is not achieved. Similar results are reported in [2].  
    - **Revisions:** We have updated section **A.6.3: Unconstrained dynamical system** with the new results.  

[1] Poli, M., Massaroli, S., Yamashita, A., Asama, H., Park, J., & Ermon, S. "TorchDyn: Implicit Models and Neural Numerical Methods in PyTorch." https://github.com/DiffEqML/torchdyn/  

[2] Rodriguez, I.D., Ames, A., & Yue, Y. (2022). LyaNet: A Lyapunov Framework for Training Neural ODEs. *International Conference on Machine Learning*.

Reviewer comment (Reproducibility):
Is training end-to-end? What are the loss functions used for training?

Author response:
- *“Is training end-to-end? What are the loss functions used for training?”*  
    - **Short answer:** Training **NCDS is end-to-end**, but the **latent NCDS is not fully end-to-end**.  
    - **Detailed answer:** Training NCDS is end-to-end, but the latent NCDS is not fully end-to-end. In the latter case, we first train the VAE (end-to-end), and then train the latent NCDS using the encoded data. In more detail, the VAE is trained using the standard evidence lower bound (ELBO):  
        
        $$\\mathcal{L} _{ELBO} = \\mathbb{E} _{q _{\\mathbf{\\xi }}(\\mathbf{z}|\\mathbf{x})}\\left[\\log(p _{\\mathbf{\\phi}}(\\mathbf{x}|\\mathbf{z}))\\right] - \\mathrm{KL}\\left(q _{\\mathbf{\\xi }}(\\mathbf{z}|\\mathbf{x})||p(\\mathbf{z})\\right) , $$  
        
        where $\mathrm{KL}$ denotes the Kullback-Leibler divergence.  
        
        Thereafter, the Jacobian network is trained according to the loss function $\mathcal{L}_{\text{Jac}}$, which measures the distance between the observed and approximated next state:  
        
        $$ \\mathcal{L} _{\\text{Jac}} = \\|\\mathbf{z} _{t+1} - (\\mathbf{z} _{t} + \\hat{\\dot{\\mathbf{z}}} _t)\\|^2,$$  
        
        where $\\mathbf{z} _{t}$*, $\\mathbf{z} _{t+1}$*, and $\\hat{\\dot{\\mathbf{z}}}_t$ represent the current and next observed latent states, along with the calculated latent velocity.  
        
    - **Paper changes:** we have updated the paper and added the loss functions and the algorithms in the Appendix A.4: Implementation details.  

---

**Algorithm 1**: *Neural Contractive Dynamical Systems (NCDS): Training in task space*  

---

**Data**: Demonstrations: $\\tau _n = \\left \\{(\\mathbf{x} _t, \\mathbf{R} _t) \\right \\},  n \\in [1, N], t \\in [1, T _n]$  

**Result**: Learned contractive dynamical system  

1. $\\mathbf{r} _{n,t} = \\operatorname{Log}(\\mathbf{R} _{n,t})$; // Obtaining skew-symmetric coefficients  
2. $\\mathbf{p} _{n, t} = [\\mathbf{x} _{n, t}, \\mathbf{r} _{n, t}]$; *//* Create new state vector  
3. $ \\operatorname{argmin}_\\xi \\mathcal{L} _{\\text{ELBO}}(\\xi^*; \mathbf{p} _{n,t})$; // Train the VAE  
4. $\\mathbf{z} _{n,t} = \\mu _ {\\mathbf{\xi}} ^{\\sim 1} (\\mathbf{p} _{n,t})$; // Encode all states using the trained VAE  
5. $\\dot{\mathbf{z}} _{n,t} = \\frac{\\mathbf{z} _{n,t+1} - \\mathbf{z} _{n,t}}{\Delta t}$; // Compute latent finite differences  
6. $ \\operatorname{argmin}_\\theta \mathcal{L} _{\\text{Jac}}(\\theta^*; \\mathbf{p} _{n,t})$; // Train the Jacobian network  

---

**Algorithm 2**: Algorithm Neural Contractive Dynamical Systems (NCDS): Robot Control Scheme  

---

**Data:** Current state of the robot end-effector at time $t$: $[\\mathbf{x} _t, \\mathbf{R} _t]$  

**Result**: Velocity of the end-effector at current time step $\\dot{\\mathbf{x}} _t$  

1. $\\mathbf{r} _{t} = \\operatorname{Log}(\\mathbf{R} _{t})$; // Obtaining skew-symmetric coefficients  
2. $\\mathbf{p} _{t} = [\\mathbf{x} _{t}, \\mathbf{r} _{t}]$; // Create new state vector  
3. $\\mathbf{z} _t = \\mu _{\\mathbf{\\xi}} ^{\\sim 1} (\\mathbf{p} _t)$; // Compute the latent state  
4. $\\hat{\\dot{\\mathbf{z}}} _t = f(\\mathbf{z} _t)$; // Compute the latent velocity  
5. $J_{\\mu _{\\mathbf{\\xi}}}(\\mathbf{z} _t) = \\frac{\\partial\\mu _{\\mathbf{\\xi}}}{\\partial \\mathbf{z} _t}$; // Compute the Jacobian of the decoder  
6. $\\dot{\\mathbf{x}} _t = J _{\\mu _{\\mathbf{\\xi}}}(\\mathbf{z} _t)\\hat{\\dot{\\mathbf{z}}} _t$; // Compute input space velocity

Reviewer comment (Reproducibility):
How are the sequential data used for training? Is each trajectory used as a whole as one data point, or cut into segments as multiple data points?

Author response:
- “How are the sequential data used for training? Is each trajectory used as a whole as one data point, or cut into segments as multiple data points? Second-order?”  

   - During training, data points from all the demonstrations undergo shuffling and are then fed into the network as a batch containing $B$ data points.

Reviewer comment (Theory):
Second-order?

Author response:
- “*In the Question 1.3, I intended to confirm whether training the Jacobian network involves second-order derivatives, as the forward propagation includes calculating Jacobian as first-order derivatives.*”  
    - **Answer:** Good question. It is important to clarify that while the forward propagation of the Jacobian network approximates the second-order derivative of the state with respect to time (acceleration), the training process involves computing the first-order derivative with respect to the state (input) during back-propagation. This means that the training of the Jacobian network does not entail any calculation of second-order derivatives. We, however, see your point that second-order information might be made available to the optimizer through derivatives of the Jacobian network. This could potentially speed up training through faster convergence. We have, however, not explored this exciting line of thinking.

Reviewer comment (Experiments):
Multi-modality

Author response:
- *“Multi-modality”*  
    - We do not quite understand the reviewer’s question here. The term "multimodality" in the context of a dynamical system could mean different things. We are happy to do our best to answer the raised question but request a bit more detail.  

- “*In the Question 2, the full question is as follows: Does contraction conflict with demonstrations that have a multimodal distribution? By "multimodal", I mean there exist multiple "modes" (potentially due to various styles or goals of the demonstrators) in the distribution of trajectories. If contractive property means prompt correction of any deviation from a template trajectory, would if ever allow for the generation of more than one modes of trajectories, as one mode of trajectories would indeed "deviate from" another mode of trajectories? For example, in the second subplot from left in the top row in Figure 5, demonstrations seem to have three branches, joining at the center point, while the generated trajectories seem to only replicate one branch when starting from the center point.*  
    - **Answer:** That is a great observation. To address this question, we can classify multimodality in two ways based on the provided clarification:  
        1. Multimodal trajectory distributions  
        2. Multimodal target distributions  
        
        In its current form, NCDS handles case 1 nicely** and this is what our reported experiment demonstrates. It is worth noting that we have made revisions to Figure 5, specifically updating the panel related to multimodal behavior. Now, upon examining the orange path integrals from the starting point of each mode, the system successfully reproduces all the modes. However, case 2 is beyond what we currently can do, and further work is needed.** Here we expect that it should be possible to do a conditional version of NCDS.  
        
    - **Revisions:** In Figure 5, the multimodal panel (first row second panel from left), has been updated to include new path integrals showing the NCDS's behavior when initiated from each mode.

Reviewer comment (Theory):
It would be nice to have the invariance of the contractivity stated formally.

Author response:
- **Reviewer’s comment:** “*It would be nice to have the invariance of the contractivity stated formally.”*
    - **Short answer:** We agree that the paper was not precise enough here.
    - **Detailed answer**: Let $\mu: \mathbb{R}^d \rightarrow \mathbb{R}^D$ be the injective decoder mapping from the low-dimensional latent space to the high-dimensional observation space. Further, let $z_t$ denote a contractive dynamical system in the latent space. Geometrically, the decoder spans a $d$-dimensional manifold embedded in $\mathbb{R}^D$ defined as $\mathcal{M} = f(\mathbb{R}^d)$. By design, the **decoder is a diffeomorphism between $\mathbb{R}^d$ and $\mathcal{M}$**; this construction is extensively discussed by Brehmer & Cranmer [1], which we build upon. **By Theorem 1 in our paper, the above implies that $f(z_t)$ is contractive.** The above further implies that the dynamical system $f(z_t)$ is along a low-dimensional manifold $\mathcal{M} \subset \mathbb{R}^D$, i.e. it does not cover all of $\mathbb{R}^D$. Such constructions are common in latent variable models. The control loop assumes that the initial robot configuration is in $\mathcal{M}$ such that it can be written $f(z_0)$**.** **Following the dynamical system moves the robot along the manifold, and the resulting motion follows a contractive system.** If the assumption is not satisfied, such that the initial configuration $x_0 \not\in \mathcal{M}$, then we have to rely on the encoder to approximate a projection onto $\mathcal{M}$, i.e. produce $z_0 = \mu^{\sim 1}(x_0)$. Note that the movement from $x_0$ to $f(z_0)$ need not be contractive, such that the distance to the nominal trajectory may increase. This is, however, a finite-time motion, after which the system is contractive.
    - **Revisions:** In Section 3.2, we have refined and formalized the correlation between injection and contraction.

Reviewer comment (Experiments):
In the discussion section (page 9), it is mentioned that the choice of integration scheme can significantly affect the behaviour of the learned model. This requires further discussion. For instance, how significantly does the computation time affect the performance of the model? Is there a choice of integrator that doesn't require adaptive step-sizes (perhaps a symplectic integrator)?

Author response:
- **Reviewer’s comment:** “*In the discussion section (page 9), it is mentioned that the choice of integration scheme can siginificantly affect the behaviour of the learned model. This requires further discussion. For instance,”*
    - “*How significantly does the computation time affect the performance of the model?*”
        - **Short answer:** Our findings show that as long as we keep the latent space low dimensional, the computation of the integration does not affect the overall performance of the system.
        - **Detailed answer**: We have extensively investigated this question in Section A.6.2: *Dimensionality and execution time*, and our findings show that **as long as we keep the latent space low dimensional, the computation of the integration does not affect the overall performance** of the system. It is important to acknowledge that while maintaining a low-dimensional latent space is favorable for real-time applications, **the input space dimensionality introduces computational challenges.** Specifically, computing the Jacobian of the injective generator at each integration step becomes more resource-intensive. Furthermore, to answer this question in a more extreme scenario **we have conducted an experiment on higher-dimensional ($44D$) human motion dataset.** The new experiment showcases the performance of injective generators in encoding complex, high-dimensional data into a low-dimensional latent space ($2D$). Certainly, the considerable dimensionality of the data space inevitably impacts the runtime, with each iteration of the control loop in Algorithm 2 requiring 121 milliseconds.
        - **Revision**: We have added a new subsection in the Appendix A.6.2 titled “Learning human motion”. We also updated Table 2 in the appendix with the new results from human motion experiments execution time.

Reviewer comment (Reproducibility):
It would be helpful if the training algorithm were stated explicitly in Section 3.

Author response:
- **Reviewer’s comment:** “*It would be helpful if the training algorithm were stated explicitly in Section 3.”*
    - **Short answer:** We agree that a training algorithm should be added to the paper.
    - **Detailed answer:** Here, we provide detailed steps of the training and robot control schemes for NCDS. The sequential steps for training the Variational Autoencoder (VAE) and Jacobian network are listed in Algorithm 1. Simultaneously, the procedural steps for employing NCDS to control a robot are shown in Algorithm 2.
    - **Revision**: A new subsection titled "Algorithms" has been included in the Appendix A.4, containing the respective algorithms for reference.
---

**Algorithm 1**: *Neural Contractive Dynamical Systems (NCDS): Training in task space*

---

**Data**: Demonstrations: $\\tau _n = \\left \\{(\\mathbf{x} _t, \\mathbf{R} _t) \\right \\},  n \\in [1, N], t \\in [1, T _n]$

**Result**: Learned contractive dynamical system

1. $\\mathbf{r} _{n,t} = \\operatorname{Log}(\\mathbf{R} _{n,t})$; // Obtaining skew-symmetric coefficients
2. $\\mathbf{p} _{n, t} = [\\mathbf{x} _{n, t}, \\mathbf{r} _{n, t}]$; *//* Create new state vector
3. $ \\operatorname{argmin}_\\xi \\mathcal{L} _{\\text{ELBO}}(\\xi^*; \mathbf{p} _{n,t})$; // Train the VAE 
4. $\\mathbf{z} _{n,t} = \\mu _ {\\mathbf{\xi}} ^{\\sim 1} (\\mathbf{p} _{n,t})$; // Encode all states using the trained VAE
5. $\\dot{\mathbf{z}} _{n,t} = \\frac{\\mathbf{z} _{n,t+1} - \\mathbf{z} _{n,t}}{\Delta t}$; // Compute latent finite differences
6. $ \\operatorname{argmin}_\\theta \mathcal{L} _{\\text{Jac}}(\\theta^*; \\mathbf{p} _{n,t})$; // Train the Jacobian network
 
---

**Algorithm 2**: Algorithm Neural Contractive Dynamical Systems (NCDS): Robot Control Scheme

---

**Data:** Current state of the robot end-effector at time $t$: $[\\mathbf{x} _t, \\mathbf{R} _t]$

**Result**: Velocity of the end-effector at current time step $\\dot{\\mathbf{x}} _t$

1. $\\mathbf{r} _{t} = \\operatorname{Log}(\\mathbf{R} _{t})$; // Obtaining skew-symmetric coefficients
2. $\\mathbf{p} _{t} = [\\mathbf{x} _{t}, \\mathbf{r} _{t}]$; // Create new state vector
3.  $\\mathbf{z} _t = \\mu _{\\mathbf{\\xi}} ^{\\sim 1} (\\mathbf{p} _t)$;  // Compute the latent state
4.  $\\hat{\\dot{\\mathbf{z}}} _t = f(\\mathbf{z}  _t)$;  // Compute the latent velocity
5.  $J_{\\mu _{\\mathbf{\\xi}}}(\\mathbf{z} _t) = \\frac{\\partial\\mu _{\\mathbf{\\xi}}}{\\partial \\mathbf{z} _t}$;  // Compute the Jacobian of the decoder
6. $\\dot{\\mathbf{x}} _t = J _{\\mu _{\\mathbf{\\xi}}}(\\mathbf{z}  _t)\\hat{\\dot{\\mathbf{z}}} _t$;  // Compute input space velocity

Reviewer comment (Theory):
This work focuses on learning contractive dynamics. However, suppose the system $\dot{x} = f(x)$ is *not* contractive. What can you say about the solution to the optimization problem (i.e. the feasibility of eqns (4) and (5)) in this scenario?

Author response:
- **Reviewer’s comment:** “*This work focuses on learning contractive dynamics. However, suppose the system $\dot{x} = f(x)$ is not contractive. What can you say about the solution to the optimization problem (i.e. the feasibility of eqns (4) and (5)) in this scenario?”*
    - **Short answer:** Good question. When the system dynamics $\dot{x}=f(x)$ are not contractive, **the dynamics described by equations may integrate to arbitrary (potentially unstable) dynamics**. The system will most likely follow the trends of the training data when evaluated near the data, and then exhibit arbitrary behavior everywhere else (thereby violating stability guarantees)**.**
    - **Detailed answer:** If the system dynamics $\dot{x}=f(x)$ are not contractive, i.e. the Jacobian of the function $f$ is not uniformly negative definite, the trajectory described by equations (4) and (5) may not exhibit stable behavior. It is worth noting that **these equations do not depend on the Jacobian being uniformly negative definite.** The optimization of equations 4 and 5 without considering the contraction formulation in equation 3 means the system might be only able to reconstruct the trajectories if started on the data support regions (not guaranteed) but the global behavior of the system outside of the data support will not be stable. **In Fig. 13 in the updated paper, you can see the differences between contractive and unconstrained dynamical systems.** As the figures show, when the resulting vector field is **not contractive, the reproduced motion follows the trend of the data only on the demonstrations region and not on the outside regions**. Overall, this system does not show any stable behavior.
    - **Revision**: We have added a new subsection in Appendix A.6.3: Unconstrained Dynamical System discussing these results.

Reviewer comment (Evaluation):
Can the authors comment on the practical effectiveness of the model in greater detail? As mentioned in the discussion section (p9), the cost of numerical integration can be extensive. Have the authors come across examples where this has been an impediment to performance?

Author response:
- **Reviewer’s comment:** “*Can the authors comment on the practical effectiveness of the model in greater detail? As mentioned in the discussion section (p9), the cost of numerical integration can be extensive. Have the authors come across examples where this has been an impediment to performance?*“
    - **Short answer:** No such examples have been encountered.
    - **Detailed answer:**  numerical integration is **one of four steps affecting performance**, where the others are: (1) encoding the current state, (2) computing the Jacobian, and (3) calculating input space velocity. Each iteration in our robot control scheme (Algorithm 2) takes up to 10 milliseconds (100﻿ Hz). Current experiments involve 7 DOF robots, encoding either pose (6-dimensional) or joint configuration  (7-dimensional), with a 2-dimensional latent space. **With respect to the cost of the numerical integration, then this depends mostly on the latent dimension and less so on the input dimension** (Table 3 in the appendix). Finally, it is worth noting that **our Python implementation is optimized for correctness rather than performance**, so there is substantial room for performance improvements.