Paper: On the Generalization and Approximation Capacities of Neural Controlled Differential Equations (ICLR-2024)

Reviewer comment (Presentation):
The presentation in Theorem 4.1 could be improved, as it is hard to see how the bound explicitly depends on depth $q$, and discretization $D$. Right now it is hidden in the constant $C_q$ and $K_1^D, K_2^D$.

Author response:
We thank the reviewer for acknowledging our contributions and results. Let us first address the two mentioned weaknesses. We agree that the presentation of Theorem 4.1 can be improved and we will try to do so within the page constraints. In particular, as this has also been highlighted by Reviewer 1, we will make the constants explicit.

Reviewer comment (Experiments):
The experiments seem very preliminary: the sampled time series only have $K=5$ points. I am not sure how interesting it is to investigate this regime as the generalization error would be poor with this few samples.

Author response:
Concerning the experimental part, we make two remarks. First, as the discretization gap $|D|$ vanishes at linear speed - i.e. $1/M$ when sampling $M$ points in a equidistant fashion - the effect of discretization is almost null when short time series are sampled at high frequency. We thus have chosen this low sampling regime to obtain a visible effect. Secondly, our article is of theoretical nature ; we have chosen to give an extensive presentation of our theoretical results, which are complex and require long developments, instead of a more developed experimental section. We are planning on conducting a follow-up work benchmarking various methods designed for supervised learning with irregular time series, which will be the empirical companion paper to this work. We also refer the reader to the growing body of applied works on NCDEs, which we extensively cite in the introduction.

Reviewer comment (Theory):
Any assumption on the continuous $x$, or its distribution, if it is random.

Author response:
We now answer the two questions raised by the reviewer. First, the underlying continuous path $x$ is supposed to be unknown and observed at a finite number of sampling times. It is continuous of bounded variation, which is required for NCDEs to be well defined (Assumptions 1 and 2). We make the simplifying assumption that the paths are Lipschitz, but this can be weakened to include paths paths of bounded variation with non linear modulus of continuity. We work in a fixed design setting, i.e. we consider our data to be fixed throughout the article. As highlighted in our response to Reviewer 2, this stands in contrast to settings in stochastic control theory, where one learns over the distribution of random paths (i.e. stochastic processes with parametrized vector fields).

Reviewer comment (Writing):
$|D|$ appears in Remark 3.5, but defined much later.

Author response:
Secondly, we thank the reviewer for catching the typo on $|D|$, which we have fixed.

Reviewer comment (Theory):
In Section 4, Theorem 4.1 the upper bound for the generalization error varies with both the width and depth of the neural network and unlike Chen et al. (2019a), where the growth w.r.t the depth and width is logarithmic, in this case it’s polynomial. This exponential blow-up in the value of bound makes the whole exercise somewhat underwhelming.

Author response:
We thank the reviewer for this thorough review of our paper and its theoretical part which raises interesting questions, and for acknowledging our contributions. Before turning to the core of the review, we agree that giving the explicit form of all constants will make Theorem 4.1. more clear, as it has also been noted by Reviewer 3. We have added references to the constants in Theorem 4.1. 

**Comparison with earlier work by Chen et al. (2020).** We first address the comparison made with the earlier results of Chen et al. ("On Generalization Bounds of a Family of Recurrent Neural Networks", 2020 - we refer to AISTATS version from now on). We would first like to emphasise that the parameter $q$, which in our setting corresponds to the depth of the neural vector field applied at each update of the latent state, is completely absent from the aforementioned article: Chen et al. only consider shallow vector fields (i.e. "vanilla RNN" in their phrasing). Their neural vector field has no hidden layers but is only parameterized by matrices $W$ and $U$ which have resp. $d_h \times d_x$ and $d_h \times d_h$ parameters. We strongly encourage the reviewer to compare the definition of Chen et al.'s vector field (p.2, left column) with ours (Equation 1, page 4) to convince himself of the difference between our models.

With this clarification in mind, we argue that the first bounds obtained by Chen et al. and ours are of identical nature. Let us take a close look at Theorem 2 in Chen et al. (p. 4, left column): one can see that the complexity term scales as the square root of the number of parameters $d$, times some log factors. We obtain a similar result (see our Theorem 4.1 on page 6): the complexity term, in our bound, grows as the square root of the number of parameters times some log factors. As stressed above, the factor $q$ appears here since having $q$ layers in the neural vector field multiplies the number of parameters by $q$. This is not a surprise since both works use the same fundamental technique (Lipschitz-based complexity measures). In the last part of their paper, Chen et al. provide a slightly improved bound at the price of stronger restrictions on the parameter space. We leave an extension of these techniques to NCDEs to future work. We thank the reviewer for noticing this precise point, which we had missed. **We have added a mathematical and precise comparison of the results in Appendix F.2**.

Reviewer comment (Theory):
A core reason why a generalization error bound is interesting is because it reveals the generalization error to be independent of some natural size parameter of the system. Most often the baseline achievement of a Rademacher bound for a neural system is to show that at a fixed depth and norm bound on the weights involved the generalization is width independent. In the bound in Theorem 4.1 in this work, there is absolutely no such independence — the bound worsens with every such parameter with which naively one would have expected worsening to happen.

Author response:
We thank the reviewer for his constructive feedback and for engaging in this interesting and stimulating discussion.  

In our opinion, analysing how the complexity of a model scales with the number of parameters is a question that is just as interesting as proving bounds that are invariant w.r.t. to a given parameter. We stress again that the results of Chen (2020) provide such an analyse: their bounds do not display independence w.r.t. to a parameter of their model, but carefully examine how strong the dependence is. As pointed out earlier, given a similar set of assumptions, our bounds are of the same nature. In the case where sampling is regular, i.e. we consider a sequence of sampling schemes with $K$ equidistant points, the discretization dependant quantities can be upper bounded by the quantities appearing in the continuous time analysis of our model (see Remark 3.5). This means that in this case, our bounds are length independent. This is close to the results of Marion (2023), who has obtained a similar independence when analysing neural ODEs and ResNets.  

We also wish to underline that our bounds taken as a whole are far from being trivial. In addition to upper bounding the Rademacher complexity of NCDEs, we have also given a rigorous theoretical framework to study them and a novel approximation bound. These bounds display dependence on the sampling gap, which is highly relevant for practical applications of NCDEs. All these results are absent from the literature and are, in our opinion, of high interest since NCDEs are an interesting model, with strong empirical results and a growing body of applications (see our Introduction).

We encourage the reviewer to consider all these contributions when rating our work and thank him again for raising an interesting series of questions.

Reviewer comment (Evaluation):
The numerical illustrations seem incomplete in the absence of any demonstration of how well the bounds hold up in the cases mentioned. At a bare minimum, in Figure 4 type experiments one would have expected to see a scatter plot of the theoretical bound and the true generalization error as the sampling gap, time, and average max path variation are changed. Such a scatter plot would have revealed if there is any correlation between the theoretical bound and the actual outcome.

Author response:
**On generalisation.** We now wish to precise what we mean by generalisation, which might differ from the reviewer's definition. In our understanding, a generalisation bound simply ensures convergence of the empirical loss towards the expected loss. It does not need to be independent of some parameters of the system, even though one can obtain such dimension-free bounds in special setups (f.i. when learning with linear models) and with certain special techniques. For instance, if one considers vanilla bounds in linear regression with deterministic non-linear transformations of the input data (see Bach, "Learning Theory from First Principles", 2021, Chapter 4 Section 5) there is no independence from the size of the parameter space: the complexity of the predictors scales with the square root of the log of the number of features. 

However, we are aware that some other types of generalisation bounds exist for deep neural networks that may achieve bounds with very weak or no dependence on some of the parameters of the network. Some of these techniques rely on a close analysis of the optimisation process and the implicit bias of optimizers, which is not the objective of our paper. See for instance Gunasekar et al. ("Implicit Regularization in Matrix Factorization", 2017) or Soudry et al. ("The Implicit Bias of Gradient Descent on Separable Data", 2017) for an example of such an analysis. Another type of bound pioneered by Golovich et al. (2018) allows indeed to obtain depth independent bounds at the price of worsened sample complexity: convergence of the empirical loss towards the expected loss is ensured at speed $n^{1/4}$ instead of the more common speed $n^{1/2}$. We believe that one could apply these bounds here to obtain a refined analysis of the complexity induced by the depth of the neural vector field at the price of degraded sample complexity. However, since NCDEs are used with very shallow neural networks i.e. $q\leq 3$, we believe that this trade-off will not improve our bounds. **We have provided a mathematical description of these points in Appendix F.3.** and leave a deeper analysis of this problem for future work.

**On experiments.** Concerning our experiments, it is unclear to us what the reviewer means by the theoretical value of the bounds. Many of the quantities provided in our bounds are unknown (i.e. data distribution dependant) constants and cannot easily be estimated. Let us emphasise again what our results show. First, experiment 1 shows that NCDEs can achieve zero loss (i.e. perfect interpolation and generalisation) with bounded parameters. This implies that the crucial assumption of bounded parameter space is satisfied in practice, and that the bounds lie in a reasonable range. A negative result would have been that NCDEs need weights with extremely high magnitude to achieve good performance. Our second result shows that the generalisation gap correlates with both the discretization gap and with a quantity that we have called the average maximum path variation.

We hope that these explanations make our contribution more clear. We would like to thank the reviewer for raising questions on these points, which lead us to add thorough discussions of the results mentioned above to our article's Appendix. We will be happy to provide more explanation and supplementary details if needed.