Paper: A trainable manifold for accurate approximation with ReLU Networks (ICLR-2024)

Reviewer comment (Theory):
The proposed optimization approach can be more useful if the reparameterization argument can be extended to high dimensional problems. It would be clearer for the reader if the authors can describe the main difficulties of such extension.

Author response:
I'm so glad you liked the paper! this was actually my first time submitting to a conference. It got buried in the appendix, but there's a proof that the differentiability constraint forces the function represented by the network to be convex. While this constraint is important for ensuring consistent performance, it also takes away a great deal of flexibility. This is part of why I had difficulty extending this method to more complex problems.

Reviewer comment (Experiments):
Given the fractal nature of the sawtooth-like function, would the proposed method demonstrate superior performance on some fractal functions? For example, the Cantor function.

Author response:
Fully connected ReLU networks are great at representing fractals. If I remember correctly, the Perekrestenko et al. citation gives a novel approximation to the Weierstrass function that exponentially improves upon previously existing methods. I think it may be the case that the ability of neural networks to represent fractals could be leveraged to provide a more flexible replacement for the differentiability constraint in this paper. Perhaps non-convex functions require nowhere-differentiable representations, but somehow this can be controlled so as to prevent jaggedness in the infinite-depth limit.