Paper: Implicit Neural Representations and the Algebra of Complex Wavelets (ICLR-2024)

Reviewer comment (Experiments):
In my view, the weakness of this paper is its small set of experiments. I also think more details can be provided to interpret the experiments and to discuss the theory.

Author response:
1. Need for more experiments. How many parameters do the models have?

To address these remarks, we have added new comparative experiments to Section 4.3 showing the interaction between the use of complex wavelets as template functions and the proposed split INR architecture. We have also added experiments to Section 5 showing the advantage of the wavelet modulus maxima initialization scheme for a dataset of images. This resulted in the figure showing the training behavior of the networks being removed to make space for more substantive results.

We have added an appendix describing the models in greater detail, including the depths and widths of the networks.

Reviewer comment (Writing):
I think providing more details and more discussions would make the paper more approachable for a broader audience.

Author response:
1. Need for more experiments. How many parameters do the models have?

To address these remarks, we have added new comparative experiments to Section 4.3 showing the interaction between the use of complex wavelets as template functions and the proposed split INR architecture. We have also added experiments to Section 5 showing the advantage of the wavelet modulus maxima initialization scheme for a dataset of images. This resulted in the figure showing the training behavior of the networks being removed to make space for more substantive results.

We have added an appendix describing the models in greater detail, including the depths and widths of the networks.

Reviewer comment (Experiments):
I would have liked to see experiments on more images. I only saw three examples. Specifically, I'd be interested to see more examples of how the method works on more images, its errors, and its possible failure modes.

Author response:
1. Need for more experiments. How many parameters do the models have?

To address these remarks, we have added new comparative experiments to Section 4.3 showing the interaction between the use of complex wavelets as template functions and the proposed split INR architecture. We have also added experiments to Section 5 showing the advantage of the wavelet modulus maxima initialization scheme for a dataset of images. This resulted in the figure showing the training behavior of the networks being removed to make space for more substantive results.

We have added an appendix describing the models in greater detail, including the depths and widths of the networks.

Reviewer comment (Experiments):
Overall, demonstrating results on more images would give a better understanding to a reader.

Author response:
1. Need for more experiments. How many parameters do the models have?

To address these remarks, we have added new comparative experiments to Section 4.3 showing the interaction between the use of complex wavelets as template functions and the proposed split INR architecture. We have also added experiments to Section 5 showing the advantage of the wavelet modulus maxima initialization scheme for a dataset of images. This resulted in the figure showing the training behavior of the networks being removed to make space for more substantive results.

We have added an appendix describing the models in greater detail, including the depths and widths of the networks.

Reviewer comment (Novelty):
I think the discussion in the context of wavelet literature could have been broader. For example, I did not see any discussions on the topic of Daubechies wavelets. Are Daubechies wavelets also progressive by the authors’ definition?

Author response:
2. I think the discussion in the context of wavelet literature could have been broader. For example, I did not see any discussions on the topic of Daubechies wavelets. Are Daubechies wavelets also progressive?

Daubechies wavelets, being real-valued in the usual case, are not progressive: we have added a comment on this in the updated paper. One can certainly convert any suitable wavelet to a progressive one by taking the Hilbert transform of the wavelet.

Reviewer comment (Evaluation):
If authors think Shearlets may have any potential here, providing a discussion might be useful. Most of the approximation error for the pictures in Figure 6 appear to be at locations where the colors change in a small neighborhood. Could a shear matrix be potentially helpful in reducing the error because of its ability to extract anisotropic features?

Author response:
3. If the authors think Shearlets may have any potential here, providing a discussion might be useful. ... Could a shear matrix be potentially helpful in reducing the error because of its ability to extract anisotropic features?

This is an interesting comment, and is indeed something that we have thought about. Empirically, architectures like WIRE have been observed by their authors to yield atoms in the first layer that resemble curvelets:

Saragadam, Vishwanath, et al. "Wire: Wavelet implicit neural representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Seeing how the shear matrix of a shearlet transforms is used in conjunction with a parabolic scaling matrix to transform a "mother wavelet," INRs like the ones we study are capable of incorporating shearing transformations via the learning of the weights in the first layer of the network, without any explicit enforcing of the form of the matrix.

However, the theoretical properties of systems of shearlets/curvelets are difficult to reconcile with INRs in the way we study them, as said properties are often asymptotic properties of a specifically designed system. Implemented naively, INRs can certainly use shearlets/curvelets in the first layer, but the higher powers of those atoms will probably not obey the relevant scaling rules that make said systems have nice theoretical properties. This is an interesting point for future research on the topic though, so we have made a note in the updated conclusion about this.

Reviewer comment (Novelty):
A relevant prior work that authors may consider citing if they see fit:  

-- Grattarola, D. and Vandergheynst, P., 2022. Generalised implicit neural representations. Advances in Neural Information Processing Systems, 35, pp.30446-30458.

Author response:
4. On some relevant works to cite from Grattarola, et. al., and Shaham, et. al.

Indeed, thank you for pointing us to these references. We have included them in the survey section of the updated paper.

Reviewer comment (Novelty):
Using wavelets to study approximation properties of neural networks, and leveraging that to design the architecture of the network, is studied in the past, but I did not see a mention of that in the paper.  

-- Shaham, U., Cloninger, A. and Coifman, R.R., 2018. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3), pp.537-557.

Author response:
4. On some relevant works to cite from Grattarola, et. al., and Shaham, et. al.

Indeed, thank you for pointing us to these references. We have included them in the survey section of the updated paper.

Reviewer comment (Theory):
Your definition of the model in eq 1 seems to have some issue to me. Psi is a function of R^d -> C but somehow W^0 r and b^0 is in R^{F_1 x d}. Thus I do not understand the model.

Author response:
1. $\psi$ is a function from $R^d$ to $C$, but somehow $W^0r$ and $b^0$ are in $R^{F_1\times d}$.

Good point: the function $\psi$ is understood to act "row-wise" on the matrix in $R^{F_1\times d}$, so that it extends to a function $\psi:R^{F_1\times d}\to C^{F_1}$. We have added a comment to this effect in the updated paper.

Reviewer comment (Writing):
In Section 4.3, what does it mean to model a signal as a sum of a linear scaling INR and a nonlinear INR?

Author response:
7. In Section 4.3, what does it mean to model a signal as a sum of a linear scaling INR and a nonlinear INR?

We have updated Section 4.3 with new comparative experiments to study the proposed architecture, along with a more precise explanation of what we mean by this. Thank you for suggesting this clarification, we feel that it has improved the presentation of this section of the paper.

Reviewer comment (Experiments):
Although the performance of the proposed method was supported by several tests and analysis in the paper, it is suggested to include some practical applications such as regression tasks on images or other high-dimensional signals to justify its practicability.

Author response:
Thank you for your comments on our work. You are indeed in agreement with the other reviewers on the need for more numerical experiments to go with the paper, which we have addressed by updating Sections 4 and 5.

We would like to remark that the main point of this paper is not strictly methodological. Rather, the suggestion of using a split architecture for INRs that use progressive wavelets as template functions is something that follows from our analysis of such networks, as determined by the Theorem (now known as Lemma 1) and the following discussion.

With this in mind, we still see the need for more practical demonstrations to show the validity of the obtained insights. In Section 4.3, you can see a comparative study between the use of real and complex wavelets, the use of nonlinear hidden layers in INRs, and using scaling networks or not (i.e., the proposed split architecture).

We have also added new experiments on a dataset of images showing the advantage of initializing a complex Gabor INR using the wavelet modulus maxima over random initialization, as well as over other choices of template function.

Reviewer comment (Theory):
The reviewer is curious if the proposed method and its theoretic analysis/intuition is valid, when the method is applied to practical applications.

Author response:
Thank you for your comments on our work. You are indeed in agreement with the other reviewers on the need for more numerical experiments to go with the paper, which we have addressed by updating Sections 4 and 5.

We would like to remark that the main point of this paper is not strictly methodological. Rather, the suggestion of using a split architecture for INRs that use progressive wavelets as template functions is something that follows from our analysis of such networks, as determined by the Theorem (now known as Lemma 1) and the following discussion.

With this in mind, we still see the need for more practical demonstrations to show the validity of the obtained insights. In Section 4.3, you can see a comparative study between the use of real and complex wavelets, the use of nonlinear hidden layers in INRs, and using scaling networks or not (i.e., the proposed split architecture).

We have also added new experiments on a dataset of images showing the advantage of initializing a complex Gabor INR using the wavelet modulus maxima over random initialization, as well as over other choices of template function.

Reviewer comment (Evaluation):
The paper provides little to no interpretation of its results and observations, which makes it confusing for readers not specialized in classical signal processing. The reviewer felt expected to independently interpret and understand results, which was not possible in many cases.

Author response:
Thank you for pointing this out. While certain terminology is quite standard to the intended audience of our paper (broadly speaking, those that are well-acquainted with Fourier analysis and its applications), we would still like to avoid using excessive jargon. The updated paper has improved explanations behind the motivations for certain results and proposed methods. Additionally, based on your comments and comments from other reviewers, we have removed Section 3.2 to make more space for new results and exposition, which seemed to be a problematic section anyway.

Reviewer comment (Theory):
It is unclear what mathematical level the authors expect of the reader. The paper states some elementary results rigorously while handling others, such as effective support and Minkowski sums, in a handwaving manner.

Author response:
Thank you for pointing this out. While certain terminology is quite standard to the intended audience of our paper (broadly speaking, those that are well-acquainted with Fourier analysis and its applications), we would still like to avoid using excessive jargon. The updated paper has improved explanations behind the motivations for certain results and proposed methods. Additionally, based on your comments and comments from other reviewers, we have removed Section 3.2 to make more space for new results and exposition, which seemed to be a problematic section anyway.

Reviewer comment (Writing):
Several technical signal processing terms (such as atoms, band-pass filters, and WIRE) are left undefined, which makes reading the paper challenging.

Author response:
Thank you for pointing this out. While certain terminology is quite standard to the intended audience of our paper (broadly speaking, those that are well-acquainted with Fourier analysis and its applications), we would still like to avoid using excessive jargon. The updated paper has improved explanations behind the motivations for certain results and proposed methods. Additionally, based on your comments and comments from other reviewers, we have removed Section 3.2 to make more space for new results and exposition, which seemed to be a problematic section anyway.

Reviewer comment (Theory):
Regarding Theorem 1, Eq (2) appears to be simply the Fourier transform of a slightly rearranged definition of the INR architecture, yet some assumptions and conclusions are unclear.

Author response:
This is a fair point, and perhaps we did not emphasize the correct aspects of the paper. The point of the work is not Theorem 1 on its own, but rather to see how it interacts with different choices of the template function, in particular template functions that constitute an algebra under multiplication. To shift the emphasis, we have "downgraded" Theorem 1 to a Lemma, and provided a comment remarking that it is really just an application of the convolution theorem.

Reviewer comment (Novelty):
It is unclear why restricting focus to polynomial activations beyond the first layer is interesting to study.

Author response:
Polynomial activations, although not usually used directly in neural network architectures, are reasonable ways to approach the harmonic analysis of more general neural network architectures. We show how polynomial activations can be extended to analytic (holomorphic) activations and beyond in Appendix B.1. For the case of real-valued networks, for instance, any continuous neural network can be uniformly approximated by polynomials over a compact domain by the Stone-Weierstrass Theorem, a fact leveraged in the study of ReLU networks in the paper

Mehmeti-Göpel, Christian HX Ali, David Hartmann, and Michael Wand. "Ringing ReLUs: Harmonic distortion analysis of nonlinear feedforward networks." International Conference on Learning Representations. 2020.

Of course, complex numbers complicate the choice of activation functions a bit in this case. For instance, the modulus function is not holomorphic, and thus can't be approximated by polynomials without using conjugation, and progressive functions are not closed under conjugation in general, so the interplay between Theorem 1 (now Lemma 1) and the algebra of progressive wavelets does not apply here. We have remarked upon this in the updated paper and in the appendix.

Reviewer comment (Theory):
It is unclear why the Fourier transform is presented by considering multiplication by a smooth function $\phi$.

Author response:
This is done for two reasons: the first is technical, the second is a matter of interpretation/emphasis.

The technical reason is to handle Fourier transforms of functions that are only locally integrable. ReLU functions, for instance, are only integrable locally, not globally, so their Fourier transform only exists in the sense of tempered distributions (i.e., via duality with the space of rapidly decaying smooth functions). Multiplying by a compactly supported smooth function assuages these issues, and can be used to reconstruct the Fourier transform of the whole signal if needed.

The interpretive reason is to emphasize the locality of the INR architecture. In the time-frequency analysis perspective that we take, it is important to understand wavelets and their powers as being localized in space. This has important interactions with the notion of a locally $\Gamma$-progressive function for some conic set $\Gamma$, since the singularities in a function might only have common direction locally. Take the indicator function of the unit disc in $R^2$, for instance. The singularities of the disc are located along the edges of the disc and point in the normal direction to the boundary. Because of this, the Fourier transform of this function has slow decay in all directions. However, if you localize to a small neighborhood of a point on the boundary of the disc via multiplication by a suitable smooth function, the Fourier transform will only exhibit slow decay in a small cone around the normal direction. This is where the notion of a conically progressive function becomes useful, as you can guarantee that all generated frequencies are in line with the direction of a desired singularity.

Reviewer comment (Theory):
The statement that the INR is composed of the sum of the template function’s “integer harmonics” does not clearly illustrate how this demonstrates the expressivity of the INR.

Author response:
This statement gives a rough bound on the type of function that can be expressed by the INR. That is to say, it dictates what functions can and cannot be represented by an INR. For instance, if the template functions in the first layer all have Fourier support in some cone (that is, they are $\Gamma$-progressive for some conic set $\Gamma$), then the INR will be unable to generate frequencies outside of $\Gamma$. Depending on what an INR user is looking for, this could be viewed as a helpful or harmful constraint, but one worth knowing about either way.

Reviewer comment (Theory):
The claim that the support of scaled and shifted atoms is preserved, so the output at coordinate $r$ is dependent only on atoms in the first layer whose support contains $r$, appears trivial and its relevance is unclear.

Author response:
Indeed, this is a simple property of the considered INRs -- that statement was meant merely to state the last part of Theorem 1 (now Lemma 1) in words. The observation becomes relevant when we consider the later sections on the use of $\Gamma$-progressive functions, as it is unreasonable to expect a multidimensional function (such as an image) to be globally approximated by a $\Gamma$-progressive function for some nontrivial conic set $\Gamma$, but it is more reasonable to expect this to hold locally in the image. See the above example of the unit disc.

Reviewer comment (Presentation):
The purpose of Section 3.2 is unclear. While the argument is understandable and agreeable, the discussion is entirely informal and contains no actual results, and the insight is not apparent.

Author response:
This is a fair critique -- based on your comments, as well as those of other reviewers asking for some additions to the paper, we have removed Section 3.2 from the updated manuscript to make room for new material.

Reviewer comment (Presentation):
In Section 4, the authors state they consider the advantages of using complex wavelets, or progressive wavelets, but this reveals that the paper title is misleading since only the algebra of progressive wavelets is considered.

Author response:
Thank you for this comment: it has revealed some aspects of the paper that we should have made more clear to begin with. In conjunction with added comparative experiments, we have updated Section 4.3 to empirically demonstrate the advantages of using progressive wavelets over their real-valued counterparts. 

The titling of the paper with the term "complex wavelet" rather than "progressive wavelet" was a stylistic choice, as "progressive wavelet" is not a universally-used term in the literature.  Many signal processing researchers will informally use the term "complex wavelet" to refer to what we call "progressive wavelets" in this paper, although it is certainly true that not all complex-valued wavelets are necessarily progressive. If the area chair and other reviewers also think that a change in title is appropriate, we are willing to do so.

We do make use of the fact that progressive wavelets form an algebra: Corollary 4 is a combination of Theorem 1 (now Lemma 1) and the algebraic closure of the space of progressive functions.

Reviewer comment (Experiments):
The authors do not demonstrate any actual advantages of using progressive wavelets despite claiming to do so.

Author response:
Thank you for this comment: it has revealed some aspects of the paper that we should have made more clear to begin with. In conjunction with added comparative experiments, we have updated Section 4.3 to empirically demonstrate the advantages of using progressive wavelets over their real-valued counterparts. 

The titling of the paper with the term "complex wavelet" rather than "progressive wavelet" was a stylistic choice, as "progressive wavelet" is not a universally-used term in the literature.  Many signal processing researchers will informally use the term "complex wavelet" to refer to what we call "progressive wavelets" in this paper, although it is certainly true that not all complex-valued wavelets are necessarily progressive. If the area chair and other reviewers also think that a change in title is appropriate, we are willing to do so.

We do make use of the fact that progressive wavelets form an algebra: Corollary 4 is a combination of Theorem 1 (now Lemma 1) and the algebraic closure of the space of progressive functions.

Reviewer comment (Theory):
The authors never make use of the fact that progressive wavelets form an algebra, even though the concept is featured in the paper title.

Author response:
Thank you for this comment: it has revealed some aspects of the paper that we should have made more clear to begin with. In conjunction with added comparative experiments, we have updated Section 4.3 to empirically demonstrate the advantages of using progressive wavelets over their real-valued counterparts. 

The titling of the paper with the term "complex wavelet" rather than "progressive wavelet" was a stylistic choice, as "progressive wavelet" is not a universally-used term in the literature.  Many signal processing researchers will informally use the term "complex wavelet" to refer to what we call "progressive wavelets" in this paper, although it is certainly true that not all complex-valued wavelets are necessarily progressive. If the area chair and other reviewers also think that a change in title is appropriate, we are willing to do so.

We do make use of the fact that progressive wavelets form an algebra: Corollary 4 is a combination of Theorem 1 (now Lemma 1) and the algebraic closure of the space of progressive functions.

Reviewer comment (Theory):
The authors spend the first two subsections defining a multivariate notion of a band-pass filter with weakly conic sets, but it is unclear why this motivates their claim in Section 4.3 that it is well-founded to approximate functions using a sum of two INRs — one for low-pass components and one for high-pass components.

Author response:
Indeed, we could have made this more clear from the start. We have updated Section 4.3 to address this in two ways. We have clarified why it is well-motivated to split the signal representation in this way by explaining that INRs using progressive wavelets are unable to easily generate low-frequency signal content. Additionally, we have noted the potential advantages of separating the low-pass and high-pass parts of the signal from a signal processing perspective. In line with this discussion, the added experiments show empirically the advantage of the split architecture with complex wavelets over alternative design choices.

Reviewer comment (Theory):
The reviewer does not follow the argument that the multivariate notion of band-pass filters justifies decomposing signals into separate high-pass and low-pass INRs.

Author response:
Indeed, we could have made this more clear from the start. We have updated Section 4.3 to address this in two ways. We have clarified why it is well-motivated to split the signal representation in this way by explaining that INRs using progressive wavelets are unable to easily generate low-frequency signal content. Additionally, we have noted the potential advantages of separating the low-pass and high-pass parts of the signal from a signal processing perspective. In line with this discussion, the added experiments show empirically the advantage of the split architecture with complex wavelets over alternative design choices.

Reviewer comment (Experiments):
The experimental section is severely lacking. Only one ablation study is conducted, comparing the suggested initialization technique to random initialization on three image reconstructions.

Author response:
Thank you for this comment. You are in agreement with the other reviewers that this work needs more empirical demonstrations. We have updated Section 4.3 with a comparative study of using the split architecture vs INRs that do not have a "scaling network" attached, as well as a discussion of why the split architecture with complex wavelets may outperform the other approaches.

We have also updated Section 5 with an experiment testing the initialization scheme on a larger dataset of images, comparing to other template functions from the literature, and showing how the initialization is helpful in denoising tasks.

Reviewer comment (Experiments):
It is unclear how the suggested signal decomposition into two INRs or the initialization technique benefits typical INR tasks in practice compared to other methods.

Author response:
Thank you for this comment. You are in agreement with the other reviewers that this work needs more empirical demonstrations. We have updated Section 4.3 with a comparative study of using the split architecture vs INRs that do not have a "scaling network" attached, as well as a discussion of why the split architecture with complex wavelets may outperform the other approaches.

We have also updated Section 5 with an experiment testing the initialization scheme on a larger dataset of images, comparing to other template functions from the literature, and showing how the initialization is helpful in denoising tasks.