Paper: On the Hidden Waves of Image (ICLR-2024)

Reviewer comment (Experiments):
Were all images resized to the same aspect ratio? The aspect ratio can impact wave speed, and it may be more natural to preserve the aspect ratio from the original images.

Author response:
During the training, the aspect ratio is randomly set between 0.75 and 1.33. During inference, all images are resized to an aspect ratio of 1:1. We also experimented with maintaining the original aspect ratio during inference, and the difference in PSNR was found to be negligible.

When changing the aspect ratio for an input image during inference, although the learned wave speeds are fixed, the contribution of these waves (with different speeds) changes accordingly. This change is reflected in the initial condition. Thus, all waves collaborate around the new initial condition to successfully reconstruct the image with the new aspect ratio.

Reviewer comment (Writing):
Conventional image compression methods like DCT transform the image into a linear composition of orthogonal basis. This is simple and provides a straightforward understanding of each coefficient. The paper effectively describes the differences, but what are the main strengths and weaknesses of this method?

Author response:
Great question. Below, we discuss the strengths and weaknesses of Hidden-Wave, compared with conventional image compression methods like DCT transform.

***Strength I: Concise Mathematical Insight.*** 
Hidden-Wave unveils a succinct mathematical principle within the latent space to describe images. It establishes a linear correlation between horizontal and vertical derivatives, expressed as:

$\frac{\partial \mathbfit{\zeta}}{\partial x}=\mathbfit{\Lambda}\frac{\partial \mathbfit{\zeta}}{\partial y}$

This formulation introduces shared wave speeds ($\mathbfit{\Lambda}$) across all images, revealing a mathematical invariance. Additionally, Hidden-Wave provides insights into how individual images differentiate from others through unique initial conditions while adhering to the same wave equations.

***Strength II: Describing the Entire Image.***
In contrast to the DCT, predominantly applied to 8x8 or 16x16 image blocks, Hidden-Wave is adept at describing the entire image.

***Strength III: Better Performance than DCT.***
Please refer to ***Conclusion II*** under **Weakness 1** in the first part of the rebuttal for more detailed information.

***Weakness: Not Straightforward.*** 
Despite its strengths, Hidden-Wave lacks the straightforwardness inherent in conventional compression methods like DCT. Its foundation on wave equations within a latent space introduces ambiguity about the meaning of each dimension. Additionally, Hidden-Wave requires an encoder and decoder to collaboratively learn, adding complexity to its implementation.

Reviewer comment (Experiments):
Is the selection of hyperparameters for training this method not very sensitive? How long does the training process take?

Author response:
We only have two training hyperparameters (learning rate and weight decay). The selection of hyperparameters for training this method is not sensitive. All models are trained with the same hyperparameters.

The training process takes varying amounts of time based on factors such as the number of channels (C), the number of special solutions (M), and the resolution of the feature map. The default setting (C=1024, M=4, feature map 64x64) takes 12 days with 8 V100 GPUs. Reducing the feature map size from 64x64 to 16x16 speeds up training to 7 days.

Reviewer comment (Novelty):
Contribution: It is stated in section 2, that the prior work FINOLA is extended in two ways: a) generalizing FINOLA to a set of one-way wave equations and b) relaxing the local constraints. Yet, a) is merely an interpretation, and b) is done by summing over multiple FINOLA solutions. I consider this to be rather simple with the increase in PSNR being very natural due to the larger latent space of the resulting model.

Author response:
Thank you for this comment. Let us clarify the contribution of Hidden-Wave over FINOLA as follows:

***I. Improved Mathematical Description:*** Hidden-Wave (one-way wave equations) enhances the mathematical description of images by generalizing FINOLA. Unlike FINOLA, which connects the partial derivatives of the feature map $\mathbfit{z}(x, y)$ along the $x$ and $y$ axes using the current feature values, Hidden-Wave directly links these derivatives through one-way wave equations:

|Method|PDEs|
|---|---|
|FINOLA| $\frac{\partial \mathbfit{z}}{\partial x}=\mathbfit{A}\mathbfit{z}_n, \frac{\partial \mathbfit{z}}{\partial y}=\mathbfit{B}\mathbfit{z}_n$ |
| **Hidden Wave (our)** | $\frac{\partial \mathbfit{z}}{\partial x}=\mathbfit{A}\mathbfit{B}^{-1}\frac{\partial \mathbfit{z}}{\partial y}$ |

This generalization is *NOT* merely interpretative. We further demonstrate that a superior solution of wave equations can be *achieved* by summing multiple special solutions of FINOLA, validating that, for ***the same feature map size***, one-way wave equations offer a more accurate reconstruction than the two PDEs in FINOLA. This core insight underscores the paper's contribution:

*"Mathematically, the feature map governed by one-way wave equations is more accurate than the one governed by the two PDEs in FINOLA."*

***II. Non-Trivial Summation over Multiple FINOLA:*** Summing over multiple FINOLA provides a simple yet elegant empirical validation of the generalized wave equations' optimality compared to the two PDEs in FINOLA. The feature map after summation stays within the solution space of wave equations through linearity, but departs from the solution space of FINOLA.

***III. Larger Latent Space:*** 
This observation is invaluable! It sheds light on why multiple FINOLA paths effortlessly navigate the solution space of wave equations for a more optimal solution. Encoding more initial conditions simplifies the approach toward the optimal solution within the wave equation space.

Moreover, multiple FINOLA paths provide a *parameter-efficient* means to adjust compression rate and reconstruction quality. In contrast to vanilla FINOLA, which achieves increased reconstruction quality by enlarging the number of channels and the sizes of matrices $\mathbfit{A}$ and $\mathbfit{B}$, multiple FINOLA achieves this by adding paths (or initial conditions) for the same number of wave equations and parameters in matrices $\mathbfit{A}$ and $\mathbfit{B}$. The table below compares Hidden-Wave with FINOLA across multiple latent sizes, ranging from 512 to 4096. Both methods achieve higher PSNR by increasing the latent size differently (by increasing paths in Hidden-Wave vs. increasing channels in FINOLA). Although Hidden-Wave is slightly behind FINOLA in terms of PSNR, it maintains a constant size for matrices $\mathbfit{A}$ and $\mathbfit{B}$, which is 256 times smaller than FINOLA at latent size 4096.

|Method|Latent Size|DIM of $\mathbfit{A}$, $\mathbfit{B}$&darr;|PSNR&uarr;|
|---|---|---|---|
|FINOLA | 512 | 512x512| **22.0** |
| **Hidden Wave (our)** | 512 (256x2) | **256x256 (x0.250)**| 21.8 |
||||
|FINOLA| 1024 | 1024x1024| **23.4**|
| **Hidden Wave (our)** | 1024 (256x4) | **256x256 (x0.063)** | 22.9 |
||||
|FINOLA| 2048 | 2048x2048| **24.8**|
| **Hidden Wave (our)** | 2048 (256x8) | **256x256 (x0.016)** | 24.4 |
||||
|FINOLA| 4096 | 4096x4096| **25.9**|
| **Hidden Wave (our)** | 4096 (256x16) | **256x256  (x0.004)** | 25.4 |

Reviewer comment (Experiments):
Insights on whether wave equations are really a common joint principle in natural images: There is no comparison to other autoencoders, no motivation on why the wave equation could be of particular importance to images, and no illustration/interpretation of the latent space encodings $q_i$ or the corresponding solutions $z_i$ of the autoregressive FINOLA features. In particular, a $256 \times 256 \times 3$ color image is represented, e.g. with a $8 \times 1024$ latent space variable and subsequently decoded (using the multi-path FINOLA) with a PSNR of 27.1. But how would approaches perform that use a latent space of similar dimension and do not exploit any autoregressive (or wave-equation-based) operation in their decoder? In order to believe that wave equations or FINOLA are a fundamental "underlying mathematical property shared by images", the proposed approach would have to perform significantly better than competing approaches. A truncated PCA, keeping the largest patch-wise DCT (or wavelet) coefficients of an image, and training a standard convolutional autoencoder would be natural very classical baselines. Currently, I am not convinced that multi-path FINOLA / wave equations are the fundamental ingredient that makes the autoencoder work exceptionally well.

Author response:
**[Weakness 2 + Q1]: Comparing with other  encoding-decoding techniques  that use a latent space of similar dimension.**

Thank you for this excellent suggestion. Below, we compare Hidden Wave with (a) DCT, (b) DWT/DTCWT, and (c) convolutional autoencoder for image reconstruction on the ImageNet validation set, with an image size of 256x256. Hidden Wave achieves a higher PSNR for the same or smaller latent size.

***Comparison with discrete cosine transform (DCT):*** The table below compares Hidden Wave with DCT. DCT is conducted per 8x8 image block, and the top-left $K$ coefficients (in zig-zag manner) are kept, while the rest are set to zero. We choose four $K$ values (1, 3, 6, 10) for comparison. Clearly, Hidden Wave achieves a higher PSNR with a small latent size. Notably, in the last row, *"mix"* refers to placing 8 initial conditions at different positions rather than overlapping at the center.

|Method|Latent Size&darr;|PSNR&uarr;|
|---|---|---|
|DCT (top-left 1) | 3072 (32x32x3) | 20.6 |
| **Hidden Wave (our)** | **2048 (1024x2)** | **24.8** |
||||
|DCT (top-left 3)| 9216 (32x32x9) | 23.5|
| **Hidden Wave (our)** | **8192 (1024x8)** | **27.1** |
||||
|DCT (top-left 6)| 18432 (32x32x18) | 25.6|
| **Hidden Wave (our)** | **16384 (2048x8)** | **27.8** |
||||
|DCT (top-left 10)| 30720 (32x32x30) | 27.5|
| **Hidden Wave (our)** | **16384 (2048x8 mix)** | **28.9** |

***Comparison with discrete wavelet transform (DWT) and dual-tree complex wavelet transform (DTCWT):*** We compare Hidden-Wave with DWT and DTCWT in the table below. Three scales are chosen for wavelet decomposition. The comparisons are organized into three groups: (a) using only the LL subband at the coarsest scale (scale 3), (b) using all subbands (LL, LH, HL, HH) at the coarsest level, and (c) using all subbands at the finer scale (scale 2). Hidden-Wave outperforms DWT and DTCWT in terms of PSNR for the first two groups, achieving a smaller latent size. In the last group, while Hidden-Wave's PSNR is lower than DTCWT, its latent size is significantly smaller (more than 10 times smaller).

|Method|Latent Size&darr;|PSNR&uarr;|
|---|---|---|
|DWT (scale-3 LL) | 3888 | 21.5 |
|DTCWT (scale-3 LL) | 12288 | 22.3 |
| **Hidden Wave (our)** | **2048 (1024x2)** | **24.8** |
||||
|DWT (scale-3 LL+LH+HL+HH)| 15552 | 24.3|
|DTCWT (scale-3 LL+LH+HL+HH) | 49152 | 25.6 |
| **Hidden Wave (our)** | **8192 (1024x8)** | **27.1** |
||||
|DWT (scale-2 LL+LH+HL+HH)| 55953 | 28.7|
|DTCWT (scale-2 LL+LH+HL+HH) | 196608 | **30.8** |
| **Hidden Wave (our)** | **16384 (2048x8 mix)** | 28.9 |

***Comparison with convolutional Autoencoder (Conv-AE):*** The table below presents a comparative analysis between our method and convolutional autoencoder (Conv-AE) concerning image reconstruction, measured by PSNR. Both approaches share the same Mobile-Former encoder and have identical latent sizes (1024x2 or 1024x8). In our method, FINOLA is initially employed to generate a 64x64 feature map, followed by a convolutional decoder to reconstruct an image with a size of 256x256. On the other hand, Conv-AE employs a deeper decoder that utilizes convolution and upsampling from the latent vector to reconstruct an image. Hidden-Wave has significantly fewer parameters in the decoder. The results highlight the superior performance of FINOLA over Conv-AE, indicating that a single-layer FINOLA is more effective than a multi-layer convolution and upsampling approach.

|Method|Latent Size&darr;|Params in Decoder&darr;|PSNR&uarr;|
|---|---|---|---|
| Conv-AE |1024x2| 35.9M | 24.6|
| **Hidden Wave (our)** |1024x2| **16.6M**|**24.8** |
||||
| Conv-AE | 1024x8 | 61.9M|26.0 |
| **Hidden Wave (our)** | 1024x8 | **16.6M**|**27.1** |

In summary, these new comparisons with three types of encoding-decoding techniques support our hypothesis that wave equations are fundamental to describe all images.

Reviewer comment (Reproducibility):
After the rebuttal, I am increasing my score due to several very encouraging additional experiments. Yet, the new version of the paper is a major revision that should undergo a full new review process. In particular, technical details (like the architecture of the convolutional autoencoder) seem to be missing. In general, in my opinion, to claim that wave equations are a mathematical property shared by all natural images more analysis is needed (including an interpretation of wave speeds, the diagonalizing basis, or experiments on what happens if the wave equation is solved differently from the special structure of the sum of FINOLAs). For claiming that a superposition of FINOLAs is one of the strongest latent compression techniques, more detailed comparisons (taking works on learned image compression into account) are needed.

Author response:
Thank you for your invaluable feedback and encouraging comments on our paper. Your time and effort in reviewing our work are highly appreciated.

**Major Revision**

We want to assure you that we are taking your feedback seriously. Currently, we are in the process of implementing a major revision to the paper. This includes incorporating the additional experiments you mentioned and integrating relevant literature on autoencoder-based image compression. The revised version will be submitted today, well before the deadline. We are committed to refining the paper to ensure a more complete and cohesive narrative, and we're pleased to hear that you find the general idea interesting and encouraging.

**Non-FINOLA-based solution**

Regarding your side note on a non-FINOLA-based solution to the wave equation, we appreciate your suggestion. We agree that it would have been a valuable addition to our study, and we plan to include it as a future perspective in the paper. The exploration of non-FINOLA-based solutions will be a focal point in our upcoming research.

Reviewer comment (Experiments):
As stated under 'Weaknesses' above, did you compare to any other encoding-decoding techniques that make the hypothesis, that wave equations are fundamental to all images more credible? The supplement compares to DCT and wavelet coding, but seemingly in terms of general aspects only, not in terms of reconstruction quality.

Author response:
**[Weakness 2 + Q1]: Comparing with other  encoding-decoding techniques  that use a latent space of similar dimension.**

Thank you for this excellent suggestion. Below, we compare Hidden Wave with (a) DCT, (b) DWT/DTCWT, and (c) convolutional autoencoder for image reconstruction on the ImageNet validation set, with an image size of 256x256. Hidden Wave achieves a higher PSNR for the same or smaller latent size.

***Comparison with discrete cosine transform (DCT):*** The table below compares Hidden Wave with DCT. DCT is conducted per 8x8 image block, and the top-left $K$ coefficients (in zig-zag manner) are kept, while the rest are set to zero. We choose four $K$ values (1, 3, 6, 10) for comparison. Clearly, Hidden Wave achieves a higher PSNR with a small latent size. Notably, in the last row, *"mix"* refers to placing 8 initial conditions at different positions rather than overlapping at the center.

|Method|Latent Size&darr;|PSNR&uarr;|
|---|---|---|
|DCT (top-left 1) | 3072 (32x32x3) | 20.6 |
| **Hidden Wave (our)** | **2048 (1024x2)** | **24.8** |
||||
|DCT (top-left 3)| 9216 (32x32x9) | 23.5|
| **Hidden Wave (our)** | **8192 (1024x8)** | **27.1** |
||||
|DCT (top-left 6)| 18432 (32x32x18) | 25.6|
| **Hidden Wave (our)** | **16384 (2048x8)** | **27.8** |
||||
|DCT (top-left 10)| 30720 (32x32x30) | 27.5|
| **Hidden Wave (our)** | **16384 (2048x8 mix)** | **28.9** |

***Comparison with discrete wavelet transform (DWT) and dual-tree complex wavelet transform (DTCWT):*** We compare Hidden-Wave with DWT and DTCWT in the table below. Three scales are chosen for wavelet decomposition. The comparisons are organized into three groups: (a) using only the LL subband at the coarsest scale (scale 3), (b) using all subbands (LL, LH, HL, HH) at the coarsest level, and (c) using all subbands at the finer scale (scale 2). Hidden-Wave outperforms DWT and DTCWT in terms of PSNR for the first two groups, achieving a smaller latent size. In the last group, while Hidden-Wave's PSNR is lower than DTCWT, its latent size is significantly smaller (more than 10 times smaller).

|Method|Latent Size&darr;|PSNR&uarr;|
|---|---|---|
|DWT (scale-3 LL) | 3888 | 21.5 |
|DTCWT (scale-3 LL) | 12288 | 22.3 |
| **Hidden Wave (our)** | **2048 (1024x2)** | **24.8** |
||||
|DWT (scale-3 LL+LH+HL+HH)| 15552 | 24.3|
|DTCWT (scale-3 LL+LH+HL+HH) | 49152 | 25.6 |
| **Hidden Wave (our)** | **8192 (1024x8)** | **27.1** |
||||
|DWT (scale-2 LL+LH+HL+HH)| 55953 | 28.7|
|DTCWT (scale-2 LL+LH+HL+HH) | 196608 | **30.8** |
| **Hidden Wave (our)** | **16384 (2048x8 mix)** | 28.9 |

***Comparison with convolutional Autoencoder (Conv-AE):*** The table below presents a comparative analysis between our method and convolutional autoencoder (Conv-AE) concerning image reconstruction, measured by PSNR. Both approaches share the same Mobile-Former encoder and have identical latent sizes (1024x2 or 1024x8). In our method, FINOLA is initially employed to generate a 64x64 feature map, followed by a convolutional decoder to reconstruct an image with a size of 256x256. On the other hand, Conv-AE employs a deeper decoder that utilizes convolution and upsampling from the latent vector to reconstruct an image. Hidden-Wave has significantly fewer parameters in the decoder. The results highlight the superior performance of FINOLA over Conv-AE, indicating that a single-layer FINOLA is more effective than a multi-layer convolution and upsampling approach.

|Method|Latent Size&darr;|Params in Decoder&darr;|PSNR&uarr;|
|---|---|---|---|
| Conv-AE |1024x2| 35.9M | 24.6|
| **Hidden Wave (our)** |1024x2| **16.6M**|**24.8** |
||||
| Conv-AE | 1024x8 | 61.9M|26.0 |
| **Hidden Wave (our)** | 1024x8 | **16.6M**|**27.1** |

In summary, these new comparisons with three types of encoding-decoding techniques support our hypothesis that wave equations are fundamental to describe all images.

Reviewer comment (Presentation):
The cited work of FINOLA by Chen et al. does not have a journal, conference or ArXiv ID. Is it published? Please complete the bibliography entry as this citation is of utmost importance for your work.

Author response:
**[Q2]: The cited work of FINOLA by Chen et al. does not have a journal, conference or ArXiv ID. Is it published? Please complete the bibliography entry as this citation is of utmost importance for your work.**

FINOLA is an arxiv paper at the link below. We will complete the bibliography entry.

*"Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance, by Chen et al."*
https://arxiv.org/abs/2305.16319.

Reviewer comment (Theory):
A major issue is the lack of mathematical clarity in presenting your model. As a consequence, certain central idea such as diagonalization does not seem to be correct.

Author response:
Thank you for pointing out the notation discrepancy and expressing concerns about the mathematical rigor in our paper.

The notation stating $\mathbfit{\zeta}(x, y) \in \mathbb{R}^C$ on top of page 4 was an oversight. We acknowledge this error and wish to clarify that $\mathbfit{\zeta}(x, y)$ is correctly defined as a complex-valued vector, denoted as $\mathbfit{\zeta}(x, y) \in \mathbb{C}^C$. This vector results from a linear projection of the real-valued feature map $\mathbfit{z}(x, y)$ using the inverse of the complex-valued eigenvalue matrix $\mathbfit{V}^{-1}$, as expressed by $\mathbfit{\zeta} = \mathbfit{V}^{-1}\mathbfit{z}$. Notably, $\mathbfit{V}^{-1}$ is itself a complex-valued matrix, as indicated by the relationship $\mathbfit{A}\mathbfit{B}^{-1} = \mathbfit{V}\mathbfit{\Lambda}\mathbfit{V}^{-1}$.

We apologize for any confusion caused by this oversight and assure you that we are taking steps to enhance the clarity and accuracy of the mathematical expressions in our paper. If you have further questions or specific areas that require clarification, please feel free to let us know. Your feedback is crucial in improving the precision of our work.

Indeed, your comment is accurate. The diagonalizability of $\mathbfit{A}\mathbfit{B}^{-1}$ is *NOT* guaranteed, even over complex values $\mathbb{C}$. In our experiments, however, this (diagonalizable) happens as a result of a learning process mainly because the set of matrices that can not be diagonalized is so small.

This led us to elaborate on the connection between the ***diagonalizability*** of $\mathbfit{A}\mathbfit{B}^{-1}$ and the ***disentanglement*** of one-way wave equations.

***Nondiagonalizable $\mathbfit{A}\mathbfit{B}^{-1}$ &rarr; Entangled One-Way Wave Equations:*** 
When $\mathbfit{A}\mathbfit{B}^{-1}$ is *not diagonalizable*, a linear partial differential equation (PDE) still holds:

$\frac{\partial \mathbfit{z}}{\partial x}=\mathbfit{A}\mathbfit{B}^{-1}\frac{\partial \mathbfit{z}}{\partial y}$.

This resembles a vectorized version of a one-way wave equation, with multiple dimensions linearly entangled.

***Diagonalizable $\mathbfit{A}\mathbfit{B}^{-1}$ &rarr; Disentangled One-Way Wave Equations:*** 
In contrast, when $\mathbfit{A}\mathbfit{B}^{-1}$ is *diagonalizable*, the wave equations can be disentangled after diagonalization. Consequently, each dimension (after projection by the inverse of the eigenvector matrix) follows a one-dimensional wave equation:

$\frac{\partial\zeta_k}{\partial x}=\lambda_k\frac{\partial \zeta_k}{\partial y}$

Therefore, the diagonalizability of $\mathbfit{A}\mathbfit{B}^{-1}$ indicates if wave equations can be disentangled. In practice, our experiments suggest that non-diagonalizable matrices rarely occur.

Reviewer comment (Theory):
In your model, A,B matrices defined in eq 2 are real-valued matrices and learnable. How do you guarantee that AB^-1 is diagonalizable such that the eigenvalues are all real-valued? (the Lambda diagonal matrix in eq 4 contains only real values along diagonal). In general, one could only assume that AB^-1 is diagonalizable such that V and Lambda are complex-valued (if AB^-1 is not symmetric).

Author response:
Thank you for your valuable feedback. Allow us to provide further clarification regarding the diagonalization process. In equation 4, it's crucial to note that we do *NOT* impose any constraints on the eigenvalues, allowing them to be complex-valued in the diagonalization of $\mathbfit{A}\mathbfit{B}^{-1}$. In other words, the wave speed can take complex values. It's important to emphasize that $\mathbfit{\Lambda}$ and $\mathbfit{V}$ are *Not* explicitly trainable; instead, they are computed from the trainable matrices $\mathbfit{A}$ and $\mathbfit{B}$ through diagonalization post-training. We meticulously examined the eigenvalues in $\mathbfit{\Lambda}$ and eigenvectors in $\mathbfit{V}$ across multiple models after training, confirming their complex nature.

In Section 2.4 of the paper, we also discuss an alternative approach wherein we constrain $\mathbfit{\Lambda}$ and $\mathbfit{V}$ to be real-valued by constraining matrices $\mathbfit{A}$ and $\mathbfit{B}$ as follows:

$\mathbfit{A}=\mathbfit{P}\mathbfit{H}_x, \quad \mathbfit{B}=\mathbfit{P}\mathbfit{H}_y$

where $\mathbfit{P}$ is the shared projection matrix for $\mathbfit{A}$ and $\mathbfit{B}$, and $\mathbfit{H}_x$ and $\mathbfit{H}_y$ are two real-valued diagonal matrices that are learnable. Consequently, the wave speed $\mathbfit{\Lambda} = \mathbfit{H}_x\mathbfit{H}_y^{-1}$ is also real-valued.

However, it's noteworthy that enforcing real values for the wave speed leads to a performance drop in reconstruction quality. Table 2 (or the table below) compares the reconstruction PSNR between complex and real speeds, showing a slight decrease from 26.1 to 24.9.

|Speed Value|\#Channels $C$|\#Solutions $M$|PSNR|
|---|---|---|---|
|Complex Value| 1024|4|**26.1**|
|Real Value| 1024|4|24.9| 

***Motivation for Studying Real-Valued Eigenvalues:***
The motivation behind our study of real-valued eigenvalues, specifically $\lambda_k$ in Table 2, stems from the question *"how the number type (real or complex) of wave speeds impacts reconstruction accuracy?"* By default, wave speeds are complex-valued, as previously discussed. To explore this, we conducted this ablation study, transitioning the number type from complex to real. The results of this investigation revealed a slight degradation in performance.

***Diagonalizable Assumption for Real-Valued Eigenvalues:***
Addressing your concern about the assumption of diagonalizability when considering real-valued eigenvalues, we affirm that ***the diagonalizable assumption holds in our model***. We constrain the FINOLA matrices $\mathbfit{A}$ and $\mathbfit{B}$ as follows:

$\mathbfit{A}=\mathbfit{P}\mathbfit{H}_x, \quad \mathbfit{B}=\mathbfit{P}\mathbfit{H}_y$

where $\mathbfit{P}$ is a real-valued matrix, and $\mathbfit{H}_x$ and $\mathbfit{H}_y$ are real-valued diagonal matrices, all of which are learnable. This results in $\mathbfit{A}\mathbfit{B}^{-1} = \mathbfit{P}\mathbfit{H}_x\mathbfit{H}_y^{-1}\mathbfit{P}^{-1}$. Importantly, the eigenvalues $\mathbfit{\Lambda} = \mathbfit{H}_x\mathbfit{H}_y^{-1}$ and eigenvector $\mathbfit{P}$ are explicitly real-valued and learnable.

We want to highlight that the only assumption in this context is that the projection matrix $\mathbfit{P}$ is invertible or full rank. Our experiments confirm that the learned $\mathbfit{P}$ matrix is indeed invertible.

Indeed, your comment is accurate. The diagonalizability of $\mathbfit{A}\mathbfit{B}^{-1}$ is *NOT* guaranteed, even over complex values $\mathbb{C}$. In our experiments, however, this (diagonalizable) happens as a result of a learning process mainly because the set of matrices that can not be diagonalized is so small.

This led us to elaborate on the connection between the ***diagonalizability*** of $\mathbfit{A}\mathbfit{B}^{-1}$ and the ***disentanglement*** of one-way wave equations.

***Nondiagonalizable $\mathbfit{A}\mathbfit{B}^{-1}$ &rarr; Entangled One-Way Wave Equations:*** 
When $\mathbfit{A}\mathbfit{B}^{-1}$ is *not diagonalizable*, a linear partial differential equation (PDE) still holds:

$\frac{\partial \mathbfit{z}}{\partial x}=\mathbfit{A}\mathbfit{B}^{-1}\frac{\partial \mathbfit{z}}{\partial y}$.

This resembles a vectorized version of a one-way wave equation, with multiple dimensions linearly entangled.

***Diagonalizable $\mathbfit{A}\mathbfit{B}^{-1}$ &rarr; Disentangled One-Way Wave Equations:*** 
In contrast, when $\mathbfit{A}\mathbfit{B}^{-1}$ is *diagonalizable*, the wave equations can be disentangled after diagonalization. Consequently, each dimension (after projection by the inverse of the eigenvector matrix) follows a one-dimensional wave equation:

$\frac{\partial\zeta_k}{\partial x}=\lambda_k\frac{\partial \zeta_k}{\partial y}$

Therefore, the diagonalizability of $\mathbfit{A}\mathbfit{B}^{-1}$ indicates if wave equations can be disentangled. In practice, our experiments suggest that non-diagonalizable matrices rarely occur.

Reviewer comment (Reproducibility):
It is not very clear in Fig 1 which part is trainable. Do you also train the encoder and decoder to recover input images? If the encoder is not trained, how is it defined? Also what is your training loss?

Author response:
***Trainable Components:*** The trainable elements in our architecture encompass both the encoder and decoder. The matrices $\mathbfit{A}$ and $\mathbfit{B}$ for the multi-path FINOLA are also trainable. Throughout training, the entire network is optimized end-to-end. It's essential to note that the wave speed $\mathbfit{\Lambda}$ and the coefficient matrices $\mathbfit{H}_x$ and $\mathbfit{H}_y$ are not explicitly trainable. Instead, they are computed post-training through the diagonalization process using the trainable matrices $\mathbfit{A}$ and $\mathbfit{B}$.

***Training of Encoder and Decoder*** Indeed, both the encoder and decoder undergo training to facilitate the reconstruction of input images.

***Training Loss:*** The training process employs mean square error over image pixels as the loss function, ensuring the end-to-end optimization of the entire network.

Reviewer comment (Evaluation):
Your decoder is linear, would that make the recovered images lose details (such as sharp edges) in images? This is not very clear from PNSR metrics. It would be better to add some visual examples.

Author response:
Thank you for your insightful recommendation. Visual examples serve as compelling evidence, offering a more nuanced perspective beyond the limitations of PSNR. Please find below a link to three visualizations:

Link to visualizations: https://tinyurl.com/yc5bb7dn

***(a) Comparison between Hidden-Wave ($M=8$) and FINOLA ($M=1$):*** 
The visual comparison vividly illustrates that Hidden-Wave, with a greater number of special solutions, achieves superior reconstruction quality by preserving finer visual details compared to FINOLA.

***(b) Reconstruction examples for varying number of wave equations ($C$) and special solutions ($M$):*** 
The visual examples provide a clear demonstration that increasing the number of special solutions consistently enhances image quality across different dimensions ($C=128$ to $C=2048$).

***(c) Reconstruction examples for complex-valued and real-valued wave speeds:*** 
This set of visualizations reveals that while complex-valued wave speed leads to superior image quality, the use of real-valued speed also results in reasonably good reconstructions.

It's essential to note that despite the linearity of the one-way wave equations, the entire decoder is *NOT* purely linear. The decoder incorporates convolutional layers during the upsampling process from the feature map (resolution 64x64) to the reconstructed images at size 256x256. This non-linearity in the decoder is crucial for capturing and preserving intricate details, such as sharp edges, in the reconstructed images.

Reviewer comment (Evaluation):
In figure 4, it is not clear if there is plateau for the reconstruction quality in terms of M & C. The curves seem to be increasing linearly, and we do not see a slowdown.

Author response:
Thanks for this insightful suggestion! We conducted additional experiments to explore the impact of increasing the number of special solutions (or the number of FINOLA paths $M$) to 16. We evaluated for three choices of the number of wave equations (or the number of channels $C$), specifically 128, 256, and 512. The PSNR results are presented in the last column of the table below. In comparison to the rate of change observed over smaller $M$ values (1→2, 2→4, 4→8), the PSNR increase from $M=8$ to $M=16$ slows down, indicating the onset of a plateau.

***Table: Reconstruction PSNR for different number of channels ($C$) and special solutions ($M$)***
|Channels|**M=1**|**M=2**|**M=4**|**M=8**|**M=16**|
|---|---|---|---|---|---|
|**C=512**|  22.0|23.2|24.5|25.8|**26.3**|
|**C=256**|  20.6|21.8|22.9|24.4|**25.5**|
|**C=128**| 19.3|20.4|21.5|22.5| **23.1**|