Paper: HOSC: Hyperbolic Oscillating Periodic Activations for Sharp Feature Preservation in Implicit Neural Representations (ICLR-2024)

Reviewer comment (Experiments):
The experiments are inadequate given the rapid progress in implicit neural representations since SIREN. The authors need to demonstrate that HOSC makes a difference on 3D objects and scenes (e.g. the NeRF dataset) and/or gigapixel images, and/or continues to yield benefits when combined with modern INR parameterizations such as DVGO, TensorRF, Instant NGP, etc.

Author response:
(2) Extended Evaluations: We have expanded our evaluations to include 3D Signed Distance Functions (3D SDFs) and Giga-Pixel images, demonstrating HOSC's versatility and quality of reconstructions. We acknowledge that periodic activations generally face challenges with NeRF applications (also SIREN is not used with NeRFs). However, the added evaluations in 3D SDFs and large-scale images provide a comprehensive understanding of HOSC's capabilities. Regarding 3D SDFs we reach reconstruction quality in the range of the recent method Dictionary Fields (please compare our results with their figure: https://apchenstu.github.io/FactorFields/img/sdfs.png)

Reviewer comment (Presentation):
Figure 3 and Figure 4 are not referenced in this paper. Please revise it.

Author response:
Figures 3 and 4 References: We have ensured that all the figures are properly referenced in the revised manuscript, providing the necessary context and explanations.

Reviewer comment (Experiments):
The experiment 1 is about fitting square wave and sawtooth wave. Maybe more results such as the loss values and fitting figures of complex or random signals in the main body could make the paper more persuasive, because the complex or random signals are more important than regular signals in the provement.

Author response:
Expanding Experiment 1: Your suggestion to include more results, such as loss values and fitting figures for complex or random signals, is well-received. We agree that demonstrating HOSC's performance on more complex or random signals would substantiate our claims further. In our revised submission, we will include additional results for these types of signals, showcasing HOSC's versatility and effectiveness beyond regular signals. We believe these revisions will enhance the clarity and persuasiveness of our paper, and we appreciate the opportunity to improve our work based on your valuable feedback.

Reviewer comment (Experiments):
The evaluations are not extensive in comparison to competing methods and across different applications.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1a. Comparisons with Other Methods: We acknowledge the need for broader comparisons, we address it by providing further experiments on 3D SDF functions and Giga-Pixel images. We use the recent results of Dictionary Fields [4] as reference. 
Due to time constraints, unfortunately we were not able to directly compare our results to “traditional” Fourier Features Positional Encoding. In fact, tuning of the parameters (especially the scale) of FF-PE turned out to be extremely cumbersome and could not be finalized in time. Nonetheless, we construct our results of 3D SDF in the same fashion as the recent work of Chen 2023b, which can be compared with their results at: https://apchenstu.github.io/FactorFields/img/sdfs.png

Reviewer comment (Experiments):
Comparisons to only SIREN and ReLU networks are not very extensive. The exclusion of positional encoding from the comparison is questionable, since an embedding layer as the first layer of a network is reasonable and commonly used in most coordinate-based applications now. No comparison to this severely limits the impact, as researchers who work with these networks will be unlikely to adopt HOSC without a detailed comparison. Furthermore, there are several other alternate coordinate-based network architectures not compared to. For example, [1] and [2] also claim improvement over ReLU and SIREN architectures. There is no reason why HOSC should not be compared to these.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1a. Comparisons with Other Methods: We acknowledge the need for broader comparisons, we address it by providing further experiments on 3D SDF functions and Giga-Pixel images. We use the recent results of Dictionary Fields [4] as reference. 
Due to time constraints, unfortunately we were not able to directly compare our results to “traditional” Fourier Features Positional Encoding. In fact, tuning of the parameters (especially the scale) of FF-PE turned out to be extremely cumbersome and could not be finalized in time. Nonetheless, we construct our results of 3D SDF in the same fashion as the recent work of Chen 2023b, which can be compared with their results at: https://apchenstu.github.io/FactorFields/img/sdfs.png

Reviewer comment (Experiments):
The comparisons are only done for "overfitting" signals, i.e., memorizing 1D/2D/3D function values, and not for utilizing these representations in inverse problems, which is how they are mainly used in research. For example, motivating the work by "radiance fields" fitting and then not demonstrating how HOSC performs in radiance field applications is an overclaim. In radiance fields, the inverse problem requires not only accurate fitting of supervised values but also accurate interpolation for novel view synthesis. Lack of comparison here limits HOSC's applicability to narrow cases like signal overfitting or compression, which is not emphasized as the efficiency of HOSC networks is not compared (e.g., showing fitting with fewer parameters).

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1b. Application to NeRFs: as the reviewer surely knows, periodic activation functions (like SIREN) are networks which learn their own Fourier-like encoding and are not used with traditional FF-PE. That particularly turned out not to work well with NeRFs and hence SIRENs are not used to NeRF encoding. This is a limitation of periodic actications and also HOSC shares it.

Reviewer comment (Experiments):
The paper makes no attempt to evaluate the "generalization" properties of coordinate-based networks using the HOSC activation function. For example, [3] extensively studies this for Fourier Features and shows that frequency affects generalization, i.e., the behavior between supervised points. This property is extremely important for coordinate-based networks in imaging and vision tasks, and it is not explored at all in this paper.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1c. Generalization Properties: We agree that a test of how HOSC performs on generalization is a very important aspect. Intuitively, we think it will perform very well on Auto Decoders like DeepSDF, as it shows superior performance on overfitting single SDFs.  Unfortunately, due to time constraints, we were not able to perform this experiment in the rebuttal version. But we will be able to perform it for the final revision of the paper.

Reviewer comment (Experiments):
The robustness of the method is questionable due to the presence of an extra hyperparameter, a, which significantly affects signal reconstruction quality. Other methods like SIREN also have hyperparameters, such as the w0 factor, which affects fit quality considerably. It is unclear why a was tuned but w0 was not. Without testing both, it is uncertain how robust HOSC is to various levels of a.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

2. Our method turns out to be very robust wrt its hyperparameters, in fact, with the extension to AdaHOSC (see main rebuttal answer), we essentially got rid of any parameter tuning. By setting a default initial value for beta=8 (parameter “a” in previous version), we were able to make the model essentially work from scratch in all setups. This is a huge improvement compared to FF-PE, where tuning of the proper scale/sigma turned out to be extremely frustrating (due to its high-sensitivity wrt the signal), and was one of the reasons why we have not included there comparisons in the revision (we will perform it for the final revision though). 

Moreover, we would like to point out that in both this and previous revision we extensively tested the “frequency” parameter “w0” of SIREN (in previous revision called “b”), now called “omega”.

Reviewer comment (Experiments):
Lack of comparisons to additional work that has been published and shown to improve upon ReLU and SIREN architectures in signal fitting is a significant weakness. Without a study on generalization within the signal domain and/or comparison on solving an inverse problem task where coordinate-based networks are actually used, such as radiance field fitting, HOSC seems extremely limited to simple problems in fitting 1D/2D/3D signals. Adding these comparisons would be critical for increasing the impact of the paper.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1a. Comparisons with Other Methods: We acknowledge the need for broader comparisons, we address it by providing further experiments on 3D SDF functions and Giga-Pixel images. We use the recent results of Dictionary Fields [4] as reference. 
Due to time constraints, unfortunately we were not able to directly compare our results to “traditional” Fourier Features Positional Encoding. In fact, tuning of the parameters (especially the scale) of FF-PE turned out to be extremely cumbersome and could not be finalized in time. Nonetheless, we construct our results of 3D SDF in the same fashion as the recent work of Chen 2023b, which can be compared with their results at: https://apchenstu.github.io/FactorFields/img/sdfs.png

1b. Application to NeRFs: as the reviewer surely knows, periodic activation functions (like SIREN) are networks which learn their own Fourier-like encoding and are not used with traditional FF-PE. That particularly turned out not to work well with NeRFs and hence SIRENs are not used to NeRF encoding. This is a limitation of periodic actications and also HOSC shares it. 

1c. Generalization Properties: We agree that a test of how HOSC performs on generalization is a very important aspect. Intuitively, we think it will perform very well on Auto Decoders like DeepSDF, as it shows superior performance on overfitting single SDFs.  Unfortunately, due to time constraints, we were not able to perform this experiment in the rebuttal version. But we will be able to perform it for the final revision of the paper.

Reviewer comment (Experiments):
Even after the additional comparisons provided in the author response, there are still too many limitations: HOSC cannot be applied to radiance fields, and there is still a lack of generalization evaluation. Here, "generalization" refers not to between the weights of different representations (as in deepSDF) but to the sense used in the Fourier Features paper, where only a subset of values is supervised and interpolation between them is evaluated. This is extremely critical for assessing any coordinate-based network.

Author response:
We appreciate the thoughtful feedback provided by the reviewer. We address the concerns as follows:

1b. Application to NeRFs: as the reviewer surely knows, periodic activation functions (like SIREN) are networks which learn their own Fourier-like encoding and are not used with traditional FF-PE. That particularly turned out not to work well with NeRFs and hence SIRENs are not used to NeRF encoding. This is a limitation of periodic actications and also HOSC shares it. 

1c. Generalization Properties: We agree that a test of how HOSC performs on generalization is a very important aspect. Intuitively, we think it will perform very well on Auto Decoders like DeepSDF, as it shows superior performance on overfitting single SDFs.  Unfortunately, due to time constraints, we were not able to perform this experiment in the rebuttal version. But we will be able to perform it for the final revision of the paper.