Paper: ASMR: Activation-Sharing Multi-Resolution Coordinate Networks for Efficient Inference (ICLR-2024)

Reviewer comment (Experiments):
While several baselines are used in all experiments, instant-NGP is not included in the experiments measuring inference cost. As Instant-NGP is known for its fast inference, I think it is not possible to know whether ASMR is really better than grid-based approaches.

Author response:
**Comparison to Grid-based Methods**

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [3] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset. 

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where direct comparison is not applicable. The main focus of our work is not to beat the state-of-the-art INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs *without* sacrificing their original reconstruction capabilities.

Reviewer comment (Presentation):
In Figure 2, left, kiloNERF always has a higher MAC count than ASMR. In Figure 2, right, kiloNERF often has a lower MAC than ASMR. From the text, I assumed these were the same trained models. This looks inconsistent to me.

Author response:
**Clarification on Figure 2**

We apologize for the unclear description and appreciate you pointing this out. The left and right figures are meant to show different sets of KiloNeRF and LoE model configurations. The intention of the left figure was to complement Table 1 (analytical relationship between MAC, \#Params, depth) with empirical data. The model configurations are chosen such that all the parameter counts are within the same range across the 4 models and the correlations between MAC and \#Params can be clearly demonstrated. On the other hand, the right figure is an extension of Table 2 (ultra-low MAC fitting results) showing the full MAC-PSNR curves of the models when having 3 to 6 layers. In our revised manuscript, we have separated the two figures and positioned them alongside the tables that they are meant to complement. Additional model configurations of the ultra-low MAC experiments are added to Appendix B accordingly.

Reviewer comment (Theory):
In Section 3.3, the paper states that the per-sample inference cost depends only on the width of the model and is independent of the depth. I am no sure I agree with that statement, as the depth depends on the choice of the basis and thus on the width of the layers. For example, as far as I understand, doubling the size of the basis also means halving the number of layers.

Author response:
**Decoupling Between MACs and Depth**

By width, we are referring to the number of hidden units in a network, which is independent of the choice of basis. If you are referring width to the number of unique inputs (coordinates) that each layer of the network needs to infer, then you are correct in that "doubling the size of the basis" would halve the number of layers *and* increase the width (number of unique inputs) by 2. However, as provided in the original manuscript, the derivation of ASMR's inference MAC is as follows:
\begin{align*}
    \text{MAC} = \sum_{i=1}^LB^iM_i \leq M\sum_{i=1}^LB^i &= MB\frac{B^L - 1}{B - 1} \approx \frac{MB}{B-1}B^L = \frac{MB}{B-1}B^{\log_B N} = \frac{B}{B-1}MN \leq 2MN
\end{align*}

where the MAC is bounded above by $2MN$ regardless of the choice of basis since $\frac{B}{B-1} < 2$ for all $B$. Hence, asymptotically, ASMR does indeed decouple its inference cost from the network depth. In fact, you may see from the right figure of Figure 2 in the original manuscript that the MAC of ASMR is not strictly but *approximately* constant in a practical setting.


We are pleased to hear that our additional experiments and amendments to the manuscript are well perceived. We would also like to express our sincere gratitude for your thoughtful comments and criticisms. It has brought fundamental improvements to our work.

We agree that adding "asymptotic" to Proposition 1 would make the statement more accurate. However, may you clarify why you believe that:
>The width and the depth of the ASMR architecture are correlated as the depth and width are constrained by the dimension and the choice of the basis

Under the assumption that parameters are NOT held fixed, we don't see a correlation between the # of hidden units and the choice of basis (hence network depth). Instead, the # of hidden units is a fully flexible hyperparameter that could be chosen arbitrarily by the user. We suspect that this confusion may be due to Figure 1(c) in the manuscript. We clarify by stating that the square in Figure 1(c) represents the feature maps of the entire image, and each "pixel" in the square is the feature generated by a forward pass of a **single** coordinate through the modulator, with length determined by the # of hidden units. That is, the choice of basis only affects the size of the square but NOT the dimension of the features

Reviewer comment (Writing):
The paper did not look polished in some places to me. For example, I am under the impression that multiple notations are used to refer to the same or very similar things, which was confusing to me. Here are a few examples:  
- a base is denoted by $|x_i$ in Figure 1 and $B_i$ in Section 3.1 (where Figure 1 is referenced)  
- the cumulative product of bases is denoted by $|x_{i...0}|$ in Figure 1 and by $C_i$ in Section 3.1.

Author response:
**Writing Issues**

We have made the following changes (highlighted in red) in the revised manuscript:
- We fixed the typo and specified that a uniform base of 2 is used in the caption of Figure 1.
- We replaced the notations of base and cumulative base in Figure 1 so that they are in line with those defined in Section 3.1.
- We fixed the labeling issue of Equation 1.

Reviewer comment (Experiments):
For the video experiment, a single video is used. I find that a bit underwhelming as recent papers have typically used video benchmarks for evaluation.

Author response:
**Additional Video Results**

We have tested ASMR on the UVG dataset, which is commonly used as a benchmark for testing video representations such as in the works of [1] and [2]. The results are consistent with our findings on the cat video presented in the original manuscript. Please refer to our General Response which contains additional experimental results on audio and 3D shapes.

Reviewer comment (Evaluation):
The number of MACs is certainly a good indicator of inference speed. It would be much more convincing to also have actual inference wall clock timing on several hardware platforms.

Author response:
**Latency**

To demonstrate the improved inference speed of our ASMR method on resource-limited devices, we measured the rendering latency of a 512 $\times$ 768 image (same as Kodak) on various hardware platforms. These platforms range from high-end to low-end CPUs, including a server-level AMD 7413 2.65GHz 24-Core CPU, a desktop 3.1 GHz Dual-Core Intel Core i5 CPU, and a Quad-core Cortex-A72 (ARM v8) 1.8GHz CPU of Raspberry Pi 4 model B. The reported latency is an average across 10 runs. The instantiation of each model is the same as the one mentioned in Section 4.3 of the manuscript, with parameter counts of around 130K. It can be observed that ASMR consistently speeds up SIREN by $>3\times$ in terms of latency, and the speed-up becomes more significant when tested on devices with reduced computation power (viz. $>4\times$ on Raspberry Pi). Processes that are too computationally intensive and result in hanging or being automatically killed due to excessive RAM usage are indicated by (Failed).

| Latency (ms) | FFN    | SIREN   | MFN      | WIRE     | ASMR   |
|--------------|--------|---------|----------|----------|--------|
| Server       | 369.5  | 441.4   | 459.9    | 2296.5   | 143.9  |
| Desktop      | 2859.2 | 4400.0  | 10291.4  | (Failed) | 957.7  |
| Raspberry Pi | 8975.4 | 13476.6 | (Failed) | (Failed) | 3202.0 |

Reviewer comment (Experiments):
It is okay to test only on image/video fitting tasks, but a study on other data types such as audio or 3D shape could be much more convincing to show the generality of the multi-resolution activation-sharing schema.

Author response:
**Other Data Modalities**

To further demonstrate ASMR's ability to generalize across multiple data modalities, we have tested on the Librispeed dataset (Audio), UVG dataset (Video), and ShapeNet dataset (3D shapes) and included the results in the general response. Consistent with our findings in the original manuscript, our new results show that ASMR successfully reduces the inference cost of a SIREN model without sacrificing (or sometimes brings an even better) reconstruction quality. In particular, we found that ASMR performs better with low-dimension data such as audio, with an exceptional improvement over vanilla SIREN. For more details, please kindly refer to the general response at the top.

Reviewer comment (Writing):
The word ‘inference bandwidth’ is kinda misused. Maybe consider ‘throughput’ or ‘latency’.

Author response:
**Writing Issues**

We have made the following changes (highlighted in red) in the revised manuscript:

* We replaced "inference bandwidth" with "inference throughput"
* We fixed the labeling issue of Equation 1.

Reviewer comment (Reproducibility):
How were the MACs counted (in terms of software implementation)?

Author response:
**MACs Calculation**

DeepSpeed Profiler is sufficient in calculating the MACs of all the models except for ASMR. To obtain the per-sample MAC, we divide the MAC outputted by the `get_model_profile()` function by the size of the input. For ASMR, we first obtained the MACs of each individual layer and modulator using DeepSpeed Profiler, then manually amortized the per-layer inference cost across the grid by the resolution of the layer.

Reviewer comment (Theory):
My understanding from Figure 1.c about the modulator M is it adds the same pattern details to coarser coordinated activations (upsampled). It is counterintuitive why the sample pattern beats SIREN. Could the authors provide some explanations or intuitions?

Author response:
**Clarification on Figure 1**

Your understanding is correct, and this is exactly why ASMR could reduce the inference cost of SIREN by orders of magnitude. ASMR does not only achieve activation-sharing on the INR backbone but also on the modulators. We would like to highlight that even though the modulations also exhibit a repeated pattern, their pattern is actually different from that of the previous-layer activation (as indicated by different colors in Figure 1). The summation of both results in activations that are unique to each coordinate. Intuitively, one may think of modulations as a set of phase shifts to the sine activations generated by the SIREN backbone. ASMR does so efficiently by finding a common set of phase shifts that can be applied repeatedly to generate a unique combination of frequency supports representing the data. 

As for the superior representation performance of ASMR over SIREN, we argue that it is because SIREN suffers from spectral bias while ASMR does not. As illustrated in Appendix G, SIREN learns the lower frequency information first, then progressively adds higher frequency details. For data with abrupt changes in value (i.e. very high frequency) such as non-analytical 3D shapes, it may even fail to converge and get stuck with a "smooth'' representation. On the other hand, by injecting coordinates that repeat themselves (i.e. periodic) in progressively higher frequencies, ASMR could tackle a much wider band of frequencies from the start of the training. Its discretized nature also permits it to learn any sort of data as long as it is within its grid resolution.

Reviewer comment (Novelty):
The overall concept is highly similar (although the detail is different) to [1].

Author response:
Thank you for the comments and suggestions. 

**Comparison to Multi-Scale INR [1]**

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.


| Method              | Coordinate Representation | Modulation | \#Params (K) | MACs (K) | PSNR (dB)$\uparrow$ | SSIM $\uparrow$ |
|---------------------|---------------------------|------------|--------------|----------|---------------------|-----------------|
| Multi-scale INR [1] | Multi-scale               | FMM        | 49.6         | 59.623   | 26.73               | 0.746           |
| Ablation \#1         | Multi-scale               | Bias-only  | 30.4         | 1.600    | 16.21               | 0.148           |
| Ablation \#2         | Coord. decomp.            | FMM        | 49.6         | 1.434    | 29.65               | 0.784           |
| ASMR                | Coord. decomp.            | Bias-only  | 30.4         | 1.429    | 29.71               | 0.795           |

We believe that there are two major differences between ASMR and [1]

**1. Multi-Scale Coordinates vs. Coordinate Decomposition**

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

**2. FMM vs. Bias-only Modulation**

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

**References**

[1] Adversarial Generation of Continuous Images, CVPR 2021

Reviewer comment (Novelty):
Using multi-scale INR is explored in [1] (also, [2] uses the same idea in the video domain). Use upsampling (bilinear, nearest, etc) to reduce the number of forward passes by sharing the activations.

Author response:
Thank you for the comments and suggestions. 

**Comparison to Multi-Scale INR [1]**

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.


| Method              | Coordinate Representation | Modulation | \#Params (K) | MACs (K) | PSNR (dB)$\uparrow$ | SSIM $\uparrow$ |
|---------------------|---------------------------|------------|--------------|----------|---------------------|-----------------|
| Multi-scale INR [1] | Multi-scale               | FMM        | 49.6         | 59.623   | 26.73               | 0.746           |
| Ablation \#1         | Multi-scale               | Bias-only  | 30.4         | 1.600    | 16.21               | 0.148           |
| Ablation \#2         | Coord. decomp.            | FMM        | 49.6         | 1.434    | 29.65               | 0.784           |
| ASMR                | Coord. decomp.            | Bias-only  | 30.4         | 1.429    | 29.71               | 0.795           |

We believe that there are two major differences between ASMR and [1]

**1. Multi-Scale Coordinates vs. Coordinate Decomposition**

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

**2. FMM vs. Bias-only Modulation**

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

**References**

[1] Adversarial Generation of Continuous Images, CVPR 2021

Reviewer comment (Novelty):
[1] also uses modulation as well.

Author response:
Thank you for the comments and suggestions. 

**Comparison to Multi-Scale INR [1]**

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.


| Method              | Coordinate Representation | Modulation | \#Params (K) | MACs (K) | PSNR (dB)$\uparrow$ | SSIM $\uparrow$ |
|---------------------|---------------------------|------------|--------------|----------|---------------------|-----------------|
| Multi-scale INR [1] | Multi-scale               | FMM        | 49.6         | 59.623   | 26.73               | 0.746           |
| Ablation \#1         | Multi-scale               | Bias-only  | 30.4         | 1.600    | 16.21               | 0.148           |
| Ablation \#2         | Coord. decomp.            | FMM        | 49.6         | 1.434    | 29.65               | 0.784           |
| ASMR                | Coord. decomp.            | Bias-only  | 30.4         | 1.429    | 29.71               | 0.795           |

We believe that there are two major differences between ASMR and [1]

**1. Multi-Scale Coordinates vs. Coordinate Decomposition**

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

**2. FMM vs. Bias-only Modulation**

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

**References**

[1] Adversarial Generation of Continuous Images, CVPR 2021

Reviewer comment (Novelty):
Due to the existence of [1], the paper should change the overall claim, e.g., the first INR to decouple MAC from its depth or using shared activations.

Author response:
Thank you for the comments and suggestions. 

**Comparison to Multi-Scale INR [1]**

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.


| Method              | Coordinate Representation | Modulation | \#Params (K) | MACs (K) | PSNR (dB)$\uparrow$ | SSIM $\uparrow$ |
|---------------------|---------------------------|------------|--------------|----------|---------------------|-----------------|
| Multi-scale INR [1] | Multi-scale               | FMM        | 49.6         | 59.623   | 26.73               | 0.746           |
| Ablation \#1         | Multi-scale               | Bias-only  | 30.4         | 1.600    | 16.21               | 0.148           |
| Ablation \#2         | Coord. decomp.            | FMM        | 49.6         | 1.434    | 29.65               | 0.784           |
| ASMR                | Coord. decomp.            | Bias-only  | 30.4         | 1.429    | 29.71               | 0.795           |

We believe that there are two major differences between ASMR and [1]

**1. Multi-Scale Coordinates vs. Coordinate Decomposition**

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

**2. FMM vs. Bias-only Modulation**

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

**References**

[1] Adversarial Generation of Continuous Images, CVPR 2021

Reviewer comment (Experiments):
A comparison with Multi-scale INR [1] is definitely needed.

Author response:
Thank you for the comments and suggestions. 

**Comparison to Multi-Scale INR [1]**

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.


| Method              | Coordinate Representation | Modulation | \#Params (K) | MACs (K) | PSNR (dB)$\uparrow$ | SSIM $\uparrow$ |
|---------------------|---------------------------|------------|--------------|----------|---------------------|-----------------|
| Multi-scale INR [1] | Multi-scale               | FMM        | 49.6         | 59.623   | 26.73               | 0.746           |
| Ablation \#1         | Multi-scale               | Bias-only  | 30.4         | 1.600    | 16.21               | 0.148           |
| Ablation \#2         | Coord. decomp.            | FMM        | 49.6         | 1.434    | 29.65               | 0.784           |
| ASMR                | Coord. decomp.            | Bias-only  | 30.4         | 1.429    | 29.71               | 0.795           |

We believe that there are two major differences between ASMR and [1]

**1. Multi-Scale Coordinates vs. Coordinate Decomposition**

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

**2. FMM vs. Bias-only Modulation**

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

**References**

[1] Adversarial Generation of Continuous Images, CVPR 2021

Reviewer comment (Novelty):
More recent modulation techniques exist in the field compared to bias modulation [3,4]. It would also be interesting if ASMR uses more advanced modulations, e.g., low-rank modulation for SIREN [3].

Author response:
**Other Modulation Methods**

We acknowledge that there exists a wide variety of methods for implementing modulation and have revised our manuscript by including the missing references (highlighted in red). However, we would like to highlight the motivation behind choosing bias modulation over other methods: its effectiveness and simplicity in terms of implementation and inference cost. We aim to introduce ASMR as a straightforward addition to a SIREN model, which can significantly reduce its inference cost without adding much complexity. Compared to advanced methods like variational modulation in [3,4] or low-rank modulation in [1], bias modulation is the best fit for these characteristics. Intuitively, bias modulation used in ASMR can be interpreted as phase shifts that describe the variation of decomposed coordinates at each hierarchical level. The above ablation study shows that bias-only modulation can effectively model such variation and even outperform the more complex low-rank modulation technique, which incurs more parameters.

Reviewer comment (Reproducibility):
The method is somewhat non-trivial to use for 3D scene rendering.

Author response:
**Other Data Modalities: Audio and 3D Shapes**

To demonstrate the versatility of ASMR in handling different modalities, we present additional experimental results on audio and 3D shape datasets (please kindly refer to the general response at the top). We found that ASMR performs exceptionally well in particular on low-dimensional data like audio. Not only does it exhibit low MACs behavior, but it also leads to faster convergence and remarkably better results vs vanilla SIREN. 

It is a valid concern that ASMR's discrete nature may make it difficult to extend to 3D data. However, we argue that as long as the data is discretizable (which is the case for almost any data including SDFs and neural fields), ASMR could effectively become a low-inference cost counterpart to a vanilla SIREN model. For instance, we trained an ASMR on 3D occupancy grids of random objects in ShapeNet, and found that matches the quality of SIREN while operating at a significantly lower inference cost (for more details, please refer to the general response at the top).

Reviewer comment (Novelty):
The authors should claim the benefit of using ASMR compared to modality-specific ones, as there have been several papers recently tackling modality-specific INRs.

Author response:
**Comparison to Modality Specific INRs**

We fully agree that each modality has its own state-of-the-art (SOTA) that excels in efficiency and quality. However, we argue that most of these SOTAs are usually overly modality-specific in the sense that their tricks are specialized to excel only in the tasks that they are intended for. For instance, [1,2] take special measures for the time dimension and grid-based methods fail to learn global latent representations. In contrast, general backbone models like SIREN are applicable to all modalities and downstream tasks. Hence, we see great values in proposing ASMR as a simple, modularized addition to SIREN that reduces its inference cost while inheriting its generalization capabilities. The modularity of ASMR also makes it orthogonal to other INR techniques such that they could be potentially combined for specific modalities or tasks.

Reviewer comment (Experiments):
The authors should show multiple modalities rather than just images (with video only showing one example). Considering Audio or Manifold [8] would be useful.

Author response:
**Other Data Modalities: Audio and 3D Shapes**

To demonstrate the versatility of ASMR in handling different modalities, we present additional experimental results on audio and 3D shape datasets (please kindly refer to the general response at the top). We found that ASMR performs exceptionally well in particular on low-dimensional data like audio. Not only does it exhibit low MACs behavior, but it also leads to faster convergence and remarkably better results vs vanilla SIREN. 

It is a valid concern that ASMR's discrete nature may make it difficult to extend to 3D data. However, we argue that as long as the data is discretizable (which is the case for almost any data including SDFs and neural fields), ASMR could effectively become a low-inference cost counterpart to a vanilla SIREN model. For instance, we trained an ASMR on 3D occupancy grids of random objects in ShapeNet, and found that matches the quality of SIREN while operating at a significantly lower inference cost (for more details, please refer to the general response at the top). 


**Additional Video Experiments**

We present additional results for the video-fitting task on the UVG dataset, which is a commonly used benchmark for testing video representations. The results are consistent with our findings on the cat video presented in the original manuscript. For details please refer to our general response at the top.

Reviewer comment (Novelty):
The authors need to discuss the benefit of using ASMR over modality-specific INRs. For instance, there are several video INR papers [6,7] showing efficiency.

Author response:
**Comparison to Modality Specific INRs**

We fully agree that each modality has its own state-of-the-art (SOTA) that excels in efficiency and quality. However, we argue that most of these SOTAs are usually overly modality-specific in the sense that their tricks are specialized to excel only in the tasks that they are intended for. For instance, [1,2] take special measures for the time dimension and grid-based methods fail to learn global latent representations. In contrast, general backbone models like SIREN are applicable to all modalities and downstream tasks. Hence, we see great values in proposing ASMR as a simple, modularized addition to SIREN that reduces its inference cost while inheriting its generalization capabilities. The modularity of ASMR also makes it orthogonal to other INR techniques such that they could be potentially combined for specific modalities or tasks.

Reviewer comment (Experiments):
ASMR seems hard to use for 3D scenes, so it has less benefit compared to SIREN or FFN.

Author response:
**Other Data Modalities: Audio and 3D Shapes**

To demonstrate the versatility of ASMR in handling different modalities, we present additional experimental results on audio and 3D shape datasets (please kindly refer to the general response at the top). We found that ASMR performs exceptionally well in particular on low-dimensional data like audio. Not only does it exhibit low MACs behavior, but it also leads to faster convergence and remarkably better results vs vanilla SIREN. 

It is a valid concern that ASMR's discrete nature may make it difficult to extend to 3D data. However, we argue that as long as the data is discretizable (which is the case for almost any data including SDFs and neural fields), ASMR could effectively become a low-inference cost counterpart to a vanilla SIREN model. For instance, we trained an ASMR on 3D occupancy grids of random objects in ShapeNet, and found that matches the quality of SIREN while operating at a significantly lower inference cost (for more details, please refer to the general response at the top).

Reviewer comment (Experiments):
Only one video sample was used for evaluation. Rather than reporting one sample, it is better to consider a dataset.

Author response:
**Additional Video Experiments**

We present additional results for the video-fitting task on the UVG dataset, which is a commonly used benchmark for testing video representations. The results are consistent with our findings on the cat video presented in the original manuscript. For details please refer to our general response at the top.

Reviewer comment (Writing):
It is hard to understand the intention of Section 5.2 (as it is somewhat trivial).

Author response:
**Comparison to Instant-NGP and Intention of Section 5.2**

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [5] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset. 

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where a direct comparison is not applicable. The main focus of our work is not to beat the SOTA INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs *without* sacrificing their original reconstruction capabilities.

Reviewer comment (Experiments):
Comparison with grid-based baselines is missing, e.g., Instant-NGP [10].

Author response:
**Comparison to Instant-NGP and Intention of Section 5.2**

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [5] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset. 

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where a direct comparison is not applicable. The main focus of our work is not to beat the SOTA INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs *without* sacrificing their original reconstruction capabilities.

Reviewer comment (Experiments):
Instant-NGP shows image, high-resolution image, and video experiments, and it is very worth comparing the efficiency as these methods are proposed for efficiency.

Author response:
**Comparison to Instant-NGP and Intention of Section 5.2**

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [5] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset. 

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where a direct comparison is not applicable. The main focus of our work is not to beat the SOTA INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs *without* sacrificing their original reconstruction capabilities.