Paper: Decodable and Sample Invariant Continuous Object Encoder (ICLR-2024)

Reviewer comment (Evaluation):
In Table 1, some metrics did not show improvement when comparing to the prior work HSurf-Net.

Author response:
We acknowledge the reviewer's observation regarding Table 1's metrics in comparison to HSurf-Net. We understand the importance of this concern and have provided a detailed analysis in the global response section. This analysis explores the potential reasons behind these findings and their broader implications.

Reviewer comment (Theory):
The encoding capacity of the proposed method might be limited.

Author response:
We recognize the concern about HDFE's limited encoding capacity for extensive input spaces, a point we also note in our conclusions. To address this, we propose partitioning the input space into disjoint subspaces and concatenating their respective encodings. While our current focus has been on low-dimensional functions, where capacity is less of an issue, expanding HDFE’s capacity for broader applications presents a significant future research direction.

Reviewer comment (Writing):
It is necessary to provide a clear definition of "implicit representation" and "explicit representation" in the manuscript. Some reviewers may intuitively refer to the fixed-length vector representation of a data sample as "implicit representation" since it may not be human-friendly. However, in this manuscript, the fixed-length vector representation is referred to as the "explicit representation."

Author response:
Thank you for highlighting the need for clearer definitions of "implicit representation" and "explicit representation" in our manuscript. We acknowledge the potential confusion among readers regarding these terms. In response to your suggestion, we have revised the definition of explicit representation in the Introduction's second paragraph: "Explicit representation refers to frameworks that generate outputs with fixed dimensions, such as fixed-length vectors." To enhance clarity, we have also included a brief comparison between explicit and implicit representations, elucidating their distinct roles and applications in our research context. This modification should provide readers with a more comprehensive understanding and eliminate ambiguities related to these key terms.

Reviewer comment (Theory):
The proposed method (HDFE) relies on the assumption that the input data follows Lipschitz continuity. While the reviewer agrees that point cloud data intuitively follows Lipschitz continuity, it would be beneficial for the manuscript to include an analysis of the types of input data that adhere to Lipschitz continuity.

Author response:
Thank you for highlighting this aspect. In response, we have included **Appendix D: Suitable Input Types for HDFE**, detailing various input types that conform to Lipschitz continuity. This addition aims to clarify the characteristics of appropriate inputs, offering both theoretical understanding and practical guidance for researchers applying our method.

Reviewer comment (Reproducibility):
As a module that doesn't require any training, it is important to provide detailed guidance on selecting hyperparameters. This includes guidance on choosing the size of the fixed-length vector representation (denoted as $N$) and determining the hyperparameters $\alpha$ and $\beta$ in Equation 5, which are influenced by the receptive field $\epsilon_0$ and the Lipschitz continuous constant $c$.

Author response:
We concur that providing guidance for hyperparameter selection is crucial for HDFE's usability. We contend, however, that hyperparameter tuning should not pose a significant barrier to users. HDFE's simplicity is characterized by minimal hyperparameter requirements. Specifically, for dimensionality, higher values invariably yield superior representations, obviating the need for intricate tuning. 
The primary hyperparameter, the receptive field, is governed by the $\alpha$ parameter in Equation 6. Typically, $\alpha$ ranges from 10 to 30 for normalized inputs between $(0,1)$, simplifying the selection process. 
Moreover, we plan to release HDFE's source code along with comprehensive documentation, including hyperparameter guidance. We also envision incorporating automated mechanisms for receptive field optimization based on user-specific datasets, further reducing the tuning burden.

Reviewer comment (Experiments):
The selection of weights ($w_i$ in Equation 1), hyperparameters ($\alpha$ and $\beta$), and the mapping functions $E_X$ and $E_Y$ are highly dependent on the dataset. This means that if the task or input data changes, all these variables need to be carefully decided and tested.

Author response:
We appreciate the opportunity to clarify potential misunderstandings:

**Task Change vs. Input Functions**: If the task changes (e.g., from classification to regression) but the input functions remain constant, the hyperparameters need not be altered. Only the downstream neural network requires modification.

**Weights $w_i$**: These are not hyperparameters but are determined through iterative refinement during input function encoding, thereby eliminating the need for user tuning.

**Hyperparameter $\beta$**: This remains constant as long as the function's range is normalized between 0 and 1. For functions normalized within $(0,1)$, we set $\beta$ to 2.5, ensuring the gradient of $\langle E_Y(y_1), E_Y(y_2)\rangle$ stays positive, as detailed in the manuscript.

**Changes in Input Functions**: The reselection of the $\alpha$ value (in Equation 4) is only necessary if there's a significant change in the input functions (e.g., shifting from 1D to 2D functions) or in the Lipschitz constant.

Please note that the hyperparameters $\alpha$, $\beta$, and the mappings $E_X$, $E_Y$ are fixed across both training and testing phases to ensure comparability of function encodings.

Reviewer comment (Evaluation):
There is a small concern regarding the experimental results on the PCPNet dataset. The proposed HDFE method is demonstrated to outperform the PCPNet model (the baseline in 2018) simply by replacing PointNet with HDFE. However, it is only comparable to the current state-of-the-art (SOTA) method, outperforming it in four out of twelve metrics, albeit with a slight drop in average performance. It would be valuable to provide insights or explanations for these observations and discuss any potential limitations or implications of the results.

Author response:
We acknowledge the reviewer's concern regarding the experimental results with the PCPNet dataset. To address this, we have conducted a thorough analysis of the HDFE's performance compared to both the baseline and current state-of-the-art methods. For a comprehensive understanding, we invite the reviewer to refer to our detailed analysis presented in the global response section.

Reviewer comment (Novelty):
In line 7 of page 2, why is the representation learned by PointNet (Qi et al., 2017a) not easily decodable? For instance, in their original paper (https://arxiv.org/pdf/1612.00593.pdf) in Figure 2, it seems possible to set m=3 and obtain normalized point clouds. Additionally, other works like [1] may also be able to 'decode' the input from the vector representation. Is there any difference between this manuscript and those works?

Author response:
Thank you for your insightful question. We claim that PointNet's representations are not easily decodable based on our experiments where PointNet, as a function encoder, struggled to minimize reconstruction loss and failed to yield reasonable reconstructions. Detailed findings are presented in **Appendix B: PointNet as Function Encoder**.

While previous works like [1] may encode implicit functions (e.g., point cloud inputs) effectively, our analysis suggests **they lack the capability for decodable representation of explicit functions**, as evidenced in Appendix B. Additionally, these methods **may not produce encoding invariant to sample distribution changes**. For instance, if trained on uniformly sampled point clouds and tested on varied sampling distributions, their performance could deteriorate, as discussed in Appendix I.4.

We are thankful for this question, which has enhanced the completeness of our manuscript, and we are open to any further inquiries regarding our experiments or arguments.

[1] Learning Representations and Generative Models for 3D Point Clouds. PMLR

Reviewer comment (Reproducibility):
When experimenting with batches, should the model visit all data samples to decide hyperparameters? According to Equation 3, the decoding step should visit all $\bm Y$.

Author response:
**The hyperparameters $\alpha$, $\beta$, and the mappings $E_X$, $E_Y$ are predetermined and remain constant during both training and testing.** The selection process, particularly for $\alpha$, involves identifying hyperparameters that minimize the function reconstruction loss. Once selected, these hyperparameters are fixed, eliminating the need for adjustments across batches. It's important to note that for meaningful comparisons of function encodings, they must be generated using consistent hyperparameters.

Reviewer comment (Theory):
By curious: why a high-dimensional input does not affect the size of the fixed-length vector representation $N$. Could the author provide further explanation, possibly an extension of the paragraph on 'Scale to high-dimensional input'?

Author response:
Our conclusion regarding the negligible impact of high-dimensional input on the size of the fixed-length vector representation is based on empirical findings. Detailed in Appendix I.6, our experiments assess the influence of input dimension and function complexity on reconstruction error. These studies reveal that **while function complexity significantly affects the reconstruction error, the input dimension appears to have a minimal impact**. This leads us to conjecture that HDFE can effectively encode high-dimensional data without incurring information loss due to increased data dimension.

We appreciate the reviewer's interest in this topic, aligning with our discussion in the "Scale to high-dimensional input" section. Our intention is to highlight a promising research avenue and provide a foundational understanding of its viability.

Reviewer comment (Theory):
In Section 2.3, the manuscript only shows the picking of $E_X$ and $E_Y$ when the function output y is scalar. Are there more cases that can be considered?

Author response:
Thank you for prompting further consideration of HDFE's application scope. To address functions with vector outputs, HDFE can be easily adapted by reshaping the $\Psi$ matrix in Equation 6. For cases where function inputs or outputs cannot be directly vectorized using Equation 6, we propose pretraining the input/output mappings $E_X$, $E_Y$ to align with the properties outlined in Section 2.1. This approach would enable HDFE to handle non-vector functions by employing customized mappings, as the principles for designing these mappings are thoroughly discussed in Section 2.1.

Reviewer comment (Evaluation):
Although HDFE is a deterministic function, is there any empirical result available to estimate the information loss from the raw input to the fixed-length vector representation?

Author response:
To address the query about HDFE's information loss, we've included **Appendix I.5: Information Loss of HDFE** in our manuscript. Our empirical analysis reveals a linear relationship between information loss (quantified by reconstruction error) and function complexity, where complexity is measured by the integral of the absolute gradient. Interestingly, we observe that the reconstruction $R^2$ value appears uncorrelated with complexity in cases of sufficiently high input dimension. We trust this analysis will be insightful for the reviewer. We are grateful for this question, which has contributed to the manuscript's comprehensiveness.

Reviewer comment (Evaluation):
One concern is the computational cost of HDFE induced by the iterative refinement process. In order to employ function representation for the downstream tasks, computational costs of HDFE is important. It may hinder application to large-scale tasks.

Author response:
We acknowledge that the need for iterative refinement, which entails additional computational effort, presents a significant challenge for large-scale tasks. However, in fact, **the refinement process can be efficiently executed in a single iteration, yielding a highly accurate estimation of the sample invariant function encoding**.

**Details:** The motivation of the iterative refinement is to balance the weights between dense and sparse samples. Such motivation can also be achieved by another cheaper one-shot refinement. After obtaining the initial function encoding by averaging the sample encoding, we compute the similarity between this initial encoding and all the sample encodings. The similarity can serve as a rough estimation of the sample density at a particular point. Therefore, if we were to balance the weights between dense and sparse samples, we can simply recompute the weights by the inverse estimated density. It turns out that **this one-shot refinement can achieve similarity of > 0.95 when sample distribution varies.** **The computational cost is only 8.0 ms on an NVIDIA Titan-X GPU.**

We kindly refer Reviewer oyrE to **Appendix I.3: Practical Consideration of Iterative Refinement** in the updated manuscript for the detailed procedure of one-shot refinement and experiments demonstrating its effectiveness. We highly appreciate your very constructive question that induces us to complete the manuscript.

Reviewer comment (Experiments):
Overall, it’s convincing that HDFE is a reasonable and general interface for processing continuous objects, supported by the experiments. However, it’s less convincing why we should use HDFE instead of other domain-specific encoding methods. The authors claim that sample invariance is a crucial property for the machine learning tasks throughout the paper, but it lacks the supporting experiment revealing HDFE’s efficacy in those scenarios (i.e., sample distributions are different in training and test dataset). It would make the paper stronger if it presents the experiments with scenarios having disparate sample distributions between training and test datasets and compares the performance of HDFE compared to the baselines.

Author response:
We heartly agree that experiments supporting HDFE's efficacy on sample invariance are important. **We will show HDFE is significantly more robust to distribution variation than PointNet through two experiments**: 1. point cloud normal estimation; 2. a synthetic function parameters regression problem.

1. **Point Cloud Normal Estimation**

   In our updated experiments, **HDFE significantly outperforms the PointNet counterpart when sample variance exists.** Specifically, HDFE achieves 4.79/7.37/0.52/0.46 lower errors than its PointNet counterpart in the setting of Density-Gradient (i.e. the density of the input point cloud varies) in the two baseline comparisons and two benchmark datasets. Through the comparison with the baseline, it is convincing that the significant improvement is attribute to HDFE's sample invariance property and the augmentation of the HDFE encoding.

   We highly appreciate your very constructive feedback and we have added a paragraph "**HDFE promotes stronger robustness to point density variance**." in the **Section 3.2: Unoriented Surface Normal Estimation** to make the analysis and discussion explicit to readers.

2. **Synthetic Function Parameters Regression**.

   We generate random functions parametrized by four coefficients and the task is to predict the four coefficients using the function samples. We artificially control the sample distribution to be varied or not varied across the training and testing phase. It turns out that **PointNet's prediction $R^2$ decreases from 0.978 to 0.513** when switching from no variation to variation, while **HDFE maintains its $R^2$ at > 0.9975** in both settings. We kindly refer the Reviewer oyrE to **Appendix I.4: Effectiveness of Sample Invariance** in the updated manuscript for the experiment details and analysis.

These experiments demonstrate HDFE's superior performance in scenarios with sample variance, thereby validating our claims about its efficacy in machine learning tasks. We highly appreciate your very constructive question that induces us to complete the manuscript.

Reviewer comment (Evaluation):
How long does the HDFE take compared to the baselines (e.g., PointNet in Experiment 3.2)? Is the iterative refinement process applicable to a large number of samples? How long does it take for convergence in the process?

Author response:
While the details are in the response to Weakness 1, we would like to mention that HDFE is much faster than PointNet. For an input of shape $5000\times 3$, assuming the complexity of multiplying an $m\times n$ matrix with an $n\times p$ matrix is $mnp$, then HDFE only has has complexity $5000\times3\times1024\approx1.5e7$. The computational cost of the one-shot refinement is negligible here. But PointNet has complexity $5000\times3\times64+5000\times64\times128+5000\times128\times1024\approx 70e7$.

Reviewer comment (Theory):
In the formulation on decoding (i.e., equation between eq. (2) and eq.(3)), can you please clarify on why orthogonality property ensures that $E_X(x_i) ⊘ E_X(x_0)$ will produce a vector orthogonal to $E_X(x_0)$ when the distance between two samples is large? Also what does the noise mean? Does it mean that it’s near zero so that it is a negligible component?

Author response:
We recognize that the noise argument may not be straight-forward from the orthogonal property. For better rigor, clarification and readability, we replace the orthogonal property as similarity-preserving property, without affecting the correctness of HDFE:

​	similarity preserving: ⟨x ⊗ y, x ⊗ z⟩ = ⟨y, z⟩.

When $d_X(x_0, x_i)$ is large, $\langle E_X(x_0)\otimes E_Y(f(x_0)), E_X(x_i)\otimes E_Y(f(x_i)) \rangle\approx 0$. Since the unbinding operation is similarity preserving, we have $\langle E_Y(f(x_0)), E_X(x_i)\otimes E_Y(f(x_i)) \oslash E_X(x_0)\rangle\approx 0$. Therefore, $\mathbf{F}\oslash E_X(x_0)$ can be decomposed into $E_Y(f(x_0))$ and the sum of vectors that are orthogonal to it. Consequently, when we search for the $y$ to maximize the $\langle \mathbf{F}\oslash E_X(x_0), E_Y(y) \rangle$, those orthogonal vectors will not bias the optimization.

Note that the Fractional Power Encoding does satisfy the similarity preserving property, so the entire HDFE formulation is still correct. We have rewritten the **decoding** section based on the new definition and put the argument in **Appendix E: Derivation of Decoding**. We highly appreciate your constructive question for helping us improve the manuscript.

Reviewer comment (Theory):
In the formulation on decoding (i.e., equation between eq. (2) and eq.(3)), it seems it misses $w_i$.

Author response:
Thanks for your catch! We have corrected the typo.

Reviewer comment (Theory):
For an unbinding operation, element-wise division of complex vectors is used. But I don't think this operation is commutative, which violates the assumption. Can you please clarify on this?

Author response:
This is a very sharp catch! We admit it is an overlook by us when we draft the manuscript. You are correct that the element-wise division of complex vectors is not commutative. Actually the commutative property is not required when deducing the decoding formulation, so the HDFE formulation is still correct. We have revised the definition of our binding and unbinding operations to address this (and Question 2) and ensure the logical consistency of our methodology.

Reviewer comment (Evaluation):
In experiment 3.1, how does the function prediction error is measured? Is it measured in embedding space? And the paper states that “when decoding is not required, our approach achieves lower error than FNO”, but how can we compare to FNO, which directly predicts the solution?

Author response:
The function prediction error is computed by the mean absolute error between the predicted function and the ground-truth function, as mentioned in the Section 3.1 -> Dataset.

We first encode the ground-truth solution into a vector and then decode the vector to reconstruct the solution. From this, we obtain a reconstruction loss $error_1$, which is visualized by the shallow red bars (Reconstruction Error) in Figure 3. Then we decode the predicted function vector to reconstruct the solution. From this, we obtain the error between the predicted solution and the ground-truth solution, denoted by $error_2$, which is visualized by the stack of Reconstruction Error and Function Prediction Error.

Our argument is, $error_2$ consists of two components. The first component arises when reconstructing the function from the encoding, which is measured by $error_1$. The second component arises when predicting the function encoding, which cannot be measured directly. But since $error_2$ only consists of two components, we can estimate the error arised from the second component by $error_2-error_1$, which is visualized as the Function Prediction Error in Figure 3.

Reviewer comment (Evaluation):
While the authors claim that HDFE is robust to point perturbation, the experiments on [PCPNet - PointNet + HDFE] in Table 1 shows that the performance boost becomes much less as the noise level increases. Can you please elaborate on this?

Author response:
We believe it is caused by the trade-off between accuracy and robustness to noises. In Table 1, we present the result obtained by $\alpha=20$, which minimizes the average error. However, this parameter, compared with other selections of $\alpha$, does not obtain lowest error at all noise levels. Kindly refer to Appendix K, where we find a larger receptive field makes HDFE more robust to noise, while a smaller receptive field makes HDFE more accurate in clean inputs. We can increase HDFE's robustness to noise by choosing a large receptive field, but we will lose some accuracy when the input is clean.

Reviewer comment (Writing):
[Possible Typo] In the last sentence in section 2.1, “appendix F.1” should be “appendix E.1”.

Author response:
Thanks for your catch! We have corrected the typo.