Paper: Reduced-Rank Online Gaussian Process Modeling With Uncertain Inputs (ICLR-2024)

Reviewer comment (Novelty):
It is unclear how the Taylor approximation in the paper differs from the one considered by Bijl et al. (2017).

Author response:
Our method is different from the one used by Bijl et al. in three different ways: 
- we do not re-estimate the distribution of the input noise, reducing the number of steps needed for the algorithm execution
- using the Taylor approximation, our development is slightly different as one of the second order terms (the third term of Equation (16)) seems to have been omitted in (17) in Bijl et al. 
- we formulated all the update equations for prediction using a reduced-rank approximation instead of a sparse method, in order to address the lack of robustness of SONIG (even when correcting the omission discussed in the previous point). The reduced-rank approximation is motivated by the fact that it enables to use a kernel of smooth eigenfunctions, with the aim of mitigating the divergence issues.

Reviewer comment (Theory):
There is no explanation of how the update equations (20)–(23) arise, and it is unclear whether they are just based on the standard sequential view of GP inference or if there is a novel derivation or motivation.

Author response:
The equations (18)-(22) are based on the updates of the data in online GP.  They follow the steps used in the book "Bayesian Filtering and Smoothing" [Sarkka2013] to transform batch updates to a sequential algorithm, adapted to our own equations.

Reviewer comment (Experiments):
It is unclear whether the baselines were implemented in C++ like the authors’ approach or in Matlab, which could lead to significant runtime differences.

Author response:
Our method is implemented in both C++ and Matlab. We will further emphasize in Section 5.1 that the comparison in terms of performance and execution time were made in Matlab using an Intel Core i7-7820HQ CPU for all the algorithms for a fair comparison. Our method is also coded in C++, which was used in Section 5.2, as it is more efficient and, thus, is the one we put in the supplementary material and we plan to make public. If the reviewer considers of interest that we also put the Matlab version, we will be very happy to.

Reviewer comment (Experiments):
For the simulated data, there is a lack of comparison across different kernels, particularly because the chosen exponential kernel benefits low-rank approximation due to its fast decaying spectrum.

Author response:
As you pointed out, it is indeed known that low rank approximation is more efficient for Gaussian kernel than for example Matern kernels, as the latter requires larger rank to efficiently approximate the inference of a given kernel [Daskalakis2022]. Our method is done for a RBF kernel only. However, we don’t consider it necessarily as a drawback as it is the most commonly used kernel. Moreover, we consider the study of other kernels to be out of the scope of our paper, which addresses the problem of absence of a method that combines efficiency, speed and robustness.

Reviewer comment (Evaluation):
In Table 1, the performance differences are within one standard deviation, making it difficult to determine whether they are systematic or artefacts.

Author response:
In Table 1, the large standard deviations are due to the fact that there can be a high variability of the quality of the GP predictions depending on the functions and the learning points (the input and the output noise are random and we learn with 200 points only, which create a high variability between the runs). As suggested by another reviewer, we added the NLPD values in order to prove that our algorithm performs better in general.

Reviewer comment (Experiments):
There is no systematic variation of hyperparameters in the simulated data experiments, and it is unclear how other hyperparameters were selected.

Author response:
Please see our response number 2 to the second reviewer for the choice of kernel hyperparameters.

Reviewer comment (Experiments):
In the magnetic map creation experiment, the paper does not compare RRONIG to another noisy-input method (such as SONIG), which is not understandable given the main contribution is a reduced-rank approach for noisy inputs.

Author response:
The other experiment, the Magnetic Map Creation, is a challenging problem and was done with real data. It does not consist only in learning a GP model, the GP algorithm here is in constant interaction with other parts of the SLAM algorithm. Adapting existing GP methods is non-trivial and no method with input noise has been proposed for this task to the best of our knowledge. The purpose of the experiment is two-folds: the relevance of taking into account the input noises and the simplicity and adaptability of our method that allows a successful implementation for this problem.

Reviewer comment (Experiments):
The paper lacks systematic ablation experiments, such as testing with Matern kernels of increasing smoothness to judge low-rank approximation effectiveness and required rank.

Author response:
We think that choosing different kernels could be an interesting perspective to broaden our understanding of the behaviour of reduced-rank GPs but we consider that it would be more appropriate in another paper as our focus was on RBF kernels.

Reviewer comment (Experiments):
No experiment evaluates the impact of rank choice or provides a strategy for selecting the rank.

Author response:
The number of eigenfunctions chosen depends on the lengthscale of the function variations and the width of the studied interval. We tried to find the best compromise between a low number of functions and enough accuracy in the predictions. For further details, [Daskalakis2022] gives theoretical bounds for the rank estimation for an efficient approximation of Gaussian kernels.

Reviewer comment (Experiments):
There is no ablation study on model mismatch to assess RRONIG performance under slightly misspecified hyperparameters, despite the absence of a model selection strategy.

Author response:
Under slightly misspecified hyperparameters, RRONIG (and the other models as well) do not show difficulties in convergence, apart from the ones already mentioned for SONIG. The hyperparameters used for learning are the ones returned by the hyperparameter optimization of NIGP and not the ground truth hyperparameters so we can consider them as slightly mismatched. Moreover, one of the secondary purposes of our real case experiment is to show the capabilities of our method with approximate hyperparameters as they are not updated for each experiment, while the settings between the three of them are different (not the same places are explored and the paths are different).

Reviewer comment (Writing):
The quality of presentation suffers due to atypical language usage, making the paper hard to understand and causing potential confusion with existing mathematical terminology.

Author response:
We took into account your grammatical remarks in the paper and thank you for them.

Reviewer comment (Writing):
Specific language issues include: using "modelize" instead of "model"; unclear distinction between "measure" as measurement versus probability measure; ambiguous expression "measure is ignored" which may mean "measurement is ignored"; using "truth function" instead of "true / latent function"; and using "conditionned to" instead of "conditioned".

Author response:
We took into account your grammatical remarks in the paper and thank you for them.

Reviewer comment (Theory):
There is no explanation or derivation for update equations (20)–(22).

Author response:
The equations (18)-(22) are based on the updates of the data in online GP.  They follow the steps used in the book "Bayesian Filtering and Smoothing" [Sarkka2013] to transform batch updates to a sequential algorithm, adapted to our own equations.

Reviewer comment (Reproducibility):
It is unclear how the number of eigenfunctions for the low-rank approximation is chosen, whether manually or automatically.

Author response:
The number of eigenfunctions chosen depends on the lengthscale of the function variations and the width of the studied interval. We tried to find the best compromise between a low number of functions and enough accuracy in the predictions. For further details, [Daskalakis2022] gives theoretical bounds for the rank estimation for an efficient approximation of Gaussian kernels.

Reviewer comment (Novelty):
The meaning of "more complete version of the Taylor approximation" compared to Bijl et al. (2017) is unclear, and exact differences to that approach are not provided.

Author response:
Our method is different from the one used by Bijl et al. in three different ways: 
- we do not re-estimate the distribution of the input noise, reducing the number of steps needed for the algorithm execution
- using the Taylor approximation, our development is slightly different as one of the second order terms (the third term of Equation (16)) seems to have been omitted in (17) in Bijl et al. 
- we formulated all the update equations for prediction using a reduced-rank approximation instead of a sparse method, in order to address the lack of robustness of SONIG (even when correcting the omission discussed in the previous point). The reduced-rank approximation is motivated by the fact that it enables to use a kernel of smooth eigenfunctions, with the aim of mitigating the divergence issues.

Reviewer comment (Experiments):
On the magnetic map creation dataset, the authors only compare their approach to methods not designed for noisy inputs (such as SONIG), which raises questions about the fairness and completeness of the comparison.

Author response:
The other experiment, the Magnetic Map Creation, is a challenging problem and was done with real data. It does not consist only in learning a GP model, the GP algorithm here is in constant interaction with other parts of the SLAM algorithm. Adapting existing GP methods is non-trivial and no method with input noise has been proposed for this task to the best of our knowledge. The purpose of the experiment is two-folds: the relevance of taking into account the input noises and the simplicity and adaptability of our method that allows a successful implementation for this problem.

Reviewer comment (Novelty):
In my opinion, the literature review is not complete. There are many approaches that deal with online regression and sparse approximations (see [1,2]).

Author response:
While our related work focused on the incorporation of the input noises, the references you mentioned are indeed relevant and enrich our related work section, we added them to the literature of our paper.

Reviewer comment (Novelty):
The paper seems to combine different known facts about uncertain input GPs. Second-order Taylor approximation and smooth basis functions are two tools that have been used already in this context. In other words, I have the feeling that the paper considers the approach of [3] with second-order approximation and usual basis function approximations. Am I right? I am not saying that the contribution is low because of this. However, I think this should be somehow emphasized.

Author response:
We agree with the fact that the second-order Taylor approximation has already been used in previous GP methods, although with minor omissions of some terms in the development. There are several reduced-rank GP methods in the literature but they do not address the problem of uncertain inputs. We needed to work on creating our own algorithm, basing ourselves on the already existing tools in the literature that we considered of interest. We reformulated our contribution as follows in order to make it clearer:
In numerous applications, real-time estimation of a model are needed (for example patients’ vital signs or telemetry data, as well as localization). Therefore a GP model that can be used online, with updates of a short computing time for each new measurement, and with a strong reliability (i.e. with a low risk of divergence) is required. The previously existing methods including uncertain inputs did not match these requirements and we created a new GP algorithm for uncertain inputs that includes ideas from a special category of GP modeling, never studied in this context: a method that uses smooth eigenfunctions in order to avoid divergences that could rise. We confirm the relevance of this approach by a comparison on simulated data between our algorithm and other noisy inputs GPs. It showed a better robustness and speed compared to the other existent online algorithm while also having a better accuracy. Moreover, to assert the practical use of this algorithm, we also adapted and evaluated it on a challenging real-life problem, a very active field in robotics: online localization in indoor environments, and obtained better results than the state of the art [Coulin2022].

Reviewer comment (Theory):
One of the main weaknesses of the paper is the explanation around equations (18)-(22). This is not straightforward to go from equations in Section 4.1 to Section 4.2. This would have deserved more explanations. Could the authors provide some more in-depth details about equations (18)-(22) and their links with equations from Section 4.1?

Author response:
The equations (18)-(22) are based on the updates of the data in online GP.  They follow the steps used in the book 
[Sarkka2013] to transform batch updates to a sequential algorithm, adapted to our own equations.

Reviewer comment (Experiments):
It would have been interesting to see some results on different simulated benchmarks datasets (heteroscedastic noise variance for instance).

Author response:
As you mentioned, it would be interesting to try the algorithm on simulated datasets with heteroscedastic noise variance. We added the results in the supplementary material. We created a simulated dataset similar to the previous ones, but this time the input noise is regularly increasing: we decided to define its standard deviation as 0.4*n/N, with n the index of each new datapoint incorporated and N the total number of datapoints. For SONIG and RRONIG, for each new update, we gave to the model the value of its corresponding input noise standard deviation. Unfortunately, it was more difficult to modify NIGP to include heteroscedastic noise, it uses a constant value of $\sigma\_x$, tuned by its hyperparameter optimization step. Our method performs better than the others (see Table 4). Please note also that on real data, the input noise variance was varying as the localization error estimated from the SLAM algorithm varies over time. RRONIG showed robustness and a strong ability to improve the position's estimation on this dataset as well.

Reviewer comment (Reproducibility):
Is the code publicly available?

Author response:
Yes, the code for our algorithm is available as supplementary material and will be public.

Reviewer comment (Experiments):
The main weakness of the work is in the experimental section. For the artificial dataset, the proposed RRONIG is compared only against one online method (SONIG, from 2017) and one batch method (NIGP, from 2011). For the real data, only a single model is used for comparison (besides visual-inertial odometry). The experimental results would be more complete with the inclusion of more recent GP methods for uncertain inputs or other online GP methods.

Author response:
$\textbf{1, Remarks on the experimental section.}$
Concerning the methods used for the comparison in the artificial dataset, we did not compare with more recent GP methods with uncertain inputs as there are unfortunately no other more recent methods that are able to incorporate them. To the best of our knowledge SONIG is the state of the art in this context.

Adapting existent GP methods to the use-case of SLAM is non-trivial and must respect certain constraints such as low computational complexity. We wanted to insert the SONIG implementation in this SLAM algorithm for comparison. However, as there are many steps of estimation and numerous matrices to be computed, contrary to RRONIG, we finally decided that the task would be too complicated. We consider the successful implementation of our method with better performance that the state of the art as strong evidence for the simplicity, the adaptability and the capabilities of our algorithm.

Reviewer comment (Experiments):
In Section 5.1, the authors did not optimize the kernel hyperparameters in the RRONIG runs. It is unclear whether this is also true for the experiments in Section 5.2. If so, it seems to be a significant drawback in the overall evaluation, since kernel hyperparameter optimization is an important aspect of GP training.

Author response:
$\textbf{2, Remarks on the kernel hyperparameters.}$
In Table 1, we chose to set the same kernel hyperparameters for the three methods for the purpose of a fair comparison, as done in [1], using the hyperparameters tuned by NIGP. If each model had its own tuning of hyperparameters, it would have been difficult to know if one model performed better only because of better hyperparameters or not. In this paper, it is the quality of the updating step of the different algorithms that we wanted to compare. The hyperparameters' optimization is a complex problem that we consider as outside of the scope of this paper: we place ourselves in the context of online GP learning, studying algorithms able to learn a model even with a few datapoints available at the beginning and improving iteratively. However the hyperparameters can only be deduced when there is a sufficiently high number of datapoints available. The optimization phase of the hyperparameters is a preliminary phase before online learning. This step is also not always required as the choice of hyperparameters in general can be based on expert knowledge (as performed in our SLAM real life application). One alternative if we do not have an idea of the hyperparameters' value would be to perform a fine tuning of the hyperparameters based on a subset of the entries before launching the online algorithm. We updated our paper with more precisions about this.

Note that in our real-life experiment in Section 5.2, we chose the kernel hyperparameters based on expert knowledge. The chosen hyperparameters proved to be very robust when varying the settings. They were the same in all the experiments presented that were conducted in different environments.

Reviewer comment (Writing):
One should always specify when using the term "robust". It should be clarified whether the proposal is robust to model misspecification, to outliers, or to numerical instabilities. From the experimental section, it seemed that the robustness is related to better numerical stability, but this should be made clear from the beginning of the paper.

Author response:
1) Indeed, the lack of robustness originates from numerical instabilities, we added the precision in the beginning of the paper.

Reviewer comment (Presentation):
In Section 2, the title "Modifying GP equations" is too generic. Almost all contributions in the GP literature require changing standard GP equations. A more specific title, such as "Input noise modeling", may be more appropriate.

Author response:
2) We agree with the fact that the suggested title is more explicit, we updated the paper to modify it.

Reviewer comment (Evaluation):
Table 1 should include the log predictive density to better evaluate the models' predicted uncertainties.

Author response:
3) We initially only put the predicted variance and the ratio as we wanted to follow the same evaluation metrics used in [Bijl2017]. However, we agree that the log predictive density metric is relevant and added the results in Table 1.

Reviewer comment (Evaluation):
Instead of reporting the average of the predicted variances and the "mean ratio", it would be more direct to simply report the mean log predictive density, which already balances the quality of the predicted means and variances.

Author response:
3) We initially only put the predicted variance and the ratio as we wanted to follow the same evaluation metrics used in [Bijl2017]. However, we agree that the log predictive density metric is relevant and added the results in Table 1.

Reviewer comment (Theory):
Page 7 states: "The variance represents the amplitude of the errors of the model that should be expected: it should be equal to the mean error." Since the variance is in squared units, is the reported error also squared?

Author response:
4) Thank you for the remark, we added the precision that it is the MSE, mean squared error, that is calculated.

Reviewer comment (Experiments):
Regarding the "reduced-rank GP" competitor considered in the SLAM experiment, is it the one from Solin et al. (2015)? In the Conclusion, it is claimed that there is "an improvement over the state of the art on this context". Was the proposal by Solin et al. (2015) considered the state of the art so far for SLAM problems? More recent strategies (even without GPs) should be considered.

Author response:
5) [Coulin2022] is the state of the art for magnetic-based localization. We improved over it and in Section 5.2, our algorithm is compared to executions of [Coulin2022]. In Table 3, we added the reference of the algorithm used on the lines that are concerned. [Solin2015] uses similar equations for magnetic mapping but its SLAM is only magneto-inertial and is not as precise as the one presented in [Coulin2022].

Reviewer comment (Experiments):
The simulated data does not seem to demonstrate online operation.

Author response:
Concerning the online operation in the simulated data, in our experiments SONIG and RRONIG learn the data one by one, as would be the case for an online execution. It means that the measurements are provided to the algorithm sequentially and it is updated at each new measurement, demonstrating online operation. We also provided indications about the computing time in Table 2 for both online methods, with their learning and prediction time.

Reviewer comment (Presentation):
The algorithm should be presented in a way to clearly indicate the steps performed.

Author response:
The algorithm used (RRONIG) is presented in Section 4.2. Thanks to your remark we restructured it in order to make a clearer separation between the different steps (initialization, update and prediction).

Reviewer comment (Writing):
In section 3.2 it is not clear what LE stands for in $B_{LE}$.

Author response:
We added an explanation of the acronym for $B\_{LE\}$. We also decided that it would be clearer to redefine its name as $B\_{IE\}$ as this refers to the random variable associated to the field B with input errors (IE).

Reviewer comment (Experiments):
What levels of input uncertainty are reasonable in practice? $\sigma_x =0.3$ appears high for an input domain $[-5,5]$.

Author response:
$\sigma_{x} = 0.3$ is not such a high value for the interval considered: such input uncertainties of the same amplitude can regularly be observed in SLAM applications when the sensor's accuracy is low and/or when the SLAM runs for a few minutes. Indeed its position error tends to increase over time if the system does not return to previously explored areas.