Paper: Efficient Recomputation of Marginal Likelihood upon Adding Training Data in Gaussian Processes and Simulator Fusion (ICLR-2024)

Reviewer comment (Experiments):
The experimental section is too weak, as it only considers synthetic datasets. It is not clear at all if the proposed method has a practical utility since no real-world problems are considered in the paper. This questions the significance of the results. The authors should give particular examples of the expected utility of the proposed approach in a real-world setting.

Author response:
In response to the concern regarding the use of real datasets, we have conducted additional experiments with actual data. The methodology and the description of these experiments are detailed in the third paragraph of Section 4.1, and the results are presented in Figure 1(c). Although the differences are not as pronounced as with synthetic data, the results show an improvement in MSE over the standard Gaussian Process. Furthermore, they demonstrate that our approach is not influenced by deviations from the true distribution, which can affect methods like SoD when using simulator data.

Reviewer comment (Novelty):
The paper lacks a solid related work section. It is not clear if this problem has already been studied in the literature and if methods have already been devised for it. In the introduction there are some related methods described; however, they seem to be methods proposed for a different setting that may be adapted to the particular setting considered by the authors.

Author response:
We appreciate the feedback regarding the related work section. Due to space constraints within the main text, we chose to briefly introduce only the most pertinent related works in the Introduction and provided a comprehensive review of the related literature in Appendix A. We acknowledge that the location of this section may not be immediately apparent to readers. To address this, we have now explicitly indicated in multiple relevant parts of the Introduction that a detailed discussion of related works can be found in Appendix A. This should guide the readers to the extended review, ensuring they are aware of the broader context and precedent for our work.

Reviewer comment (Experiments):
The proposed method has a very large computational cost that is cubic with respect to the number of training points or the points to be added to the training set. This is a limitation since only a few thousand points may be considered at most. The authors should try to scale the method to larger experimental settings, considering, for example, approaches for sparse GPs.

Author response:
We acknowledge the reviewer's concerns regarding the computational expense of our method. As depicted in Figure 4, the computational time for our proposed method, even with 7000 training points, is under three hours. It's important to clarify that this computation is not recurrent for every prediction, but is a one-time cost incurred during the data selection phase from the simulator. Given this context, we consider the computational demand to be practical for the intended applications.

Reviewer comment (Novelty):
The use of Cholesky factors that are updated efficiently is not new within the GP literature.

Author response:
The efficient update of Cholesky factors is indeed a known technique within GP literature, as (\cite{osborne2010bayesian}) have previously demonstrated the computation of $\mathbf{L}\_{m+1}$ from $\mathbf{L}\_{m}$ in $O(m^2)$ time. Our method, however, introduces an efficient computation for deriving $\mathbf{V}\_{m+1}$ from $\mathbf{V}\_{m}$ with a complexity of $O(mN+N^2)$, where $\boldsymbol{\Sigma}\_{m+1}=\mathbf{V}\_{m+1}\mathbf{V}\_{m+1}^\mathsf{T}$. To communicate this effectively, we have detailed this contribution in footnote 5. We believe our computational technique, which is elaborated in Section 3.2, is not trivial.

Reviewer comment (Experiments):
Why do not each method in Fig. 1 start from the same initial value?

Author response:
The horizontal axis in Figure 1 represents the number of true training data points, not the generated data. The data point at the left end of Figure 1 corresponds to the MSE when there are 50 true training data points. Since each method selects a different number and subset of generated data to use, the MSEs vary accordingly. This explains the difference in starting points across the methods depicted in the figure.

Reviewer comment (Experiments):
Could your approach be extended to take advantage of sparse GPs approaches to scale to large datasets?

Author response:
Yes, our approach can indeed be extended to leverage sparse Gaussian processes for scalability to larger datasets. The most straightforward adaptation would be to first select the data from the simulator to be added to the training set using our proposed method, and then treat the combined dataset from the selected data and the original training data indistinctly when applying sparse GPs approaches. There is also potential for a more sophisticated integration of our method with sparse GPs, which could further enhance efficiency and scalability.

Reviewer comment (Novelty):
The idea of accepting data to be added as part of the training set by improving the marginal likelihood was previously explored by Titsias M. in "Variational Learning of Inducing Variables in Sparse Gaussian Processes" section 3.1. In the context of Titsias' work was used to generated pseudo-inputs (or inducing points).

Author response:
I had not noticed that in the paper. Thank you for pointing it out.
While it is true that (Titsias2009a) utilizes the lower bound of the marginal likelihood as a metric, our method differs in that it directly employs the marginal likelihood itself. (Titsias2009a) aimed to develop sparse GPs for reducing training data by replacing the dependence of function values $\mathbf{f}$ on the training data $\mathbf{X}$ with $p(\mathbf{f}|\mathbf{f}^m)$, thus altering the model's marginal likelihood from Equation 4 in (Titsias2009b) to Equation 10 in the same work. Consequently, the marginal likelihood could not be quickly computed, leading to the use of its lower bound as a metric (as shown in Equation 8 in (Titsias2009a) or Equation 13 in (Titsias2009b)). In contrast, our proposed application allows for the use of all training data, thereby eliminating the need to modify the marginal likelihood. Our acceleration method enables the use of the log likelihood itself as the metric. We have added (Titsias2009a) and other papers employing log marginal likelihood to the related works in Appendix A.2 and clarified these differences as described above.

Reviewer comment (Writing):
The introduction does not properly motives the research problem to engage the reader with the work. Also, the introduction lacks of references to better support different phrases or claims. The methodology sections are generally well written and not difficult to follow, though they present some inconsistencies in the mathematical notation.

Author response:
Thank you for your detailed feedback regarding the Abstract, Introduction, and Methodology sections as outlined in the Questions section. We have taken your comments into careful consideration and made appropriate revisions to address these concerns.

Reviewer comment (Writing):
In Abstract, it sounds contradictory to say that we rely on knowledge that is unreliable: "construct models based on their knowledge of the modeling target and, as training data increases, choose more flexible models with reduced dependence on that knowledge if that knowledge is unreliable." Maybe the last sentence would be better understood if read as:"...if that knowledge becomes unreliable" or simply get rid of last part and leave: "construct models based on their knowledge of the modeling target and, as training data increases, choose more flexible models with reduced dependence on that knowledge."

Author response:
Thank you for your insightful feedback and specific suggestions regarding the phrasing in the Abstract. You are correct in pointing out that the term "unreliable" may not be the most appropriate, as the context is not limited to situations where the knowledge is incorrect. We have taken your advice and revised the sentence to the second option you proposed: "construct models based on their knowledge of the modeling target and, as training data increases, choose more flexible models with reduced dependence on that knowledge.

Reviewer comment (Writing):
In Abstract, it is not clear what it is the intention of "We propose a faster method considering the Cholesky factor and matrix element dependencies." There is something missing to properly connect with all the previous text.

Author response:
We appreciate the feedback on the clarity of our abstract. To clarify the motivation for developing a faster method, we have revised the text as follows: "On the other hand, the log marginal likelihood is a theoretically grounded metric when viewed as a model selection criterion for incorporating data generated from a simulator into the training data. Calculating this metric for GPs is computationally expensive due to the necessity of inverting large covariance matrices. To address this challenge, our proposed method accelerates the process by efficiently updating the Cholesky decomposition and considering the dependencies between matrix elements. This improvement not only expedites computations but also maintains the theoretical robustness of the model selection process."

Reviewer comment (Writing):
In the Introduction, there is probably a sentence missing at the very beginning regarding modelling issues or modelling challenges than allows the reader understand where the idea or problem of bias-variance trade-off comes from. Also, it is necessary to include a strong reference regarding "bias-variance trade-off" to support the text.

Author response:
Thank you for the constructive feedback. Indeed, the initial motivation for considering the bias-variance trade-off was not clearly stated. To rectify this, we have added an introductory sentence to the Introduction that outlines the general aim of regression models, smoothly transitioning to the discussion of the bias-variance trade-off. The revised text is as follows:

"One of the ultimate objectives of machine learning models is to reduce generalization loss. According to the bias-variance trade-off, the generalization loss, when employing Mean Squared Error (MSE) as the loss function, can be decomposed into terms of bias and variance."
		
In addition, we have simplified the explanation of the bias-variance trade-off and have incorporated a strong reference to support the concept by adding PRML (Bishop2006), which discusses the same in the context of Equation 3.44.
		
(Bishop2006) Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. Springer, 2006.

Reviewer comment (Writing):
In the introduction, the phrase that reads: "On the other hand, the method of selectively adding generated simulator data to the training data only requires that data can be generated from the simulator" seems ambiguous or needs rewording.

Author response:
Indeed, the sentence in question could have been misinterpreted due to its redundant structure. We have revised it for clarity and conciseness as follows:
"On the other hand, the method of selectively adding generated data requires only the ability to produce data from the simulator."

Reviewer comment (Writing):
Please include references to support: "The criterion for selecting important data is the diversity of the training data. Various methods to measure this diversity have been proposed."

Author response:
We have included relevant references of "the various methods".

Reviewer comment (Theory):
Please include references to support: "The negative log marginal likelihood is a metric that measures the model’s ﬁt to the training data and has a theoretical foundation that it matches, on average, the KL divergence between the true distribution and the model’s distribution."

Author response:
We have included references that detail the relationship between the negative log marginal likelihood and the KL divergence. Additionally, we offer a thorough explanation in Appendix G, H.

Reviewer comment (Writing):
In Eq. (1), the Covariance matrices $\mathbf{K}_{N,m^*}$ and $\mathbf{K}^{\top}_{N,m^*}$ might be swapped of quadrant. It is more intuitive to think that the pair $N,m^*$ refers to rows,columns respectively. Add period "." at the end of the equation. Putting the $\mathbf{x}_1...\mathbf{x}_N$ and $\mathbf{x}_{1^*}$ ... $\mathbf{x}_{m^*}$ inside the equation looks strange as if a vector were multiplying the covariance matrix. Maybe a footnote should be added to avoid confusion.

Author response:
The following content has been added to Footnote 3 as indicated. Thank you for bringing this to our attention. $\mathbf{x}\_{1^*}\dots\mathbf{x}\_{m^*}, \mathbf{x}\_{1}\dots\mathbf{x}\_{N}$ represent the input variables for the kernel function that constructs the covariance matrix.

Reviewer comment (Theory):
In Eq (3) the negative sign is not applied, previously it was introduced $F_{m+1^*}=-\log \mathcal{N}...$. Also, as per Eq. (4) the operation in Eq. (3) should be $(\mathbf{y}^{m+1}-\boldsymbol{\mu}_{m+1})$ instead of $(\mathbf{y}^N-\boldsymbol{\mu}_{m+1})$.

Author response:
Indeed, the notation $\mathbf{y}^N$ as presented is accurate. To clarify the distribution aspect of $F\_{m+1^*}$ with respect to $\mathbf{y}^N$, we have now explicitly stated in the preceding sentence that $F_{m+1^*} = -\log p(\mathbf{y}^N|\ldots)$. This amendment should eliminate any confusion regarding the distribution of $\mathbf{y}^N$ in relation to $F_{m+1^*}$.

Reviewer comment (Writing):
In section 3: write $(m+1)\times(m+1)$ instead of $m+1 \times m+1$. Indeed, in the equations should be better to write, say, $\mathbf{y}_{(m+1)^*}$ or $\mathbf{K}_{(m+1)^*}$.

Author response:
Acknowledging your point, we have decided not to add parentheses around $\mathbf{K}_{N, (m+1)^*}$ to avoid cluttering the notation. The current format should be clear enough to avoid any misunderstanding.

Reviewer comment (Presentation):
In the figure 3: it is not possible to visualise the Training data (brown-ish colour) for SoD.

Author response:
Indeed, due to the order in which the data points were plotted, the Training data (in a brown-ish color) for SoD was obscured in Figure 3. We have corrected this issue and adjusted the plotting order to ensure that all points are now clearly visible in the figure. Thank you for pointing out this oversight.

Reviewer comment (Writing):
In the conclusion: "the algorithm we proposed is specialized for regression models", not regression models in general, but a regression model particularly with a Gaussian likelihood.

Author response:
Indeed, our algorithm is tailored for regression models where the observation noise is Gaussian. To make this clear, we have explicitly stated this limitation in the conclusion as follows:		
"As a limitation, the algorithm we proposed is specialized for regression models with Gaussian likelihood, and its extension to classification models and other likelihood models is not straightforward."

Reviewer comment (Theory):
Why is there a distribution $q(\mathbf{X}^N)$ in appendix H for Eq. (21)? Aren't we saying in $KL(q(.|\mathbf{X}^N)||p(.|\mathbf{X}^N))$ that $\mathbf{X}^N$ is given? I do not think the Eq. (21) is correct.

Author response:
We require $q(\mathbf{X}^N)$ to ensure that the distance between our model $p(\mathbf{y}^N|\mathbf{X}^N, \mathbf{X}^{m*}, \mathbf{y}^{m*})$ and the true distribution $q(\mathbf{y}^N|\mathbf{X}^N)$ is minimized on average over the true distribution $q(\mathbf{X}^N)$, rather than for a specific training dataset $\mathbf{X}^N$. This definition aligns with the conditional KL divergence as defined in (Poczos2012) Definition 5, which includes $q(\mathbf{X}^N)$. To avoid confusion, we have made the following modifications:

- Cited (Poczos2012) right after mentioning "Conditional KL divergence".
- Revised the description before Equation 20 from $q(\mathbf{y}^N|\mathbf{X}^N)$ to $q(\mathbf{y}^N|\mathbf{X}^N)q(\mathbf{X}^N)$ to reflect the true distribution of the dataset.
- Adjusted Equations 20 and 21 to align with (Poczos2012)’s definition by bringing the integral over $\mathbf{X}^N$ to the outermost layer.

(Poczos2012) Barnabas Poczos and Jeff Schneider. Nonparametric estimation of conditional information and divergences. In Artificial Intelligence and Statistics, 2012.

Reviewer comment (Experiments):
We fit the GP hyperparameters with the training data, but are those hyperparameters tuned again when adding simulator data?

Author response:
The hyperparameters of the Gaussian Process are determined from the initial training data and are not modified post-learning.
"The hyperparameters of the input kernel function are pre-optimized using the initial training data $(\mathbf{X}^N, \mathbf{y}^N)$ and remain fixed throughout the execution of Algorithm 1. While re-optimization of the kernel's hyperparameters using the selected generated data $(\mathbf{X}^{M^*}, \mathbf{y}^{M^*})$ in conjunction with the original training data $(\mathbf{X}^N, \mathbf{y}^N)$ could be considered after the selection process, this was not performed in the current study."
Further, while an iterative approach to hyperparameter optimization in conjunction with data selection, as suggested by (titsias2009), could be contemplated, such an exploration is reserved for future work.

Reviewer comment (Experiments):
The experiments shown seem to have an appropriate number of N data observation so that the GP model fits quite well for the range of input data $\mathbf{x}$, so due to the conditioning properties of a Normal distribution it is expected to only accept data that could improve the conditional distribution $p(\mathbf{y}^N|\mathbf{X}^N,\mathbf{y}^{m^*},\mathbf{X}^{m^*})$. What would it happen if the GP has a smaller number of data observation, or lack of data in regions such that the predictive distribution was less uncertain? How would the acceptance and rejection would behave in such a region?

Author response:
In regions where the true training data is sparse, adding generated data to the training set is likely to have a minimal impact on the predictions for true training data points that are distant from the added data, due to the properties of kernel methods. Consequently, the value of $F_{m+1}$is expected to change very little, whether it increases or decreases, compared to $F_m$. Therefore, the inclusion of such generated data in these sparse areas would likely be random. We have not conducted experiments specifically to support this hypothesis.

Reviewer comment (Evaluation):
It seems that the Log marginal likelihood metric gives priority to the model fitting, so when do we trust the simulator?

Author response:
When it comes to prioritizing data over domain knowledge, our method indeed favors the log marginal likelihood, or the data. As mentioned in the Introduction, model assumptions can be adjusted automatically from data, which inherently places priority on the data. Trust in the simulator would require manual intervention to incorporate domain knowledge into the model assumptions. For instance, one might manually add all data from a trusted simulator region to the training set.

Reviewer comment (Evaluation):
What ways to measure a trade-off, as mentioned in the introduction, to achieve an appropriate bias-variance in our last model that contains training and simulator data?

Author response:
Bias-variance trade-off cannot be directly measured because we do not have knowledge of the true underlying distribution. Instead, we rely on indirect metrics. One such metric is the log marginal likelihood, which assesses the model's fit to the data and the balance of complexity (as discussed in Bishop, 2006, Section 3.4). Optimizing the log marginal likelihood indirectly contributes to achieving a favorable balance between bias and variance. We have added the motivation for using log marginal likelihood to the 6th paragraph of the Introduction to provide a clearer explanation.

Reviewer comment (Experiments):
If I fit the GP and generate data from such a GP and use it as simulator data, wouldn't I expect to achieve improvements in the Log marginal likelihood? Wouldn't the fitted GP be simply the best data simulator?

Author response:
This is an intriguing consideration. The improvement of log marginal likelihood using data generated from a GP fitted to a limited dataset is not a given. For outlier values of $\mathbf{x}$ that are less likely to be sampled from the true distribution $q(y∣\mathbf{x})q(\mathbf{x})$, whether the generated data pairs $(\mathbf{x},y)$ from the GP's predictive distribution will enhance the log marginal likelihood is uncertain.

Reviewer comment (Experiments):
The work is missing to show a real world application to additionally assess the performance of the approach, for instance an example as claimed in appendix C.

Author response:
In response to the concern regarding the use of real datasets, we have conducted additional experiments with actual data. The methodology and the description of these experiments are detailed in the third paragraph of Section 4.1, and the results are presented in Figure 1(c). Although the differences are not as pronounced as with synthetic data, the results show an improvement in MSE over the standard Gaussian Process. Furthermore, they demonstrate that our approach is not influenced by deviations from the true distribution, which can affect methods like SoD when using simulator data.

Reviewer comment (Experiments):
What would be the effect of using different data simulators? For instance, a simulator less similar to the real distribution.

Author response:
We have investigated the effect of using simulators that diverge from the true distribution in Section 4.2, as depicted in Figure 2. The parameter $a$ increases the dissimilarity between the simulator's distribution and the true distribution. Our findings indicate that even as the simulator becomes less similar to the real distribution, the MSE of our proposed method remains relatively stable.

Reviewer comment (Reproducibility):
For the practitioner, how is a data simulator generally built or where does it come from?

Author response:
Practitioners generally build data simulators by translating the knowledge they currently possess about the relationship between y and x into a model. Our method is applicable as long as the simulator can generate pairs of $(y,x)$, regardless of whether the simulator describes the $(y,x)$ relationship analytically or generates $(y,x)$ hrough numerical computation.

Reviewer comment (Theory):
What if the dataset we are fitting presents a heteroscedastic noise, how could this affect the method approach for accepting training data candidates?

Author response:
Process model assumes homoscedastic noise, so employing it on data with heteroscedastic noise implies using an incorrect model. In areas where the heteroscedastic noise variance is less than that assumed by our GP model, simulator data with variability within this range may not be sufficiently penalized in terms of the negative log marginal likelihood, potentially leading to incorrect acceptance. Conversely, in areas where the heteroscedastic noise variance is greater, our method might overly penalize simulator-generated data, resulting in excessive rejection.

Reviewer comment (Novelty):
The novelty of GP on this topic is a bit limited. GP is not a new method at all. The algorithm that makes it faster is more interesting but no major breakthrough.

Author response:
As discussed in the third paragraph of the Introduction, Gaussian Processes (GPs) offer a unique advantage in incorporating the evolution of deep learning by allowing the kernel function to be represented as a neural network architecture. This adaptability ensures that GPs remains a relevant and powerful tool in the field. Moreover, the ability of GP to quantify predictive uncertainty is a significant contribution that enhances the capabilities of deep learning models. Hence, we contend that GP continues to be of substantial importance and relevance in our domain.

Reviewer comment (Presentation):
Why gray points are not adopted in Figure 3?

Author response:
Thank you for bringing this to our attention. The omission of the gray points from the legend in Figure 3 was an oversight. We have now updated the figure to include these points in the legend.