Original: GAR: Generalized Autoregression for Multi-Fidelity Fusion
Revised: GAR: Generalized Autoregression for Multi-Fidelity Fusion

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In many scientiﬁc research and engineering applications, where repeated simulations of complex systems are conducted, a surrogate is commonly adopted toquickly estimate the whole system.
Revised: In many scientiﬁc research and engineering applications, where repeated simulations of complex systems are conducted, a surrogate is commonly adopted toquickly estimate the whole system.

Original: To reduce the expensive cost of generatingtraining examples, it has become a promising approach to combine the results oflow-ﬁdelity (fast but inaccurate) and high-ﬁdelity (slow but accurate) simulations.
Revised: To reduce the expensive cost of generatingtraining examples, it has become a promising approach to combine the results oflow-ﬁdelity (fast but inaccurate) and high-ﬁdelity (slow but accurate) simulations.

Original: Despite the fast developments of multi-ﬁdelity fusion techniques, most existingmethods require particular data structures and do not scale well to high-dimensionaloutput.
Revised: Despite the fast developments of multi-ﬁdelity fusion techniques, most existingmethods require particular data structures and do not scale well to high-dimensionaloutput.

Original: To resolve these issues, we generalize the classic autoregression (AR),which is wildly used due to its simplicity, robustness, accuracy, and tractability,and propose generalized autoregression (GAR) using tensor formulation and latentfeatures.
Revised: To resolve these issues, we generalize the classic autoregression (AR),which is wildly used due to its simplicity, robustness, accuracy, and tractability,and propose generalized autoregression (GAR) using tensor formulation and latentfeatures.

Original: GAR can deal with arbitrary dimensional outputs and arbitrary multi- ﬁdelity data structure to satisfy the demand of multi-ﬁdelity fusion for complexproblems; it admits a fully tractable likelihood and posterior requiring no approxi- mate inference and scales well to high-dimensional problems.
Revised: GAR can deal with arbitrary dimensional outputs and arbitrary multi- ﬁdelity data structure to satisfy the demand of multi-ﬁdelity fusion for complexproblems; it admits a fully tractable likelihood and posterior requiring no approxi- mate inference and scales well to high-dimensional problems.

Original: Furthermore, we prove the autokrigeability theorem based on GAR in the multi-ﬁdelity case anddevelop CIGAR, a simpliﬁed GAR with the exact predictive mean accuracy withcomputation reduction by a factor of d 3 , where d is the dimensionality of theoutput.
Revised: Furthermore, we prove the autokrigeability theorem based on GAR in the multi-ﬁdelity case anddevelop CIGAR, a simpliﬁed GAR with the exact predictive mean accuracy withcomputation reduction by a factor of d 3 , where d is the dimensionality of theoutput.

Original: In experiments of canonical PDEs and scientiﬁc computational examples,the proposed method consistently outperforms the SOTA methods with a largemargin (up to 6x improvement in RMSE) with only a couple high-ﬁdelity trainingsamples.
Revised: In experiments of canonical PDEs and scientiﬁc computational examples,the proposed method consistently outperforms the SOTA methods with a largemargin (up to 6x improvement in RMSE) with only a couple high-ﬁdelity trainingsamples.

Original:  Introduction
Revised:  Introduction

Original: The design, optimization, and control of many systems in science and engineering can rely heavilyon the numerical analysis of differential equations, which is generally computationally intense.
Revised: The design, optimization, and control of many systems in science and engineering can rely heavilyon the numerical analysis of differential equations, which is generally computationally intense.

Original: Inthis case, a data-driven surrogate model is used to approximate the system based on the input-outputdata of the numerical solver and to help improve convergence efﬁciency where repeated simulationsare required, e.g., in Bayesian optimization (BO) [1] and uncertainty analysis (UQ) [2].
Revised: Inthis case, a data-driven surrogate model is used to approximate the system based on the input-outputdata of the numerical solver and to help improve convergence efﬁciency where repeated simulationsare required, e.g., in Bayesian optimization (BO) [1] and uncertainty analysis (UQ) [2].

Original: With the surrogate model in place, the remaining challenge is that executing the high-ﬁdelity numericalsolver to generate training data can still be very expensive.
Revised: With the surrogate model in place, the remaining challenge is that executing the high-ﬁdelity numericalsolver to generate training data can still be very expensive.

Original: To further reduce the computationalburden, it is possible to combine low-ﬁdelity results to make high-ﬁdelity predictions [3].
Revised: To further reduce the computationalburden, it is possible to combine low-ﬁdelity results to make high-ﬁdelity predictions [3].

Original: Morespeciﬁcally, low-ﬁdelity solvers are normally based on simpliﬁed PDEs (e.g., reducing the levels ofphysical detail) or simple solver setup (e.g., using a coarse mesh, a large time step, a lower order of anapproximating basis, and a higher error tolerance).
Revised: Morespeciﬁcally, low-ﬁdelity solvers are normally based on simpliﬁed PDEs (e.g., reducing the levels ofphysical detail) or simple solver setup (e.g., using a coarse mesh, a large time step, a lower order of anapproximating basis, and a higher error tolerance).

Original: They provide fast but inaccurate solutions to theoriginal problems where as the high-ﬁdelity solver is accurate yet expensive.
Revised: They provide fast but inaccurate solutions to theoriginal problems where as the high-ﬁdelity solver is accurate yet expensive.

Original: The multi-ﬁdelity fusiontechnique works similar to transfer learning to utilize many low-ﬁdelity observations to improvethe accuracy when using only a few high-ﬁdelity samples.
Revised: The multi-ﬁdelity fusiontechnique works similar to transfer learning to utilize many low-ﬁdelity observations to improvethe accuracy when using only a few high-ﬁdelity samples.

Original: In general, it involves constructionsof surrogates for different ﬁdelities and a cross-ﬁdelity transition process.
Revised: In general, it involves constructionsof surrogates for different ﬁdelities and a cross-ﬁdelity transition process.

Original: Due to its efﬁciency, multi-ﬁdelity has attracted increasing attention in BO [4, 5], UQ [6, 7], and surrogate modeling [8, 9]l.
Revised: Due to its efﬁciency, multi-ﬁdelity has attracted increasing attention in BO [4, 5], UQ [6, 7], and surrogate modeling [8, 9]l.

Original: We refer to [10, 11] for a recent review.
Revised: We refer to [10, 11] for a recent review.

Original: Despite the success of many state-of-the-art (SOTA) approaches, they normally assume that (1)the output dimension is the same and aligned across all ﬁdelities, which generally does not holdfor multi-ﬁdelity simulation where the output are quantities at nodes that are not aligned; (2) thehigh-ﬁdelity samples’ corresponding inputs form a subset of the low-ﬁdelity inputs; and (3) the outputdimension is small, which in not practical for scientiﬁc computing where dimension can be 1-million(for a 100 ⇥ 100 ⇥ 100 spatial-temporal ﬁeld).
Revised: Despite the success of many state-of-the-art (SOTA) approaches, they normally assume that (1)the output dimension is the same and aligned across all ﬁdelities, which generally does not holdfor multi-ﬁdelity simulation where the output are quantities at nodes that are not aligned; (2) thehigh-ﬁdelity samples’ corresponding inputs form a subset of the low-ﬁdelity inputs; and (3) the outputdimension is small, which in not practical for scientiﬁc computing where dimension can be 1-million(for a 100 ⇥ 100 ⇥ 100 spatial-temporal ﬁeld).

Original: These assumptions seriously hinder their applicationsfor modern high-dimensional problems, which is common in scientiﬁc computing for solving PDEsand real-world datasets, e.g., MRI imaging.
Revised: These assumptions seriously hinder their applicationsfor modern high-dimensional problems, which is common in scientiﬁc computing for solving PDEsand real-world datasets, e.g., MRI imaging.

Original: To resolve these challenges, previous work either uses interpolation to align the dimension [12, 9] orrelies on approximate inference with brutal simpliﬁcation [8, 13], leading to inferior performance.
Revised: To resolve these challenges, previous work either uses interpolation to align the dimension [12, 9] orrelies on approximate inference with brutal simpliﬁcation [8, 13], leading to inferior performance.

Original: Wenotice that the classic autoregression (AR), which is popularly used due to its simplicity, robustness,accuracy, and tractability, consistently shows robust and top-tier performance for different datasetsin the literature, despite its incapability for high-dimensional problems.
Revised: Wenotice that the classic autoregression (AR), which is popularly used due to its simplicity, robustness,accuracy, and tractability, consistently shows robust and top-tier performance for different datasetsin the literature, despite its incapability for high-dimensional problems.

Original: Thus, instead of proposinganother ad-hoc model with pre-processing and simpliﬁcation (leading to models that are difﬁcult totune and generalize poorly), we generalize AR with tensor algebra and latent features and proposegeneralized autoregression (GAR), which can deal with arbitrary high-dimensional Problems withoutthe subset multi-ﬁdelity data structure.
Revised: Thus, instead of proposinganother ad-hoc model with pre-processing and simpliﬁcation (leading to models that are difﬁcult totune and generalize poorly), we generalize AR with tensor algebra and latent features and proposegeneralized autoregression (GAR), which can deal with arbitrary high-dimensional Problems withoutthe subset multi-ﬁdelity data structure.

Original: GAR is a fully tractable Bayesian model with scalability toextremely high-dimensional outputs, without requiring any approximate inference.
Revised: GAR is a fully tractable Bayesian model with scalability toextremely high-dimensional outputs, without requiring any approximate inference.

Original: The novelty ofthis work is as follows,
Revised: The novelty ofthis work is as follows,

Original: Generalization of the AR for arbitrary non-structured high-dimensional outputs.
Revised: Generalization of the AR for arbitrary non-structured high-dimensional outputs.

Original: With tensor
Revised: With tensor

Original: algebra and latent
Revised: algebra and latent

Original: features, GAR
Revised: features, GAR

Original: allows
Revised: allows

Original: effective knowledge transfer
Revised: effective knowledge transfer

Original: in closed-
Revised: in closed-

Original: form
Revised: form

Original: and is
Revised: and is

Original: scalable to extreme
Revised: scalable to extreme

Original: high dimensional problems
Revised: high dimensional problems

Original: Generalization to non-subset multi-ﬁdelity data.
Revised: Generalization to non-subset multi-ﬁdelity data.

Original: To the best of our knowledge, we are the
Revised: To the best of our knowledge, we are the

Original: ﬁrst to generalize the closed form solution of subset data to non-subset cases for the AR and
Revised: ﬁrst to generalize the closed form solution of subset data to non-subset cases for the AR and

Original: the proposed GAR.
Revised: the proposed GAR.

Original: For the ﬁrst time, we reveal the autokrigeability [14] for the multi-ﬁdelity fusion withinan AR structure, based on which we derive conditional independent GAR (CIGAR), anefﬁcient implementation of GAR with the exact accuracy in posterior mean predictions.
Revised: For the ﬁrst time, we reveal the autokrigeability [14] for the multi-ﬁdelity fusion withinan AR structure, based on which we derive conditional independent GAR (CIGAR), anefﬁcient implementation of GAR with the exact accuracy in posterior mean predictions.

Original:  Backgronud
Revised:  Backgronud

Original: 2.1 Statement of the problem
Revised: 2.1 Statement of the problem

Original: Given multi-ﬁdelity data D i
Revised: Given multi-ﬁdelity data D i

Original: = { x in , y in } N i n =1 , for i = 1 , . . . , ⌧ , where x 2 R l denotes the systeminputs (a vector of parameters that appear in the system of equations and/or in the initial-boundaryconditions for a simulation);
Revised: = { x in , y in } N i n =1 , for i = 1 , . . . , ⌧ , where x 2 R l denotes the systeminputs (a vector of parameters that appear in the system of equations and/or in the initial-boundaryconditions for a simulation);

Original: y i 2 R d i denotes the corresponding outputs, where d i is the dimensionfor i ﬁdelity; ⌧ is the total number of ﬁdelities.
Revised: y i 2 R d i denotes the corresponding outputs, where d i is the dimensionfor i ﬁdelity; ⌧ is the total number of ﬁdelities.

Original: Generally speaking, higher-ﬁdelity are closer tothe ground truth and are expensive to obtain, and thus we have fewer samples for high ﬁdelity, i.e.,
Revised: Generally speaking, higher-ﬁdelity are closer tothe ground truth and are expensive to obtain, and thus we have fewer samples for high ﬁdelity, i.e.,

Original: N 1 > N 2 > · · · > N ⌧ .
Revised: N 1 > N 2 > · · · > N ⌧ .

Original: The dimensionality d i is not necessary the same or aligned across differentﬁdelities.
Revised: The dimensionality d i is not necessary the same or aligned across differentﬁdelities.

Original: In most work e.g., [15, 16, 11], the system inputs of higher-ﬁdelity are chosen to bethe subset of the lower-ﬁdelity, i.e., X ⌧ ⇢ · · · ⇢
Revised: In most work e.g., [15, 16, 11], the system inputs of higher-ﬁdelity are chosen to bethe subset of the lower-ﬁdelity, i.e., X ⌧ ⇢ · · · ⇢

Original: X 2 ⇢
Revised: X 2 ⇢

Original: X 1 .
Revised: X 1 .

Original: We call this the subset structure fora multi-ﬁdelity dataset, as opposed to arbitrary data structures, which we will resolve in Sectionsection 3.3 with a closed-form solution and extend it to the classic AR.
Revised: We call this the subset structure fora multi-ﬁdelity dataset, as opposed to arbitrary data structures, which we will resolve in Sectionsection 3.3 with a closed-form solution and extend it to the classic AR.

Original: Our goal is to estimate thefunction f ⌧ : R l !
Revised: Our goal is to estimate thefunction f ⌧ : R l !

Original: R d ⌧ given the observation data across different ﬁdelities { D i } ⌧ i =1 .
Revised: R d ⌧ given the observation data across different ﬁdelities { D i } ⌧ i =1 .

Original: 2.2 Autoregression
Revised: 2.2 Autoregression

Original: For the sake of clarity, we consider a two-ﬁdelity case with superscript h indicating high-ﬁdelity andl low-ﬁdelity.
Revised: For the sake of clarity, we consider a two-ﬁdelity case with superscript h indicating high-ﬁdelity andl low-ﬁdelity.

Original: Nevertheless, the formulation can be generalized to problems with multiple ﬁdelitiesrecursively.
Revised: Nevertheless, the formulation can be generalized to problems with multiple ﬁdelitiesrecursively.

Original: Considering a simple scalar output for all ﬁdelities, the AR [3] assumes
Revised: Considering a simple scalar output for all ﬁdelities, the AR [3] assumes

Original: [Equation]
Revised: [Equation]

Original: where ⇢ is a factor transferring knowledge from the low ﬁdelity in a linear fashion whereas f r ( x )tries to capture the residual information.
Revised: where ⇢ is a factor transferring knowledge from the low ﬁdelity in a linear fashion whereas f r ( x )tries to capture the residual information.

Original: If we assume a zero mean Gaussian process (GP) prior
Revised: If we assume a zero mean Gaussian process (GP) prior

Original: [17] (see Appendix A for a brief description) for f l ( x ) and f r ( x ) , i.e., f l ( x ) ⇠ N (0 , k l ( x , x 0 ))
Revised: [17] (see Appendix A for a brief description) for f l ( x ) and f r ( x ) , i.e., f l ( x ) ⇠ N (0 , k l ( x , x 0 ))

Original: andf r ( x ) ⇠ N (0 , k r ( x , x 0 )) , the high-ﬁdelity function also follows a GP.
Revised: andf r ( x ) ⇠ N (0 , k r ( x , x 0 )) , the high-ﬁdelity function also follows a GP.

Original: This gives an elegant joint GPfor the joint observations y =
Revised: This gives an elegant joint GPfor the joint observations y =

Original: [ y l ; y h ] T ,
Revised: [ y l ; y h ] T ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where y l 2 R N l ⇥ 1 is the low-ﬁdelity observations corresponding to input X l 2 R N l ⇥ L and y h 

Revised: where y l 2 R N l ⇥ 1 is the low-ﬁdelity observations corresponding to input X l 2 R N l ⇥ L and y h

Original: R N h ⇥ 1 is the high-ﬁdelity observations; [ K l ( X l , X l )] ij = k l ( x i , x j ) is the covariance matrix ofthe low-ﬁdelity inputs x i , x j 2 X l ; [ K r ( X l , X l )] ij = k r ( x i , x j ) is for the high-ﬁdelity inputsx i , x j 2 X h ; [ K l ( X l , X h )] ij = k l ( x i , x j ) is the cross-ﬁdelity covariance matrix of the low-ﬁdelityinputs x i 2 X l and high-ﬁdelity inputs x j 2 X h , and K l ( X h , X l ) =
Revised: R N h ⇥ 1 is the high-ﬁdelity observations; [ K l ( X l , X l )] ij = k l ( x i , x j ) is the covariance matrix ofthe low-ﬁdelity inputs x i , x j 2 X l ; [ K r ( X l , X l )] ij = k r ( x i , x j ) is for the high-ﬁdelity inputsx i , x j 2 X h ; [ K l ( X l , X h )] ij = k l ( x i , x j ) is the cross-ﬁdelity covariance matrix of the low-ﬁdelityinputs x i 2 X l and high-ﬁdelity inputs x j 2 X h , and K l ( X h , X l ) =

Original: ( K l ( X l , X h )) T .
Revised: ( K l ( X l , X h )) T .

Original: Oneimmediate advantage of the AR is that the joint Gaussian form allows not only joint training for alllow- and high-ﬁdelity data but also predictions for any given new inputs by conditional Gaussian(as the posterior is derived in a standard GP [17]).
Revised: Oneimmediate advantage of the AR is that the joint Gaussian form allows not only joint training for alllow- and high-ﬁdelity data but also predictions for any given new inputs by conditional Gaussian(as the posterior is derived in a standard GP [17]).

Original: Furthermore, Le Gratiet [15] derive Lemma 1 toreduce the complexity from O (( N l + N h ) 3 ) to O (( N l ) 3 + ( N h ) 3 ) with a subset data structure.
Revised: Furthermore, Le Gratiet [15] derive Lemma 1 toreduce the complexity from O (( N l + N h ) 3 ) to O (( N l ) 3 + ( N h ) 3 ) with a subset data structure.

Original: Lemma 1.
Revised: Lemma 1.

Original: [15] If X h ⇢ X l , the joint likelihood and predictive posterior of the AR can be decomposed into two independent parts corresponding to the low- and high-ﬁdelity data.
Revised: [15] If X h ⇢ X l , the joint likelihood and predictive posterior of the AR can be decomposed into two independent parts corresponding to the low- and high-ﬁdelity data.

Original:  Generalized Autoregression
Revised:  Generalized Autoregression

Original: Let’s now consider the high-dimensional case.
Revised: Let’s now consider the high-dimensional case.

Original: A naive approach is to simply convert the multidimensional output into a scalar value by attaching a dimension index to the input.
Revised: A naive approach is to simply convert the multidimensional output into a scalar value by attaching a dimension index to the input.

Original: However, the
Revised: However, the

Original: AR will end up with a joint GP with a covariance matrix of the size of ( N l d l + N h d h ) 2 , making itinfeasible for large dimensional problems.
Revised: AR will end up with a joint GP with a covariance matrix of the size of ( N l d l + N h d h ) 2 , making itinfeasible for large dimensional problems.

Original: 3.1 Tensor Factorized Generalization with Latent Features
Revised: 3.1 Tensor Factorized Generalization with Latent Features

Original: To resolve the scalability issue, we rearrange all the output into a multidimensional space (i.e., atensor space) and introduce latent coordinate features to index the outputs to capture their correlationsas in HOGP [18].
Revised: To resolve the scalability issue, we rearrange all the output into a multidimensional space (i.e., atensor space) and introduce latent coordinate features to index the outputs to capture their correlationsas in HOGP [18].

Original: More speciﬁcally, we organize the low-ﬁdelity outputs as a M -mode tensor,
Revised: More speciﬁcally, we organize the low-ﬁdelity outputs as a M -mode tensor,

Original: Z l 2 R d l 1 ⇥ ··· ⇥ d lM , where the output dimension d l =
Revised: Z l 2 R d l 1 ⇥ ··· ⇥ d lM , where the output dimension d l =

Original: Q
Revised: Q

Original: Mm =1 d lm .
Revised: Mm =1 d lm .

Original: The element Z l is indexed based onits coordinates
Revised: The element Z l is indexed based onits coordinates

Original: c
Revised: c

Original: = ( c 1 , . .
Revised: = ( c 1 , . .

Original: . , c M ) ( 1 6 c k 6 d k for k = 1 , . . .
Revised: . , c M ) ( 1 6 c k 6 d k for k = 1 , . . .

Original: , M ).
Revised: , M ).

Original: If the original data indeed admita multi-array structure, we can use their original index with actual meaning to index the coordinates.
Revised: If the original data indeed admita multi-array structure, we can use their original index with actual meaning to index the coordinates.

Original: For instance, a 2D spatial dataset can use its original spatial coordinate to index a single location(pixel).
Revised: For instance, a 2D spatial dataset can use its original spatial coordinate to index a single location(pixel).

Original: To improve our model ﬂexibility, we do not have to limit ourselves from using the originalindex, particularly for the cases where the original data does not admit a multi-array structure orthe multi-array structure is of too large size.
Revised: To improve our model ﬂexibility, we do not have to limit ourselves from using the originalindex, particularly for the cases where the original data does not admit a multi-array structure orthe multi-array structure is of too large size.

Original: In such case, we can use arbitrary tensorization and alatent feature vector v lc m (whose values are inferred in model training) for each coordinate c m inmode m .
Revised: In such case, we can use arbitrary tensorization and alatent feature vector v lc m (whose values are inferred in model training) for each coordinate c m inmode m .

Original: This way, the element Z l is indexed by the vector ( v lc 1 , . .
Revised: This way, the element Z l is indexed by the vector ( v lc 1 , . .

Original: , v lc M ) .
Revised: , v lc M ) .

Original: Following the lineartransformation of Eq.
Revised: Following the lineartransformation of Eq.

Original: (1), we ﬁrst introduce the tensor-matrix product [19] at mode m ,
Revised: (1), we ﬁrst introduce the tensor-matrix product [19] at mode m ,

Original: [Equation]
Revised: [Equation]

Original: where F h ( x ) denotes target function f h ( x ) with its output organized into multi-array Z h , and thesame concept applies to F l ( x ) and F r ( x ) ; ⇥ m denotes the tensor-matrix product at mode m .
Revised: where F h ( x ) denotes target function f h ( x ) with its output organized into multi-array Z h , and thesame concept applies to F l ( x ) and F r ( x ) ; ⇥ m denotes the tensor-matrix product at mode m .

Original: To give aconcrete example, considering an arbitrary tensor Z l 2 R d l 1 ⇥ ··· ⇥ d lM and a matrix W m 2 R s ⇥ d m , the⇥ m product is calculated as [ Z l ⇥ m W m ] i 1 ,...
Revised: To give aconcrete example, considering an arbitrary tensor Z l 2 R d l 1 ⇥ ··· ⇥ d lM and a matrix W m 2 R s ⇥ d m , the⇥ m product is calculated as [ Z l ⇥ m W m ] i 1 ,...

Original: [Equation]
Revised: [Equation]

Original: i m
Revised: i m

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ...,i
Revised: ...,i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ,which becomes an d l 1 ⇥ · · · ⇥ d lm  1 ⇥ s ⇥ d lm +1 ⇥ · · · ⇥ d lM tensor.
Revised: ,which becomes an d l 1 ⇥ · · · ⇥ d lm  1 ⇥ s ⇥ d lm +1 ⇥ · · · ⇥ d lM tensor.

Original: We can further denote the group 124 of M linear transformation matrixes as a Tucker tensor W =
Revised: We can further denote the group 124 of M linear transformation matrixes as a Tucker tensor W =

Original: [ W 1 , . . .
Revised: [ W 1 , . . .

Original: , W M ] and represent Eq.
Revised: , W M ] and represent Eq.

Original: (3)compactly using a Tucker operator [19], F l ( x ) ⇥ W , which has an important property:
Revised: (3)compactly using a Tucker operator [19], F l ( x ) ⇥ W , which has an important property:

Original: [Equation]
Revised: [Equation]

Original: Inspired by the AR of Eq.
Revised: Inspired by the AR of Eq.

Original: (1), we place a tensor-variate GP (TGP) prior [20] for the low-ﬁdelitytensor function F l ( x ) and the residual tensor function F r ( x ) :
Revised: (1), we place a tensor-variate GP (TGP) prior [20] for the low-ﬁdelitytensor function F l ( x ) and the residual tensor function F r ( x ) :

Original: [Equation]
Revised: [Equation]

Original: where S im 2 R d m ⇥ d m are the output correlation matrix with [ S im ] jk = ˜
Revised: where S im 2 R d m ⇥ d m are the output correlation matrix with [ S im ] jk = ˜

Original: k
Revised: k

Original: im ( v ic i , v ic k ) and ˜ k
Revised: im ( v ic i , v ic k ) and ˜ k

Original: im ( · , · ) being the kernel function (with unknown hyperparameters).
Revised: im ( · , · ) being the kernel function (with unknown hyperparameters).

Original: A TGP is a generalization of a multivariate
Revised: A TGP is a generalization of a multivariate

Original: GP that essentially represents a joint GP prior vec( Y l ) ⇠ N ⇣ 0 , K l ( X l , X l )
Revised: GP that essentially represents a joint GP prior vec( Y l ) ⇠ N ⇣ 0 , K l ( X l , X l )

Original: N
Revised: N

Original: Mm =1 S m ⌘ .
Revised: Mm =1 S m ⌘ .

Original: Similar tothe joint probability of (2), we can derive the joint probability for y =
Revised: Similar tothe joint probability of (2), we can derive the joint probability for y =

Original: [vec( Y l ) T , vec( Y h ) T ] T basedon Tucker transformation of (3)
Revised: [vec( Y l ) T , vec( Y h ) T ] T basedon Tucker transformation of (3)

Original: ; we keep the the proof in the Appendix for clarity.
Revised: ; we keep the the proof in the Appendix for clarity.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given the tensor GP priors for Y l ( x , x 0 ) and Y r ( x , x 0 ) and the Tucker transformation of(3) , the joint probability for y =
Revised: Given the tensor GP priors for Y l ( x , x 0 ) and Y r ( x , x 0 ) and the Tucker transformation of(3) , the joint probability for y =

Original: [vec( Y l ) T , vec( Y h ) T ] T is y ⇠ N ( 0 , ⌃ ) , where ⌃ =
Revised: [vec( Y l ) T , vec( Y h ) T ] T is y ⇠ N ( 0 , ⌃ ) , where ⌃ =

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 admits any arbitrary outputs (living in different space, having different dimensions and/ormode, and being unaligned) at different ﬁdelity.
Revised: Lemma 2 admits any arbitrary outputs (living in different space, having different dimensions and/ormode, and being unaligned) at different ﬁdelity.

Original: Also, it does not require a subset dataset to hold.
Revised: Also, it does not require a subset dataset to hold.

Original: Corollary 3.0.1.
Revised: Corollary 3.0.1.

Original: Lemma 2 can be applied to data with a different number of mode at each ﬁdelity,i.e., M l 6 =
Revised: Lemma 2 can be applied to data with a different number of mode at each ﬁdelity,i.e., M l 6 =

Original: M h , if we pad the output with fewer modes with a mode having only one latent index suchthat all outputs have the same M number of modes.
Revised: M h , if we pad the output with fewer modes with a mode having only one latent index suchthat all outputs have the same M number of modes.

Original: Lemma 2 deﬁnes our GAR model, a generalized AR with special tensor structures.
Revised: Lemma 2 deﬁnes our GAR model, a generalized AR with special tensor structures.

Original: The covariance for low-ﬁdelity is cov( Z l c ( x ) , Z l c 0 ( x 0 ))
Revised: The covariance for low-ﬁdelity is cov( Z l c ( x ) , Z l c 0 ( x 0 ))

Original: =
Revised: =

Original: k l ( x , x 0 )
Revised: k l ( x , x 0 )

Original: Q
Revised: Q

Original: Mm =1 ˜ k lm ( v mc m , v mc 0 m ) , cross-ﬁdelitycov( Z l c ( x ) , Z h c 0 ( x 0 ))
Revised: Mm =1 ˜ k lm ( v mc m , v mc 0 m ) , cross-ﬁdelitycov( Z l c ( x ) , Z h c 0 ( x 0 ))

Original: =
Revised: =

Original: k l ( x , x 0 )
Revised: k l ( x , x 0 )

Original: Q
Revised: Q

Original: Mm =1 ˜ k lm ( v lc m , v lc 0 m ) w mc,c 0 (where w mc,c 0 is the c, c 0 -th elementof W m ), and high-ﬁdelity cov( Z h c ( x ) , Z h c 0 ( x 0 ))
Revised: Mm =1 ˜ k lm ( v lc m , v lc 0 m ) w mc,c 0 (where w mc,c 0 is the c, c 0 -th elementof W m ), and high-ﬁdelity cov( Z h c ( x ) , Z h c 0 ( x 0 ))

Original: =
Revised: =

Original: k l ( x , x 0 )
Revised: k l ( x , x 0 )

Original: Q
Revised: Q

Original: Mm =1 ˜ k lm ( v lc m , v lc 0 m )( w mc,c 0 ) 

Revised: Mm =1 ˜ k lm ( v lc m , v lc 0 m )( w mc,c 0 )

Original: +k h ( x , x 0 )
Revised: +k h ( x , x 0 )

Original: Q
Revised: Q

Original: Mm =1 ˜ k rm ( u mc m , u mc 0 m )( w mc,c 0 ) 2 .
Revised: Mm =1 ˜ k rm ( u mc m , u mc 0 m )( w mc,c 0 ) 2 .

Original: The complex between-ﬁdelity output correlations are captured using latent features { V m , U m } Mm =1 with arbitrary kernel function ˜ k
Revised: The complex between-ﬁdelity output correlations are captured using latent features { V m , U m } Mm =1 with arbitrary kernel function ˜ k

Original: im whereas the crossﬁdelity output correlations are captured in a composite manner.
Revised: im whereas the crossﬁdelity output correlations are captured in a composite manner.

Original: This combination overcomes thesimple linear correlations assumed in previous work that simply decomposes the output as a dimensionreduction preprocess [12].
Revised: This combination overcomes thesimple linear correlations assumed in previous work that simply decomposes the output as a dimensionreduction preprocess [12].

Original: When the dimensionality aligns for Z l and Z h and thus d lm =
Revised: When the dimensionality aligns for Z l and Z h and thus d lm =

Original: d hm , we canshare the same latent features across the two ﬁdelity data by letting v mj
Revised: d hm , we canshare the same latent features across the two ﬁdelity data by letting v mj

Original: =
Revised: =

Original: u mj while keeping the kernelfunctions different.
Revised: u mj while keeping the kernelfunctions different.

Original: This way, the latent features are more resistant to overﬁtting.
Revised: This way, the latent features are more resistant to overﬁtting.

Original: For non-aligned datawith explicit indexing, we can use kernel interpolation [21] for the same purpose.
Revised: For non-aligned datawith explicit indexing, we can use kernel interpolation [21] for the same purpose.

Original: To further encouragesparsity in the latent feature, we impose a Laplace prior
Revised: To further encouragesparsity in the latent feature, we impose a Laplace prior

Original: , i.e., v mj ⇠ Laplace(  ) / exp(  
Revised: , i.e., v mj ⇠ Laplace(  ) / exp(

Original: ||
Revised: ||

Original: v mj || 1 ) .
Revised: v mj || 1 ) .

Original: 3.2 Efﬁcient Model Inference for
Revised: 3.2 Efﬁcient Model Inference for

Original: Subset Data Structure
Revised: Subset Data Structure

Original: With the model fully deﬁned, we can now train the model to obtain all unknown model parameters.
Revised: With the model fully deﬁned, we can now train the model to obtain all unknown model parameters.

Original: For compactness, we use the following compact notation S l =
Revised: For compactness, we use the following compact notation S l =

Original: N
Revised: N

Original: Mm =1 S lm , S h =
Revised: Mm =1 S lm , S h =

Original: N
Revised: N

Original: Mm =
Revised: Mm =

Original: [Equation]
Revised: [Equation]

Original: S hm , W =
Revised: S hm , W =

Original: N
Revised: N

Original: Mm =1 W m , K l = K l ( X l , X l ) , K lh = K l ( X l , X h ) , K hl =
Revised: Mm =1 W m , K l = K l ( X l , X l ) , K lh = K l ( X l , X h ) , K hl =

Original: K l ( X h , X l ) , K lr =
Revised: K l ( X h , X l ) , K lr =

Original: K l ( X h , X h ) ,and, K r =
Revised: K l ( X h , X h ) ,and, K r =

Original: K r
Revised: K r

Original: ( X h , X h ) (with a slight abuse of notation).
Revised: ( X h , X h ) (with a slight abuse of notation).

Original: Lemma 3.
Revised: Lemma 3.

Original: Tensor generalization of Lemma 1.
Revised: Tensor generalization of Lemma 1.

Original: If X h ⇢ X l , the joint likelihood L for y =[vec( Y l ) T , vec( Y h ) T ]
Revised: If X h ⇢ X l , the joint likelihood L for y =[vec( Y l ) T , vec( Y h ) T ]

Original: T admits two independent separable likelihoods L =
Revised: T admits two independent separable likelihoods L =

Original: L l + L r , where
Revised: L l + L r , where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆ W =
Revised: where ˆ W =

Original: [ E , W 1 , . . .
Revised: [ E , W 1 , . . .

Original: , W M ] is a Tucker tensor with selection matrix
Revised: , W M ] is a Tucker tensor with selection matrix

Original: E T X l = X h .
Revised: E T X l = X h .

Original: We preserve the proof in Appendix for clarity.
Revised: We preserve the proof in Appendix for clarity.

Original: Note that L l and L r are HOGP likelihoods for Y land
Revised: Note that L l and L r are HOGP likelihoods for Y land

Original: residual Y h
Revised: residual Y h

Original:  Y l ⇥ ˆ W , respectively.
Revised:  Y l ⇥ ˆ W , respectively.

Original: Since the computational of L l and L r are independent, themodel training can be conducted efﬁciently in parallel.
Revised: Since the computational of L l and L r are independent, themodel training can be conducted efﬁciently in parallel.

Original: Predictive posterior.
Revised: Predictive posterior.

Original: Similarly, we can derive the concrete predictive posterior for high-ﬁdelityoutputs by integrating out the latent functions after some tedious linear algebra (see Appendix), which is also Gaussian, vec( Z h ⇤ )
Revised: Similarly, we can derive the concrete predictive posterior for high-ﬁdelityoutputs by integrating out the latent functions after some tedious linear algebra (see Appendix), which is also Gaussian, vec( Z h ⇤ )

Original: ⇠ N (vec(¯ Z h ⇤ ) , S h ⇤ ) , where
Revised: ⇠ N (vec(¯ Z h ⇤ ) , S h ⇤ ) , where

Original: [Equation]
Revised: [Equation]

Original: where k l ⇤ =
Revised: where k l ⇤ =

Original: k l ( x ⇤ , X l ) is the vector of covariance between the give input x ⇤ and low-ﬁdelityobservation inputs X l and similarly, k l ⇤⇤ =
Revised: k l ( x ⇤ , X l ) is the vector of covariance between the give input x ⇤ and low-ﬁdelityobservation inputs X l and similarly, k l ⇤⇤ =

Original: k l ( x ⇤ , x ⇤ ) , k r ⇤ =
Revised: k l ( x ⇤ , x ⇤ ) , k r ⇤ =

Original: k r ( x ⇤ , X h ) , k r
Revised: k r ( x ⇤ , X h ) , k r

Original: ⇤⇤
Revised: ⇤⇤

Original: =
Revised: =

Original: k r ( x ⇤ , x ⇤ ) .
Revised: k r ( x ⇤ , x ⇤ ) .

Original: 3.3 Generalization for Non-subset Data: Efﬁcient Model Inference and Prediction
Revised: 3.3 Generalization for Non-subset Data: Efﬁcient Model Inference and Prediction

Original: In practice, it is sometimes difﬁcult to ask the multi-ﬁdelity data to preserve a subset structure,particularly in the case of multi-ﬁdelity Bayesian optimization [22, 23].
Revised: In practice, it is sometimes difﬁcult to ask the multi-ﬁdelity data to preserve a subset structure,particularly in the case of multi-ﬁdelity Bayesian optimization [22, 23].

Original: This presents the challengefor most SOTA multi-ﬁdelity models e.g., NAR [16], ResGP [9], stochastic collocation [24].
Revised: This presents the challengefor most SOTA multi-ﬁdelity models e.g., NAR [16], ResGP [9], stochastic collocation [24].

Original: Incontrast, the advantage of AR is that even if the multi-ﬁdelity data does not admit a subset datastructure, the model can still be trained using all available data based on the joint likelihood of (2).
Revised: Incontrast, the advantage of AR is that even if the multi-ﬁdelity data does not admit a subset datastructure, the model can still be trained using all available data based on the joint likelihood of (2).

Original: However, this method lacks scalability due to the inversion of the large joint covariance matrix ⌃ .
Revised: However, this method lacks scalability due to the inversion of the large joint covariance matrix ⌃ .

Original: The situation will get worse if we are dealing with multi-ﬁdelity data with more than two ﬁdelities.
Revised: The situation will get worse if we are dealing with multi-ﬁdelity data with more than two ﬁdelities.

Original: Toresolve this issue, we propose a fast inference method based on imaginary subset.
Revised: Toresolve this issue, we propose a fast inference method based on imaginary subset.

Original: More speciﬁcally,considering the missing low-ﬁdelity data as latent variables ˆ Y l , the joint likelihood function is
Revised: More speciﬁcally,considering the missing low-ﬁdelity data as latent variables ˆ Y l , the joint likelihood function is

Original: [Equation]
Revised: [Equation]

Original: where p ( Y h | ˆ Y l , Y l ) is the likelihood in Lemma 3 given the complementary imaginary subset, andp (ˆ Y l | Y l ) ⇠ N (¯ Y l , ˆ S l ⌦ S l ) is the imaginary posterior with the given low-ﬁdelity observations Y l .
Revised: where p ( Y h | ˆ Y l , Y l ) is the likelihood in Lemma 3 given the complementary imaginary subset, andp (ˆ Y l | Y l ) ⇠ N (¯ Y l , ˆ S l ⌦ S l ) is the imaginary posterior with the given low-ﬁdelity observations Y l .

Original: The integral can be calculated using Gaussian quadrature or other sampling methods as in [8, 25],which is slow and inaccurate.
Revised: The integral can be calculated using Gaussian quadrature or other sampling methods as in [8, 25],which is slow and inaccurate.

Original: Lemma 4.
Revised: Lemma 4.

Original: The joint likelihood of GAR for non-subset (and also unaligned) data can be decomposed 185 into two independent GPs’
Revised: The joint likelihood of GAR for non-subset (and also unaligned) data can be decomposed 185 into two independent GPs’

Original: [Equation]
Revised: [Equation]

Original: where L l is the likelihood for low-ﬁdelity data Y l ,

Original: ˜ W

Original: = I N h ⌦ W ˆ Y h is the collection of high-ﬁdelityobservations corresponding to the imaginary low-ﬁdelity outputs ˆ Y l ; ˇ Y h is the complement (withselection matrix ˇ X 
Revised: where L l is the likelihood for low-ﬁdelity data Y l , ˆ Y h is the collection of high-ﬁdelity observationscorresponding to the imaginary low-ﬁdelity outputs ˆ Y l ; ˇ Y h is the complement (with selection matrixˇ X h =

Original: h = E T X l ) corresponding to low-ﬁdelity outputs ˇ Y l , i.e., Y h
Revised: E T X l ), i.e., Y h

Original: =
Revised: =

Original: [ˇ Y h , ˆ Y h ] ; andˆ X h
Revised: [ˇ Y h , ˆ Y h ] ; and ˆ X h

Original: = ˆ E T X h are the selection matrix for ˆ Y l .
Revised: = ˆ E T X h are the selection matrix for ˆ Y l .

Original: We preserve the proof in the appendix.
Revised: We preserve the proof in the appendix.

Original: Notice that
Revised: Notice that

Original: ˆ E ˆ S
Revised: ˆ E ˆ S

Original: l ˆ E T ⌦ W T S l W =
Revised: l ˆ E T ⌦ W T S l W =

Original: ✓   ˆ S l ◆ ⌦ W T S l Wis the low-right block of the predictive variance for the missing low-ﬁdelity observations
Revised: ✓   ˆ S l ◆ ⌦ W T S l Wis the low-right block of the predictive variance for the missing low-ﬁdelity observations

Original: vec(ˆ Y ) ; Wecan easily understand the last part of the likelihood as a GP with accumulated uncertainty(variance)added to the corresponding missing points.
Revised: vec(ˆ Y ) ; Wecan easily understand the last part of the likelihood as a GP with accumulated uncertainty(variance)added to the corresponding missing points.

Original: Lemma 4 naturally applied to the AR by when the outputsis a scaler, where W = ⇢ , S l = 1 , and S r = 

Revised: Lemma 4 naturally applied to the AR by when the outputsis a scaler, where W = ⇢ , S l = 1 , and S r =

Original: Predictive posterior.
Revised: Predictive posterior.

Original: Surprisingly, the posterior also turns out to be a Gaussian distribution,p
Revised: Surprisingly, the posterior also turns out to be a Gaussian distribution,

Original: [Equation]
Revised: [Equation]

Original: where and  and the mean of the predictive posterior ¯ Z ⇤ are given as follows,
Revised: where and  and the mean of the predictive posterior ¯ Z ⇤ are given as follows,

Original: [Equation]
Revised: [Equation]

Original: where E m and E n are the selection matrices that ˆ X h
Revised: where E m and E n are the selection matrices that ˆ X h

Original: = E Tm [ X l , ˆ X h ] , X h
Revised: = E Tm [ X l , ˆ X h ] , X h

Original: = E Tn [ X l , ˆ X h ] , ˆ W =

Original: E Tn ⌦ W and ˆ K l is the covariance matrix that ˆ K l = K l

Original: [Equation]
Revised: [Equation]

Original:  [ X l [Equation] X h 
Revised: E Tn [ X l , ˆ X h ] , ˆ W =

Original: [Equation]
Revised: E Tn ⌦ W and ˆ K l is the covariance matrix that ˆ K l

Original: X
Revised: = K

Original: [Equation]
Revised: [Equation]

Original: 3.4 Autokrigeability, Complexity, and Further Acceleration
Revised: 3.4 Autokrigeability, Complexity, and Further Acceleration

Original: For subset structured data, the computational complexity of GAR is decomposed into two independent TGPs for likelihood and predictive posterior.
Revised: For subset structured data, the computational complexity of GAR is decomposed into two independent TGPs for likelihood and predictive posterior.

Original: Thanks to the tensor algebra (mainly( K ⌦ S )  1 = K  1 ⌦ S  1 ), the complexity of the i -ﬁdelity kernel matrix inversion is reduced to
Revised: Thanks to the tensor algebra (mainly( K ⌦ S )  1 = K  1 ⌦ S  1 ), the complexity of the i -ﬁdelity kernel matrix inversion is reduced to

Original: O (
Revised: O (

Original: P
Revised: P

Original: Mm =1 ( d im ) 3 + ( N i ) 3 ) instead of O (( N i d i ) 3 ) .
Revised: Mm =1 ( d im ) 3 + ( N i ) 3 ) instead of O (( N i d i ) 3 ) .

Original: For the non-subset case, the computational complexity in Eq.
Revised: For the non-subset case, the computational complexity in Eq.

Original: (8) is unfortunately O (( N im d i ) 3 ) where N m is the number of the imaginary low-ﬁdelitypoints.
Revised: (8) is unfortunately O (( N im d i ) 3 ) where N m is the number of the imaginary low-ﬁdelitypoints.

Original: Nevertheless, due to the tensor structure, we can still use conjugate gradient [26] to solve thelinear system efﬁciently.
Revised: Nevertheless, due to the tensor structure, we can still use conjugate gradient [26] to solve thelinear system efﬁciently.

Original: Notice that the mean prediction ¯ Z h ⇤ in Eq.
Revised: Notice that the mean prediction ¯ Z h ⇤ in Eq.

Original: (A.60) and Eq.
Revised: (6) and Eq.

Original: (9) does not depend on any outputcovariance matrixes { S hm , S lm } Mm =1 , which reassemble the autokrigeability (no knowledge transferin noiseless cases for mean predictions) [14, 9] based on the GAR framework.
Revised: (9) does not depend on any output covariancematrixes { S hm , S lm } Mm =1 , which reassemble the autokrigeability (no knowledge transfer in noiselesscases for mean predictions) [14, 9] based on the GAR framework.

Original: For applicationswhere the predictive variation is not of interest, we can introduce a conditional independent outputcorrelation, i.e., S hm
Revised: For applications where the predictive variation is not of interest, we can introduce a conditional independent output-correlation, i.e., S hm

Original: = I , S
Revised: = I , S

Original: lm
Revised: lm

Original: =
Revised: =

Original: I and orthogonal weight matrixes, i.e., W Tm W m =
Revised: I and orthogonal weight matrixes, i.e., W Tm W m =

Original: I , to reduce thecomputationally complexity further down to O (( N i ) 3 ) (see Appendix for detailed proof).
Revised: I , to reduce the computationallycomplexity further down to O (( N i ) 3 ) (see Appendix for detailed proof).

Original: We callthis CIGAR as an abbreviation for conditional independent GAR.
Revised: We call this CIGAR as anabbreviation for conditional independent GAR.

Original: In practice, CIGAR is slightlyworse than GAR due to the difﬁculty of ensuring W Tm W m =
Revised: In practice, CIGAR is slightly worse than GAR dueto the difﬁculty of ensuring W Tm W m =

Original: I .
Revised: I .

Original:  Related Work
Revised:  Related Work

Original: GP for high-dimensional outputs is an important model in many applications such as spatial datamodeling and uncertainty quantiﬁcation.
Revised: GP for high-dimensional outputs is an important model in many applications such as spatial datamodeling and uncertainty quantiﬁcation.

Original: For an excellent review, the readers are referred to [27].
Revised: For an excellent review, the readers are referred to [27].

Original: Linear model of coregionalization (LMC) [28, 29] might be the most general framework for highdimensional GP developed in the geostatistic community.
Revised: Linear model of coregionalization (LMC) [28, 29] might be the most general framework for highdimensional GP developed in the geostatistic community.

Original: LMC assumes that the full covariancematrix as a sum of constant matrixes timing input-dependent kernels.
Revised: LMC assumes that the full covariancematrix as a sum of constant matrixes timing input-dependent kernels.

Original: To reduce model complexity,semiparametric latent factor models (SLFM) [30] simplify LMC by assuming that the matrixes arerank-1 matrixes.
Revised: To reduce model complexity,semiparametric latent factor models (SLFM) [30] simplify LMC by assuming that the matrixes arerank-1 matrixes.

Original: [31] further simpliﬁes SLFM using singular value decomposition (SVD) on theoutput collection to ﬁx the bases for the rank-1 matrixes.
Revised: [31] further simpliﬁes SLFM using singular value decomposition (SVD) on theoutput collection to ﬁx the bases for the rank-1 matrixes.

Original: To overcome the linear assumptions of
Revised: To overcome the linear assumptions of

Original: LMC, the (implicit) bases can be constructed in a nonlinear fashion using manifold learning, e.g.,
Revised: LMC, the (implicit) bases can be constructed in a nonlinear fashion using manifold learning, e.g.,

Original: KPCA [32] and IsoMap [33] or process convolution [34–36].
Revised: KPCA [32] and IsoMap [33] or process convolution [34–36].

Original: Other approaches include multi-task
Revised: Other approaches include multi-task

Original: GP, which considers the outputs as dependent tasks [37–39] in a framework similar to LMC and GPregression network (GPRN)
Revised: GP, which considers the outputs as dependent tasks [37–39] in a framework similar to LMC and GPregression network (GPRN)

Original: [40, 41], which proposes products of GPs to model nonlinear outputs,leading to nontractable models.
Revised: [40, 41], which proposes products of GPs to model nonlinear outputs,leading to nontractable models.

Original: Despite their success, the complicity of the above approaches areat best O
Revised: Despite their success, the complicity of the above approaches areat best O

Original: ( N 3 + D 3 ) whereas some are O ( N 3 D 3 ) , which cannot scale well to high-dimensionaloutputs for scientiﬁc data where D can be, says, 1 million.
Revised: ( N 3 + D 3 ) whereas some are O ( N 3 D 3 ) , which can not scale well to high-dimensionaloutputs for scientiﬁc data where D can be, says, 1 million.

Original: This problem can be well resolved byintroducing tensor algebra [42] to form HOGP [18] or scalable model inference, e.g., in GPRN [43].
Revised: This problem can be well resolved byintroducing tensor algebra [42, 43] into the GP directly to form HOGP [18] or to the model inference,e.g., in GPRN [44].

Original: Multi-ﬁdelity has become a promising approach to further reduce the data demands in building asurrogate model [13] and Bayesian optimization.
Revised: Multi-ﬁdelity has become a promising approach to further reduce the data demands in building asurrogate model [13] and Bayesian optimization.

Original: The seminal autoregressive (AR) model of Kennedy[3] introduce a linear transformation to univariate high-ﬁdelity outputs.
Revised: The seminal autoregressive (AR) model of Kennedy[3] introduce a linear transformation to univariate high-ﬁdelity outputs.

Original: This model was enhancedby Le Gratiet [15] by adopting a deterministic parametric form of linear transformation for theefﬁcient training scheme as introduced previously.
Revised: This model was enhancedby Le Gratiet [15] by adopting a deterministic parametric form of linear transformation for theefﬁcient training scheme as introduced previously.

Original: However, it is unclear how AR can deal withhigh-dimensional outputs.
Revised: However, it is unclear how AR can deal withhigh-dimensional outputs.

Original: To overcome the linearity of AR, Perdikaris et al.
Revised: To overcome the linearity of AR, Perdikaris et al.

Original: [16] proposes nonlinear
Revised: [16] proposes nonlinear

Original: AR (NAR).
Revised: AR (NAR).

Original: It ignores the output distributions and directly uses the low-ﬁdelity solution as an inputfor the high-ﬁdelity GP model, which is essentially a concatenating GP structure known as deep
Revised: It ignores the output distributions and directly uses the low-ﬁdelity solution as an inputfor the high-ﬁdelity GP model, which is essentially a concatenating GP structure known as deep

Original: GP [44].
Revised: GP [45].

Original: To propagate uncertainty through the multi-ﬁdelity model, Cutajar et al.
Revised: To propagate uncertainty through the multi-ﬁdelity model, Cutajar et al.

Original: [25] uses expensiveapproximation inference, which makes it prone to overﬁtting and incapable of dealing with verylarge dimensional problems.
Revised: [25] uses expensiveapproximation inference, which makes it prone to overﬁtting and incapable of dealing with very large dimensional problems.

Original: For multi-ﬁdelity Bayesian optimization (MFBO), Poloczek et al.
Revised: For multi-ﬁdelity Bayesian optimization (MFBO), Poloczek et al.

Original: [4] and Kandasamy et al.
Revised: [4]and Kandasamy et al.

Original: [45] approximate each ﬁdelity with a GP independently; Zhang et al.
Revised: [46] approximate each ﬁdelity with a GP independently; Zhang et al.

Original: [46] useconvolution kernel, similar to the process convolution [34, 36] to learn the ﬁdelity correlations; Songet al.
Revised: [47] useconvolution kernel, similar to the process convolution [34, 36] to learn the ﬁdelity correlations; Songet al.

Original: [5] combine all ﬁdelity data into one single GP to reduce uncertainty.
Revised: [5] combine all ﬁdelity data into one single GP to reduce uncertainty.

Original: However, most MFBOsurrogates do not scale to high-dimensional problems because they are designed for one target (or atmost a couple).
Revised: However, most MFBOsurrogates do not scale to high-dimensional problems because they are designed for one target (or atmost a couple).

Original: To deal with large dimensional outputs, e.g., spatial-temporal ﬁelds, Xing et al.
Revised: To deal with large dimensional outputs, e.g., spatial-temporal ﬁelds, Xing et al.

Original: [9] extend AR byassuming a simple additive structure and replacing the simple GPs with scalable multi-output GPs atthe cost of losing the power for capturing the output correlations, leading to inferior performanceand inaccurate uncertainty estimates; Xing et al.
Revised: [9] extend AR byassuming a simple additive structure and replacing the simple GPs with scalable multi-output GPs atthe cost of losing the power for capturing the output correlations, leading to inferior performanceand inaccurate uncertainty estimates; Xing et al.

Original: [12] propose Deep coregionalization to extend
Revised: [12] propose Deep coregionalization to extend

Original: NAR by learning the latent process [30, 29] extracted from embedding the high-dimensional outputsonto a residual latent space using a proposed residual PCA; Wang et al.
Revised: NAR by learning the latent process [30, 29] extracted from embedding the high-dimensional outputsonto a residual latent space using a proposed residual PCA; Wang et al.

Original: [8] further introduce basespropagation along with latent process propagation in a deep GP to increase model ﬂexibility atthe cost of signiﬁcant growth in the number of model parameters and a few simpliﬁcations in theapproximated inference.
Revised: [8] further introduce basespropagation along with latent process propagation in a deep GP to increase model ﬂexibility atthe cost of signiﬁcant growth in the number of model parameters and a few simpliﬁcations in theapproximated inference.

Original: Parussini et al.
Revised: Parussini et al.

Original: [6] generalize NAR to high-dimensional problems.
Revised: [6] generalize NAR to high-dimensional problems.

Original: However,these methods lack a systematic way for joint model training, leading to instability and poor ﬁttingfor small datasets.
Revised: However,these methods lack a systematic way for joint model training, leading to instability and poor ﬁtting forsmall datasets.

Original: Wu et al.

Original: [47] extend GP using neural process to model high-dimensional and nonsubset problem effectively.

Original: In scientiﬁc computing, multi-ﬁdelity fusion has been implemented usingstochastic collocation (SC) method [24] for high-dimensional problems, which provides closed-formsolutions and efﬁcient design of experiments for the multi-ﬁdelity problem.
Revised: In scientiﬁc computing, multi-ﬁdelity fusion has been implemented using stochasticcollocation (SC) method [24] for high-dimensional problems, which provides closed-form solutionsand efﬁcient design of experiments for the multi-ﬁdelity problem.

Original: Xing et al.
Revised: Xing et al.

Original: [7] showedthat SC is essentially a special case of AR and proposed active learning to select the best subset forthe high-ﬁdelity experiments.
Revised: [7] showed that SCis essentially a special case of AR and proposed active learning to select the best subset for thehigh-ﬁdelity experiments.

Original: To take the advances of deep learning neural network (NN) and being compatible with arbitrarymulti-ﬁdelity data (i.e., non-subset structure), Perrone et al.
Revised: To take the advances of deep learning neural network (NN) and being compatible with arbitrarymulti-ﬁdelity data (i.e., non-subset structure), Perrone et al.

Original: [22] propose an NN-based multi-taskmethod that can naturally extend to MFBO.
Revised: [22] propose an NN-based multi-taskmethod that can naturally extend to MFBO.

Original: Li et al.
Revised: Li et al.

Original: [23] further extend it as a Bayesian neuralnetwork (BNN) to MFBO.
Revised: [23] further extend it as a Bayesian neuralnetwork (BNN) to MFBO.

Original: Meng and Karniadakis [48] add a physics regularization layer, whichrequires an explicit form of the problem PDEs, to improve prediction accuracy.
Revised: Meng and Karniadakis [48] add a physics regularization layer, whichrequires an explicit form of the problem PDEs, to improve prediction accuracy.

Original: To scale for highdimensional problems with arbitrary dimensions in each ﬁdelity, Li et al.
Revised: To scale for highdimensional problems with arbitrary dimensions in each ﬁdelity, Li et al.

Original: [13] propose a Bayesiannetwork approach to multi-ﬁdelity fusion with active learning techniques for efﬁciency improvement.
Revised: [13] propose a Bayesiannetwork approach to multi-ﬁdelity fusion with active learning techniques for efﬁciency improvement.

Original: Except for multi-ﬁdelity fusion, AR can be used for model multi-variate problem [49, 50], where
Revised: Except for multi-ﬁdelity fusion, AR can be used for model multi-variate problem [49, 50], where

Original: GAR can also ﬁnd its applications.
Revised: GAR can also ﬁnd its applications.

Original: GAR is a general framework for GP-based multi-ﬁdelity fusionof high-dimensional outputs.
Revised: GAR is a general framework for GP-based multi-ﬁdelity fusionof high-dimensional outputs.

Original: Speciﬁcally, AR is a special case when setting W = ⇢ I and using aseparable kernel; ResGP is a special case of GAR by setting W
Revised: Speciﬁcally, AR is a special case when setting W = ⇢ I and using aseparable kernel; ResGP is a special case of GAR by setting W

Original: = I and S
Revised: = I and S

Original: =
Revised: =

Original: I ; NAR is a special caseof integrating out W with a normal prior and using a separable kernel; DC is a special case of GARif it only uses one latent process, integrating out W as in NAR with a separable kernel; MF-BNN is aﬁnite case of GAR if only one hidden layer is used.
Revised: I ; NAR is a special caseof integrating out W with a normal prior and using a separable kernel; DC is a special case of GARif it only uses one latent process, integrating out W as in NAR with a separable kernel; MF-BNN is aﬁnite case of GAR if only one hidden layer is used.

Original: See Appendix C for the comparison between
Revised: See Appendix C for the comparison between

Original: SOTA methods.
Revised: SOTA methods.

Original:  Experimental Results
Revised:  Experimental Results

Original: To access GAR and CIGAR, we compare them with the SOTA multi-ﬁdelity fusion methods forhigh-dimensional outputs.
Revised: To access GAR and CIGAR, we compare them with the SOTA multi-ﬁdelity fusion methods forhigh-dimensional outputs.

Original: Particularly.
Revised: Particularly.

Original: we compare to the following methods: (1) AR [3], (2) NAR[16], (3) ResGP [9], (4) DC 1 [12], and, (5) MF-BNN 2 [13].
Revised: we compare to the following methods: (1) AR [3], (2) NAR[16], (3) ResGP [9], (4) DC 1 [12], and, (5) MF-BNN 2 [13].

Original: All GPs use an RBF kernel for a faircomparison.
Revised: All GPs use an RBF kernel for a faircomparison.

Original: Because the ARD kernel is separable, the AR and NAR are accelerated using the
Revised: Because the ARD kernel is separable, the AR and NAR are accelerated using the

Original: Kronecker product structure as in GAR for a feasible computation.
Revised: Kronecker product structure as in GAR for a feasible computation.

Original: The original DC with residual
Revised: The original DC with residual

Original: PCA cannot deal with unaligned outputs, but it can do so by using an independent PCA, which wecalled
Revised: PCA cannot deal with unaligned outputs, but it can do so by using an independent PCA, which wecalled

Original: DC-I.
Revised: DC-I.

Original: Both DCs preserve 0.99% energy for dimension reductions.
Revised: Both DCs preserve 0.99% energy for dimension reductions.

Original: MF-BNN is conductedusing its default setting.
Revised: MF-BNN is conductedusing its default setting.

Original: GAR, CIGAR, AR, NAR, and ResGP are implemented using Pytorch 3 .
Revised: GAR, CIGAR, AR, NAR, and ResGP are implemented using Pytorch 3 .

Original: Allexperiments are run on a workstation with an AMD 5950x CPU and 32 GB RAM.
Revised: Allexperiments are run on a workstation with an AMD 5950x CPU and 32 GB RAM.

Original: 5.1 Multi-Fidelity Fusion for Canonical PDEs
Revised: 5.1 Multi-Fidelity Fusion for Canonical PDEs

Original: We ﬁrst access GAR in canonical PDE simulation benchmarks, which produces high-dimensionalspatial/spatial-temporal ﬁelds as model outputs.
Revised: We ﬁrst access GAR in canonical PDE simulation benchmarks, which produces high-dimensionalspatial/spatial-temporal ﬁelds as model outputs.

Original: Speciﬁcally, we test on Burgers’ , Poisson’s and Heatequations as in [12, 51–53].
Revised: Speciﬁcally, we test on Burgers’ , Poisson’s and Heatequations as in [12, 51–53].

Original: The high ﬁdelity results are obtained by solving these equations usingﬁnite difference on a 32 ⇥ 32 mesh, whereas low ﬁdelity by a 8 ⇥ 8 coarse mesh.
Revised: The high ﬁdelity results are obtained by solving these equations usingﬁnite difference on a 32 ⇥ 32 mesh, whereas low ﬁdelity by a 8 ⇥ 8 coarse mesh.

Original: The solutions onthese grid points are recorded and vectorized as outputs.
Revised: The solutions onthese grid points are recorded and vectorized as outputs.

Original: Because the mesh differs, the dimensionalityvaries.
Revised: Because the mesh differs, the dimensionalityvaries.

Original: To compare with the standard multi-ﬁdelity method that can only deal with aligned outputs,we use interpolation to upscale the low-ﬁdelity and record them at the high-ﬁdelity grid nodes.
Revised: To compare with the standard multi-ﬁdelity method that can only deal with aligned outputs,we use interpolation to upscale the low-ﬁdelity and record them at the high-ﬁdelity grid nodes.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: corresponding inputs are PDE parameters and parameterized initial or boundary conditions.
Revised: corresponding inputs are PDE parameters and parameterized initial or boundary conditions.

Original: Detailedexperimental setups can be found in Appendix E.

Revised: Detailedexperimental setups can be found in Appendix E.

Original: We uniformly generate 128 samples for testing and 32 for training.
Revised: We uniformly generate 256 samples for testing and 32 for training.

Original: We increase the high-ﬁdelitytraining samples to the number of low-ﬁdelity training samples of 32.
Revised: We increase the high-ﬁdelitytraining samples to the number of low-ﬁdelity training samples of 32.

Original: The comparisons are conductedﬁve times with shufﬂed samples.
Revised: The comparisons are conductedﬁve times with shufﬂed samples.

Original: The statistical results (mean and std) of the RMSE are reported in
Revised: The statistical results (mean and std) of the RMSE are reported in

Original: Fig.
Revised: Fig.

Original: GAR and CIGAR outperform competitors with a signiﬁcant margin, up to 6x reduction in
Revised: GAR and CIGAR outperform competitors with a signiﬁcant margin, up to 6x reduction in

Original: RMSE and also reaching the optimal with maximum 8 high-ﬁdelity samples, indicating a successfultransferring from low- to high-ﬁdelity.
Revised: RMSE and also reaching the optimal with maximum 8 high-ﬁdelity samples, indicating a successfultransferring from low- to high-ﬁdelity.

Original: CIGAR is slightly worse than GAR possibility due to lackof hard constraint on the orthogonality of its weight matrixes during implementation.
Revised: CIGAR is slightly worse than GAR possibility due to lackof hard constraint on the orthogonality of its weight matrixes during implementation.

Original: As we havediscovered in the literature, AR consistently performs well.
Revised: As we havediscovered in the literature, AR consistently performs well.

Original: With a ﬂexible linear transformation, GARoutperforms the AR while inheriting its robustness, leading to the best performance.
Revised: With a ﬂexible linear transformation, GARoutperforms the AR while inheriting its robustness, leading to the best performance.

Original: For the unalignedoutput, MF-BNN showed slightly worse performance than in the aligned cases, highlighting thechallenges of the unaligned outputs.
Revised: For the unalignedoutput, MF-BNN showed slightly worse performance than in the aligned cases, highlighting thechallenges of the unaligned outputs.

Original: In contrast, GAR and CIGAR show almost identical performancefor both cases.
Revised: In contrast, GAR and CIGAR show almost identical performancefor both cases.

Original: Nevertheless, MF-BNN also shows good performance compared to the rest of the othermethods, which is consistent with the ﬁnding in [13].
Revised: Nevertheless, MF-BNN also shows good performance compared to the rest of the othermethods, which is consistent with the ﬁnding in [13].

Original: It is interesting to see that for the non-subsetdata, the capable methods show better performances than in the subset cases.
Revised: It is interesting to see that for the non-subsetdata, the capable methods show better performances than in the subset cases.

Original: GAR and CIGAR stilloutperforms the competitors with a clear margin.
Revised: GAR and CIGAR stilloutperforms the competitors with a clear margin.

Original: To approximately assess the performance under an active learning process.
Revised: To approximately assess the performance under an active learning process.

Original: We instead generatetraining samples in a Sobol sequence [54].
Revised: We instead generatetraining samples in a Sobol sequence [54].

Original: The results are shown in Appendix E.2 where GAR and
Revised: The results are shown in Appendix E.2 where GAR and

Original: CIGAR also outperform the other methods by a large margin.
Revised: CIGAR also outperform the other methods by a large margin.

Original: 5.2 Multi-Fidelity Fusion for Real-World Applications
Revised: 5.2 Multi-Fidelity Fusion for Real-World Applications

Original: Optimal topology structure is the optimized layout of materials, e.g., alloy and concrete, given somedesign speciﬁcations, e.g., external force and angle.
Revised: Optimal topology structure is the optimized layout of materials, e.g., alloy and concrete, given somedesign speciﬁcations, e.g., external force and angle.

Original: This topology optimization is a key technique in mechanical designs, but it is also known for its high computational cost, which renders the need formulti-ﬁdelity fusion.
Revised: This topology optimization is a key technique in mechanical designs, but it is also known for its high computational cost, which renders the need formulti-ﬁdelity fusion.

Original: We consider the topology optimization of a cantilever beam with the location ofpoint load, the angle of point load, and the ﬁlter radius [55] as system inputs.
Revised: We consider the topology optimization of a cantilever beam with the location ofpoint load, the angle of point load, and the ﬁlter radius [55] as system inputs.

Original: The low-ﬁdelity usea 16 ⇥ 16 regular mesh for the ﬁnite element solver, whereas the high-ﬁdelity 64 ⇥ 64 .
Revised: The low-ﬁdelity usea 16 ⇥ 16 regular mesh for the ﬁnite element solver, whereas the high-ﬁdelity 64 ⇥ 64 .

Original: Please see
Revised: Please see

Original: Appendix E.3 for detailed setup.
Revised: Appendix E.3 for detailed setup.

Original: As in the previous experiment, the RMSE statistics against an increasing number of high-ﬁdelitytraining samples are shown in Fig.
Revised: As in the previous experiment, the RMSE statistics against an increasing number of high-ﬁdelitytraining samples are shown in Fig.

Original: It is clear that GAR outperforms other competitors with a largemargin consistently.
Revised: It is clear that GAR outperforms other competitors with a largemargin consistently.

Original: CIGAR can be as good as GAR when the number of training samples is large.
Revised: CIGAR can be as good as GAR when the number of training samples is large.

Original: Steady-state 3D solid oxide fuel cell which solves complex coupled PDEs including Ohm’s law,
Revised: Steady-state 3D solid oxide fuel cell which solves complex coupled PDEs including Ohm’s law,

Original: Navier-Stokes equations, Brinkman equation, and Maxwell-Stefan diffusion and convection simultaneously, is a key model for modern fuel cell optimization.
Revised: Navier-Stokes equations, Brinkman equation, and Maxwell-Stefan diffusion and convection simultaneously, is a key model for modern fuel cell optimization.

Original: The model was solved using ﬁnite elementin COMSOL.
Revised: The model was solved using ﬁnite elementin COMSOL.

Original: The inputs were taken to be the electrode porosities, cell voltage, temperature, andpressure in the channels.
Revised: The inputs were taken to be the electrode porosities, cell voltage, temperature, andpressure in the channels.

Original: The low-ﬁdelity experiment is conducted using 3164 elements and relativetolerance of 0.1, whereas the high-ﬁdelity uses 37064 elements and relative tolerance of 0.001.
Revised: The low-ﬁdelity experiment is conducted using 3164 elements and relativetolerance of 0.1, whereas the high-ﬁdelity uses 37064 elements and relative tolerance of 0.001.

Original: Theoutputs are the coupled ﬁelds of electrolyte current density (ECD) and ionic potential (IP) in thex  z plane located at the center of the channel.
Revised: Theoutputs are the coupled ﬁelds of electrolyte current density (ECD) and ionic potential (IP) in thex  z plane located at the center of the channel.

Original: The RMSE statistics are shown in Fig.
Revised: The RMSE statistics are shown in Fig.

Original: 3a, which, again, highlights the superiority of the proposedmethod with only four high-ﬁdelity data points.
Revised: 3a, which, again, highlights the superiority of the proposedmethod with only four high-ﬁdelity data points.

Original: To further assess the model capacity for nonstructured outputs, we keep only the ECD (Fig. 3b) and IP (Fig. 3c) in the low-ﬁdelity training datato rise the challenges of predicting high ﬁdelity ECD+IP ﬁelds.
Revised: To further assess the model capacity for nonstructured outputs, we keep only the ECD (Fig. 3b) and IP (Fig. 3c) in the low-ﬁdelity training datato rise the challenges of predicting high ﬁdelity ECD+IP ﬁelds.

Original: We can see that removing somelow-ﬁdelity information indeed increases the difﬁculties, especially when removing ECD, where
Revised: We can see that removing somelow-ﬁdelity information indeed increases the difﬁculties, especially when removing ECD, where

Original: MF-BNN outperforms GAR and CIGAR with a small number of training data.
Revised: MF-BNN outperforms GAR and CIGAR with a small number of training data.

Original: As soon as the trainingnumber increases, GAR and CIGAR become superior again.
Revised: As soon as the trainingnumber increases, GAR and CIGAR become superior again.

Original: We have conducted experiments of relative low-dimensional outputs, i.e., Plasmonic nanoparticlearrays .
Revised: We have conducted experiments of relative low-dimensional outputs, i.e., Plasmonic nanoparticlearrays .

Original: GAR and CIGAR consistently outperform other methods by a large margin (Appendix E.5).
Revised: GAR and CIGAR consistently outperform other methods by a large margin (Appendix E.5).

Original: [Figure]
Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We propose GAR, the ﬁrst AR generalization to arbitrary outputs and non-subset multi-ﬁdelity datawith a closed-form solution,and CIGAR, an efﬁcient implementation by revealing the autokrigeabilityin the AR.
Revised: We propose GAR, the ﬁrst AR generalization to arbitrary outputs and non-subset multi-ﬁdelity datawith a closed-form solution,and CIGAR, an efﬁcient implementation by revealing the autokrigeabilityin the AR.

Original: Limitation of this work is scalability w.r.t to samples, lack of active learning [13] and theapplications to the broader problems of time series and transfer learning[49, 50]
Revised: Limitation of this work is scalability w.r.t to samples, lack of active learning [13] and theapplications to the broader problems of time series and transfer learning[49, 50]

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to 533 answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to 523 answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You arestrongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of 535 your paper or providing a brief inline description.
Revised: You arestrongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of 525 your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•
Revised: [Yes] See Section ?? .•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.•
Revised: [No] The code and the data are proprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]Please do not modify the questions and only use the provided macros for your answers.
Revised: [N/A] Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the Checklist 540 section does not count towards the page limit.
Revised: Note that the Checklist 530 section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?

Original: [Yes] See contributions, abstract, and introduction
Revised: [Yes] See contributions, abstract, and introduction

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See the conclusion and complexity analysis
Revised: [Yes] See the conclusion and complexity analysis

Original: section
Revised: section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do not see an obvisou negative societal impacts as it is fundamental and quite theoretical. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/

Original: [Yes]
Revised: [Table]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Please see Appendix3.
Revised: [Yes Please see Appendix

Original: If you ran experiments...
Revised: If you experiments

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Please see supplementary
Revised: [Yes] Please see supplementary

Original: materials
Revised: materials

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Please see the Appendix
Revised: [Yes] Please see the Appendix

Original: (c) Did you report error bars (e g., with respect to the random seed after running experimentsmultiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Please see the experimental section
Revised: [Yes] Please see the experimental section

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see the experimental section
Revised: [Yes] Please see the experimental section

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Please see the experimental
Revised: [Yes] Please see the experimental

Original: section
Revised: section

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [Yes] see experiment senction and the Appendix
Revised: [Yes] see experiment senction and the Appendix

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: informa-
Revised: informa-

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original: We do not use such data
Revised: We do not use such data

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?
Revised: approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?
Revised: participant compensation?

Original: [N/A]
Revised: [N/A]