Original: Meta-Learning Dynamics Forecasting Using Task Inference
Revised: Meta-Learning Dynamics Forecasting Using Task Inference

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Current deep learning models for dynamics forecasting struggle with generalization.
Revised: Current deep learning models for dynamics forecasting struggle with generalization.

Original: They can only forecast in a specific domain and fail when applied to systemswith different parameters, external forces, or boundary conditions.
Revised: They can only forecast in a specific domain and fail when applied to systemswith different parameters, external forces, or boundary conditions.

Original: We proposea model-based meta-learning method called DyAd which can generalize acrossheterogeneous domains by partitioning them into different tasks.
Revised: We proposea model-based meta-learning method called DyAd which can generalize acrossheterogeneous domains by partitioning them into different tasks.

Original: DyAd has twoparts: an encoder which infers the time-invariant hidden features of the task withweak supervision, and a forecaster which learns the shared dynamics of the entiredomain.
Revised: DyAd has twoparts: an encoder which infers the time-invariant hidden features of the task withweak supervision, and a forecaster which learns the shared dynamics of the entiredomain.

Original: The encoder adapts and controls the forecaster during inference usingadaptive instance normalization and adaptive padding.
Revised: The encoder adapts and controls the forecaster during inference usingadaptive instance normalization and adaptive padding.

Original: Theoretically, we provethat the generalization error of such procedure is related to the task relatednessin the source domain, as well as the domain differences between source andtarget.
Revised: Theoretically, we provethat the generalization error of such procedure is related to the task relatednessin the source domain, as well as the domain differences between source andtarget.

Original: Experimentally, we demonstrate that our model outperforms state-of-the-artapproaches on forecasting complex physical dynamics including turbulent flow,real-world sea surface temperature and ocean currents.
Revised: Experimentally, we demonstrate that our model outperforms state-of-the-artapproaches on forecasting complex physical dynamics including turbulent flow,real-world sea surface temperature and ocean currents.

Original:  Introduction
Revised:  Introduction

Original: Modeling dynamical systems with deep learning has shown great success in a wide range of systemsfrom climate science, Internet of Things (IoT) to infectious diseases [23, 64, 8, 33, 42, 22].
Revised: Modeling dynamical systems with deep learning has shown great success in a wide range of systemsfrom climate science, Internet of Things (IoT) to infectious diseases [23, 64, 8, 33, 42, 22].

Original: However,the main limitation of previous works is limited generalizability.
Revised: However,the main limitation of previous works is limited generalizability.

Original: Most approaches only focus on aspecific system and train on past data in order to predict the future.
Revised: Most approaches only focus on aspecific system and train on past data in order to predict the future.

Original: Thus a new model must be trainedto predict a system with different dynamics.
Revised: Thus a new model must be trainedto predict a system with different dynamics.

Original: Consider, for example, learning fluid dynamics; shownin Fig.
Revised: Consider, for example, learning fluid dynamics; shownin Fig.

Original: 1 are two fluid flows with different degrees of turbulence.
Revised: 1 are two fluid flows with different degrees of turbulence.

Original: Even though the flows are governedby the same equations, the difference in buoyant forces would require two separate deep learningmodels to forecast.
Revised: Even though the flows are governedby the same equations, the difference in buoyant forces would require two separate deep learningmodels to forecast.

Original: Therefore, it is imperative to develop generalizable deep learning models fordynamical systems that can learn and predict well over a large heterogeneous domain.... time ?
Revised: Therefore, it is imperative to develop generalizable deep learning models fordynamical systems that can learn and predict well over a large heterogeneous domain.... time ?

Original: [Figure]
Revised: [Figure]

Original: [63, 7, 13], or learning to learn,improves generalization by learning multipletasks from the environment.
Revised: [63, 7, 13], or learning to learn,improves generalization by learning multipletasks from the environment.

Original: Recent developments in meta-learning have been successfullyapplied to few-shot classification [46, 62], active learning [77], and reinforcement learning[18, 20].
Revised: Recent developments in meta-learning have been successfullyapplied to few-shot classification [46, 62], active learning [77], and reinforcement learning[18, 20].

Original: However, meta-learning in the context of forecasting high-dimensional physical dynamics has not been really studied before.
Revised: However, meta-learning in the context of forecasting high-dimensional physical dynamics has not been really studied before.

Original: Thechallenges with meta-learning dynamical systems are unique in that (1) we need to efficientlyinfer the latent representation of the dynamical system given observed time series data, (2) we need to account for changes in unknown initial andboundary conditions, and (3) we need to model the temporal dynamics across heterogeneous domains.
Revised: Thechallenges with meta-learning dynamical systems are unique in that (1) we need to efficientlyinfer the latent representation of the dynamical system given observed time series data, (2) we need to account for changes in unknown initial andboundary conditions, and (3) we need to model the temporal dynamics across heterogeneous domains.

Original: Our approach is inspired by the fact that similar dynamical systems may share time-invariant hiddenfeatures.
Revised: Our approach is inspired by the fact that similar dynamical systems may share time-invariant hiddenfeatures.

Original: Even the slightest change in these features may lead to vastly different phenomena.
Revised: Even the slightest change in these features may lead to vastly different phenomena.

Original: Forexample, in climate science, fluids are governed by a set of differential equations called NavierStokes equations.
Revised: Forexample, in climate science, fluids are governed by a set of differential equations called NavierStokes equations.

Original: Some features such as kinematic viscosity and external forces (e.g. gravity), aretime-invariant and determine the flow characteristics.
Revised: Some features such as kinematic viscosity and external forces (e.g. gravity), aretime-invariant and determine the flow characteristics.

Original: By inferring this latent representation, we canmodel diverse system behaviors from smoothly flowing water to atmospheric turbulence.
Revised: By inferring this latent representation, we canmodel diverse system behaviors from smoothly flowing water to atmospheric turbulence.

Original: Inspired by neural style transfer [26], we propose a model-based meta-learning method, called DyAd ,which can rapidly adapt to systems with varying dynamics.
Revised: Inspired by neural style transfer [26], we propose a model-based meta-learning method, called DyAd ,which can rapidly adapt to systems with varying dynamics.

Original: DyAd has two parts, an encoder g anda forecaster f .
Revised: DyAd has two parts, an encoder g anda forecaster f .

Original: The encoder maps different dynamical systems to time-invariant hidden featuresrepresenting constants of motion, boundary conditions, and external forces which characterize thesystem.
Revised: The encoder maps different dynamical systems to time-invariant hidden featuresrepresenting constants of motion, boundary conditions, and external forces which characterize thesystem.

Original: The forecaster f then takes the learnt hidden features from the encoder and the past systemstates to forecast the future system state.
Revised: The forecaster f then takes the learnt hidden features from the encoder and the past systemstates to forecast the future system state.

Original: Controlled by the time-invariant hidden features, theforecaster has the flexibility to adapt to a wide range of systems with heterogeneous dynamics.
Revised: Controlled by the time-invariant hidden features, theforecaster has the flexibility to adapt to a wide range of systems with heterogeneous dynamics.

Original: Unlike gradient-based meta-learning techniques such as MAML [13], DyAd automatically adaptsduring inference using an encoder and does not require any retraining.
Revised: Unlike gradient-based meta-learning techniques such as MAML [13], DyAd automatically adaptsduring inference using an encoder and does not require any retraining.

Original: Similar to model-basedmeta-learning methods such as MetaNets [46], we employ a two-part design with an adaptable learnerwhich receives task-specific weights.
Revised: Similar to model-basedmeta-learning methods such as MetaNets [46], we employ a two-part design with an adaptable learnerwhich receives task-specific weights.

Original: However, for time series forecasting, since input and outputcome from the same domain, a support set of labeled data is unnecessary to define the task.
Revised: However, for time series forecasting, since input and outputcome from the same domain, a support set of labeled data is unnecessary to define the task.

Original: Theencoder can infer the task directly from query input.
Revised: Theencoder can infer the task directly from query input.

Original: Our contributions include:
Revised: Our contributions include:

Original: • A novel model-based meta-learning method ( DyAd ) for dynamic forecasting in a large
Revised: • A novel model-based meta-learning method ( DyAd ) for dynamic forecasting in a large

Original: heterogeneous domain.
Revised: heterogeneous domain.

Original: • An encoder capable of extracting the time-invariant hidden features of a dynamical systemusing time-shift invariant model structure and weak supervision.•
Revised: • An encoder capable of extracting the time-invariant hidden features of a dynamical systemusing time-shift invariant model structure and weak supervision.•

Original: A new adaptive padding layer (AdaPad), designed for adapting to boundary conditions.• Theoretical guarantees for DyAd on the generalization error of task inference in the sourcedomain as well as domain adaptation to the target domain.•
Revised: A new adaptive padding layer (AdaPad), designed for adapting to boundary conditions.• Theoretical guarantees for DyAd on the generalization error of task inference in the sourcedomain as well as domain adaptation to the target domain.•

Original: Improved generalization performance on heterogeneous domains such as fluid flow and seatemperature forecasting, even to new tasks outside the training distribution.
Revised: Improved generalization performance on heterogeneous domains such as fluid flow and seatemperature forecasting, even to new tasks outside the training distribution.

Original:  Methods
Revised:  Methods

Original: 2.1 Meta-learning in dynamics forecasting
Revised: 2.1 Meta-learning in dynamics forecasting

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be a d -dimensional state of a dynamical system governed by parameters ψ .
Revised: d be a d -dimensional state of a dynamical system governed by parameters ψ .

Original: The problemof dynamics forecasting is that given a sequence of past states ( x 1 , . . .
Revised: The problemof dynamics forecasting is that given a sequence of past states ( x t − l +1 , . . .

Original: , x t ) , we want to learn a mapf such that: f : ( x 1 , . . .
Revised: , x t ) , we want to learn amap f such that: f : ( x t − l +1 , . . .

Original: , x t ) (cid:55)−→ ( x t +1 , . . .
Revised: , x t ) (cid:55)−→ ( x t +1 , . . .

Original: x t + k ) .
Revised: x t + h ) .

Original: Here l is the length of the input series, and h is the forecasting horizon in the output.
Revised: Here l is the length of the input series, and h is the forecasting horizon in the output.

Original: Existingapproaches for dynamics forecasting only predict future data for a specific system as a single task.
Revised: Existingapproaches for dynamics forecasting only predict future data for a specific system as a single task.

Original: Here a task refers to forecasting for a specific system with a given set of parameters.
Revised: Here a task refers to forecasting for a specific system with a given set of parameters.

Original: The resultingmodels often generalize poorly to different system dynamics.
Revised: The resultingmodels often generalize poorly to different system dynamics.

Original: Thus a new model must be trained topredict for each specific system.
Revised: Thus a new model must be trained topredict for each specific system.

Original: To perform meta-learning, we identify each forecasting task by some parameters c ⊂ ψ , such asconstants of motion, external forces, and boundary conditions.
Revised: To perform meta-learning, we identify each forecasting task by some parameters c ⊂ ψ , such asconstants of motion, external forces, and boundary conditions.

Original: We learn multiple tasks simultaneouslyand infer the task from data.
Revised: We learn multiple tasks simultaneouslyand infer the task from data.

Original: Here we use c for a subset of system parameters ψ , because we usuallydo not have the full knowledge of the system dynamics.
Revised: Here we use c for a subset of system parameters ψ , because we usuallydo not have the full knowledge of the system dynamics.

Original: In the turbulent flow example, the state x tis the velocity field at time t .
Revised: In the turbulent flow example, the state x tis the velocity field at time t .

Original: Parameters c can represent Reynolds number, mean vorticity, meanmagnitude, or a vector of all three.
Revised: Parameters c can represent Reynolds number, mean vorticity, meanmagnitude, or a vector of all three.

Original: Let µ be the data distribution over X × Y representing the function f : X → Y where X = R d × tand Y =
Revised: Let µ be the data distribution over X × Y representing the function f : X → Y where X = R d × land Y =

Original: R d × k .
Revised: R d × h .

Original: Our main assumption is that the domain X can be partitioned into separate tasks
Revised: Our main assumption is that the domain X can be partitioned into separate tasks

Original: X = ∪ c ∈C X c , where X c is the domain for task c and C is the set of all tasks.
Revised: X = ∪ c ∈C X c , where X c is the domain for task c and C is the set of all tasks.

Original: The data in the sametask share the same set of parameters.
Revised: The data in the sametask share the same set of parameters.

Original: Let µ c be the conditional distribution over X c × Y for task c .
Revised: Let µ c be the conditional distribution over X c × Y for task c .

Original: During training, the model is presented with data drawn from a subset of tasks { ( x, y ) : ( x, y ) ∼µ c , c ∼ C } .
Revised: During training, the model is presented with data drawn from a subset of tasks { ( x, y ) : ( x, y ) ∼µ c , c ∼ C } .

Original: Our goal is to learn the function f : X → Y over the whole domain X which can thusgeneralize across all tasks c ∈ C .
Revised: Our goal is to learn the function f : X → Y over the whole domain X which can thusgeneralize across all tasks c ∈ C .

Original: To do so, we need to learn the map g : X → C taking x ∈ X c to cin order to infer the task with minimal supervision.
Revised: To do so, we need to learn the map g : X → C taking x ∈ X c to cin order to infer the task with minimal supervision.

Original: 2.2 DyAd : Dynamic Adaptation Network
Revised: 2.2 DyAd : Dynamic Adaptation Network

Original: We propose a model-based meta-learning approach for dynamics forecasting.
Revised: We propose a model-based meta-learning approach for dynamics forecasting.

Original: Given multiple forecasting tasks, we propose to learn the function f in two stages, that is, by first inferring the task
Revised: Given multiple forecasting tasks, we propose to learn the function f in two stages, that is, by first inferring the task

Original: c fromthe input x , and then adapting to a specialized forecaster f c : X c → Y for each task.
Revised: c fromthe input x , and then adapting to a specialized forecaster f c : X c → Y for each task.

Original: An alternativeis to use a single deep neural network to directly model f in one step over the whole domain.
Revised: An alternativeis to use a single deep neural network to directly model f in one step over the whole domain.

Original: Butthis requires the training set to have good and uniform coverage of the different tasks.
Revised: Butthis requires the training set to have good and uniform coverage of the different tasks.

Original: If the datadistribution µ is highly heterogeneous or the training set is not sampled i.i.d.
Revised: If the datadistribution µ is highly heterogeneous or the training set is not sampled i.i.d.

Original: from the whole domain,then a single model may struggle with generalization.
Revised: from the whole domain,then a single model may struggle with generalization.

Original: We hypothesize that by partitioning the domain into different tasks, the model would learn to pickup task-specific features without requiring uniform coverage of the training data.
Revised: We hypothesize that by partitioning the domain into different tasks, the model would learn to pickup task-specific features without requiring uniform coverage of the training data.

Original: Furthermore, byseparating task inference and forecasting into two stages, we allow the forecaster to rapidly adapt tonew tasks that never appeared in the training set.
Revised: Furthermore, byseparating task inference and forecasting into two stages, we allow the forecaster to rapidly adapt tonew tasks that never appeared in the training set.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, our model consists of two parts: an encoder g and a forecaster f .
Revised: 2, our model consists of two parts: an encoder g and a forecaster f .

Original: We introducez c as a time-invariant hidden feature for task c .
Revised: We introducez c as a time-invariant hidden feature for task c .

Original: We assume that c depends linearly on the hiddenfeature for simplicity and easy interpretation.
Revised: We assume that c depends linearly on the hiddenfeature for simplicity and easy interpretation.

Original: We design the encoder to infer the hidden feature z cgiven the input x .
Revised: We design the encoder to infer the hidden feature z cgiven the input x .

Original: We then use z c to adapt the forecaster f to the specific task, i.e., model y =
Revised: We then use z c to adapt the forecaster f to the specific task, i.e., model y =

Original: f c ( x )as y = f
Revised: f c ( x )as y = f

Original: ( x, z c ) .
Revised: ( x, z c ) .

Original: As the system dynamics are encoded in the input sequence x , we can feed the sameinput sequence x to a forecaster and generate predictions ˆ y =
Revised: As the system dynamics are encoded in the input sequence x , we can feed the sameinput sequence x to a forecaster and generate predictions ˆ y =

Original: f c ( x ) .
Revised: f c ( x ) .

Original: 2.3 Encoder Network
Revised: 2.3 Encoder Network

Original: The encoder maps the input x to the hidden features z c that are time-invariant.
Revised: The encoder maps the input x to the hidden features z c that are time-invariant.

Original: To enforce thisinductive bias, we encode time-invariance both in the architecture and in the training objective.
Revised: To enforce thisinductive bias, we encode time-invariance both in the architecture and in the training objective.

Original: Time-Invariant Encoder The encoder is implemented using 4 Conv 3D layers, each followed by
Revised: Time-Invariant Encoder The encoder is implemented using 4 Conv 3D layers, each followed by

Original: BatchNorm , LeakyReLU , and max-pooling.
Revised: BatchNorm , LeakyReLU , and max-pooling.

Original: Note that theoretically, max-pooling is not perfectly shiftinvariant since 2x2x2 max-pooling is equivariant to shifts of size 2 and only approximately invariantto shifts of size 1.
Revised: Note that theoretically, max-pooling is not perfectly shiftinvariant since 2x2x2 max-pooling is equivariant to shifts of size 2 and only approximately invariantto shifts of size 1.

Original: But standard convolutional architectures often include max-pooling layers to boostperformance.
Revised: But standard convolutional architectures often include max-pooling layers to boostperformance.

Original: We convolve both across spatial and temporal dimensions.
Revised: We convolve both across spatial and temporal dimensions.

Original: After that, we use a global mean-pooling layer and a fully connected layer to estimate the hiddenfeature ˆ z c .
Revised: After that, we use a global mean-pooling layer and a fully connected layer to estimate the hiddenfeature ˆ z c .

Original: The task parameter depends linearly on the hidden feature.
Revised: The task parameter depends linearly on the hidden feature.

Original: We use a fully connected layer to compute the parameter estimate ˆ c .
Revised: We use a fully connected layer to compute the parameter estimate ˆ c .

Original: Since convolutions are equivariant to shift (up to boundaryframes) and mean pooling is invariant to shift, the encoder is shift-invariant.
Revised: Since convolutions are equivariant to shift (up to boundaryframes) and mean pooling is invariant to shift, the encoder is shift-invariant.

Original: In practice, shifting thetime sequence one frame forward will add one new frame at the beginning and drop one frame at theend.
Revised: In practice, shifting thetime sequence one frame forward will add one new frame at the beginning and drop one frame at theend.

Original: This creates some change in output value of the encoder.
Revised: This creates some change in output value of the encoder.

Original: Thus, practically, the encoder is onlyapproximately shift-invariant.
Revised: Thus, practically, the encoder is onlyapproximately shift-invariant.

Original: Encoder Training The encoder network g is trained first.
Revised: Encoder Training The encoder network g is trained first.

Original: To combat the loss of shift invariance fromthe change from the boundary frames, we train the encoder using a time-invariant loss.
Revised: To combat the loss of shift invariance fromthe change from the boundary frames, we train the encoder using a time-invariant loss.

Original: Given twotraining samples ( x ( i ) , y ( i ) ) and ( x ( j ) , y ( j ) ) and their task parameters c , we have loss
Revised: Given twotraining samples ( x ( i ) , y ( i ) ) and ( x ( j ) , y ( j ) ) and their task parameters c , we have loss

Original: [Equation]
Revised: [Equation]

Original: where ˆ z ( i ) =
Revised: where ˆ z ( i ) =

Original: g ( x ( i ) ) and ˆ z ( j ) = g ( x ( j ) ) and ˆ c ( i ) =
Revised: g ( x ( i ) ) and ˆ z ( j ) = g ( x ( j ) ) and ˆ c ( i ) =

Original: W ˆ
Revised: W ˆ

Original: z
Revised: z

Original: ( i ) c
Revised: ( i ) c

Original: + b is an affine transformation of z c .
Revised: + b is an affine transformation of z c .

Original: The first term ∥ ˆ c − c ∥ 2 uses weak supervision of the task parameters whenever they are available.
Revised: The first term ∥ ˆ c − c ∥ 2 uses weak supervision of the task parameters whenever they are available.

Original: Such weak supervision helps guide the learning of hidden feature z c for each task.
Revised: Such weak supervision helps guide the learning of hidden feature z c for each task.

Original: While not allparameters of the dynamical system are known, we can compute approximate values in the datumc ( i ) based on our domain knowledge.
Revised: While not allparameters of the dynamical system are known, we can compute approximate values in the datumc ( i ) based on our domain knowledge.

Original: For example, instead of the Reynolds number of the fluid flow,we can use the average vorticity as a surrogate for task parameters.
Revised: For example, instead of the Reynolds number of the fluid flow,we can use the average vorticity as a surrogate for task parameters.

Original: The second term ∥ ˆ z ( i ) c − ˆ z ( j ) c ∥ 2 is the time-shift invariance loss, which penalizes the changes in latentvariables between samples from different time steps.
Revised: The second term ∥ ˆ z ( i ) c − ˆ z ( j ) c ∥ 2 is the time-shift invariance loss, which penalizes the changes in latentvariables between samples from different time steps.

Original: Since the time-shift invariance of convolution isonly approximate, this loss term drives the time-shift error even lower.
Revised: Since the time-shift invariance of convolution isonly approximate, this loss term drives the time-shift error even lower.

Original: The third term |∥ ˆ z ( i ) c
Revised: The third term |∥ ˆ z ( i ) c

Original: ∥ − m | 2( m is a positive value) prevents the encoder from generating small ˆ z ( i ) c
Revised: ∥ − m | 2( m is a positive value) prevents the encoder from generating small ˆ z ( i ) c

Original: due to time-shift invarianceloss.
Revised: due to time-shift invarianceloss.

Original: It also helps the encoder to learn more interesting z , even in the absence of weak supervision.
Revised: It also helps the encoder to learn more interesting z , even in the absence of weak supervision.

Original: Hidden Features The encoder learns time-invariant hidden features.
Revised: Hidden Features The encoder learns time-invariant hidden features.

Original: These hidden features resemblethe time-invariant dimensionless parameters [30] in physical modeling, such as Reynolds number influid mechanics.
Revised: These hidden features resemblethe time-invariant dimensionless parameters [30] in physical modeling, such as Reynolds number influid mechanics.

Original: The hidden features may also be viewed as partial disentanglement of the systemstate.
Revised: The hidden features may also be viewed as partial disentanglement of the systemstate.

Original: As suggested by [37, 47], our disentanglement method is guided by inductive bias and trainingobjectives.
Revised: As suggested by [37, 47], our disentanglement method is guided by inductive bias and trainingobjectives.

Original: Unlike complete disentanglement, as in e.g. [39], in which the latent representation isfactored into time-invariant and time-varying components, we focus only on time-shift-invariance.
Revised: Unlike complete disentanglement, as in e.g. [39], in which the latent representation isfactored into time-invariant and time-varying components, we focus only on time-shift-invariance.

Original: Nonetheless, the hidden features can control the forecaster which is useful for generalization.
Revised: Nonetheless, the hidden features can control the forecaster which is useful for generalization.

Original: 2.4 Forecaster Network
Revised: 2.4 Forecaster Network

Original: The forecaster incorporates the hidden feature z c from the encoder and adapts to the specific forecasting task f c = f ( · , z c ) .
Revised: The forecaster incorporates the hidden feature z c from the encoder and adapts to the specific forecasting task f c = f ( · , z c ) .

Original: In what follows, we use z for z c .
Revised: In what follows, we use z for z c .

Original: We use two specialized layers, adaptiveinstance normalization ( AdaIN ) and adaptive padding ( AdaPad ).
Revised: We use two specialized layers, adaptiveinstance normalization ( AdaIN ) and adaptive padding ( AdaPad ).

Original: AdaIN has been used in neuralstyle transfer [26, 21] to control generative networks.
Revised: AdaIN has been used in neuralstyle transfer [26, 21] to control generative networks.

Original: Here, AdaIN may adapt for specific coefficientsand external forces.
Revised: Here, AdaIN may adapt for specific coefficientsand external forces.

Original: We also introduce a new layer, AdaPad( x, z ) , which is designed for encodingthe boundary conditions of dynamical systems.
Revised: We also introduce a new layer, AdaPad( x, z ) , which is designed for encodingthe boundary conditions of dynamical systems.

Original: The backbone of the forecaster network can be anysequence prediction model.
Revised: The backbone of the forecaster network can be anysequence prediction model.

Original: We use a design that is similar to ResNet for spatiotemporal sequences.𝒙
Revised: We use a design that is similar to ResNet for spatiotemporal sequences.𝒙

Original: FC z
Revised: FC z

Original: AdaPad 𝒚
Revised: AdaPad 𝒚

Original: AdaIN We employ AdaIN to adapt the forecaster network.
Revised: AdaIN We employ AdaIN to adapt the forecaster network.

Original: Denotethe channels of input x by x i and let µ ( x i ) and σ ( x i ) be the mean andstandard deviation of channel i .
Revised: Denotethe channels of input x by x i and let µ ( x i ) and σ ( x i ) be the mean andstandard deviation of channel i .

Original: For each AdaIN layer, a particularstyle is computed s =
Revised: For each AdaIN layer, a particularstyle is computed s =

Original: ( µ i , σ i ) i =
Revised: ( µ i , σ i ) i =

Original: Az + b , where the linear map Aand bias b are learned weights.
Revised: Az + b , where the linear map Aand bias b are learned weights.

Original: Adaptive instance normalization isthen defined as y i
Revised: Adaptive instance normalization isthen defined as y i

Original: =
Revised: =

Original: σ i x i − µ ( x i ) σ ( x i ) + µ i .
Revised: σ i x i − µ ( x i ) σ ( x i ) + µ i .

Original: In essence, the channels arerenormalized to the style s .
Revised: In essence, the channels arerenormalized to the style s .

Original: For dynamics forecasting, the hidden feature z encodes data analogousto the coefficients of a differential equation and external forces onthe system.
Revised: For dynamics forecasting, the hidden feature z encodes data analogousto the coefficients of a differential equation and external forces onthe system.

Original: In numerical simulation of a differential equation thesecoefficients enter as scalings of different terms in the equation and theexternal forces are added to the combined force equation [24].
Revised: In numerical simulation of a differential equation thesecoefficients enter as scalings of different terms in the equation and theexternal forces are added to the combined force equation [24].

Original: Thus inour context AdaIN , which scales channels and adds a global vector, iswell-suited to injecting this information.
Revised: Thus inour context AdaIN , which scales channels and adds a global vector, iswell-suited to injecting this information.

Original: AdaPad To complement AdaIN , we introduce AdaPad , which encodes the boundary conditions of each specific dynamical system.
Revised: AdaPad To complement AdaIN , we introduce AdaPad , which encodes the boundary conditions of each specific dynamical system.

Original: Gen- erally when predicting dynamical systems, error is introduced along the boundaries since it isunknown how the dynamics interact with the boundary of the domain, and there may be unknowninflows or outflows.
Revised: Gen- erally when predicting dynamical systems, error is introduced along the boundaries since it isunknown how the dynamics interact with the boundary of the domain, and there may be unknowninflows or outflows.

Original: In our method, the inferred hidden feature z may contain the boundary information.
Revised: In our method, the inferred hidden feature z may contain the boundary information.

Original: AdaPad uses the hidden features to compute the boundary conditions via a linear layer.
Revised: AdaPad uses the hidden features to compute the boundary conditions via a linear layer.

Original: Then itapplies the boundary conditions as padding immediately outside the spatial domain in each layer, asshown in Fig.
Revised: Then itapplies the boundary conditions as padding immediately outside the spatial domain in each layer, asshown in Fig.

Original: Forecaster Training The forecaster is trained separately after the encoder.
Revised: Forecaster Training The forecaster is trained separately after the encoder.

Original: The kernels of theconvolutions and the mappings of the AdaIN and AdaPad layers are all trained simultaneously asthe forecaster network is trained.
Revised: The kernels of theconvolutions and the mappings of the AdaIN and AdaPad layers are all trained simultaneously asthe forecaster network is trained.

Original: Denote the true state as y and the predicted state as ˆ y , we computethe loss per time step ∥ ˆ y − y ∥ 2 for each example.
Revised: Denote the true state as y and the predicted state as ˆ y , we computethe loss per time step ∥ ˆ y − y ∥ 2 for each example.

Original: We accumulate the loss over different time stepsand generate multi-step forecasts in an autoregressive fashion.
Revised: We accumulate the loss over different time stepsand generate multi-step forecasts in an autoregressive fashion.

Original: 2.5 Theoretical Analysis
Revised: 2.5 Theoretical Analysis

Original: The high-level idea of our method is to learn a good representation of the dynamics that generalizeswell across a heterogeneous domain, and then adapt this representation to make predictions on newtasks.
Revised: The high-level idea of our method is to learn a good representation of the dynamics that generalizeswell across a heterogeneous domain, and then adapt this representation to make predictions on newtasks.

Original: Our model achieves this by learning on multiple tasks simultaneously and then adapting tonew tasks with domain transfer.
Revised: Our model achieves this by learning on multiple tasks simultaneously and then adapting tonew tasks with domain transfer.

Original: We prove that learning the tasks simultaneously as opposed toindependently results in better generalization (Proposition B.3 in Appendix).
Revised: We prove that learning the tasks simultaneously as opposed toindependently results in better generalization (Proposition B.3 in Appendix).

Original: We also provide atheoretical decomposition for the generalization error.
Revised: We also provide atheoretical decomposition for the generalization error.

Original: Specifically, our hypothesis space has the form{ x (cid:55)→ f θ ( x, g ϕ ( x )) } where ϕ and θ are the weights of the encoder and forecaster respectively.
Revised: Specifically, our hypothesis space has the form{ x (cid:55)→ f θ ( x, g ϕ ( x )) } where ϕ and θ are the weights of the encoder and forecaster respectively.

Original: Letϵ X be the error over the entire domain X , that is, for all c .
Revised: Letϵ X be the error over the entire domain X , that is, for all c .

Original: Let ϵ enc ( g ϕ ) =
Revised: Let ϵ enc ( g ϕ ) =

Original: E x ∼X
Revised: E x ∼X

Original: ( L 1 ( g ( x ) , g ϕ ( x ))be the encoder error where g : X → C is the ground truth.
Revised: ( L 1 ( g ( x ) , g ϕ ( x ))be the encoder error where g : X → C is the ground truth.

Original: Let G =
Revised: Let G =

Original: { g ϕ : X → C} be the encoderhypothesis space.
Revised: { g ϕ : X → C} be the encoderhypothesis space.

Original: Denote the empirical risk of g ϕ by ˆ ϵ enc ( g ϕ ) .
Revised: Denote the empirical risk of g ϕ by ˆ ϵ enc ( g ϕ ) .

Original: W 1 denotes the Wasserstein distancebetween tasks.
Revised: W 1 denotes the Wasserstein distancebetween tasks.

Original: R ( G ) and R ( F ) represent the Rademacher complexities of encoder and forecaster.
Revised: R ( G ) and R ( F ) represent the Rademacher complexities of encoder and forecaster.

Original: The following Proposition decomposes the generalization error in terms of forecaster error, encodererror, and distance between tasks.
Revised: The following Proposition decomposes the generalization error in terms of forecaster error, encodererror, and distance between tasks.

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: Assume the forecaster c (cid:55)→ f θ ( · , c ) is Lipschitz continuous with Lipschitz constantγ uniformly in θ and l ≤ 1 / 2 .
Revised: Assume the forecaster c (cid:55)→ f θ ( · , c ) is Lipschitz continuous with Lipschitz constantγ uniformly in θ and l ≤ 1 / 2 .

Original: Let λ c = min f ∈F
Revised: Let λ c = min f ∈F

Original: ( ϵ c ( f ) + 1 /K (cid:80)
Revised: ( ϵ c ( f ) + 1 /K (cid:80)

Original: Kk
Revised: Kk

Original: =1 ϵ c k ( f )) .
Revised: =1 ϵ c k ( f )) .

Original: For large n andprobability ≥ 1 − δ ,
Revised: For large n andprobability ≥ 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: This result helps to quantify the trade-offs with respect to different terms in the error bound in ourtwo-part architecture and the error of the model can be controlled by minimizing the empirical errorsof the encoder and forecaster.
Revised: This result helps to quantify the trade-offs with respect to different terms in the error bound in ourtwo-part architecture and the error of the model can be controlled by minimizing the empirical errorsof the encoder and forecaster.

Original: See the Appendix Proposition B.6 for full proofs.
Revised: See the Appendix Proposition B.6 for full proofs.

Original:  Related Work
Revised:  Related Work

Original: Learning Dynamical Systems.
Revised: Learning Dynamical Systems.

Original: Deep learning models are gaining popularity for learning dynamicalsystems [60, 8, 29, 5, 73, 64, 51].
Revised: Deep learning models are gaining popularity for learning dynamicalsystems [60, 8, 29, 5, 73, 64, 51].

Original: An emerging topic is physics-informed deep learning [53, 6, 11,69, 4, 3, 68, 12, 10] which integrates inductive biases from physical systems to improve learning.
Revised: An emerging topic is physics-informed deep learning [53, 6, 11,69, 4, 3, 68, 12, 10] which integrates inductive biases from physical systems to improve learning.

Original: For example, [45, 6] incorporated Koopman theory into the architecture.
Revised: For example, [45, 6] incorporated Koopman theory into the architecture.

Original: [43] used deep neuralnetworks to solve PDEs with physical laws enforced in the loss functions.
Revised: [43] used deep neuralnetworks to solve PDEs with physical laws enforced in the loss functions.

Original: [17] and [9] build modelsupon Hamiltonian and Lagrangian mechanics that respect conservation laws.
Revised: [17] and [9] build modelsupon Hamiltonian and Lagrangian mechanics that respect conservation laws.

Original: [68] proposed a hybridapproach by marrying two well-established turbulent flow simulation techniques with deep learning toproduce better prediction of turbulence.
Revised: [68] proposed a hybridapproach by marrying two well-established turbulent flow simulation techniques with deep learning toproduce better prediction of turbulence.

Original: [10] propose a physics-informed GAN architecture that usesphysics knowledge to inform the learning of both the generator and discriminator models.
Revised: [10] propose a physics-informed GAN architecture that usesphysics knowledge to inform the learning of both the generator and discriminator models.

Original: However,these approaches deal with a specific system dynamics instead of meta-learning.
Revised: However,these approaches deal with a specific system dynamics instead of meta-learning.

Original: Multi-task learning and Meta-learning.
Revised: Multi-task learning and Meta-learning.

Original: Multi-task learning [65] focuses on learning sharedrepresentations from multiple related tasks.
Revised: Multi-task learning [65] focuses on learning sharedrepresentations from multiple related tasks.

Original: Architecture-based MTL methods can be categorized intoencoder-focused [35] and decoder-focused [74].
Revised: Architecture-based MTL methods can be categorized intoencoder-focused [35] and decoder-focused [74].

Original: There are also optimization-based MTL methods,such as task balancing methods [27].
Revised: There are also optimization-based MTL methods,such as task balancing methods [27].

Original: But MTL assumes tasks are known a priori instead of inferring the task from data.
Revised: But MTL assumes tasks are known a priori instead of inferring the task from data.

Original: On the other hand, the aim of meta-learning [63] is to leverage the sharedrepresentation to fast adapt to unseen tasks.
Revised: On the other hand, the aim of meta-learning [63] is to leverage the sharedrepresentation to fast adapt to unseen tasks.

Original: Based on how the meta-level knowledge is extractedand used, meta-learning methods are classified into model-based [46, 1, 50, 59, 78], metric-based[67, 61] and gradient-based [13, 57, 16, 76].
Revised: Based on how the meta-level knowledge is extractedand used, meta-learning methods are classified into model-based [46, 1, 50, 59, 78], metric-based[67, 61] and gradient-based [13, 57, 16, 76].

Original: Most meta-learning approaches are not designed forforecasting with a few exceptions.
Revised: Most meta-learning approaches are not designed forforecasting with a few exceptions.

Original: [72] proposed to train a domain classifier with weak supervisionto help domain adaptation but focused on low-dimensioal time series classification problems.
Revised: [72] proposed to train a domain classifier with weak supervisionto help domain adaptation but focused on low-dimensioal time series classification problems.

Original: [50]designed a residual architecture for time series forecasting with a meta-learning parallel.
Revised: [50]designed a residual architecture for time series forecasting with a meta-learning parallel.

Original: [1] proposeda modular meta-learning approach for continuous control.
Revised: [1] proposeda modular meta-learning approach for continuous control.

Original: But forecasting physical dynamics posesunique challenges to meta-learning as we seek ways to encode physical knowledge into our model.
Revised: But forecasting physical dynamics posesunique challenges to meta-learning as we seek ways to encode physical knowledge into our model.

Original: Style Transfer.
Revised: Style Transfer.

Original: Our approach is inspired by neural style transfer techniques.
Revised: Our approach is inspired by neural style transfer techniques.

Original: In style-transfer, agenerative network is controlled by an external style vector though adaptive instance normalizationbetween convolutional layers.
Revised: In style-transfer, agenerative network is controlled by an external style vector though adaptive instance normalizationbetween convolutional layers.

Original: Our hidden representation bears affinity with the “style” vector instyle transfer techniques.
Revised: Our hidden representation bears affinity with the “style” vector instyle transfer techniques.

Original: Rather than aesthetic style in images, our hidden representation encodestime-invariant features.
Revised: Rather than aesthetic style in images, our hidden representation encodestime-invariant features.

Original: Style transfer initially appear in non-photorealistic rendering [31].
Revised: Style transfer initially appear in non-photorealistic rendering [31].

Original: Recently,neural style transfer [25] has been applied to image synthesis [15, 36], videos generation [56], andlanguage translation [52].
Revised: Recently,neural style transfer [25] has been applied to image synthesis [15, 36], videos generation [56], andlanguage translation [52].

Original: For dynamical systems, [58] adapts texture synthesis to transfer the styleof turbulence for animation.
Revised: For dynamical systems, [58] adapts texture synthesis to transfer the styleof turbulence for animation.

Original: [28] studies unsupervised generative modeling of turbulent flows but forsuper-resolution reconstruction rather than forecasting.
Revised: [28] studies unsupervised generative modeling of turbulent flows but forsuper-resolution reconstruction rather than forecasting.

Original: Video Prediction.
Revised: Video Prediction.

Original: Our work is also related to video prediction.
Revised: Our work is also related to video prediction.

Original: Conditioning on the historic observedframes, video prediction models are trained to predict future frames, e.g., [41, 14, 75, 66, 49, 14, 70,71, 32, 40, 34].
Revised: Conditioning on the historic observedframes, video prediction models are trained to predict future frames, e.g., [41, 14, 75, 66, 49, 14, 70,71, 32, 40, 34].

Original: There is also conditional video prediction [48] which achieves controlled synthesis.
Revised: There is also conditional video prediction [48] which achieves controlled synthesis.

Original: Many of these models are trained on natural videos from unknown physical processes.
Revised: Many of these models are trained on natural videos from unknown physical processes.

Original: Our work issubstantially different because we do not attempt to predict object or camera motions.
Revised: Our work issubstantially different because we do not attempt to predict object or camera motions.

Original: However, ourmethod can be potentially combined with video prediction models to improve generalization.
Revised: However, ourmethod can be potentially combined with video prediction models to improve generalization.

Original:  Experiments
Revised:  Experiments

Original: [Table]
Revised: [Table]

Original: 4.1 Datasets
Revised: 4.1 Datasets

Original: We experiment on three datasets: synthetic turbulent flows, real-world sea surface temperature andocean currents data.
Revised: We experiment on three datasets: synthetic turbulent flows, real-world sea surface temperature andocean currents data.

Original: These are difficult to forecast using numerical methods due to unknown externalforces and complex dynamics not fully captured by simplified mathematical models.
Revised: These are difficult to forecast using numerical methods due to unknown externalforces and complex dynamics not fully captured by simplified mathematical models.

Original: Turbulent Flow with Varying Buoyancy.
Revised: Turbulent Flow with Varying Buoyancy.

Original: We generate a synthetic dataset of turbulent flows with anumerical simulator, PhiFlow 1 .
Revised: We generate a synthetic dataset of turbulent flows with anumerical simulator, PhiFlow 1 .

Original: It contains 64 × 64 velocity fields of turbulent flows in which we vary the buoyant force acting on the fluid from 1 to 25.
Revised: It contains 64 × 64 velocity fields of turbulent flows in which we vary the buoyant force acting on the fluid from 1 to 25.

Original: Each buoyant force corresponds to a forecastingtask and there are 25 tasks in total.
Revised: Each buoyant force corresponds to a forecastingtask and there are 25 tasks in total.

Original: We use the mean vorticity of each task as partial supervision c aswe can directly calculate it from the data.
Revised: We use the mean vorticity of each task as partial supervision c aswe can directly calculate it from the data.

Original: Vorticity can characterize formation and circular motion ofturbulent flows.
Revised: Vorticity can characterize formation and circular motion ofturbulent flows.

Original: Sea Surface Temperature.
Revised: Sea Surface Temperature.

Original: We evaluate on a real-world sea surface temperature data generatedby the NEMO ocean engine [38] 2 .
Revised: We evaluate on a real-world sea surface temperature data generatedby the NEMO ocean engine [38] 2 .

Original: We select an area from Pacific ocean range from 01/01/2018to 12/31/2020.
Revised: We select an area from Pacific ocean range from 01/01/2018to 12/31/2020.

Original: The corresponding latitude and longitude are (-150 ∼ -120, -20 ∼ -50).
Revised: The corresponding latitude and longitude are (-150 ∼ -120, -20 ∼ -50).

Original: This area isthen divided into 25 64 × 64 subregions, each is a task since the mean temperature varies a lot alonglongitude and latitude.
Revised: This area isthen divided into 25 64 × 64 subregions, each is a task since the mean temperature varies a lot alonglongitude and latitude.

Original: For the encoder training, we use season as an additional supervision signalbesides the mean temperature of each subregion.
Revised: For the encoder training, we use season as an additional supervision signalbesides the mean temperature of each subregion.

Original: In other words, the encoder should be able to inferthe mean temperature of the subregion as well as to classify four seasons given the temperature series.
Revised: In other words, the encoder should be able to inferthe mean temperature of the subregion as well as to classify four seasons given the temperature series.

Original: Ocean Currents.
Revised: Ocean Currents.

Original: We also experiment with the velocity fields of ocean currents from the same regionand use the same task division as the sea surface temperature data set.
Revised: We also experiment with the velocity fields of ocean currents from the same regionand use the same task division as the sea surface temperature data set.

Original: Similar to the turbulent flowdata set, we use the mean vorticity of each subregion as the weak-supervision signal.
Revised: Similar to the turbulent flowdata set, we use the mean vorticity of each subregion as the weak-supervision signal.

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We include several SoTA baselines from meta-learning, as well as common methods for dynamicsforecasting.
Revised: We include several SoTA baselines from meta-learning, as well as common methods for dynamicsforecasting.

Original: • ResNet [19]: A widely adopted video prediction model [69].
Revised: • ResNet [19]: A widely adopted video prediction model [69].

Original: • U-net
Revised: • U-net

Original: [55]: Originally developed for biomedical image segmentation, adapted for dynamicsforecasting [11].
Revised: [55]: Originally developed for biomedical image segmentation, adapted for dynamicsforecasting [11].

Original: • ResNet/Unet-c : Above ResNet and Unet with an additional final layer that generates taskparameter c and trained with weak-supervision and forecasting loss altogether.
Revised: • ResNet/Unet-c : Above ResNet and Unet with an additional final layer that generates taskparameter c and trained with weak-supervision and forecasting loss altogether.

Original: • PredRNN [70]: A state-of-art RNNs-based spatiotemporal forecasting model.• VarSepNet [12]: A convolutional dynamics forecasting model based on spatiotemporal disentanglement.• Mod-attn [1]: A modular meta-learning method which combines the outputs of modules to generalize to new tasks using attention.• Mod-wt : A modular meta-learning variant which uses attention weights to combine the parametersof the convolutional kernels in different modules for new tasks.• MetaNet [46]: A model-based meta-learning method which requires a few samples from test tasksas a support set to adapt.• MAML [13]: A optimization-based meta-learning approach.
Revised: • PredRNN [70]: A state-of-art RNNs-based spatiotemporal forecasting model.• VarSepNet [12]: A convolutional dynamics forecasting model based on spatiotemporal disentanglement.• Mod-attn [1]: A modular meta-learning method which combines the outputs of modules to generalize to new tasks using attention.• Mod-wt : A modular meta-learning variant which uses attention weights to combine the parametersof the convolutional kernels in different modules for new tasks.• MetaNet [46]: A model-based meta-learning method which requires a few samples from test tasksas a support set to adapt.• MAML [13]: A optimization-based meta-learning approach.

Original: We replaced the original classifier witha ResNet for regression.
Revised: We replaced the original classifier witha ResNet for regression.

Original: [Table]
Revised: [Table]

Original: Model future
Revised: Model future

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Note that both ResNet-c and Unet-c have access to task parameters c .
Revised: Note that both ResNet-c and Unet-c have access to task parameters c .

Original: Modular-attn has aconvolutional encoder f that takes the same input x as each module M to generate attention weights, 282 (cid:80) ml
Revised: Modular-attn has aconvolutional encoder f that takes the same input x as each module M to generate attention weights, 282 (cid:80) ml

Original: =
Revised: =

Original: exp
Revised: exp

Original: [ f ( x )( l )] (cid:80) mk
Revised: [ f ( x )( l )] (cid:80) mk

Original: =1 exp[ f ( x )( k )]
Revised: =1 exp[ f ( x )( k )]

Original: M l ( x ) .
Revised: M l ( x ) .

Original: Modular-wt also has the same encoder but to generate weights forcombining the convolution parameters of all modules.
Revised: Modular-wt also has the same encoder but to generate weights forcombining the convolution parameters of all modules.

Original: MetaNet requires samples from test tasks as asupport set and MAML needs adaptation retraining on test tasks, while other models do not need anyinformation from the test domains.
Revised: MetaNet requires samples from test tasks as asupport set and MAML needs adaptation retraining on test tasks, while other models do not need anyinformation from the test domains.

Original: Thus, we use additional samples of up to 20% of the test set fromtest tasks.
Revised: Thus, we use additional samples of up to 20% of the test set fromtest tasks.

Original: MetaNet uses these as a support set.
Revised: MetaNet uses these as a support set.

Original: MAML is retrained on these samples for 10 epoch foradaptation.
Revised: MAML is retrained on these samples for 10 epoch foradaptation.

Original: To demonstrate the generalizability of DyAd , we experimented with ResNet , U-net anda big U-net as our forecaster.
Revised: To demonstrate the generalizability of DyAd , we experimented with ResNet , U-net anda big U-net as our forecaster.

Original: 4.3 Experiments Setup
Revised: 4.3 Experiments Setup

Original: For all datasets, we use a sliding window approach to generate samples of sequences.
Revised: For all datasets, we use a sliding window approach to generate samples of sequences.

Original: For test-future,we train and test on the same tasks but different time steps.
Revised: For test-future,we train and test on the same tasks but different time steps.

Original: For test-domain, we train and teston different tasks with an 80-20 split.
Revised: For test-domain, we train and teston different tasks with an 80-20 split.

Original: All models are trained to make next step prediction giventhe previous steps as input.
Revised: All models are trained to make next step prediction giventhe previous steps as input.

Original: We forecast in an autoregressive manner to generate multi-step aheadpredictions.
Revised: We forecast in an autoregressive manner to generate multi-step aheadpredictions.

Original: All results are averaged over 3 runs with random initialization.
Revised: All results are averaged over 3 runs with random initialization.

Original: Apart from the root mean square error (RMSE), we also report the energy spectrum error (ESE)for ocean current prediction which quantifies the physical consistency.
Revised: Apart from the root mean square error (RMSE), we also report the energy spectrum error (ESE)for ocean current prediction which quantifies the physical consistency.

Original: ESE indicates whether thepredictions preserve the correct statistical distribution and obey the energy conservation law, which isa critical metric for physical consistency.
Revised: ESE indicates whether thepredictions preserve the correct statistical distribution and obey the energy conservation law, which isa critical metric for physical consistency.

Original: See details about energy spectrum and complete experimentsdetails in Appendix A.1.
Revised: See details about energy spectrum and complete experimentsdetails in Appendix A.1.

Original:  Results
Revised:  Results

Original: 5.1 Prediction Performance.
Revised: 5.1 Prediction Performance.

Original: input
Revised: input

Original: Table 1 shows the RMSE of multi-step predictions on
Revised: Table 1 shows the RMSE of multi-step predictions on

Original: Turbulent Flows (20 steps), Sea Surface Temperature (10steps), and Ocean Currents (10 step) in two testing scenarios.
Revised: Turbulent Flows (20 steps), Sea Surface Temperature (10steps), and Ocean Currents (10 step) in two testing scenarios.

Original: We observe that DyAd makes the most accurate predictions in both scenarios across all datasets.
Revised: We observe that DyAd makes the most accurate predictions in both scenarios across all datasets.

Original: Comparing
Revised: Comparing

Original: ResNet/Unet-c with DyAd , we observe the clear advantage of task inference with separate training.
Revised: ResNet/Unet-c with DyAd , we observe the clear advantage of task inference with separate training.

Original: VarSepNetachieves competitive performances on Ocean Currents(second best) through spatiotemporal disentanglement butcannot adapt to future data.
Revised: VarSepNetachieves competitive performances on Ocean Currents(second best) through spatiotemporal disentanglement butcannot adapt to future data.

Original: Table 1 also reports ESEson real-world Ocean Currents.
Revised: Table 1 also reports ESEson real-world Ocean Currents.

Original: DyAd not only has small
Revised: DyAd not only has small

Original: RMSE but also obtains the smallest ESE, suggesting itcaptures the statistical distribution of ocean currents well.
Revised: RMSE but also obtains the smallest ESE, suggesting itcaptures the statistical distribution of ocean currents well.

Original: Figure 4 shows the target and the predicted velocity normfields ( √ u 2 + v 2 ) by Unet-c , Modular-wt and DyAd attime step 1, 5, 10 for Turbulent Flows with buoyancyfactors 9 and 21 respectively.
Revised: Figure 4 shows the target and the predicted velocity normfields ( √ u 2 + v 2 ) by Unet-c , Modular-wt and DyAd attime step 1, 5, 10 for Turbulent Flows with buoyancyfactors 9 and 21 respectively.

Original: We can see that DyAd can generate realistic flows with the corresponding characteristics while the baselines have trouble understanding and disentangling the buoyancy factor.
Revised: We can see that DyAd can generate realistic flows with the corresponding characteristics while the baselines have trouble understanding and disentangling the buoyancy factor.

Original: Figure 5 shows DyAd , ResNet , U-net , PredRNNpredictions on an ocean current sample in the future test set, and we see the shape of predictions by
Revised: Figure 5 shows DyAd , ResNet , U-net , PredRNNpredictions on an ocean current sample in the future test set, and we see the shape of predictions by

Original: DyAd is closest to the target.
Revised: DyAd is closest to the target.

Original: These results demonstrate that DyAd not only forecasts well but alsoaccurately captures the physical characteristics of the system.
Revised: These results demonstrate that DyAd not only forecasts well but alsoaccurately captures the physical characteristics of the system.

Original: 5.2 Controllable Forecast.
Revised: 5.2 Controllable Forecast.

Original: DyAd infers the hidden features from data, which allows direct control of the latent space in theforecaster.
Revised: DyAd infers the hidden features from data, which allows direct control of the latent space in theforecaster.

Original: We vary the encoder input while keeping the forecaster input fixed.
Revised: We vary the encoder input while keeping the forecaster input fixed.

Original: Figure 6 rightshows the forecasts from DyAd when the encoder is fed with flows having different buoyancy factorsc = 5 , 15 , 25 .
Revised: Figure 6 rightshows the forecasts from DyAd when the encoder is fed with flows having different buoyancy factorsc = 5 , 15 , 25 .

Original: As expected, with higher buoyancy factors, the predictions from the forecasterbecome more turbulent.
Revised: As expected, with higher buoyancy factors, the predictions from the forecasterbecome more turbulent.

Original: This demonstrates that the encoder can successfully disentangle the latentrepresentation of difference tasks, and control the predictions of the forecaster.
Revised: This demonstrates that the encoder can successfully disentangle the latentrepresentation of difference tasks, and control the predictions of the forecaster.

Original: 5.3 Ablation Study.
Revised: 5.3 Ablation Study.

Original: We performed an ablation study of DyAd on the turbulence dataset to understand the contribution ofeach component.
Revised: We performed an ablation study of DyAd on the turbulence dataset to understand the contribution ofeach component.

Original: Ablation of the Model Architecture We first performed ablation study of the model architecture,shown in Table 2.
Revised: Ablation of the Model Architecture We first performed ablation study of the model architecture,shown in Table 2.

Original: We first remove the encoder from DyAd while keeping the same forecasternetwork ( No_enc ).
Revised: We first remove the encoder from DyAd while keeping the same forecasternetwork ( No_enc ).

Original: The resulting model degrades but still outperforms ResNet .
Revised: The resulting model degrades but still outperforms ResNet .

Original: This demonstratesthe effectiveness of AdaIN and AdaPad for forecasting.
Revised: This demonstratesthe effectiveness of AdaIN and AdaPad for forecasting.

Original: We also tested DyAd with AdaIN only( No_AdaPad ), and the performance without AdaPad was slightly worse.
Revised: We also tested DyAd with AdaIN only( No_AdaPad ), and the performance without AdaPad was slightly worse.

Original: Another notable feature of our model is the ability to infer tasks with weakly supervised signals c .
Revised: Another notable feature of our model is the ability to infer tasks with weakly supervised signals c .

Original: Itis important to have a c that is related to the task domain.
Revised: Itis important to have a c that is related to the task domain.

Original: As an ablative study, we fed the encoder in
Revised: As an ablative study, we fed the encoder in

Original: DyAd with a random c , leading to Wrong_enc .
Revised: DyAd with a random c , leading to Wrong_enc .

Original: We can see that having the totally wrong supervisionmay slightly hurt the forecasting performance.
Revised: We can see that having the totally wrong supervisionmay slightly hurt the forecasting performance.

Original: We also trained the encoder and the forecaster in
Revised: We also trained the encoder and the forecaster in

Original: DyAd altogether ( End2End ) but observed worse performance.
Revised: DyAd altogether ( End2End ) but observed worse performance.

Original: This validates our hypothesis about thesignificance of domain partitioning and separate training strategy.
Revised: This validates our hypothesis about thesignificance of domain partitioning and separate training strategy.

Original: Ablation of the Encoder Training Losses.
Revised: Ablation of the Encoder Training Losses.

Original: We performed an ablation study of three training lossterms for the encoder to show the necessity of each term, shown in Table 5 in appendix A.
Revised: We performed an ablation study of three training lossterms for the encoder to show the necessity of each term, shown in Table 5 in appendix A.

Original: Wetried training the encoder only with the supervision of task parameters ( Sup_c ).
Revised: Wetried training the encoder only with the supervision of task parameters ( Sup_c ).

Original: We also testedtraining with the first two terms but without the magnitude loss ( Sup_c+time_inv ).
Revised: We also testedtraining with the first two terms but without the magnitude loss ( Sup_c+time_inv ).

Original: We can see thatmissing any of three loss terms would make the encoder fail to learn the correct task-specific andtime-invariant features.
Revised: We can see thatmissing any of three loss terms would make the encoder fail to learn the correct task-specific andtime-invariant features.

Original: Alternatives to AdaIN.
Revised: Alternatives to AdaIN.

Original: We also tested 5 different alternatives to AdaIN for injecting the hiddenfeature z c into the forecaster, and reported the results in Table 4 in Appendix A. We tested concatenating z c to the input of the forecaster, using z c as the kernels of the forecaster, concatenating z c tothe hidden states, adding z c to the hidden states, using z c as the biases in the convolutional layers inthe forecaster.
Revised: We also tested 5 different alternatives to AdaIN for injecting the hiddenfeature z c into the forecaster, and reported the results in Table 4 in Appendix A. We tested concatenating z c to the input of the forecaster, using z c as the kernels of the forecaster, concatenating z c tothe hidden states, adding z c to the hidden states, using z c as the biases in the convolutional layers inthe forecaster.

Original: AdaIN worked better than any alternative we tested.
Revised: AdaIN worked better than any alternative we tested.

Original: The implementation code of DyAd and baselines are included in the supplementary material.
Revised: The implementation code of DyAd and baselines are included in the supplementary material.

Original: Thereadme file includes the instructions of downloading PhiFlow and generating the turbulent flowdataset.
Revised: Thereadme file includes the instructions of downloading PhiFlow and generating the turbulent flowdataset.

Original: The ocean currents and sea temperature can be downloaded manually from the link providedin the Experiments section.
Revised: The ocean currents and sea temperature can be downloaded manually from the link providedin the Experiments section.

Original:  Conclusion
Revised:  Conclusion

Original: We propose a model-based meta-learning method, DyAd to forecast physical dynamics.
Revised: We propose a model-based meta-learning method, DyAd to forecast physical dynamics.

Original: DyAd uses anencoder to infer the parameters of the task and a prediction network to adapt and forecast giving theinferred task.
Revised: DyAd uses anencoder to infer the parameters of the task and a prediction network to adapt and forecast giving theinferred task.

Original: Our model can also leverage any weak supervision signals that can help distinguishdifferent tasks, allowing the incorporation of additional domain knowledge.
Revised: Our model can also leverage any weak supervision signals that can help distinguishdifferent tasks, allowing the incorporation of additional domain knowledge.

Original: On challenging turbulentflow prediction and real-world ocean temperature and currents forecasting tasks, we observe superiorperformance of our model across heterogeneous dynamics.
Revised: On challenging turbulentflow prediction and real-world ocean temperature and currents forecasting tasks, we observe superiorperformance of our model across heterogeneous dynamics.

Original: Future work would consider non-griddata such as flows on a graph or a sphere.
Revised: Future work would consider non-griddata such as flows on a graph or a sphere.

Original: A Additional Results
Revised: A Additional ResultsA.1 Experiments Details

Original: A.1 Experiments Details

Original: We use a sliding window approach to generate samples of sequences.
Revised: We use a sliding window approach to generate samples of sequences.

Original: For test-future, we train and teston the same tasks but different time steps.
Revised: For test-future, we train and teston the same tasks but different time steps.

Original: For test-domain, we train and test on different tasks withan 80%-20% split.
Revised: For test-domain, we train and test on different tasks withan 80%-20% split.

Original: All models are trained to make next step prediction given the previous steps asinput.
Revised: All models are trained to make next step prediction given the previous steps asinput.

Original: We forecast in an autoregressive manner to generate multi-step ahead predictions.
Revised: We forecast in an autoregressive manner to generate multi-step ahead predictions.

Original: All resultsare averaged over 3 runs with random initialization.
Revised: All resultsare averaged over 3 runs with random initialization.

Original: We tuned learning rate (1e-2 ∼ 1e-5), batch size (16 ∼ 64), the number of unrolled prediction steps forbackpropogation (2 ∼ 6), and hidden size (64 ∼ 512).
Revised: We tuned learning rate (1e-2 ∼ 1e-5), batch size (16 ∼ 64), the number of unrolled prediction steps forbackpropogation (2 ∼ 6), and hidden size (64 ∼ 512).

Original: We fixed the number of historic input frames as20.
Revised: We fixed the number of historic input frames as20.

Original: The models that use ResNet, including MAML , MetaNets and DyAd+ResNet , have same numberof residual blocks for fair comparison.
Revised: The models that use ResNet, including MAML , MetaNets and DyAd+ResNet , have same numberof residual blocks for fair comparison.

Original: When we trained the encoder on turbulence and sea surfacetemperature, we used α = 1 and β = 1 in the Equ. 1.
Revised: When we trained the encoder on turbulence and sea surfacetemperature, we used α = 1 and β = 1 in the Equ. 1.

Original: For ocean currents, we used α = 0 .
Revised: For ocean currents, we used α = 0 .

Original: 2 andβ = 0 .
Revised: 2 andβ = 0 .

Original: We performed all experiments on 4 V100 GPUs.
Revised: We performed all experiments on 4 V100 GPUs.

Original: Table 3 in Appendix A displays thenumber of parameters of each fine-tuned model on the turbulence dataset.
Revised: Table 3 in Appendix A displays thenumber of parameters of each fine-tuned model on the turbulence dataset.

Original: We compare our model with a series of baselines on the multi-step forecasting with different dynamics.
Revised: We compare our model with a series of baselines on the multi-step forecasting with different dynamics.

Original: We consider two testing scenarios: (1) dynamics with different initial conditions (test-future) and (2)dynamics with different parameters such as external force (test-domain).
Revised: We consider two testing scenarios: (1) dynamics with different initial conditions (test-future) and (2)dynamics with different parameters such as external force (test-domain).

Original: The first scenario evaluatesthe models’ ability to extrapolate into the future for the same task.
Revised: The first scenario evaluatesthe models’ ability to extrapolate into the future for the same task.

Original: The second scenario is to estimatethe capability of the models to generalize across different tasks.
Revised: The second scenario is to estimatethe capability of the models to generalize across different tasks.

Original: Apart from the root mean square error (RMSE), we also report the energy spectrum error (ESE)for ocean current prediction which quantifies the physical consistency.
Revised: Apart from the root mean square error (RMSE), we also report the energy spectrum error (ESE)for ocean current prediction which quantifies the physical consistency.

Original: ESE indicates whether thepredictions preserve the correct statistical distribution and obey the energy conservation law, which isa critical metric for physical consistency.
Revised: ESE indicates whether thepredictions preserve the correct statistical distribution and obey the energy conservation law, which isa critical metric for physical consistency.

Original: The turbulence kinetic energy spectrum E ( k ) is related tothe mean turbulence kinetic energy as
Revised: The turbulence kinetic energy spectrum E ( k ) is related tothe mean turbulence kinetic energy as

Original: [Equation]
Revised: [Equation]

Original: where the k is the wavenumber and t is the time step.
Revised: where the k is the wavenumber and t is the time step.

Original: The spectrum can describe the transfer ofenergy from large scales of motion to the small scales and provides a representation of the dependenceof energy on frequency.
Revised: The spectrum can describe the transfer ofenergy from large scales of motion to the small scales and provides a representation of the dependenceof energy on frequency.

Original: Thus, the ESE can indicate whether the predictions preserve the correctstatistical distribution and obey the energy conservation law.
Revised: Thus, the ESE can indicate whether the predictions preserve the correctstatistical distribution and obey the energy conservation law.

Original: A trivial example that can illustrate whywe need ESE is that if a model simply outputs moving averages of input frames,
Revised: A trivial example that can illustrate whywe need ESE is that if a model simply outputs moving averages of input frames,

Original: the accumulated
Revised: the accumulated

Original: RMSE of predictions might not be high but the ESE would be really big because all the small or evenmedium eddies are smoothed out.
Revised: RMSE of predictions might not be high but the ESE would be really big because all the small or evenmedium eddies are smoothed out.

Original: A.2 Addtional Experiments Results
Revised: A.2 Addtional Experiments Results

Original: [Table]
Revised: [Table]

Original: B Theoretical Analysis
Revised: B Theoretical Analysis

Original: [Table]
Revised: [Table]

Original: The high-level idea of our method is to learn a good representation of the dynamics that generalizeswell across a heterogeneous domain, and then adapt this representation to make predictions on newtasks.
Revised: The high-level idea of our method is to learn a good representation of the dynamics that generalizeswell across a heterogeneous domain, and then adapt this representation to make predictions on newtasks.

Original: Our model achieves this by learning on multiple tasks simultaneously and then adaptingto new tasks with domain transfer.
Revised: Our model achieves this by learning on multiple tasks simultaneously and then adaptingto new tasks with domain transfer.

Original: We prove that learning the tasks simultaneously as opposed
Revised: We prove that learning the tasks simultaneously as opposed

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: to independently results in tighter generalization bounds.
Revised: to independently results in tighter generalization bounds.

Original: As these are upper bounds, they do notnecessarily imply better generalization, although empirically, this is what we observe.
Revised: As these are upper bounds, they do notnecessarily imply better generalization, although empirically, this is what we observe.

Original: Althoughwe are not able to provide a proof of our empirical finding that an encoder-forecaster architectureoutperforms a monolithic architecture, we do provide an analysis of the error, decomposing it intoerror due to the forecaster, encoder, and domain adaptation.
Revised: Althoughwe are not able to provide a proof of our empirical finding that an encoder-forecaster architectureoutperforms a monolithic architecture, we do provide an analysis of the error, decomposing it intoerror due to the forecaster, encoder, and domain adaptation.

Original: This decomposition helps to quantify thetrade-offs involved in our two-part architecture
Revised: This decomposition helps to quantify thetrade-offs involved in our two-part architecture.

Original: [Table] Suppose we have K tasks { c k } Kk =1 ∼ C , each of which is sampled from a continuous parameterspace C .
Revised: Suppose we have K tasks { c k } Kk =1 ∼ C , each of which is sampled from a continuous parameterspace C .

Original: Here c k are the parameters of the task, which can be inferred by the encoder.
Revised: Here c k are the parameters of the task, which can be inferred by the encoder.

Original: For each taskc k , we have a collection of data ˆ µ c k of size n , sampled from µ k , a shorthand for µ c k .
Revised: For each taskc k , we have a collection of data ˆ µ c k of size n , sampled from µ k , a shorthand for µ c k .

Original: B.1 Multi-Task Learning Error
Revised: B.1 Multi-Task Learning Error

Original: We want to bound the true loss ϵ using the empirical loss ˆ ϵ and Rademacher complexity of thehypothesis class F .
Revised: We want to bound the true loss ϵ using the empirical loss ˆ ϵ and Rademacher complexity of thehypothesis class F .

Original: We can use the classic results from [2].
Revised: We can use the classic results from [2].

Original: Define empirical Rademacher complexityfor all tasks as
Revised: Define empirical Rademacher complexityfor all tasks as

Original: [Equation]
Revised: [Equation]

Original: where { σ ( i ) k } are independent binary variables σ ( i ) k ∈ {− 1 , 1 } .
Revised: where { σ ( i ) k } are independent binary variables σ ( i ) k ∈ {− 1 , 1 } .

Original: The true Rademacher complexity isthen defined R ( F ) =
Revised: The true Rademacher complexity isthen defined R ( F ) =

Original: E X
Revised: E X

Original: ( ˆ R X ( F )) .
Revised: ( ˆ R X ( F )) .

Original: The following theorem restates the main result from [2].
Revised: The following theorem restates the main result from [2].

Original: We simplify the statement by using
Revised: We simplify the statement by using

Original: Rademacher complexity rather than the set cover number argument used in the original proof.
Revised: Rademacher complexity rather than the set cover number argument used in the original proof.

Original: Theassumption of bounded loss is reasonable given a prescribed data domain and data preprocessing.
Revised: Theassumption of bounded loss is reasonable given a prescribed data domain and data preprocessing.

Original: Theorem B.1.
Revised: Theorem B.1.

Original: ([2]) Assume the loss is bounded l ≤ 1 / 2 .
Revised: ([2]) Assume the loss is bounded l ≤ 1 / 2 .

Original: Given n samples each from K differentforecasting tasks µ 1 , · · · , µ k , then with probability at least 1 − δ , the following inequality holds foreach f ∈ F in the hypothesis class:
Revised: Given n samples each from K differentforecasting tasks µ 1 , · · · , µ k , then with probability at least 1 − δ , the following inequality holds foreach f ∈ F in the hypothesis class:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider { x ( i ) k } as independent random variables.
Revised: Consider { x ( i ) k } as independent random variables.

Original: For a function ϕ that satisfies| ϕ ( x (1) , · · · , x ( i ) , · · · x ( n ) ) − ϕ ( x (1) , · · · , ˜x ( i ) , · · · x ( n ) ) | ≤ c iby McDiarmid’s inequality, we have
Revised: For a function ϕ that satisfies| ϕ ( x (1) , · · · , x ( i ) , · · · x ( n ) ) − ϕ ( x (1) , · · · , ˜x ( i ) , · · · x ( n ) ) | ≤ c iby McDiarmid’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Applying this inequality to the max difference Q ( X ) =
Revised: Applying this inequality to the max difference Q ( X ) =

Original: sup f ∈F [ ϵ ( f ) − ˆ ϵ ( f, X )] , then with probabilityat least 1 − δ , we have
Revised: sup f ∈F [ ϵ ( f ) − ˆ ϵ ( f, X )] , then with probabilityat least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where C is a constant depending on the bounds
Revised: where C is a constant depending on the bounds

Original: c i .
Revised: c i .

Original: If the loss l ≤ 1 / 2 , then | Q | ≤ 1 / 2 and so we cantake c
Revised: If the loss l ≤ 1 / 2 , then | Q | ≤ 1 / 2 and so we cantake c

Original: i = 1 leading to C = 1 / √ 2 .
Revised: i = 1 leading to C = 1 / √ 2 .

Original: A standard computation (see [44] Theorem 3.3) using the law oftotal expectation shows E X [ Q ( X )] ≤ 2 R ( F ) , which finishes the proof.
Revised: A standard computation (see [44] Theorem 3.3) using the law oftotal expectation shows E X [ Q ( X )] ≤ 2 R ( F ) , which finishes the proof.

Original: The following inequality compares the performance for multi-task learning to learning individualtasks.
Revised: The following inequality compares the performance for multi-task learning to learning individualtasks.

Original: Let R k ( F ) be the Rademacher complexity for F over µ k .
Revised: Let R k ( F ) be the Rademacher complexity for F over µ k .

Original: Lemma B.2.
Revised: Lemma B.2.

Original: The Rademacher complexity for multi-task learning is bounded R ( F ) ≤(1 /K )
Revised: The Rademacher complexity for multi-task learning is bounded R ( F ) ≤(1 /K )

Original: (cid:80) Kk =1 R k ( F ) .
Revised: (cid:80) Kk =1 R k ( F ) .

Original: Proof.
Revised: Proof.

Original: We compute the empirical Rademacher complexity,
Revised: We compute the empirical Rademacher complexity,

Revised: [Equation]

Revised: (cid:17)(cid:17)(cid:33)(cid:35)

Revised: [Equation]

Revised: (cid:17)(cid:17)(cid:33)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: Proposition B.3.
Revised: Proposition B.3.

Original: Assume n = n k for all tasks k
Revised: Assume n = n k for all tasks k

Original: and the loss l is bounded l ≤ 1 / 2 , then thegeneralization bound given by considering each task individually is
Revised: and the loss l is bounded l ≤ 1 / 2 , then thegeneralization bound given by considering each task individually is

Original: [Equation]
Revised: [Equation]

Original: which is weaker than the bound of Theorem B.1.
Revised: which is weaker than the bound of Theorem B.1.

Original: Proof.
Revised: Proof.

Original: Applying the classical analog of Theorem B.1 to a single task, we find with probability greaterthan 1 − δ ,
Revised: Applying the classical analog of Theorem B.1 to a single task, we find with probability greaterthan 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Averaging over all tasks yields
Revised: Averaging over all tasks yields

Original: [Equation]
Revised: [Equation]

Original: Since the loss l is bounded l ≤ 1 / 2 , we can take C =
Revised: Since the loss l is bounded l ≤ 1 / 2 , we can take C =

Original: C k = 1 / √ 2 , giving the generalization upperbound for the joint error of Equation 3.
Revised: C k = 1 / √ 2 , giving the generalization upperbound for the joint error of Equation 3.

Original: By Lemma B.2 and the fact 1 / √ 2 nK ≤ 1 / √ 2 n , the boundin Theorem B.1 is strictly tighter.
Revised: By Lemma B.2 and the fact 1 / √ 2 nK ≤ 1 / √ 2 n , the boundin Theorem B.1 is strictly tighter.

Original: The upper bound in Theorem B.1 is strictly tighter than that of Proposition B.3 as the first terms ˆ ϵ ( f )are equal, the second term is smaller by Lemma B.2 and the third is smaller since 1 / √ 2 nK ≤ 1 / √ 2 n .
Revised: The upper bound in Theorem B.1 is strictly tighter than that of Proposition B.3 as the first terms ˆ ϵ ( f )are equal, the second term is smaller by Lemma B.2 and the third is smaller since 1 / √ 2 nK ≤ 1 / √ 2 n .

Original: This helps explain why our multitask learning forecaster has better generalization than learning eachtask independently.
Revised: This helps explain why our multitask learning forecaster has better generalization than learning eachtask independently.

Original: The shared data tightens the generalization bound.
Revised: The shared data tightens the generalization bound.

Original: Ultimately, though a tighterupper bound suggests lower error, it does not strictly imply it.
Revised: Ultimately, though a tighterupper bound suggests lower error, it does not strictly imply it.

Original: We further verify this theory in ourexperiments by comparison to baselines which learn each task independently.
Revised: We further verify this theory in ourexperiments by comparison to baselines which learn each task independently.

Original: B.2 Domain Adaptation Error
Revised: B.2 Domain Adaptation Error

Original: Since we test on c ∼ C outside the training set { c k } , we incur error due to domain adaptation fromthe source domains µ c 1 , . .
Revised: Since we test on c ∼ C outside the training set { c k } , we incur error due to domain adaptation fromthe source domains µ c 1 , . .

Original: . , µ c K to target domain µ c with µ being the true distribution.
Revised: . , µ c K to target domain µ c with µ being the true distribution.

Original: Denote thecorresponding empirical distributions of n samples per task by ˆ µ c = 1 n c (cid:80) n c i =

Revised: Denote thecorresponding empirical distributions of n samples per task by ˆ µ c = 1 n c (cid:80) n c i =

Original: δ x ( i ) c .
Revised: δ x ( i ) c .

Original: For different c and c ′ , the domains µ c and µ c ′ may have largely disjoint support, leading to veryhigh KL divergence.
Revised: For different c and c ′ , the domains µ c and µ c ′ may have largely disjoint support, leading to veryhigh KL divergence.

Original: However, if c and c ′ are close, samples x c ∼ µ c and x c ′ ∼ µ c ′ may be closein the domain X with respect to the metric ∥ · ∥ X .
Revised: However, if c and c ′ are close, samples x c ∼ µ c and x c ′ ∼ µ c ′ may be closein the domain X with respect to the metric ∥ · ∥ X .

Original: For example, if the external forces c and c ′ areclose, given x c ∼ µ c there is likely x c ′ ∼ µ c ′
Revised: For example, if the external forces c and c ′ areclose, given x c ∼ µ c there is likely x c ′ ∼ µ c ′

Original: such that the distance between the velocity fields∥ x
Revised: such that the distance between the velocity fields∥ x

Original: c − x c ′ ∥ is small.
Revised: c − x c ′ ∥ is small.

Original: This implies the distributions µ c and µ c ′ may be be close in Wasserstein distance
Revised: This implies the distributions µ c and µ c ′ may be be close in Wasserstein distance

Original: W 1 ( µ c , µ c ′ ) .
Revised: W 1 ( µ c , µ c ′ ) .

Original: The bound from [54] applies well to our setting as such:
Revised: The bound from [54] applies well to our setting as such:

Original: Theorem B.4.
Revised: Theorem B.4.

Original: Let λ c = min f ∈F (cid:16) ϵ c ( f ) + 1 /K (cid:80)
Revised: Let λ c = min f ∈F (cid:16) ϵ c ( f ) + 1 /K (cid:80)

Original: Kk
Revised: Kk

Original: =1 ϵ c k
Revised: =1 ϵ c k

Original: ( f ) (cid:17) .
Revised: ( f ) (cid:17) .

Original: There is N = N (dim( X )) suchthat for n > N , for any hypothesis f , with probability at least 1 − δ ,
Revised: There is N = N (dim( X )) suchthat for n > N , for any hypothesis f , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We apply [54] Theorem 2 to target domain µ T = µ c and joint source domain µ S =1 /K
Revised: We apply [54] Theorem 2 to target domain µ T = µ c and joint source domain µ S =1 /K

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =1 µ c k with empirical samples ˆ µ T =
Revised: =1 µ c k with empirical samples ˆ µ T =

Original: ˆ µ c and ˆ µ S = 1 /K (cid:80)
Revised: ˆ µ c and ˆ µ S = 1 /K (cid:80)

Original: Kk
Revised: Kk

Original: =

Revised: =

Original: ˆ µ c k .
Revised: ˆ µ c k .

Original: B.3 Encoder versus Prediction Network Error
Revised: B.3 Encoder versus Prediction Network Error

Original: Our goal is to learn a joint hypothesis h over the entire domain X in two steps, first inferring the taskc and then inferring x t +1 conditioned on c .
Revised: Our goal is to learn a joint hypothesis h over the entire domain X in two steps, first inferring the taskc and then inferring x t +1 conditioned on c .

Original: Error from DyAd may result from either the encoder g ϕor the prediction network f θ .
Revised: Error from DyAd may result from either the encoder g ϕor the prediction network f θ .

Original: Our hypothesis space has the form { x
Revised: Our hypothesis space has the form { x

Original: (cid:55)→ f θ ( x, g ϕ ( x )) } where ϕ andθ are the weights of the encoder and prediction network respectively.
Revised: (cid:55)→ f θ ( x, g ϕ ( x )) } where ϕ andθ are the weights of the encoder and prediction network respectively.

Original: Let ϵ X be the error over theentire domain X , that is, for all c .
Revised: Let ϵ X be the error over theentire domain X , that is, for all c .

Original: Let ϵ enc ( g ϕ ) =
Revised: Let ϵ enc ( g ϕ ) =

Original: E x ∼X
Revised: E x ∼X

Original: ( L 1 ( g ( x ) , g ϕ ( x )) be the encoder error whereg :
Revised: ( L 1 ( g ( x ) , g ϕ ( x )) be the encoder error whereg :

Original: X → C is the ground truth.
Revised: X → C is the ground truth.

Original: We state a result that decomposes the final error into that attributableto the encoder and that to the forecaster.
Revised: We state a result that decomposes the final error into that attributableto the encoder and that to the forecaster.

Original: Proposition B.5.
Revised: Proposition B.5.

Original: Assume c
Revised: Assume c

Original: (cid:55)→ f θ ( · , c ) is Lipschitz continuous with Lipschitz constant γ uniformly inθ .
Revised: (cid:55)→ f θ ( · , c ) is Lipschitz continuous with Lipschitz constant γ uniformly inθ .

Original: Then we bound
Revised: Then we bound

Original: [Equation]
Revised: [Equation]

Original: where the first term is the error due to the encoder incorrectly identifying the task and the secondterm is the error due the prediction network alone.
Revised: where the first term is the error due to the encoder incorrectly identifying the task and the secondterm is the error due the prediction network alone.

Original: The hypothesis in the second term consists of the prediction network combined with the ground truthtask label x
Revised: The hypothesis in the second term consists of the prediction network combined with the ground truthtask label x

Original: (cid:55)→ f θ ( x, g ( x )) .
Revised: (cid:55)→ f θ ( x, g ( x )) .

Original: In practice, the Lipschitz constant γ of the encoder may be indirectlyminimized through weight decay.
Revised: In practice, the Lipschitz constant γ of the encoder may be indirectlyminimized through weight decay.

Original: Proof.
Revised: Proof.

Original: By the triangle inequality and linearity of expectation,
Revised: By the triangle inequality and linearity of expectation,

Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Original: By Lipschitz continuity,
Revised: By Lipschitz continuity,

Original: [Equation]
Revised: [Equation]

Original: which, since g ( x ) = c and by linearity of expectation,
Revised: which, since g ( x ) = c and by linearity of expectation,

Original: [Equation]
Revised: [Equation]

Original: by definition of ϵ enc and ϵ c , = γϵ enc ( g ϕ )
Revised: by definition of ϵ enc and ϵ c , = γϵ enc ( g ϕ )

Original: + E c ∼C [ ϵ c
Revised: + E c ∼C [ ϵ c

Revised: [Equation]

Original: ( f θ ( x, c ))] .
Revised: f θ ( x, c ))] .

Original: By combining Theorem B.1, Proposition B.5, and Theorem B.4, we can bound the generalizationerror in terms of the empirical error of the prediction network on the source domains, the Wassersteindistance between the source and target domains, and the empirical error of the encoder.
Revised: By combining Theorem B.1, Proposition B.5, and Theorem B.4, we can bound the generalizationerror in terms of the empirical error of the prediction network on the source domains, the Wassersteindistance between the source and target domains, and the empirical error of the encoder.

Original: Let G =
Revised: Let G =

Original: { g ϕ : X → C} be the task encoder hypothesis space.
Revised: { g ϕ : X → C} be the task encoder hypothesis space.

Original: Denote the empirical risk of the encoder g ϕwith respect to X by ˆ ϵ enc ( g ϕ ) .
Revised: Denote the empirical risk of the encoder g ϕwith respect to X by ˆ ϵ enc ( g ϕ ) .

Original: Proposition B.6.
Revised: Proposition B.6.

Original: Assuming the hypotheses of Theorem B.1, Proposition B.5, and Theorem B.4,
Revised: Assuming the hypotheses of Theorem B.1, Proposition B.5, and Theorem B.4,

Original: [Equation]
Revised: [Equation]

Original:  Proof.
Revised: 744 Proof.

Original: We provide an upper bound for the right-hand side of (4).
Revised: We provide an upper bound for the right-hand side of (4).

Original: By Theorem B.1 or [44], Theorem3.3, we can bound
Revised: By Theorem B.1 or [44], Theorem3.3, we can bound

Original: [Equation]
Revised: [Equation]

Original: In order to apply Theorem B.1 to the risk ϵ c and relate it to the empirical risk [Equation] we first relate the erroron the target domain back to the source domain of our empirical samples.
Revised: In order to apply Theorem B.1 to the risk ϵ c and relate it to the empirical risk, we first relate the erroron the target domain back to the source domain of our empirical samples.

Original: By Theorem B.4,
Revised: By Theorem B.4,

Original: [Equation]
Revised: [Equation]

Original: Applying Theorem B [Equation] [Equation] this is
Revised: Applying Theorem B.1, this is

Original: [Equation]
Revised: [Equation]

Original: Substituting (5) and (7) into (4) gives
Revised: Substituting (5) and (7) into (4) gives

Original: [Equation]
Revised: [Equation]

Original: (cid:88) k

Original: =

Original: [Equation]

Original: f

Original: [Equation]

Original: F

Original: [Equation]

Original: 2 nK

Original: [Equation]

Original: K

Original: [Equation]

Original: (cid:35) .

Original: Using linearity of the expectation over c , removing it where there is no dependence on c , andrearranging terms.
Revised: Using linearity of the expectation over c , removing it where there is no dependence on c , andrearranging terms.

Original: Although this result does not settle either the question of end-to-end versus pre-training or encoderforecaster versus monolithic model, it quantifies the trade-offs these choices depend on.
Revised: Although this result does not settle either the question of end-to-end versus pre-training or encoderforecaster versus monolithic model, it quantifies the trade-offs these choices depend on.

Original: We empirically consider both questions in the experiments section.
Revised: We empirically consider both questions in the experiments section.