NOMAD: Nonlinear Manifold Decoders for Operator Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Supervised learning in function spaces is an emerging area of machine learningresearch with applications to the prediction of complex physical systems such asfluid flows, solid mechanics, and climate modeling.
By directly learning maps(operators) between infinite dimensional function spaces, these models are able tolearn discretization invariant representations of target functions.
A common approach is to represent such target functions as linear combinations of basis elementslearned from data.
However, there are simple scenarios where, even though thetarget functions form a low dimensional submanifold, a very large number of basiselements is needed for an accurate linear representation.
Here we present NOMAD,a novel operator learning framework with a nonlinear decoder map capable oflearning finite dimensional representations of nonlinear submanifolds in functionspaces.
We show this method is able to accurately learn low dimensional representations of solution manifolds to partial differential equations while outperforminglinear models of larger size.
Additionally, we compare to state-of-the-art operatorlearning methods on a complex fluid dynamics benchmark and achieve competitiveperformance with a significantly smaller model size and training cost.
Introduction
[Figure]
Machine learning techniqueshave been applied to great success for modeling functions between finite dimensional vectorspaces.
For example, in computer vision (vectors of pixel values) and natural language processing (vectors of word embeddings) these methods have produced state-of-the-art results inimage recognition [14] and translation tasks [36].
However, notall data has an obvious and faithful representation as finite dimensional vectors.
In particular, functional data is mathematically represented as a vector in an infinite dimensional vector space.
Thiskind of data appears naturally in problems coming from physics, where scenarios in fluid dynamics,solid mechanics, and kinematics are described by functions of continuous quantities.
Supervised learning in the infinite dimensional setting can be considered for cases where we wantto map functional inputs to target functional outputs.
For example, we might wish to predict the velocity of a fluid as function of time given an initial velocity field, or predict the pressure field across the surface of the Earth given temperature measurements.
This is similar to a finite dimensionalregression problem, except that we are now interested in learning an operator between spaces offunctions.
We refer to this as a supervised operator learning problem : given a data-set of N pairsof functions { ( u 1 , s 1 ) , . . .
, ( u N , s N ) } , learn an operator F which maps input functions to outputfunctions such that F ( u i )
=
s i , ∀ i .
One approach to solve the supervised operator learning problem is to introduce a parameterizedoperator architecture and train it to minimize a loss between the model’s predicted functions andthe true target functions in the training set.
One of the first operator network architectures waspresented in [6] with accompanying universal approximation guarantees in the uniform norm.
Theseresults were adapted to deep networks in [24] and led to the DeepONet architecture and its variants[39, 25, 15].
The Neural Operator architecture, motivated by the composition of linear and nonlinearlayers in neural networks, was proposed in [21].
Using the Fourier convolution theorem to computethe integral transform in Neural Operators led to the Fourier Neural Operator [22].
Other recentarchitectures include approaches based on PCA-based representations [1], random feature approaches[27], wavelet approximations to integral transforms [13], and attention-based architectures [17].
A common feature shared among many of these approaches is that they aim to approximate anoperator using three maps: an encoder, an approximator, and a decoder, see Figure 1 and Section 3for more details.
In all existing approaches embracing this structure, the decoder is constructed as alinear map.
In doing so, the set of target functions is being approximated with a finite dimensionallinear subspace in the ambient target function space.
Under this setting, the universal approximationtheorems of [6, 18, 19] guarantee that there exists a linear subspace of a large enough dimensionwhich approximates the target functions to any prescribed accuracy.
However, as with finite dimensional data, there are scenarios where the target functional dataconcentrates on a low dimensional nonlinear submanifold.
We refer to the phenomenon of data infunction spaces concentrating on low dimensional submanifolds as the Operator Learning Manifold
Hypothesis , see Figure 1.
For example, it is known that certain classes of parametric partial differentialequations admit low dimensional nonlinear manifolds of solution functions [7].
Although linearrepresentations can be guaranteed to approximate these spaces, their required dimension can becomevery large and thus inefficient in capturing the true low dimensional structure of the data.
In this paper, we are motivated by the Operator Learning Manifold Hypothesis to formulate a newclass of operator learning architectures with nonlinear decoders.
Our key contributions can besummarized as follows.
• Limitations of Linear Decoders: We describe in detail the shortcomings of operator
learning methods with linear decoders and present some fundamental lower bounds alongwith an illustrative operator learning problem which is subject to these limitations.
• Nonlinear Manifold Decoders (NOMAD):
This motivates a novel operator learning framework with a nonlinear decoder that can find low dimensional representations for finitedimensional nonlinear submanifolds in function spaces.
• Enhanced Dimensionality Reduction: A collection of numerical experiments involvinglinear transport and nonlinear wave propagation shows that, by learning nonlinear submanifolds of target functions, we can build models that achieve state-of-the-art accuracy whilerequiring a significantly smaller number of latent dimensions.
• Enhanced Computational Efficiency: As a consequence, the resulting architectures contain
a significantly smaller number of trainable parameters and their training cost is greatlyreduced compared to competing linear approaches.
We begin our presentation in Section 2 by providing a taxonomy of representative works in theliterature.
In Section 3 we formally define the supervised operator learning problem and discussexisting approximation strategies, with a focus on highlighting open challenges and limitations.
In
Section 4 we present the main contributions of this work and illustrate their utility through the lensof a pedagogical example.
In Section 5 we provide a comprehensive collection of experimentsthat demonstrate the performance of using NOMAD against competing state-of-the-art methods foroperator learning.
Section 6 summarizes our main findings and discusses lingering limitations andbroader impact.
Additional details on architectures, hyperparameter selection, and training details areprovided in the Supplemental Materials.
Related Work in Dimensionality Reduction
Low Dimensional Representations in Finite Dimensional Vector Spaces
: Finding low dimensional representations of high dimensional data has a long history, going back to 1901 with theoriginal formulation of principal components analysis (PCA) [28].
PCA is a linear method that worksbest when data concentrates on low dimensional subspaces.
When data instead concentrates onlow dimensional nonlinear spaces, kernelized PCA [31] and manifold learning techniques such as
Isomap and diffusion maps [34, 9] can be effective in finding nonlinear low dimensional structure,see [35] for a review.
The recent popularity of deep learning has introduced new methods for findinglow dimensional structure in high dimensional data-sets, most notably using auto-encoders [40, 4]and deep generative models [10, 16].
Relevant to our work, such techniques have found success inapproximating submanifolds in vector spaces corresponding to discretized solutions of parametricpartial differential equations (PDEs) [32, 30, 12], where a particular need for nonlinear dimensionreduction arises in advection-dominated problems common to fluid mechanics and climate science[20, 26].
Low Dimensional Representations in Infinite Dimensional Vector Spaces: The principles behind PCA generalize in a straightforward way to functions residing in low dimensional subspaces of infinitedimensional Hilbert spaces [38].
In the field of reduced order modeling of PDEs this is sometimesreferred to as proper orthogonal decomposition [5] (see [23] for an interesting exposition of thediscrete version and connections to the Karhunen-Loève decomposition).
Affine representations ofsolution manifolds to parametric PDEs and guarantees on when they are effective using the notionof linear n -widths [29] have been explored in [7].
As in the case of finite dimensional data, usinga kernel to create a feature representation of a set of functions, and then performing PCA in theassociated Reproducing Kernel Hilbert Space can give nonlinear low dimensional representations[33].
The theory behind optimal nonlinear low dimensional representations for sets of functions isstill being developed, but there has been work towards defining what “optimal” should mean in thiscontext and how it relates to more familiar geometric quantities [8].
Operator Learning
Notation: Let us first set up some notation and give a formal statement of the supervised operatorlearning problem.
We define C ( X ; R d ) as the set of continuous functions from a set X to R d .
When
X ⊂ R n , we define the Hilbert space,
[Equation]
This is an infinite dimensional vector space equipped with the inner product ⟨ f, g ⟩ = (cid:82) X f ( x ) g
( x ) dx .
When X is compact, we have that C ( X ; R d ) ⊂
L 2 ( X ; R d ) .
We now can present a formal statementof the supervised operator learning problem.
Problem Formulation: Suppose we are given a training data-set of N pairs of functions ( u i , s i ) ,where u i ∈ C
( X ; R d u ) with compact X ⊂ R d x , and s
i ∈ C
( Y ; R d s ) with compact Y
⊂ R d y .
Assume there is a ground truth operator G : C ( X ; R d u ) →
C ( Y ; R d s ) such that G ( u i ) =
s i andthat the u
i are sampled i.i.d.
from a probability measure on C ( X ; R d u ) .
The goal of the supervisedoperator learning problem is to learn a continuous operator F : C ( X ; R d x ) → C ( Y ; R d s ) to 128 approximate G .
To do so, we will attempt to minimize the following empirical risk over a class ofoperators F θ , with parameters θ ∈ Θ
⊂ R d θ ,
[Equation]
An Approximation Framework for Operators: A popular approach to learning an operator
G : L [Equation] on a probability measure µ on L 2 ( X ) is to construct an approximation 132 out of three maps [19] (see Figure 1),
[Equation]
The first map, E : L 2 ( X ) → R m is known as the encoder.
It takes an input function and mapsit to a finite dimensional feature representation.
For example, E could take a continuous functionto its point-wise evaluations along a collection of m sensors, or project a function onto m basisfunctions.
The next map A : R m → R n is known as the approximation map.
This can be interpretedas a finite dimensional approximation of the action of the operator G .
Finally, the image of theapproximation map is used to create the output functions in L 2 ( Y ) by means of the decoding map
D : R n → L 2 ( Y ) .
We will refer to the dimension, n , of the domain of the decoder as the latent 140 dimension .
The composition of these maps can be visualized in the following diagram.
[Equation]
Linear Decoders: Many successful operator learning architectures such as the DeepONet [24],the (pseudo-spectral) Fourier Neural Operator in [18], LOCA [17], and the PCA-based method in[1] all use linear decoding maps D .
A linear D can be defined by a set of functions τ
i ∈
L 2 ( Y ) ,i = 1 , . .
. , n , and acts on a vector β ∈ R n as
[Equation]
For example, the functions τ i can be built using trigonometric polynomials as in the Ψ -FNO [18], beparameterized by a neural network as in DeepONet [24], or created as the normalized output of akernel integral transform as in LOCA [17].
Limitations of Linear Decoders: We can measure the approximation accuracy of the operator Fwith two different norms.
First is the L 2 ( µ ) operator norm,
[Equation]
Note that the empirical risk used to train a model for the supervised operator learning problem (see(1)) is a Monte Carlo approximation of the above population loss.
The other option to measure theapproximation accuracy is the uniform operator norm,
[Equation]
When a linear decoder is used for F = D ◦ A ◦ E , a data-dependent lower bound to each of these 154 errors can be derived.
L 2 lower bound: When the pushforward measure has a finite second moment, its covarianceoperator Γ : L 2 ( Y ) → L 2 ( Y ) is self-adjoint, positive semi-definite, and trace-class, and thus admitsan orthogonal set of eigenfunctions spanning its image, { ϕ 1 , ϕ 2 , . . . } with associated decreasing 158 eigenvalues λ 1 ≥ λ 2 ≥ ... .
The decay of these eigenvalues indicates the extent to which samples 159 from G # µ concentrate along the leading finite-dimensional eigenspaces.
It was shown in [19] that forany choice of E and A , these eigenvalues give a fundamental lower bound to the expected squared L 2error of the operator learning problem with architectures as in (3) using a linear decoder D ,
[Equation]
This result can be further refined to show that the optimal choice of functions τ i (see equation (4))
fora linear decoder are given by the leading n eigenfunctions of the covariance operator { ϕ 1 , . . . , ϕ n } .
The interpretation of this result is that the best way to approximate samples from G # µ with ann -dimensional subspace is to use the subspace spanned by the first n “principal components” of theprobability measure G # µ .
The error incurred by using this subspace is determined by the remainingprincipal components, namely the sum of their eigenvalues (cid:80) k>n λ k .
The operator learning literaturehas noted that for problems with a slowly
Uniform lower bound:
In the reduced order modelling of PDEs literature [7, 8, 20] there existsa related notion for measuring the degree to an n -dimensional subspace can approximate a set offunctions S ⊂
L 2 ( Y ) .
This is known as the Kolmogorov n -width [29], and for a compact set S is 174 defined as
[Equation]
This measure of how well a set of functions can be approximated by a linear subspace in the uniformnorm leads naturally to a lower bound for the uniform error (6).
To see this, first note that for anyu ∈ U , the error from F ( u ) to G ( u ) is bounded by the minimum distance from G ( u ) to the image of
F .
For a linear decoder D : R n → L 2 ( Y ) , define the (at most) n -dimensional V n = im( D ) ⊂
L 2 ( Y ) .
179 Note that im(
F ) ⊆
V n , and we may write
[Equation]
Taking the supremum of both sides over u ∈ U , and then the infimum of both sides over alln -dimensional subspaces V n gives
[Equation]
The quantity on the right is exactly the Kolmogorov n -width of G ( U ) .
We have thus proved thefollowing complementary statement to (7) when the error is measured in the uniform norm.
Proposition 1 Let U ∈
L 2 ( X ) be compact and consider an operator learning architecture as in (3) , 185 where [Equation] 2 ( Y ) is a linear decoder.
Then, for any E : L 2 ( X ) → R p and A : R p → R n , the 186 uniform norm error of F [Equation]
= D [Equation] A ◦ E satisfies the lower bound
[Equation]
Therefore, we see that in both the L 2 ( µ ) and uniform norm, the error for an operator learning problemwith a linear decoder is fundamentally limited by the extent to which the space of output functions“fits” inside a finite dimensional linear subspace.
In the next section we will alleviate this fundamentalrestriction by allowing decoders that can learn nonlinear embeddings of R n into L 2 ( Y ) .
Nonlinear Decoders for Operator Learning
A Motivating Example: Consider the problem of learning the antiderivative operator mappingfunctions to their first-order derivative [Equation]
[Equation]
acting on a set of input functions
[Equation]
The set of output functions is given by G ( U ) = { sin(2 πtx ) | 0 < t 0 < t < T } .
This is a one- 196 dimensional curve of functions in L 2 ([0 , 1]) parameterized by a single number t .
However, we wouldnot be able to represent this set of functions with a one-dimensional linear subspace.
In Figure 2bwe perform PCA on the functions in this set evaluated on a uniform grid of values of t .
We seethat the first 20 eigenvalues are nonzero and relatively constant, suggesting that an operator learningarchitecture with a linear or affine decoder would need a latent dimension of at least 20 to effectivelyapproximate functions from G ( U ) .
Figure 2a gives a visualization of this curve of functions projectedonto the first three PCA components.
We will return to this example in Section 5, and see that anarchitecture with a nonlinear decoder can in fact approximate the target output functions with superioraccuracy compared to the linear case, using a single latent dimension that can capture the underlyingnonlinear manifold structure.
[Figure]
Operator Learning Manifold Hypothesis: We now describe an assumption under which a nonlinear decoder is expected to be effective, and use this to formulate the NOMAD architecture.
To thisend, let µ be a probability measure on L 2 ( X ) and G : L 2 ( X ) → L 2 ( Y ) .
We assume that there exists 209 an n -dimensional manifold M ⊆
L 2 ( Y ) and an open subset
O ⊂ M such that
[Equation]
In connection with the manifold hypothesis in deep learning [3, 2], we refer to this as the Operator
Learning Manifold Hypothesis .
There are scenarios where it is known this assumption holds, such asin learning solutions to parametric PDEs [26].
This assumption motivates the construction of a nonlinear decoder for the architecture in (3) asfollows.
For each u , choose v ( u ) ∈
O such that
[Equation]
[Equation]
[Figure]
Let ϕ : O → R n be a coordinate chart for
O ⊂ M .
We can represent v ( u ) ∈ O by its 217 coordinates ϕ ( v ( u ))
∈ R n . Consider a choice 218 of encoding and approximation maps such that
A ( E ( u )) gives the coordinates for v ( u ) .
If the 220 decoder were chosen as D := ϕ − 1 then by construction, the operator F :=
D ◦ A ◦ E will 222 satisfy
[Equation]
Therefore, we interpret a learned decoding mapas attempting to give a finite dimensional coordinate system for the solution manifold.
Considera generalized decoder of the following form
[Equation]
This induces a map from D : R n → L 2 ( Y ) ,as D ( β ) =
˜ D ( β, · ) .
If the solution manifold
M is a finite dimensional linear subspace in
L 2 ( Y ) spanned by { τ i } ni =1 , we would want a 231 decoder to use the coefficients along the basisas a coordinate system for M .
A generalized decoder could learn this basis as the output of a deepneural network to act as
[Equation]
However, if the solution manifold is not linear, then we should learn a nonlinear coordinate systemgiven by a nonlinear D .
A nonlinear version of ˜ D can be parameterized by using a deep neuralnetwork f
: R n × Y → R which jointly takes as arguments ( β, y ) ,
[Equation]
When used in the context of an operator learning architecture of the form (3), we call a nonlineardecoder from (17) NOMAD (NOnlinear MAnifold Decoder).
Figure 3 presents a visual comparisonbetween linear and nonlinear decoders.
Summary of NOMAD: Under the assumption of the Operator Learning Manifold Hypothesis , wehave proposed a fully nonlinear decoder (17) to represent target functions using architectures of theform (3).
We next show that using a decoder of the form (17) results in operator learning architectureswhich can learn nonlinear low dimensional solution manifolds.
Additionally, we will see that whenthese solution manifolds do not “fit” inside low dimensional linear subspaces, architectures withlinear decoders will either fail or require a significantly larger number of latent dimensions.
Results
In this section we investigate the effect of using a linear versus nonlinear decoders as building blocksof operator learning architecture taking the form (3).
In all cases, we will use an encoder E whichtakes point-wise evaluations of the input functions, and an approximator map A given by a deepneural network.
The linear decoder parametrizes a set of basis functions that are learned as the outputsof an MLP network.
In this case, the resulting architecture exactly corresponds to the DeepONetmodel from [24].
We will compare this against using NOMAD where the nonlinear decoder is builtusing an MLP network that takes as inputs the concatenation of β ∈ R n and a given query pointy ∈ Y .
All models are trained with by performing stochastic gradient descent on the loss function in(1).
The reported errors are measured in the relative L 2 ( Y ) norm by averaging over all functionalpairs in the testing data-set.
More details about architectures, hyperparameters settings, and trainingdetails are provided in the Supplemental Materials.
Learning the Antiderivative Operator: First, we revisit the motivating example from Section4, where the goal is to learn the antidervative operator (10) acting on the set of functions (11).
In
Figure 2c we see the performance of a model with a linear decoder and NOMAD over a range oflatent dimensions n .
For each choice of n , 10 experiments with random initialization seeds wereperformed, and the mean and standard deviation of testing errors are reported.
We see that
the
NOMAD architecture consistently outperforms the linear one (by one order of magnitude), and caneven achieve a 10% relative prediction error using only n = 1 .
Solution Operator of a Parametric Advection PDE: Here we consider the problem of learningthe solution operator to a PDE describing the transport of a scalar field with conserved energy, [Equation]
[Equation]
over a domain ( x, t ) ∈ [0 , 2] × [0 , 1] .
The solution operator maps an initial condition s ( x, 0)
=
u ( x )to the solution at all times s ( x, t ) which satisfies (18).
We consider a training data-set of initialconditions taking the form of radial basis functions with a very small fixed lengthscale centered atrandomly chosen locations in the interval [0 , 1] .
We create the output functions by evolving theseinitial conditions forward in time for 1 time unit according to the advection equation (18) (see
Supplemental Materials for more details).
Figure 4a gives an illustration of one such solution plottedover the space-time domain.
Performing PCA on the solution functions generated by these initial conditions shows a very slowdecay of eigenvalues (see Figure 4b), suggesting that methods with linear decoders will requirea moderately large number of latent dimensions.
However, since the data-set was constructed byevolving a set of functions with a single degree of freedom (the center of the initial conditions), wewould expect the output functions to form a solution manifold of very low dimension.
In Figure 4c we compare the performance of a linear decoder and NOMAD as a function of the latentdimension n .
Linear decoders yield poor performance for small values of n , while NOMAD appearsto immediately discover a good approximation to the true solution manifold.
[Figure]
Propagation of Free-surface Waves:
As a more challenging benchmark we consider the shallowwater equations; a set of hyperbolic equations that describe the flow below a pressure surface in afluid [37].
The underlying PDE system takes the form
[Equation]
[Equation] where, [Equation] where ρ ( x, y, t ) the fluid height from the free surface, g is the gravity acceleration, and v 1 ( x, y, t ) ,v 2 ( x, y, t ) denote the horizontal and vertical fluid velocities, respectively.
We consider reflectiveboundary conditions and random initial conditions corresponding to a random droplet falling intoa still fluid bed (see Supplemental Materials).
In Figure 5a we show the average testing error of amodel with a linear and nonlinear decoder as a function of the latent dimension.
Figure 5b showssnapshots of the predicted surface height function on top of a plot of the errors to the ground truth forthe best, worst, median, and a random sample from the testing data-set.
We additionally use this example to compare the performance of a model with a linear decoderand NOMAD to other state-of-the-art operator learning architectures (see Supplemental Materialfor details).
In Table 1, we present the mean relative error and its standard deviation for differentoperator learning methods, as well as the prediction that provides the worst error in the testing data-setwhen compared against the ground truth solution.
For each method we also report the number ofits trainable parameters, the number of its latent dimension n , and the training wall-clock time inminutes.
Since the general form of the FNO [22] does not neatly fit into the architecture given by(3), there is not a directly comparable measure of latent dimension for it.
We also observe that,although the model with NOMAD closely matches the performance of LOCA [17], its required latentdimension, total number of trainable parameters, and total training time are all significantly smaller.
Discussion
Summary: We have presented a novel framework for supervised learning in function spaces.
Theproposed methods aim to address challenging scenarios where the manifold of target functions haslow dimensional structure, but is embedded nonlinearly into its associated function space.
Such casescommonly arise across diverse functional observables in the physical and engineering sciences (e.g.turbulent fluid flows, plasma physics, chemical reactions), and pose a significant challenge to theapplication of most existing operator learning methods that rely on linear decoding maps, forcingthem to require an excessively large number of latent dimensions to accurately represent targetfunctions.
To address this shortcoming we put forth a fully nonlinear framework that can effectivelylearn low dimensional representations of nonlinear embeddings in function spaces, and demonstratedthat it can achieve competitive accuracy to state-of-the-art operator learning methods while using
[Figure]
[Table]
a significantly smaller number of latent dimensions, leading to lighter model parametrizations andreduced training cost.
Limitations: Our proposed approach relies on the Operator Learning Manifold Hypothesis (seeequation (12)), suggesting that cases where a low dimensional manifold structure does not exist willbe hard to tackle (e.g. target function manifolds with fractal structure, solutions to evolution equationswith strange attractors).
Moreover, even when the manifold hypothesis holds, the underlying effectivelatent embedding dimension is typically not known a-priori, and may only be precisely found viacross-validation.
Another direct consequence of replacing linear decoders with fully nonlinear maps isthat the lower bound in (9) needs to be rephrased in terms of a nonlinear n -width, which in general canbe difficult to quantify.
Finally, in this work we restricted ourselves to exploring simple architecturebuilding blocks such as MLPs, and further investigation is needed to quantify the improvements thatcould be brought by considering more contemporary deep learning architectures.
Potential Negative Societal Impacts: Our contributions enable advances in accelerating themodeling, simulation, and optimization of physical and engineering systems.
As with any tool thatfurthers our understanding and ability to predict the outcomes of complex systems, there may beill-intentioned use cases, but we do not expect any specific negative impact from this work. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will upload
the codes and data after the review process is completed on a public repository and we
will include the codes and data to the Supplemental Material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We report this information in the Supplemental Material.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We include this information in the
Supplemental Material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]