L EARNING S IMILARITY M ETRICS FOR V OLUMETRIC S IMULATIONS WITH M ULTISCALE CNN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Simulations that produce three-dimensional data are ubiquitous in science, ranging from ﬂuid ﬂows to plasma physics.
We propose a similarity model based on entropy, which allows for the creation of physically meaningful ground truth distances for the similarity assessment of scalar and vectorial data, produced from transport and motion-based simulations.
Utilizing two data acquisition methods derived from this model, we create collections of ﬁelds from numerical PDE solvers and existing simulation data repositories and highlight the importance of an appropriate data distribution for an effective training process.
Furthermore, a multiscale CNN architecture that computes a volumetric similarity metric ( VolSiM ) is proposed.
To the best of our knowledge this is the ﬁrst learning method inherently designed to address the challenges arising for the similarity assessment of high-dimensional simulation data.
Additionally, the tradeoff between a large batch size and an accurate correlation computation for correlation-based loss functions is investigated and the metric’s invariance with respect to rotation and scale operations is analyzed.
Finally, the robustness and generalization of VolSiM is evaluated on a large range of test data, as well as a particularly challenging turbulence case study, that is close to potential real-world applications
I NTRODUCTION
Making comparisons is a fundamental operation that is essential for any kind of computation.
This is especially true for the simulation of physical phenomena, as we are often interested in comparing simulations against other types of models or measurements from the physical system.
Mathematically, such comparisons require metric functions that determine scalar distance values as a similarity assessment.
A fundamental problem is that traditional comparisons are typically based on simple, element-wise metrics like the L 1 or L 2 distances, due to their computational simplicity and a lack of alternatives.
Such metrics can work reliably for systems with few elements of interest, e.g. if we want to analyze the position of a moving object at different points in time, matching our intuitive understanding of distances.
However, more complex physical problems often exhibit large numbers of degrees of freedom, and strong dependencies between elements in their solutions.
Those coherences should be considered when comparing physical data, but element-wise operations by deﬁnition ignore such interactions between elements.
With the curse of dimensionality this situation becomes signiﬁcantly worse for systems that are modeled with dense grid data, as the number of interactions grows exponentially with a linearly increasing number elements.
Such data representations are widely used, e.g. for medical blood ﬂow simulations (Olufsen et al., 2000), over climate and weather predictions (Stocker et al., 2014), to the famous unsolved problem of turbulence (Holmes et al., 2012).
Another downside of element-wise metrics is that each element is weighted equally, which is typically suboptimal: E.g. in the context of ﬂuids, smoke plumes behave differently along the vertical dimension due to gravity or buoyancy, and small key features like vortices are more indicative of the ﬂuid’s general behavior than large areas of near constant ﬂow (Pope, 2000).
In the domain of natural images, neural networks have been successfully employed for similarity models that can consider larger structures, typically via learning from the semantic meaning of the images through class labels or data that encodes our human perception.
Similarly, physical systems exhibit spatial and temporal coherence due to the underlying laws of physics, from which we can derive a similarity model.
To robustly learn similarity assessments of scalar and vectorial volumetric data from such a model, our work makes the following contributions:
• We propose a novel similarity model based on the entropy of physical systems.
It is employed to synthesize sequences of volumetric physical ﬁelds suitable for metric learning.
• We show that our Siamese, multiscale feature network results in a stable metric that successfully generalizes to new physical phenomena.
To the best of our knowledge this is the ﬁrst learned metric inherently designed for the similarity assessment of volumetric ﬁelds.
• The metric is employed to analyze turbulence in a case study, and its invariance to rotation and scale are evaluated.
In addition, we analyze correlation-based loss functions with respect to their tradeoff between batch size and accuracy of correlation computation.
The central application of the proposed VolSiM metric is the similarity assessment of new physical simulation methods, numerical or learning-based, against a known ground truth.
This ground truth can take the form of measurements, higher resolution simulations, or existing models.
Furthermore, the trained metric can be used as a differentiable similarity loss for physical learning problems, similar to perceptual losses for computer vision tasks.
R ELATED W ORK
Apart from simple L n distances, the two metrics PSNR and SSIM (Wang et al., 2004) are commonly used across disciplines for the similarity assessment of data.
Similar to the underlying L 2 distance,
PSNR shares the issues of element-wise metrics (Huynh-Thu & Ghanbari, 2008; 2012).
SSIM computes a more intricate function, but it was shown to be closely related to PSNR (Hor ´ e & Ziou, 2010) and thus has similar problems (Nilsson & Akenine-M¨oller, 2020).
Furthermore, statistical tools like the Pearson correlation coefﬁcient PCC (Pearson, 1920) and Spearman’s rank correlation coefﬁcient SRCC (Spearman, 1904) can be employed as a simple similarity measurement.
There are several learning-based metrics specialized for different domains such as rendered (Andersson et al., 2020) and natural images (Bosse et al., 2016), interior object design (Bell & Bala, 2015), audio (Avgoustinakis et al., 2020), and haptic signals (Kumari et al., 2019).
Especially for images, similarity measurements have been studied in multiple settings, mainly by combining deep embeddings as perceptually more accurate metrics (Prashnani et al., 2018;
Talebi & Milanfar, 2018).
Similarly, such metrics can be employed for various image related tasks like super-resolution (Johnson et al., 2016) or image generation (Dosovitskiy & Brox, 2016).
We additionally study the behavior of invariance and equivariance to transformations, which was targeted previously for rotational symmetries (Weiler et al., 2018; Chidester et al., 2019) and improved generalization (Wang et al., 2021).
Traditional metric learning for natural image domains typically works in one of two ways: Either, the training is directly supervised by learning from manually created labels, e.g. via two-alternative forced choice where humans pick the most similar option to a reference (Zhang et al., 2018).
Or, the training is indirectly semi-supervised through images with class labels and a contrastive loss (Chopra et al., 2005;
Hadsell et al., 2006).
In that case, triplets of reference, same class image, and other class images are sampled, and the corresponding latent space representations are pulled together or pushed apart.
We refer to Roth et al.
for an overview of different image metric training strategies.
Similarity metrics for simulation data have not been studied extensively yet.
For smoke ﬂow synthesis, Siamese networks for ﬁnding similar ﬂuid descriptors have been applied (Chu & Thuerey, 2017).
Um et al.
(2017; 2021) used crowd-sourced user studies for the similarity assessment of liquid simulations.
Simulation data was compared via a learned metric (Kohl et al., 2020) using a feature extractor with a structure similar to a classiﬁcation CNN, however only considering scalar 2D data and a simple, linear distance model.
Deep learning techniques also receive attention for improving simulations, e.g. for predicting states over time (Morton et al., 2018;
Kim et al., 2019), for learning via physical laws (Watters et al., 2017;
Greydanus et al., 2019), and for the dynamics of complex materials (Ummenhofer et al., 2020;
Sanchez-Gonzalez et al., 2020).
While these approaches have potential to be combined with our metric, we focus on traditional numerical methods to acquire data in the following.
M ODELING S IMILARITY OF S IMULATIONS
To formulate our methodology for learning similarity metrics that target dissipative physical systems, we turn to the fundamental quantity of entropy.
The second law of thermodynamics states that the entropy S of a physical system never decreases, thus ∆ S ≥ 0 .
In the following, we make the reasonable assumption that the behavior of the system is continuous and non-oscillating,
[Equation]
and that ∆ S > 0 .
On the right, Eq. 1 is the Boltzmann equation from statistical mechanics (Boltzmann, 1866), that describes S in terms of the Boltzmann constant k b and the number of microstates
W of a system (here an ideal gas).
1 Since entropy only depends on a single system state, it can be reformulated to take the relative change between two states into account: given a sequence of states s 0 , s 1 , . . .
, s n , the relative entropy ˜ S ( s ) is deﬁned in Eq.
2, where w s is the monotonically increasing, relative number of microstates deﬁned as 0 for s
and as 1 for s n .
c > 0 is a system-dependent factor that determines how quickly the number of microstates increases.
Next, we shift Eq.
2 such that the lower bound is 0 to derive a ﬁrst similarity model D .
This offset is necessary, since the properties of similarity metrics dictate that distances are always non-negative and only zero for identical states.
Finally, we choose the system-dependent constant k appropriately, as relative similarities do not change through linear transformations, and thus k does not inﬂuence the general behavior of D .
Choosing k = 1 / (log c + 1) leads to the full similarity model D ( s ) in Eq.
4, that is bound to the range [0 , 1] , and directly predicts the level of similarity between the reference s 0 and any state s from a sequence.
[Figure]
Δ Δ inf high low
S i m il a r i t
y t
o s
Fig.
1 illustrates this connection between the logarithmic increase of entropy within a system and the proposed similarity model.
In addition, trajectories for different ∆ through this space are shown.
This ∆ between states corresponds to c in the similarity model in Eq. 4, and is directly proportional to it: For large changes ∆ inf to the system, the entropy quickly increases and the states s 1 , . . .
, s n contain little information, hence are quickly very dissimilar to s 0 (red dotted curve).
For the smallest possible changes ∆ (cid:15) , entropy slowly increases while similarity linearly decreases, due to the continuous nature of the system (green dashed line).
Ideally, a sequence of states employed for learning tasks should evenly exhibit both regimes as well as intermediate ones, as indicated by the black curve.
For practicality, this should not rely on an extremely large number of states, as would be necessary for ∆ (cid:15) .
In the following, we will refer to this property of a sequence being informative with respect to a similarity analysis as difﬁculty of a sequence with a ﬁxed length n .
The red area in Fig.
1 generally corresponds to sequences that are too difﬁcult, i.e. that do not bare any similarity to s 0 very quickly, while the green area generally corresponds to sequences that are too easy, i.e. that barely change.
s 0 , s 1 , . . .
, s n .
for i = 1 to n do q .append (1 − PCC ( s 0 , s i ))
q ← normalize q to [0 , 1] c ← ﬁt log (10 c q +1) log (10 c +1) for c for i = 0 to n − 1 do for j = i + 1 to n do d .append ( m ( s i , s j ) )
w
[Equation]
n )
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
The resulting formulation for D represents an idealized similarity model and does not directly work as a metric due to two reasons: 1) a metric only takes two states as an input, so estimating c would be difﬁcult, and 2) both inputs do not necessarily have to come from the same system, for example when comparing a velocity observation to a simulated ground truth.
To overcome this, we formulate a semi-supervised learning problem by creating sequences s 0 , s 1 , . . .
, s n of different physical systems, and training a neural network m to predict the distance d given the reference distances g as computed via
D .
This technique incorporates the underlying physical behavior, compared to adding variations in a post-process (as it is often done in the domain of images, e.g. by Ponomarenko et al.
Alg.
1 shows how d and g are computed: First, we estimate c by evaluating Pearson’s distance that indicates how quickly the states become uncorrelated, normalizing it to [0 , 1] , and optimizing Eq.
4 for c via a standard least-squares optimization.
Then, g directly follows from Eq.
4 through the ﬁt of c and the linearly increasing number of microstates w s , while d is computed by a pairwise evaluation of m .
S EQUENCE C REATION
To create sequences s 0 , s 1 , . .
, s n within a controlled environment, we make use of the knowledge about the underlying physical processes: We either employ direct changes, based on spatial or temporal coherences to s 0 , or use changes to the initial conditions of the process that lead to s 0 .
The central challenge now becomes how to ﬁnd suitable sequences via the magnitude of ∆ , without knowing a priori which degree of non-linearity in the similarity model provides the most information for each setup.
However, given a value of ∆ and a corresponding sequence, we can compute how strongly it differs from a sequence with ∆ (cid:15) , as depicted in Fig.
This is achieved by a proxy similarity function, via computing the PCC between this proxy and a simple linearly decreasing similarity as predicted by ∆ (cid:15) .
If the resulting correlation is high, the sequence is close to a similarity behavior as predicted by ∆ (cid:15) , while lower correlations indicate stronger deviations.
Finally, we sample sequences based on simple correlation thresholds to acquire a suitable difﬁculty level.
By using the mean squared error (MSE) as a proxy, we empirically determined that correlations between 0 .and 0 .
85 work well for all cases we considered.
Hence, we propose two semi-automatic iterative methods (see Fig. 2) to create data, depending on the method to introduce variations to a given state.
Both methods iteratively sample a small set of sequences to calibrate ∆ to a suitable magnitude and use that for the full data set. Compared to a stricter sampling of every single sequence, this method avoids potential biases in the data and is computationally more efﬁcient as less sampling is needed.
[Figure] [Table]
[Figure] [Table]
[Figure]
I
[Figure]
c
[Figure]
[A] Variations based on Initial Conditions of Simulations Given a numerical PDE solver and a set of initial conditions or parameters p , the solver computes a solution to the PDE over the time steps t 0 , t 1 , . . .
, t t .
To create a larger number of different sequences, we make the systems nondeterministic by adding noise to a simulation ﬁeld and randomly generating the initial conditions from a given range.
Adjusting one of the parameters p i in steps with a small perturbation ∆ i , allows for the creation of a sequence s 0 , s 1 , . . .
, s n with decreasing similarity to the unperturbed simulation s 0 .
This is repeated for every suitable parameter in p , and the corresponding ∆ is updated individually until the targeted MSE correlation range is reached.
The global noise strength factor also inﬂuences the difﬁculty to some degree and can be updated.
[B] Variations based on Spatio-temporal Coherences For a source D of volumetric spatialtemporal data without access to a solver, we rely on a larger spatial and/or temporal dimension than the one required for a sequence.
We start at a random spatio-temporal position p to extract a cubical spatial area s 0 around it.
p can be repeatedly translated in space and/or time by ∆ t,x,y,z to create a sequence s 0 , s 1 , . . .
, s n of decreasing similarity.
Using a different position leads to new sequences, as long as the repository features enough diverse data.
It is possible to add a global amount of random perturbations q with a chosen strength to the positions, to further increase the difﬁculty.
Data Sets To create training data with method [A], we utilize solvers for a basic AdvectionDiffusion model ( Adv ), Burger’s equation ( Bur ) with an additional viscosity term, and the full
Navier-Stokes equations via a Eulerian smoke simulation ( Smo ) and a hybrid Eulerian-Lagrangian liquid simulation ( Liq ).
The corresponding validation sets are generated with a separate set of random seeds.
Furthermore, we use adjusted versions of the noise integration for two test sets, by adding noise to the density instead of the velocity ﬁeld in the Advection-Diffusion model ( AdvD ), and overlayed noise to the background area of the liquid simulation ( LiqN ).
We create seven test sets via method [B].
Four come from the Johns Hopkins Turbulence Database JHTDB (Perlman et al., 2007) that contains a large amount of direct numerical simulation (DNS) data, where each is based on a subset of the JHTDB and features different characteristics:
isotropic turbulence ( Iso ), a channel ﬂow ( Cha ), magneto-hydrodynamic turbulence ( Mhd ), and a transitional boundary layer ( Tra ).
Since turbulence contains structures of interest across all length scales, we additionally randomly stride or interpolate the query points for scale factors in [0 . 25 , 4] to create sequences of different physical size.
One additional test set ( SF ) via temporal translations is based on ScalarFlow (Eckert et al., 2019), consisting of 3D reconstructions of real smoke plumes.
Furthermore, method [B] is slightly modiﬁed for two synthetic test sets: Instead of using a data repository, we procedurally synthesize spatial ﬁelds.
For that, we employ linearly moving randomized shapes ( Sha ), and randomized damped waves ( Wav ) of the general form f ( x ) = cos ( x ) ∗ e − x .
All data was gathered in sequences with n = 10 at resolution 128 3 , and downsampled to 64 3 for computational efﬁciency during training and evaluations.
Fig.
3 shows parts of multiple example sequences via a mean projection to 2D, and App. B and C contain further details on each data set.
Data Distribution There are many independent factors that inﬂuence the difﬁculty of the created data sequences, and measuring the similarity in the generation process with a proxy function introduces additional uncertainties.
In addition, our iteration schemes only calibrate ∆ to a suitable magnitude, instead of enforcing stricter bounds on every single sequence.
As a result, the computed
PCC values from the MSE (see Fig. 2) on the full data sets exhibit a more natural, smooth distribution with controllable difﬁculty, instead of introducing an artiﬁcial distribution with hard cutoffs.
Due to the central limit theorem, we expect a normal distribution of the PCC values that indicate the sequence difﬁculty.
However, correlations have a ﬁxed upper bound of 1 , meaning a truncated normal distribution is a more accurate prediction.
Intuitively, this corresponds to trajectories with different curvature in the similarity model in Fig.
In fact, we empirically determined that only training data distributions which reasonably closely follow this prediction, result in a successful training of our model.
We assume, that signiﬁcantly different distributions indicate unwanted biases in the data, which prevented effective learning in our experiments.
Fig.
4 shows the correlation histograms of our training and validation data sets, all of which roughly follow this desired truncated normal distribution.
[Figure]
L EARNING A D ISTANCE F UNCTION
For our method, we generally follow the established Siamese structure, that was originally proposed for 2D domains (Zhang et al., 2018;
Kohl et al., 2020):
First, two inputs are embedded in a latent space using a CNN as feature extractor.
The Siamese network structure means that the weights are shared, which ensures the mathematical requirements for a pseudo-metric.
Next, the features are normalized and compared with an element-wise comparison like an absolute or squared difference.
Finally, the feature difference is aggregated with sum, mean, and learned weighted average functions.
To compute the proposed VolSiM metric, that operates on inherently more complex 3D data, the following changes to this framework are proposed.
[Figure]
Multiscale Network Scale is important for a reliable similarity assessment, since physical systems often exhibit self-similar behavior that does not signiﬁcantly change across scales, as indicated by the large number of dimensionless quantities in physics.
Generally, scaling a data pair should not alter its similarity, and networks can learn such an invariance to scale most effectively by processing data at different scales.
One example where this is crucial is the energy cascade in turbulence (Pope, 2000), which is also analyzed in our case study below.
For learned image metrics, this invariance is also useful (but less crucial), and often introduced with large strides and kernels in the convolutions, e.g. via a feature extractor based on AlexNet (Zhang et al., 2018).
In fact, our experiments with similar architectures showed, that models with large strides and kernels generally perform better than models that modify the scale over the course of the network to a lesser extent.
However, we propose to directly encode this scale structure in a multiscale architecture for a more accurate similarity assessment, and a network with a smaller resource footprint.
Fig.
5 shows the proposed fully convolutional network:
Four scale blocks individually process the input on increasingly smaller scales, where each block follows the same layer structure, but deeper blocks effectively cover a signiﬁcantly larger volume due to the reduced input resolutions.
Furthermore, deeper architectures can model complex functions more easily, so we additionally include resolution connections from each scale block to the next resolution level via concatenation.
Effectively, the network learns a mixture of connected deep features and similar representations across scales as a result.
Training and Evaluation To increase the model’s robustness during training, we used the following data augmentations for each sequence: the data is normalized to [ − 1 , 1] , and together randomly ﬂipped and rotated in increments of 90° around a random axis.
The velocity channels are randomly swapped to prevent directional biases from some simulations, while scalar data is extended to the three input channels via repetition.
For inference, only the normalization operation and the repetition of scalar data is performed.
The ﬁnal metric model was trained with the Adam optimizer with a learning rate of 10 − 4 for 30 epochs via early stopping, further details can be found in App.
A.
To train the metric end-to-end, Alg. 1 is employed to compute the predicted distances d and the ground truth g according to the similarity model.
Both are passed to the correlation loss function below, that compares them and provides gradients.
To determine the accuracy of a metric during inference, we compute the SRCC between d and w s , where a value closer to 1 indicates a better reconstruction.
Loss Function Given predicted distances d and a ground truth g of size n , we train our metric networks with the loss in Eq.
It consists of a weighted combination of an MSE and an inverted correlation term r , where ¯ d and ¯ g denote the mean.
While the formulation follows existing work (Kohl et al., 2020), it is important to note that g is computed by our similarity model from Sec. 3, and below we introduce a slicing technique to apply this loss formulation to high-dimensional data sets.
For the training process via stochastic gradient descent, Eq. 5 requires a trade-off: A large batch size b is useful to improve training stability via less random gradients for optimization.
Similarly, a sufﬁciently large value of n is required to keep the correlation values accurate and stable.
[Equation]

 (5)
However, with ﬁnite amounts of memory, choosing large values for n and b is not possible in practice.
Especially so for 3D cases, where a single sample can already be memory intensive.
In general, n is implicitly determined by the length of the created sequences, via the number of possible pairs within a sequence.
Thus, we provide an analysis how the correlation can be approximated in multiple steps for a ﬁxed n , to allow for increasing b in return.
In the following, the batch dimension is not explicitly shown, but all expressions can be extended with a vectorized ﬁrst dimension.
[Figure]
The full distance vectors d and g are split in slices with v elements, where v should be a proper divisor of n .
For any slice k , we can compute a partial correlation r k with r k = (cid:80) k + v i = k
[Equation]
Note that this is only an approximation, and choosing larger values of v for a given b is always beneﬁcial, if sufﬁcient memory is available.
For all slices, the gradients are accumulated during backpropagation since other aggregations would required a computational graph of the original, impractical size.
Eq.
6 still requires the computation of the means ¯
d
and ¯ g as a pre-process over all samples.
Both can be approximated with the running means ˜ d and ˜ g for efﬁciency ( RM ).
For small values of v , the slicing results in very coarse, unstable correlation results.
To alleviate that, it is possible to use a running mean over all previous values ˜
r k = (1 /k )
( r k + (cid:80) k − 1 l =1 r l ) .
This aggregation ( AG ) can stabilize the gradients of individual r k as they converge to the true correlation value during training, as App.
D shows in more detail.
Fig.
6 displays the resulting performance on our data, when training with different combinations of b , v , RM , and AG .
All models exhibit similar levels of memory consumption and were trained with the same random training seed.
When comparing models with and without RM (blue vs red and purple vs orange) both are on par in most cases, even though computation times for a running mean are about 20% lower.
The results of models with and without AG (blue vs purple and red vs orange) conﬁrm the expected stabilization effect, where reduced ﬂuctuations in the training translate to better generalizing metrics.
Overall, this experiment demonstrates that choosing larger v consistently leads to better results (marker shape), so more accurate correlations are beneﬁcial over a large batch size b in memory limited scenarios. Thus, we use b = 1 and v = 55 for the ﬁnal model.
R ESULTS
We compare the proposed VolSiM metric to a variety of existing methods in the upper section of
Tab. 1.
All metrics were evaluated on the volumetric data from Sec. 4, which contain a wide range of test sets that differ strongly from the training data.
VolSiM consistently reconstructs the ground truth distances from the entropy-based similarity model more reliably than other approaches on most data sets.
As expected, this effect is most apparent on the validation sets since their distribution is closest to the training data.
But even on the majority of test sets with a very different distribution,
VolSiM is the best performing or close to the best performing metric.
Metrics without deep learning often fall short, indicating that they were initially designed for different use cases, like SSIM (Wang et al., 2004) for images, or variation of information VI (Meil˘a, 2007) for clustering.
The strictly element-wise metrics MSE and PSNR exhibit almost identical performance, and both work poorly on a variety of data sets.
The learning-based methods LPIPS (Zhang et al., 2018) for images, and
LSiM (Kohl et al., 2020) are only available for two-dimensional data.
Hence, they can not correlate structures along all dimensions.
To evaluate them in our setting, the 3D data was sliced along all three dimensions to avoid biases against one direction, and the resulting distances are averaged over all slices.
As shown in Tab. 1, this results in a reasonable performance overall. [Table] [Table]
Johns Hopkins
[Table]
(Perlman et al., 2007)
[Table]
(Eckert et al., 2019)
[Table]
The bottom half of Tab. 1 contains an ablation study of the proposed architecture MS , and a simple
CNN model.
This model is similar to an extension of the convolution layers of AlexNet (Krizhevsky et al., 2017) to three dimension, and does not utilize a multiscale structure.
Even though VolSiM has more than 80% fewer weights compared to CNN trained , it can ﬁt the training data more easily and works better or equally well on every single dataset in Tab. 1, indicating the strengths of the proposed multiscale architecture.
The performance of untrained models CNN rand and MS rand conﬁrm the ﬁndings from Zhang et al.
(2018), who also report a surprisingly strong performance of random networks.
Using no resolution skip connections for MS no skip (see Fig. 5) slightly lowers the metrics generalization across most data sets.
Removing the last resolution scale block for MS 3 scales overly reduces the capacity of the model, while adding another block for MS 5 scales only leads to minor differences.
In addition, we also investigate two slightly different training setups: for MS added Iso we integrate extra sequences created like the Iso data in the training, while MS only Iso is exclusively trained on such sequences.
MS added Iso only slightly improves upon the baseline, and even the turbulence-speciﬁc MS only Iso model does not consistently improve the results on the JHTDB data sets.
Both cases indicate a high level of generalization for VolSiM , as it was not trained on any turbulence data.
App.
E contains more details for each model and App. F features further ablations.
Transformation Invariance Physical systems are often characterized by Galilean invariance (McComb, 1999), i.e. identical laws of motion across inertial frames. Likewise, a metric should be invariant to transformations of the input, meaning a constant distance output when translating, scaling, or rotating both inputs.
Element-wise metrics fulﬁll these properties by construction, but our
Siamese network structure requires an equivariant feature representation that changes along with input transformations to achieve them. As CNN features are translation equivariant by design, we only empirically examine rotation and scale invariance for our multiscale metric and a standard Conv+ReLU model on a ﬁxed set of 8 random data pairs from each data set.
For the rotation experiment, we rotate the pairs in steps of 5° around a random coordinate axis.
The empty volume inside the original frame is ﬁlled with a value of 0 , and data outside the frame is cut to prevent any scaling.
For scaling, the data is bilinearly up- or downsampled according to the scale factor, and processed fully convolutionally by each network.
In Fig.
7, the resulting distance deviation from the mean of the predicted distances is plotted for rotation and scaling operations.
The optimal result would be a perfectly equal distance with zero deviation across all transformations.
Compared to the model CNN trained , it can be observed that VolSiM produces less deviations overall, and leads to signiﬁcantly smoother distance curves.
Note that we observe scale equivariance rather than invariance for VolSiM , i.e. a mostly linear scaling of the distances according to the input size. This is not as problematic as the unstable behavior of CNN trained , as the general order is preserved and data is typically not compared across resolutions without resampling.
The main reason is that the fully convolutional features are larger and thus lead to scale equivariant distances when compared. With a normalization for the physical size of the system, this effect could be further reduced in future work.
Case Study: Turbulence Analysis As a particularly challenging test for generalization, we further perform a detailed case study on forced isotropic turbulence.
This study resembles a potential realworld scenario for our metric.
For this purpose, fully resolved raw data over a long temporal interval from the isotropic turbulence data set from JHTDB is utilized (see bottom of Fig.
The 1024domain is ﬁltered and reduced to a size of 128 3 with strides of 8, meaning VolSiM is applied in a fully convolutional manner, and has to generalize beyond the training resolution of 64 3 .
Three different time spans of the simulation are investigated, where the long span also uses a temporal stride of 20.
Traditionally, turbulence research makes use of two-point correlations to study such cases (Pope, 2000).
Since we are interested in a comprehensive spatial analysis instead of two single points, we can make use of Pearson’s distance on the full ﬁelds to obtain reference values for an evaluation.
App.
G contains further details on the implementation of this case study.
[Figure]
At the top of Fig. [Figure] we ﬁnd that our similarity model from Sec.
3 (gray dotted) closely follows this correlation-based trajectory (red dashed) over time, as also indicated by the value SRCC a between both cases.
Even though there are smaller ﬂuctuations, the proposed VolSiM metric (blue) results in a very similar distance trajectory to the similarity model (see SRCC b ) across all time spans.
Our similarity metric faithfully recovers the correlation-based reference, despite not having seen any turbulence data at training time.
Hence, this experiment indicates the accuracy of the similarity model [Figure] and the capabilities for generalization of the multiscale metric network to new cases.
C ONCLUSION
We presented the multiscale CNN architecture VolSiM [Figure] and demonstrated its capabilities as a similarity metric for volumetric simulation data.
A similarity model based on the behavior of entropy in physical systems was proposed and utilized to learn a robust, physical similarity assessment.
Different methods to compute correlations inside a loss function were analyzed [Figure] and the invariance to scale and rotation transformations investigated.
The proposed metric potentially has an impact on a broad range of disciplines where volumetric simulation data arises.
An interesting area for future work is designing a metric speciﬁcally for turbulence simulations, ﬁrst steps towards which were taken with our case study.
Additionally, investigating learning-based methods with features that are by construction equivariant to rotation and scaling may lead to further improvements in the future.
E THICS S TATEMENT
Since we target the fundamental problem of the similarity assessment of numerical simulations, we do not see any direct negative ethical implications of our work.
However, there could be indirect negative effects since this work can act as a tool for more accurate and/or robust numerical simulation methods in the future, for which a military relevance exists.
A further indirect issue could be explainability, e.g. when simulations in an engineering process yield unexpected inaccuracies.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of our work we took the following steps: An extensive appendix with details and hyperparameters regarding the implementation, data sets and sequence creation is provided below.
Furthermore, the trained VolSiM model and source code to evaluate it are provided alongside this submission.
The complete data and implementation of this work will be published upon acceptance.
This includes the source code to create the data sets as well as the full data sets themselves, the source code to recreate all ﬁgures and the source code train the metric end-to-end and evaluate it.
APPENDIX
In the following, additional details for the proposed VolSiM metric are provided: App.
A contains implementation details regarding the training and the metric model setup, and App.
B features generation details for all our data sets.
Next, additional examples of the created data sequences are displayed in App.
C, and the training stability for the loss experiment is investigated in App.
D. Details for the experimental setup of the ablation study models are given in App. E, and App. F contains additional ablation studies.
Finally, details regarding the turbulence case study can be found in App.
G.
A I MPLEMENTATION D ETAILS
The training and evaluation process of the metric was implemented in PyTorch, while the data was simulated and collected with specialized solvers and data interfaces as described in App B.
The data acquisition, training, and metric evaluation was performed on a server with an Intel i7- (3.60Ghz) CPU and an NVIDIA GeForce GTX 1080 Ti GPU.
It took about 38 hours of training to fully optimize the ﬁnal VolSiM model for the data sequences with a spatial resolution of 64 3 .
In addition to the multiscale feature extractor network, the following operations were used for the
Siamese architecture of the metric: Each feature map is normalized via a mean and standard deviation normalization to a standard normal distribution.
The mean and standard deviation of each feature map is computed in a pre-processing step for the initialization of the network over all data samples.
Both values are ﬁxed for training the metric afterwards.
To compare both sets of feature maps in the latent space, a simple element-wise, squared difference is employed.
To keep the mathematical metric properties, this also requires a square root operation before the ﬁnal distance output.
The spatial squared feature map differences are then aggregated along all dimensions into a scalar distance output.
Here, we used a single learned weight with dropout for every feature map, to combine them to a weighted average per network layer.
The activations of the average feature maps are spatially combined with a simple mean, and summed over all network layers afterwards.
This process of normalizing, comparing, and aggregating the feature maps computed by the feature extractor follows previous work (Kohl et al., 2020; Zhang et al., 2018).
The weights to adjust the inﬂuence of each feature map are initialized to 0 .
1 , all other weights of the multiscale feature extractor are initialized with the default PyTorch initialization.
For the ﬁnal loss, the MSE term was weighted with λ 1 = 1 . 0 , while the correlation term was weighted with λ 2 = 0 .
B D ATA S ET D ETAILS
In the following sections, the details underlying each data set are described.
Tab. 3 contains a summary of simulator, simulation setup, varied parameters, noise integration, and used ﬁelds for the simulated and generated data sets.
Tab. 4 features a summary of the collected data sets, with repository details, jitter and cutout settings, and spatial and temporal ∆ values.
Both tables also contain the number of sequences created for training, validation, and testing for every data source.
These values only apply for the general metric setup, and changes for the ablation study models can be found in App.
E below.
B. A DVECTION -D IFFUSION AND B URGER ’ S E QUATION
In its simplest form, the transport of matter in a ﬂow can be described by the two phenomena of advection and diffusion.
Advection describes the movement of a passive quantity inside a velocity ﬁeld over time, and diffusion describes the process of dissipation of this quantity due to the second law of thermodynamics.
[Equation]
Eq.
7 is the simpliﬁed Advection-Diffusion equation with constant diffusivity and no sources or sinks, where u denotes the velocity, d is a scalar passive quantity that is transported, and ν is the diffusion coefﬁcient or kinematic viscosity.
Burger’s Equation in Eq.
8 is similar to the Advection-Diffusion equation, but it describes how the velocity ﬁeld itself changes over time with advection and diffusion.
The diffusion term can also be interpreted as a viscosity, that models the resistance of the material to deformations.
Furthermore, this variation can develop discontinuities (also called shock waves).
Here, u also denotes the velocity and ν the kinematic viscosity or diffusion coefﬁcient.
[Equation]
To solve both PDEs, the differentiable ﬂuid framework PhiFlow (Holl et al., 2020) was used.
The solver utilizes a Semi-Lagrangian advection scheme, and we chose periodic domain boundary conditions to allow for the usage of a Fourier space diffusion solver.
We introduced additional continuous forcing to the simulations by adding a force term f to the velocity after every simulation step.
Thus, f depends on the time steps t , that is normalized by division of the simulation domain size beforehand.
For Adv , Bur , and AdvD , we initialized the ﬁelds for velocity, density, and force with multiple layered parameterized sine functions.
This leads to a large range of patterns across multiple scales and frequencies when varying the sine parameters.
[Equation]
where c = (1 , 1 , 0 . 4 , 0 . 3) (9)
[Equation]
where ˜ p = p + f x 7 ∗ 0 .
5 + f x 7 ∗ sin (3 t ) and c = (0 , 1 , 1 , 0 . 7)
[Equation]
[Equation]
Eq. 9, 10, and 11 show the layered sine functions in u x ( p ) , f x ( p ) , and d ( p ) for a spatial grid position p ∈ R 3 .
The sum operation denotes a sum of all vector elements here, and all binary operations on vectors and scalars use broadcasting of the scalar value to match the dimensions.
Eq.
12 shows the deﬁnition of the function parameters used above, all of which are randomly sampled based on the simulation seed for more diverse simulations.
Note that ν was multiplied by 0 .
1 for Bur .
The remaining velocity and force components u y ( p ) , u z ( p ) , f y ( p ) , and f z ( p ) and corresponding parameters are omitted for brevity here, since they follow the same initialization pattern as u x ( p ) and f x ( p ) .
Tab. 3 shows the function parameters that were varied, by using the random initializations and adjusting one of them in linear steps to create a sequence.
[Equation]
The main difference between the Advection-Diffusion training data and the test set is the method of noise integration: For Adv it is integrated into the simulation velocity, while for AdvD it is added to the density ﬁeld instead.
The amount of noise added to the velocity for Bur and Adv and to the density for AdvD was varied in isolation as well.
B. N AVIER -S TOKES E QUATIONS
The Navier-Stokes Equations fully describe the behavior of ﬂuids like gases and liquids, via modelling advection, viscosity, and pressure effects, as well as mass conservation.
Pressure can be interpreted as the force exerted by surrounding ﬂuid mass at a given point, and the conservation of mass means that the ﬂuid resists compression.
[Equation]
Eq.
13 describes the conservation of momentum, and Eq.
14 describes mass conservation.
Again, u denotes the velocity, P is the pressure, ρ is the ﬂuids density, ν is the kinematic viscosity, and g denotes external forces like gravity.
Smoke
To create the smoke data set Smo , the ﬂuid framework MantaFlow that provides a gridbased Eulerian smoke solver for the Navier-Stokes Equations was used.
It is based on a SemiLagrangian advection scheme and on the conjugate gradient method as a pressure solver.
The simulation setup consists of a cylindrical smoke source at the bottom of the domain with a ﬁxed noise pattern initialization to create more diverse smoke plumes.
Furthermore, a constant spherical force ﬁeld ff is positioned over the source.
This setup allows for a variation of multiple simulation parameters, like the smoke buoyancy, the source position and different force ﬁeld settings.
They include position, rotation, radius and strength.
In addition, the amount of added noise to the velocity can also be varied in isolation.
Liquid Both liquid data sets, Liq and LiqN , were created with a liquid solver in MantaFlow.
It utilizes the hybrid Eulerian-Lagrangian ﬂuid implicit particle method (Zhu & Bridson, 2005), that combines the advantages of particle and grid-based liquid simulations for reduced numerical dissipation.
The simulation setup consists of two liquid cuboids of different shapes, similar to the common breaking dam setup.
After 25 simulation time steps a liquid drop is added near the top of the simulation domain, and it falls down on the water surface that is still moving.
Here, the external gravity force as well as the drops position and radius are varied to create similarity sequences.
As for the smoke data, a modiﬁcation of the amount of noise added to the velocity was also employed as a varied parameter.
The main difference between the liquid training data and the test set is the method of noise integration: For Liq it is integrated into the simulation velocity, while for LiqN it is overlayed on the simulation background.
B. G ENERATED D ATA
To create the shape data set Sha and the wave data set Wav , a random number of straight paths are created by randomly generating a start and end point inside the domain.
It is ensured that both are not too close to the boundaries and that the path has a sufﬁcient length.
The intermediary positions for the sequence are a result of linearly interpolating on these paths.
The positions on the path determine the center for the generated objects that are added to a occupancy marker grid.
For both data sets, overlapping shapes and waves are combined additively, and variations with and without overlayed noise to the marker grid were created.
Shapes For Sha , random shapes (box or sphere) are added to the positions, where the shape’s size is a random fraction of the path length, with a minimum and maximum constraint.
The created shapes are then applied to the marker grid either with or without smoothed borders.
Waves For Wav , randomized volumetric damped cosine waves are added around the positions instead.
The marker grid value m at point p for a single wave around a center c is deﬁned as
[Equation]
where ˜ p = (cid:107) p
[Equation]
Here, r is the radius given by the randomized size that is computed as for Sha , and w ∼ U (0 . 1 , 0 . 3) is a randomized waviness value, that determines the frequency of the damped wave.
B. C OLLECTED D ATA
The collected data sets Iso , Cha , Mhd , and Tra are based on different subsets from the Johns
Hopkins Turbulence Database JHTDB (Perlman et al., 2007), that contain different types of data from direct numerical simulations (DNS).
In these simulations, all spatial scales of turbulence up to the dissipative regime are resolved.
The data set SF is based on the ScalarFlow data (Eckert et al., 2019), that contains dense 3D velocity ﬁelds of real buoyant smoke plumes, created via multi-view reconstruction technique.
ScalarFlow
Since 100 reconstructions of different smoke plumes are provided in ScalarFlow, there is no need to add additional randomization to create multiple test sequences.
Instead, we directly use each reconstruction sequence to create one similarity sequence in equal temporal steps.
The only necessary pre-processing step is cutting off the bottom part of domain that contains the smoke inﬂow, since it is frequently not fully reconstructed.
Afterwards, the data is interpolated to the full spatial size of 128 3 to match the other data sets.
JHTDB The JHTDB subsets typically contain a single simulation with a very high spatial and temporal resolution and a variety of ﬁelds.
We focus on the velocity ﬁelds, since turbulent ﬂow data is especially complex and potentially beneﬁts most from a better similarity assessment.
We can mainly rely on using temporal sequences, and only need to add spatial jitters in some cases to increase the difﬁculty.
As turbulence generally features structures of interest across all length scales, we create sequences of different spatial scales for each subset.
To achieve this, we randomly pick a cutout scale factor s .
If s = 1 , we directly use the native spatial discretization provided by the database.
For s > 1 we stride the spatial query points of the normal cubical cutout by s after ﬁltering the data.
For s < 1 the size of the cubical cutout is reduced by a factor of s in each dimension, and the cutout is interpolated to the full spatial size of 128 3 afterwards.
Among other details, Tab.shows the cutout scale factors, as well as the corresponding random weights.
C A DDITIONAL E XAMPLE S EQUENCES
Fig.
10, 11, and 12 show multiple full example sequences from all our data sets.
In every sequence, the leftmost image is the baseline ﬁeld.
Moving further to the right, the change of one initial parameter increases for simulated data sets, and the spatio-temporal position offset increases for generated and collected data.
To plot the sequences, the 3D data is projected along the z-axis to 2D via a simple mean operation.
This means, noise that was added to the data or the simulation is typically signiﬁcantly less obvious due to statistical averaging in the projection.
Velocity data is directly mapped to
RGB color channels, and scalar data is shown via different shades of gray.
Unless note otherwise, the data is jointly normalized to [0 , 1] for all channels at the same time, via the overall minimum and maximum of the data ﬁeld.
D T RAINING S TABILITY FOR L OSS E XPERIMENT
[Figure]
To further analyze the resulting stability of different variants of the loss investigated in Sec. 5, Fig.shows training trajectories of multiple models.
Displayed are the direct training loss over the training iterations, i.e. one loss value per batch, and a smoothed version via an exponential weighted moving average computed with α = 0 .
05 , corresponding to a window size of 40.
The models are colorcoded according to Fig. 6, and were trained with different batch sizes b , slicing values v , usage of the running sample mean RM , and usage of the correlation aggregation AG .
Note that the models were generally trained with the same training seed, meaning the order of samples is should be very similar across all runs (only with potential deviations due to race conditions and GPU processing).
It can be observed that models trained with lower batch sizes generally achieve lower training losses, leading to the better generalization on test and validation sets in Fig.
Using no AG makes the training procedure less stable and occasionally large loss spikes occur, leading worse generalization as a result.
E A BLATION S TUDY D ETAILS
In the following, details for the ablation study models in Sec.
6 are provided.
The proposed VolSiM metric uses the multiscale architecture MS described in Sec.
5 as a feature extractor.
For all models, the general training setup mentioned in App.
A stays identical, apart from: 1) changes to the feature extractor, 2) a different feature extractor, 3) the training amount, or 4) the training data.
Changes to MS Architecture For MS no skip the resolution skip connections (red dashed arrows in
Fig. 5) are entirely removed.
This means each resolution is isolated, and sharing information across features of different levels more difﬁcult.
For MS 3 scales the last scale block is removed, meaning the architecture from Fig.
5 ends with the 1 / 4 resolution level, while the 1 / 8 resolution level is omitted.
For the MS 5 scales model, one additional scale block is added.
In Fig.
5, this corresponds to a 1 /resolution level with a 16 × 16 × 16 AvgPool and a scale block with 64 channels.
Simple CNN Feature Extractor
The CNN trained model employs an entirely different feature extractor network, similar to the classical AlexNet architecture (Krizhevsky et al., 2017).
It consists of 5 layers, each with a 3D convolution followed by a ReLU activation.
The kernel sizes (12 - 5 - 33 - 3) decrease along with the strides (4 - 1 - 1 - 1 - 1), while the number of features ﬁrst increase, then decreases again (32 - 96 - 192 - 128 - 128).
To create some spatial reductions, two 4 × 4 ×
MaxPools with stride 2 are included before the second and third convolution.
The normalization and aggregation of the resulting feature maps is performed as described in App.
A, in the same way as for the proposed feature extractor network.
As a result, CNN trained can also be applied in a fully convolutional manner for the scaling invariance experiment in Fig.
Random Models For the random models CNN rand and MS rand , the corresponding feature extractor along with the aggregation weights are only initialized and not trained.
However, to normalize every feature to a standard normal distribution, the training data is processed once to determine the feature mean and standard deviation before evaluating the models, as detailed in App. A.
Different Training Data For the MS added Iso model, we created 400 additional sequences from the JHTDB according to the Iso column in Tab. 4, that are added to the other training data.
Note that they utilize different random seeds than the 60 test sequences, so the Iso data set becomes an additional validation set in Tab. 1.
Similarly for the MS only Iso model, 1000 sequences according to the Iso column in Tab. 4 were collected.
They replace all other training data from the original model, meaning the Iso data set becomes a validation set, and Adv , Bur , Liq , and Smo become further test sets in Tab. 1.
For consistency, the All column in Tab. 1 still reports the combined values of the original test sets for both cases.
F A DDITIONAL A BLATIONS
To further investigate our method, we perform three additional ablations in the following.
The ﬁrst ablation focuses on the beneﬁts of our entropy-based similarity model, by replacing it with a simpliﬁed identity transformation.
The other two replace the multiscale aspect in different ways, to demonstrate the robustness and memory efﬁciency of the proposed architecture.
Tab. 2 shows the resulting SRCC values for these models and the corresponding baselines on our data sets, which are computed as for Tab. 1.
Simpliﬁed Similarity Model We compare the VolSiM model as proposed in the main paper with
MS identity that does not make use of the similarity model based on entropy during training.
For that, the logarithmic transformation of w s is replaced with an identity transformation, corresponding to linear ground truth distances, i.e. g ← w s in Alg. 1.
The results in the ﬁrst block in Tab. 2 indicate the beneﬁts of the entropy-based ground truth, with improvements across most data sets.
Note that the simpliﬁed network version is slightly more efﬁcient as the computation of the parameter c in
Alg. 1 can be omitted.
However, pre-computing these values for all sequences is computationally very light compared to training the full metric network.
Singlescale Model To investigate the multiscale aspect of our architecture, we removed all scale blocks at lower resolutions, such that only the component at the native input resolution remains (see
Fig. 5).
To compensate for the lower number of network parameters, we scale the number channels in each layer by a factor of 11 , leading to the rather shallow but wide network MS 1 scale with around 360 k weights.
The original network structure MS 4 scales with the four scale blocks reconstructs the ground truth more accurate for almost all data sets as displayed in the middle block of Tab. 2.
Furthermore, the proposed structure requires about ﬁve times less memory during training.
No Pooling Layers A different approach to analyze the multiscale aspect, is to eliminate all AvgPool layers and set all convolution strides to 1 (also see Fig.5).
Here, no further adjustments to the network are required as neither pooling layers nor strides alter the number of network weights.
The resulting MS no pool model is only slightly worse compared to the baseline, as indicated by bottom block in Tab. 2.
However, the MS with pool model needs about ten times less memory compared to the adjust variant during training due to the signiﬁcantly smaller feature sizes.
Note that for both ablations on the multiscale structure, the training and test data resolution was reduced to 32 3 .
Furthermore, the number of channels in each layer for the MS with pool and MS no pool models was reduced by a factor of 0 . 5 .
Both choices are motivated by memory limitations and are the reason for the slightly different results across the three baseline models in Tab. 2.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
85 MS identity 0.
MS 4scales 0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
84 MS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MS withpool 0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
.83 MS nopool 0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] G T URBULENCE C ASE S TUDY D ETAILS For the case study in Sec. 6, the velocity ﬁeld from the isotropic1024coarse data set from the JHTDB (Perlman et al., 2007) is utilized.
Instead of creating adjusted sequences according to the similarity model, the data is directly converted to three sequences of different time spans without any randomization in this case.
A spatial stride of 8 is employed is employed for all cases, and to reduce memory consumption an additional temporal stride of 20 is used for the long time span only.
The resulting sequences exhibit a spatial resolution of 128 3 , with 40 frames for the short span, 400 frames for medium span, and 200 frames for the long span.
Before further processing, each frame is individually normalized to [ − 1 , 1] .
To create the results in Fig.
8, the ﬁrst simulation frame t 1 is compared to all following simulation frames t i via Pearson’s distance 1 − PCC ( t 1 , t i ) (red dashed trajectory) and VolSiM ( t 1 , t i ) (blue solid trajectory).
Note that VolSiM is applied fully convolutionally here, as it was trained on 64 3 data.
For both cases, the resulting distances are normalized to [0 , 1] to visually compare them more easily.
The gray dotted curve, is the direct application of the proposed similarity model via Alg. 1 by ﬁtting c for each time span, as described in Sec. 3.
The examples from the sequences are visualized as described in App.
C, while a projecting along the x-axis here.
[Table]
http
[Table]
//mantaflow
[Table]
com [Table] [Table]
[Figure]
Burger’s Equation ( 2 × velocity
[Figure]
density
[Figure]
leveset
[Figure]
[Figure]
Iso : Isotropic turbulence ( 3 × velocity) Cha : Channel ﬂow ( 3 × velocity)
Mhd : Magneto-hydrodynamic turbulence ( 3 × velocity)
Tra : Transitional boundary layer ( 3 × velocity)