Knowledge-Guided Transfer Learning for Modeling Subsurface Phenomena Under Data Paucity
Anonymous Author(s)
AffiliationAddress email
Abstract
Knowledge transfer from machine learning (ML) models, pre-trained on largecorpuses has been leveraged effectively in domains like natural language processingand computer vision to improve model generalization.
The knowledge transferprowess of ML and especially deep learning (DL) models has been demonstratedto be especially effective under data paucity of the target task.
Many scientificphenomena require the execution of costly simulations to estimate the process ofinterest.
Predicting molecular configuration of fluids confined in porous mediais one such problem of extreme relevance in many scientific applications, thestudy of which requires the execution of expensive Molecular Dynamics (MD)simulations.
However, due to the cost of MD, large scale simulations becomeintractable.
Hence, in this work, we develop a novel science-guided deep learningframework NanoNet-SG to emulate MD simulations.
Our proposed NanoNetSG model leverages scientific domain knowledge in conjunction with knowledgefrom pre-trained knowledge bases for estimating molecular configuration of fluidmixtures.
Through rigorous experimentation, we demonstrate that our proposed
NanoNet-SG model yields good generalization performance improvements (i.e.,overall performance improvement of greater than 16 % over baselines) and yieldspredictions that are consistent with known scientific domain rules despite beingtrained on a low volume of MD simulation data (i.e., data paucity).
Introduction
An especially popular use of ML models (primarily deep learning models) is to function as relativelyinexpensive surrogates to emulate the behavior of costly scientific processes.
Owing to their abilityto model complex functions, various ML (surrogate) models have been proposed for modeling fluidflows Brunton et al.
[2019], Wang et al.
[2020], Raissi et al.
Deep learning models, which are usually capable of effectively modeling complex functions, areplagued by their inability to learn good representations under data paucity.
However, certain scientificmodeling contexts inherently suffer from data paucity owing to expensive simulation and datacollection costs.
One such scientific application, which is the focus of this work is modeling fluidproperties in nano-porous media Amini and Mohaghegh [2019], Monteiro et al.
Fluidproperties in nano-porous media are important in applications like petroleum engineering Amini and
Mohaghegh [2019]
Lubbers et al.
Santos et al.
[2020], carbon sequestration Li et al.
[2020a] and energy conservation.
With the increasing energy demands and technological advances likehydraulic fracking, unconventional resources like shale formations are being leveraged as potential sources of energy.
Hydrocarbons like methane and ethane which are abundant in nano-pores in largeshale formations, serve as feasible and effective candidates to address increasing energy demands.
One of the main challenges to modeling fluid properties (e.g., viscosity, density, adsorption patterns)in shale reservoirs is that the size of the pores comprising the hydrocarbons of interest are often lessthan 50 nanometers (nm) in width.
It has been observed that fluid properties in such small pores (i.e.,under nano-confinement) significantly deviate from their properties under bulk conditions, i.e. inlarger spaces where interactions with the surface are rare.
Thus, modeling of fluid properties undernano-confinement cannot be faithfully represented using classical fluid models (e.g. Navier-Stokes
Equations), and so often researchers employ more expensive molecular dynamics (MD) simulations.
Owing to the high computational demands of modeling fluid properties via MD, it is infeasible tocompute fluid properties directly for all possible pore spaces.
In this work we develop NanoNet-SG , a novel science-guided framework to emulate MD simulationsfor fluid mixtures.
Specifically, to alleviate the cost of expensive MD simulations of fluid mixtures,
NanoNet-SG leverages limited fluid mixture simulation data and relies mostly on knowledge gainedfrom relatively simpler (and cheaper)
MD simulations of singular fluids in related experimentalcontexts.
This knowledge from modeling singular fluid simulations (with relatively large datavolumes) to modeling fluid mixtures (problem of interest) with low data volumes is performed via
Science-Guided Knowledge Transfer in our proposed NanoNet-SG model.
Our contributions are as follows:
• We develop a novel and light-weight inductive transfer learning framework NanoNet-SG formodeling the density of fluid mixtures under nano-confinement.•
Our NanoNet-SG model leverages scientific domain knowledge to influence training dynamics.•
We demonstrate through rigorous experimentation, the generalization ability of NanoNet-SG .
• We also verify the improved domain consistency of NanoNet-SG relative to variants without science
guidance.
Background
To counter the effects of modeling under data paucity, we leverage the well studied transfer learningparadigm to develop our NanoNet-SG model.
2.1 Transfer Learning
Essentially, the paradigm of transfer learning (effectively used in various natural language Ruderet al.
[2019], computer vision Li et al.
[2020b] and healthcare Gupta et al.
[2020], Chen et al.
[2019])relaxes the fundamental assumption in traditional ML paradigms of distributional similarity in trainingand testing domains.
Instead, transfer learning approaches focus on developing ML frameworkswith the ability to apply knowledge and skills learned on previous tasks to a related but novel taskwherein data might not necessarily share distributional similarity with previously encountered tasks.
Transfer learning is especially effective in training ML models on tasks with low volumes of data, byfirst training the model on (i.e., leveraging knowledge from) a related task with relatively larger datavolumes.
We now re-state relevant preliminaries detailed in Pan and Yang [2009] specific to transferlearning.
Let us assume a domain D comprising a feature space X and a marginal distribution P ( x ) wherex ∈ X is a particular data instance.
In a specific domain D = {X , P( x ) } , let us define a task T tocontain a label space Y and a prediction function f( · ) to be learned from training data.
For an instancex i ∈ X , f( x i ) yields the corresponding estimate y i ∈ Y .
Consider a source domain D S (and source learning task T S ) and a target domain D T (target learningtask T T ).
Let C S
= { ( x 1 S , y 1 S ) , .., ( x nS , y nS ) } and C T = { ( x 1 T , y 1 S ) , .., ( x mT , y mT ) } represent the training data corpus in the source and target domains, respectively, such that | C T | << | C S | , where| · | indicates cardinality of the data corpus (i.e., available training data volume).
Definition 1 Given a source domain and task ( D S , T S ) and a related but distinct target domain andtask ( D T , T T ) such that | C T | << | C S | , transfer learning aims to improve the learned predictivefunction f T (C T ) by leveraging knowledge from ( D S , T S ) .
Following from Def. 1, a setting
in which T S ̸ = T T is specifically characterized as Inductive Transfer
Learning (ITL)
Pan and Yang [2009].
2.2 Fluid Constitution Under Nano-Confinement
The overall structure of a density profile of a fluid confined in a nano-pore comprises of two regions.
Peak Region .
The density distribution of the fluid closest to the pore walls which exhibit increasedconcentration (i.e., peaks ) due to adsorption onto the pore walls.
Bulk Region .
Intuitively, the bulkregion corresponds to any part of the density profile away from the walls where the adsorption effectsdue to the pore walls is negligible.
For simplicity, we assume that the bulk region corresponds to thedensity distribution in and near the center of the pore.
Problem Formulation
Setup.
We propose a novel ML emulator that leverages the power of inductive transfer learning (ITL)and scientific domain knowledge to develop a novel science-guided architecture (NanoNet-SG ) forfluid density estimation of a mixture M of fluids under nano-confinement.
Large scale MD simulations for fluid mixtures are expensive to setup and execute, leading to datapaucity in the target domain D T .
To alleviate the effect of data paucity in D T , we leverage ITLand pre-train models to encode knowledge from the relatively simpler context of single fluid nanoconfinement for each of the constituent fluids in mixture M .
We now detail the setting of interest.
3.1 Target Domain Description
Our problem comprises of a target domain D T , target task T T and function f T : R
In → R Out thatestimates fluid density profiles for a mixture of ‘k’ constituent fluids M = { F 1 , F 2 , .., F k } .
f T ( · ) istrained on a limited data corpus C T =
{ ( x 1 T , Y 1 T ) , ..., ( x mT , Y mT ) } where x iT corresponds to the i thset of input features and Y iT ∈ R | M |× l (where l is the estimation horizon of the density profiles) isthe corresponding set of target density profiles.
3.1.1 Target Task Feature Description.
Each set of input features x iT = { e i , υ i , ˆ y iS 1 , ˆ y iS 2 , .., ˆ y iS k } consists of three components (i)
theexperimental setting e i (ii) the spatial pore encoding υ
i and (iii) the set of predictions ˆ y iS j frompre-trained single fluid models in the source domain D S j for each constituent fluid F j ∈ M .
Experimental Setting.
In e i = { w i , t i , p i , ψ i 1 , .., ψ ik } .
w i ∈ R 1 × 1 indicates the width of the porebeing modeled, where the mixture of fluids is confined (measured in Angstroms), t i ∈ R 1 × 1 signifiesthe temperature in the pore (measured in Kelvins) and p i ∈
R 1 × 1 the total pressure within the pore(measured in Atmospheres).
Each ψ ij ∈ R 1 × 1 value indicates the partial pressure for each constituentfluid F j ∈ M such that p i = | M | (cid:80) j
=1 ψ ij .
Spatial Pore Encoding.
Each spatial pore encoding υ i ∈ R
d × l comprises ‘d’ different spatialencodings of every point in the pore at which the density profile is to be estimated (called theestimation horizon of size ‘ l ’) 1 for each constituent fluid.
Note that υ
i is the same for all fluids in the mixture for a particular instance ‘ i ’.
In our experiments d = 2 , consisting of two separate encodingsfor every point in the estimation horizon.
First encoding being a binary representation indicating ifeach point in the estimation horizon is internal to the pore or outside (i.e. pore wall).
The secondencoding is a continuous representation of the inverse distance of each point in the pore from thenearest pore wall.
Pre-Trained Density Estimates.
ˆ y iS j ∈ R 1 ×
l represents the prediction (by pre-trained models in thesource domain) of fluid density for fluid F j at each point in the estimation horizon for experimentalsetting e i and spatial encoding υ i .
3.2 Source Domain Description
Owing to the data paucity due to simulation cost and complexity of MD simulations in the targetdomain D T , we leverage knowledge from a related (albeit simpler) paradigm of modeling fluiddensity under nano-confinement of singular fluids.
For each fluid F j ∈ M , our source context is( D S j , T S j , f S j : R In S → R Out S ) where D S j is the source domain comprising of the training data
C S j
= { ( x 1 S j , y 1 S j ) , .., ( x nS j , y nS j ) } used to train a learning model to estimate function f S j for thesingle fluid density estimation task T S j .
x iS j = { e i , υ i } and y iS j ∈ R 1 × l .
The experimental settingfor the source domain comprises of e i = { w i , t i , p i } without the partial pressures as the settingcomprises only of a single fluid.
υ i ∈ R
d × l is encoded same as in the target domain.
The set of tasks
T S =
{ T S 1 , .., T S | M | } results in | M | pre-trained models, each trained to estimate fluid density for aparticular fluid F j ∈ M .
3.3 Domain Adaptation with NanoNet
[Figure]
Fig.
1 outlines the inductive transfer learning architecture of our proposed NanoNet-SG model.
Themodel architecture comprises
Encoding module
module captures relevant
The Adaptation module combines outputs of the interaction , spatial encoding to yield the Adapted
Mixture Prediction .
Finally, the Mapper module which is conditioned on experimental setting e i ,also comprising partial pressures of each constituent fluid in the mixture being modeled producesoutputs used to correct the Adapted Mixture Predictions to yield the final output of density profilepredictions.
Finally, the adapted mixture prediction is combined with appropriate corrections toyield the final predicted output of the fluid density for each constituent fluid in the mixture M .
The
Interaction Encoding, Spatial Encoding & Adaptation modules leverage 1 × 1 convolutions Lin et al.[2013] while the Mapper module is a set of fully connected blocks.
3.4 Science-Guided Domain Adaptation with NanoNet-SG
The overall NanoNet-SG training loss is a combination of the data-driven and the science guided lossfunctions and is represented in Eq.
[Equation]
Data Loss ( L Data ).
The NanoNet-SG model is trained on data corpus C T with data-driven meansquared error loss (Eq. 2).
Here, y ijT ∈ R 1 × l represents the density profile of the i th instance forfluid F j ∈ M , and ˆ y ijT ∈ R 1 × l represents the corresponding density profile prediction made by
NanoNet-SG model for input x iT .
[Equation]
Science-Guided Loss ( L SG ).
In order to improve model generalization and produce domain consistent representations under data paucity, we augment the NanoNet-SG training pipeline with scienceguidance derived from theoretical knowledge about fluids under nano-confinement.
Specifically, weadopt the following well understood and accepted phenomenon.
Remark
We draw from thermodynamics that total density ‘ ξ ’ of a fluid under nano-confinement ismonotonically non-decreasing with increase in pressure (p) for a given pore width (w), temperature(t).
Let ξ ij represent the total density, calculated as the area under a predicted density curve ˆ y ijT .
Further,without loss of generality, we represent ξ ijp to be the total density for an experimental setting e i withpressure p .
The monotonically non-decreasing relationship between ξ and pressure is incorporated directly intothe training loss to yield the novel NanoNet-SG model as defined in Eq.
[Equation]
We choose the step-size ∆ resulting from uniformly dividing our entire domain of interest (i.e.,pressure range of interest as described in Table. 1) into 100 bins.
Experimental Setup
4.1 Dataset Description
As described in Section.
3, we employ two separate types of datasets.
The first type (i.e., the targetdomain) being the molecular dynamics simulations of fluid mixtures (i.e., a 2 fluid mixture ofmethane and ethane) and the source domain(s) being data corpuses from MD simulations of eachof the constituent fluids (i.e., separate datasets of methane and ethane simulated in isolation).
Thepressure and temperature ranges as well as the size of each dataset are described in Table 1.
Ourexperimental setups comprise distinctly of fluid mixture and single fluid modeling tasks.
In this work,we focus on modeling the two hydrocarbons methane and ethane both in the single fluid and mixturesettings.
The experimental setup ranges and data generation processes are primarily governed bydomain experts in petroleum engineering.
Although we focus on the task of fluid density estimation,the proposed NanoNet-SG model can be easily adapted to estimate other properties like viscosity .
[Table]
4.2 Baseline Model Description
[Table]
Single-Fluid CNN (SF-CNN ) .
A fully convolutional neural network architecture 3 with kernel sizeof 13 (large enough to capture the spatial span of adsorption phenomena as determined by domainexperts).
This model accepts input features in the source domain D S for a particular fluid and yieldsthe density profile at every point in the estimation horizon as specified in the inputs.
Two-Fluid CNN (TF-CNN ) .
Similar architecture3 0 to the Single-Fluid CNN model, except theoutputs yield density profiles for multiple fluids (i.e., all constituent fluids) in a mixture M beingmodeled.
Additionally the experimental settings added as input to the model also comprise the partialpressures of each constituent fluid in M .
4.3 Training & Evaluation Pipeline
In each of our single fluid and two fluid experimentation settings we employ 64% of the data fortraining and 36% for testing (data sizes i.e., number of simulations in Table 1).
Each model is trainedfor 15000 epochs with the Adam optimizer, a learning rate of 1e − 4 and mean squared error loss 4 .
Results & Discussion
We inspect our proposed NanoNet-SG framework in multiple qualitative and quantitative contextsto better understand model performance.
Specifically, we are interested in dissecting the followingaspects of model performance.
(1) The need for Domain Adaptation.
(2) Overall generalization performance.
(3) Effect of scienceguidance on domain consistency.
5.1 The Need for Domain Adaptation
To investigate this aspect, we leverage Single-Fluid CNN models (i.e., trained on single fluid simula- tions) to directly yield predictions in the target domain (i.e., without re-training).
Fig.
2 depicts a qualitative sample 5 (pore width = 20) of the prediction performance of the Single-Fluid CNN models
(trained on large corpus of single fluid simulations in similar experimental settings, separately formethane & ethane) tested on fluid density prediction in the target domain.
We can notice distinct deviations in prediction performance in the Single-Fluid CNN model trainedon individual fluids.
In the case of the methane prediction curve, we notice that the Single-Fluid
CNN model is able to predict the bulk behavior of the fluid accurately while overestimating theadsorption effects at the pore boundaries.
However, in the case of ethane prediction, we notice thatthe bulk density behavior of the fluid is overestimated by the corresponding Single-Fluid CNN modelwhile the overestimation of the primary peaks (i.e., the highest peak at each end of the pore boundary)is relatively minimal.
Thus, Fig.
2 reiterates the need for adapting the predictions of the Single-Fluid
CNN model to also account for non-trivial mixture effects on each constituent fluid.
5.2 Overall generalization performance
In an effort to effectively model fluid mixtures by accounting for mixture effects under data paucity,we have developed the NanoNet-SG framework.
Here, we present generalization results of theproposed framework compared to the Single-Fluid CNN and Two-Fluid CNN models (detailed insec. 4.2).
In Table 2, we notice that NanoNet (variant of NanoNet-SG without science guided loss)
[Table]
222 and NanoNet-SG outperform other baselines thereby exhibiting superior generalization performance.
Specifically we notice that the models exhibit better generalization performance overall as well as inthe context of unseen pore widths and pressures.
We also notice that NanoNet & NanoNet-SG modelsyield more consistent predictions (i.e., low standard deviations) across all inspected generalizationcontexts.
From Table 2 , we can infer that inductive transfer has a significant effect in domain adaptationand enables improved generalization performance .
We further infer that NanoNet-SG model is thebest performing overall (also with the least overall standard deviation - Table. 2) primarily due toscience-guidance incorporated during model training leading to improved prediction consistency.
Further baseline comparison (with state of the art regression baselines) are included in the appendix.
Qualitative Performance Characterization
Fig.
3 demonstrates (qualitatively) 6 the improvements in fluid density estimation obtained for each ofthe two fluids (i.e., methane and ethane).
Specifically, we notice from Fig.
3a, Fig.
3b, a significantimprovement in the prediction of the peak densities and bulk densities.
This is in contrast to theprediction results by the SF-CNN models without science guidance or ITL wherein peak and bulkdensities were significantly overestimated.
Hence, based on results in Fig. 3 and result in Table 2wemay infer the effectiveness of our proposed NanoNet-SG model.
methane
3b
We notice significant improvements in bulk and peak overestimation compared to SF-CNN (Fig. 2).
5.3 Effect of Science Guidance on Domain Consistency
In addition to prediction performance characterization, we also investigate domain consistency of
NanoNet-SG .
In Fig.
4 we plot the evolution of cumulative 7 total density over pressure (across theentire range of interest of pressure) at different pore widths and temperatures.
Each subplot Fig. 4(a)
-(c) compares the total cumulative density derived from the density prediction of NanoNet (yellowcurve) and NanoNet-SG (blue curve).
The phenomenon of interest is to ensure that the evolution oftotal cumulative density depicts monotonically non-decreasing behavior according to known scientifictheory (Remark 1).
In each plot in Fig.
4 we notice that the evolution of the total cumulative densityderived from the NanoNet predictions violate the monotonically non-decreasing constraint at multiplepoints (indicated with while the regions exhibit
Conclusion
In this work, we have presented NanoNet-SG , a novel model for science-guided learning under datapaucity via.
inductive transfer learning.
Our contributions are the novel thermodynamic constraintencoded into NanoNet-SG and the inductive transfer learning framework for modeling density of fluidmixtures under nano-confinement and data paucity.
Through rigorous analysis, we have demonstratedthat NanoNet-SG yields significantly better predictions than baselines for fluid density estimation.
The superior generalization performance of NanoNet-SG holds in both the bulk and peak regions 8of
the density profile and also in the case of seen and unseen settings (i.e., unseen pore widths andpressures).
Finally, we also demonstrate that NanoNet-SG yields more domain-consistent predictionsas a function of the novel science-guided training mechanism.  Baseline Model Architecture Description
We employ the same basic architecture w.r.t tasks (i.e., density profile shape prediction, densityprofile bulk prediction) for our Single-Fluid CNN and Two-Fluid CNN models as detailed in Fig.
It must be noted that NanoNet and NanoNet-SG models employ predictions from the Single-Fluid
CNN models as pre-trained baselines in their pipeline.
[Figure]
Spatial Density Profile Component.
In Fig.
1, we notice that our model comprises a multi-layer fullyconvolutional component (depicted as yellow rectangles) which accepts the spatial pore encodingυ i (see section 3.1 for details) as well as the experimental setting
e
i (see section 3.1) as input andyields ˆ ρ s ∈ R | M |× l the spatial component of the density profile estimate for each fluid in the mixture
M and each point in the estimation horizon l .
We employ a constant kernel size of 13 and a constantnumber of channels 32 for both the Single-Fluid CNN and Two-Fluid CNN models.
Bulk Density Profile Component.
Further, the second component of the model (Fig. 1) is amulti-layer fully connected network (depicted as purple boxes) which is conditioned only upon theexperimental setup e i to yield an estimate of ˆ ρ b ∈ R
| M |× 1 the bulk component of the density profileestimate for each fluid in the mixture M .
The outputs ˆ ρ s and ˆ ρ b from the spatial and bulk components are summed together (point-wise) toyield the final density profile estimate y i ∈ R | M |× l .
Note, in the case of Single-Fluid CNN (i.e., single fluid predictions) | M | = 1 and the experimentalsetup does not comprise of partial pressures (i.e., e i =
[w i , t i , p i ] ).
Both Single-Fluid CNN and
Two-Fluid CNN models have about ∼
000 learnable parameters.
The Single-Fluid CNN and
Two-Fluid CNN model architectures have been developed as part of a parallel work by our group andthe publication (in preparation) will be cited in the current paper upon acceptance.
1.1 NanoNet-SG Model Architecture Details
The Spatial Encoding Module (sec. 3.3) and Interaction Encoding Module (sec. 3.3) both compriseof 2 input channels and 16 output channels.
The outputs of the Spatial Encoding and Interaction
Encoding modules are concatenated (channel-wise) and the 32 channel input is operated on by the
Adaptation Module (sec. 3.3) comprising two convolutional blocks sequentially connected.
Firstof the two convolutional blocks has 32 input channels and 16 output channels while the secondconvolutional block has 16 input channels followed by | M | (i.e., the number of fluids in the mixturebeing predicted) output channels 1 .
The | M | channel output of the Adaptation Module is corrected bythe output of the Mapper module (which is a | e i | × l ∗ | M | dimensional fully connected network) toproduce an | M | channel output to correct each point in the prediction horizon ‘ l ’ obtained from the
Adaptation Module .
Through this process, NanoNet-SG architecture is able to adapt the predictionsobtained from pre-trained single fluid models to the complex fluid-mixture modeling context withonly 5108 learnable parameters while also yielding improved domain consistency and superiorgeneralization ability relative to models that are agnostic to science guidance.
Qualitative Figures2.1 Single-Fluid CNN Model on Two-Fluid Prediction Task
We notice from Fig.
2, Fig.
3, that the behavior of the Single-Fluid CNN model is consistent with thatpresented in the main paper for other sample prediction scenarios in the target domain.
Essentially,we once again notice an overestimation of the bulk density in the context of ethane profile predictionand overestimation of the peak in the methane profile prediction context.
2.2 NanoNet-SG Model Predictions
We also include qualitative samples of the NanoNet-SG density profile predictions.
Fig.
4 and
Fig.
5 demonstrate that NanoNet-SG is able to yield consistent accurate predictions across all thepredictions contexts sans any underestimation or overestimation behavior as observed in the case ofthe Single-Fluid CNN models.
[Figure]
Baseline Performance Comparison
3.1 Baseline Model Description
Random Forest (RF-Reg.).
We train a random forest regressor (with 100 estimators) for the task ofdensity profile prediction.
Specifically, for a particular experimental setup e i , and a particular porelocation ‘q’, the model is trained to predict ˆ y i T ( q ) ∈ R 1 × 1 (i.e., the fluid density at location ‘q’).
Themodel is conditioned upon e i and υ i ( q ) where υ i ( q ) represents the spatial inputs at pore location ‘q’.
Gradient Boosting (GB-Reg.).
The problem setup for the GB-Reg.
model (with 100 estimators,squared error loss function, learning rate of 0.1) is identical to that of the RF-Reg.
model (i.e.,predicting density of a fluid at each location ‘q’ conditioned upon e i , υ i ( q ) ).
Linear Regression (Lin-Reg.).
The problem setup for the Lin-Reg.
model is identical to the RF-Reg.,
GB-Reg.
setups.
Single-Fluid CNN (SF-CNN ) .
A fully convolutional neural network architecture 2 with kernel sizeof 13 (large enough to capture the spatial span of adsorption phenomena as determined by domainexperts).
This model accepts input features in the source domain D S for a particular fluid and yieldsthe density profile at every point in the estimation horizon as specified in the inputs.
Two-Fluid CNN (TF-CNN ) .
Similar architecture2 0 to the Single-Fluid CNN model, except theoutputs yield density profiles for multiple fluids (i.e., all constituent fluids) in a mixture M being modeled.
Additionally the experimental settings added as input to the model also comprise the partialpressures of each constituent fluid in M .
Performance Comparison.
To further investigate the performance of NanoNet-SG , we compare
NanoNet (orange), NanoNet-SG (red) models yield lowest errors and standard deviations.
67 the prediction performance to state-of-the-art regression baselines (details in sec. 3.1).
In Fig.
6 wenotice that NanoNet and NanoNet-SG models yield significantly improved prediction performance onboth the methane and ethane profile prediction tasks in the fluid mixture density profile predictioncontext.
We specifically report methane and ethane profile prediction errors for each model in Fig.
6and notice that NanoNet and NanoNet-SG outperform all other models in both prediction tasks (i.e.,methane, ethane density profile prediction).
[Table]
We once again notice that NanoNet-SG yields best overall performance due to knowledge transfer coupled with science-guidance.
In Table 1, we further showcase a task breakdown of the generalization performance observed in
Table 1 (main paper) by inspecting generalization errors in the bulk and profile prediction contextper fluid for each model.
We can infer from Table 1 that models NanoNet and NanoNet-SG yieldsignificant performance improvement owing to the effect of the knowledge transfer architectureand the science guidance in the case of NanoNet-SG .
Specifically, we notice that science guidanceresults in improved overall generalization as well as bulk and profile generalization in both the caseof Methane and Ethane.
Science guidance leads to NanoNet-SG gaining a 16.26% performanceimprovement over its variant without science guidance (i.e., NanoNet ) and significantly larger gainsin performance over SF-CNN and TF-CNN models.
Qualitative Performance Characterization.
We notice from Fig.
7 that NanoNet-SG yields highlyaccurate density estimates across the entire range of the prediction spectrum of fluid density for eachof the fluids in the mixture.
Also apparent from the plots is the ability of the model to fit the multipledistributional modes in the density profile owing to the density in the peak and bulk regions due tothe presence and absence of adsorption effects respectively.
These multiple modes are indicated bydisjoint clusters of points in each plot in Fig.
We also report the R 2 statistic for density predictionof NanoNet-SG which indicates consistent performance across the two fluids with R 2 =0.991 formethane density prediction and R 2 =0.998 for ethane density prediction.
P o i n t D
en s i
t y
P o i n t D
en s i
t y
Methane
Ethane (a) (b)
3.2 Effect of Science Guidance on Domain Consistency
Further, in Fig.
8 (a), 8(b) we depict evolution of total cumulative densities for NanoNet and NanoNetSG at a sample pore width of 25 at various temperatures.
We once again notice that the evolution for
NanoNet showcases significant deviations from the monotonically non-decreasing domain constraintwhereas the NanoNet-SG model predictions result in total density curves that adhere to the scientificdomain knowledge much more consistently.
From Fig. ??
and Fig.
8, we can infer that the NanoNet-
SG model significantly domain results owing science-guided trainingcriterion
Related Work
Transfer Learning -Transfer learning as a model training paradigm has been effectively leveraged inapplications like natural language processing Ruder et al.
[2019], computer vision Li et al.
[2020],healthcare Gupta et al.
[2020], Chen et al.
[2019] and many other contexts.
However, few efforts haveemployed transfer learning under data paucity in scientific applications.
To the best of our knowledge,our NanoNet-SG is the first science-guided model to employ inductive transfer learning to counterthe effects of data paucity in scientific applications.
Fluid-Flow Modeling.
Various ML models have been proposed for modeling fluid flows Bruntonet al.
[2019], Wang et al.
[2020], Raissi et al.
However all these models (unlike NanoNet-SG )either assume complete knowledge of the physics (e.g., PDE, ODE, initial, boundary conditions)governing the system or employ large datasets in target domain.
Modeling Under Nano-confinement.
The works Santos et al.
[2020], Lubbers et al.
[2020] areclosest to ours.
In Santos et al.
[2020], authors propose an active-learning framework for training a MLemulator of MD simulations and Lubbers et al.
[2020] develop an ML model to bridge fine-grained(MD) and coarse-grained (Lattice-Boltzman) simulations of nano-confined fluids.
However, neither of these works tackle the complex problem of modeling fluid mixtures and neither work employsexplicit science guidance in the loss function for modeling under data paucity.