Knowledge-Guided Transfer Learning for Modeling Subsurface Phenomena Under Data Paucity
Abstract
Knowledge transfer from machine learning (ML) models, pre-trained on large corpuses has been leveraged effectively in domains like natural language processing and computer vision to improve model generalization.
The knowledge transfer prowess of ML and especially deep learning (DL) models has been demonstrated to be especially effective under data paucity of the target task.
Many scientiﬁc phenomena require the execution of costly simulations to estimate the process of interest.
Predicting molecular conﬁguration of ﬂuids conﬁned in porous media is one such problem of extreme relevance in many scientiﬁc applications, the study of which requires the execution of expensive Molecular Dynamics (MD) simulations.
However, due to the cost of MD, large scale simulations become intractable.
Hence, in this work, we develop a novel science-guided deep learning framework NanoNet-SG to emulate MD simulations.
Our proposed NanoNetSG model leverages scientiﬁc domain knowledge in conjunction with knowledge from pre-trained knowledge bases for estimating molecular conﬁguration of ﬂuid mixtures.
Through rigorous experimentation, we demonstrate that our proposed
NanoNet-SG model yields good generalization performance improvements (i.e., overall performance improvement of greater than 16 % over baselines) and yields predictions that are consistent with known scientiﬁc domain rules despite being trained on a low volume of MD simulation data (i.e., data paucity).
Introduction
An especially popular use of ML models (primarily deep learning models) is to function as relatively inexpensive surrogates to emulate the behavior of costly scientiﬁc processes.
Owing to their ability to model complex functions, various ML (surrogate) models have been proposed for modeling ﬂuid ﬂows Brunton et al.
[2019], Wang et al.
[2020], Raissi et al.
Deep learning models, which are usually capable of effectively modeling complex functions, are plagued by their inability to learn good representations under data paucity.
However, certain scientiﬁc modeling contexts inherently suffer from data paucity owing to expensive simulation and data collection costs.
One such scientiﬁc application, which is the focus of this work, is modeling ﬂuid properties in nano-porous media Amini and Mohaghegh [2019], Monteiro et al.
Fluid properties in nano-porous media are important in applications like petroleum engineering Amini and
Mohaghegh [2019], Lubbers et al.
[2020], Santos et al.
[2020], carbon sequestration Li et al.
[2020a] and energy conservation.
With the increasing energy demands and technological advances like hydraulic fracking, unconventional resources like shale formations are being leveraged as potential sources of energy.
Hydrocarbons like methane and ethane which are abundant in nano-pores in large shale formations, serve as feasible and effective candidates to address increasing energy demands.
One of the main challenges to modeling ﬂuid properties (e.g., viscosity, density, adsorption patterns) in shale reservoirs is that the size of the pores comprising the hydrocarbons of interest are often less than 50 nanometers (nm) in width.
It has been observed that ﬂuid properties in such small pores (i.e., under nano-conﬁnement) signiﬁcantly deviate from their properties under bulk conditions, i.e. in larger spaces where interactions with the surface are rare.
Thus, modeling of ﬂuid properties under nano-conﬁnement cannot be faithfully represented using classical ﬂuid models (e.g. Navier-Stokes
Equations), and so often researchers employ more expensive molecular dynamics (MD) simulations.
Owing to the high computational demands of modeling ﬂuid properties via MD, it is infeasible to compute ﬂuid properties directly for all possible pore spaces.
In this work we develop NanoNet-SG , a novel science-guided framework to emulate MD simulations for ﬂuid mixtures.
Speciﬁcally, to alleviate the cost of expensive MD simulations of ﬂuid mixtures,
NanoNet-SG leverages limited ﬂuid mixture simulation data and relies mostly on knowledge gained from relatively simpler (and cheaper)
MD simulations of singular ﬂuids in related experimental contexts.
This knowledge from modeling singular ﬂuid simulations (with relatively large data volumes) to modeling ﬂuid mixtures (problem of interest) with low data volumes is performed via
Science-Guided Knowledge Transfer in our proposed NanoNet-SG model.
Our contributions are as follows:
• We develop a novel and light-weight inductive transfer learning framework NanoNet-SG for modeling the density of ﬂuid mixtures under nano-conﬁnement.
• Our NanoNet-SG model leverages scientiﬁc domain knowledge to inﬂuence training dynamics.
• We demonstrate through rigorous experimentation, the generalization ability of NanoNet-SG .
• We also verify the improved domain consistency of NanoNet-SG relative to variants without science guidance.
Background
To counter the effects of modeling under data paucity, we leverage the well studied transfer learning paradigm to develop our NanoNet-SG model.
2.1 Transfer Learning
Essentially, the paradigm of transfer learning (effectively used in various natural language Ruder et al.
[2019], computer vision Li et al.
[2020b] and healthcare Gupta et al.
[2020], Chen et al.
[2019]) relaxes the fundamental assumption in traditional ML paradigms of distributional similarity in training and testing domains.
Instead, transfer learning approaches focus on developing ML frameworks with the ability to apply knowledge and skills learned on previous tasks to a related but novel task wherein data might not necessarily share distributional similarity with previously encountered tasks.
Transfer learning is especially effective in training ML models on tasks with low volumes of data, by ﬁrst training the model on (i.e., leveraging knowledge from) a related task with relatively larger data volumes.
We now re-state relevant preliminaries detailed in Pan and Yang [2009] speciﬁc to transfer learning.
Let us assume a domain D comprising a feature space X and a marginal distribution P ( x ) where x 2 X is a particular data instance.
In a speciﬁc domain D = {X , P( x ) } , let us deﬁne a task T to contain a label space Y and a prediction function f( · ) to be learned from training data.
For an instance x i 2 X , f( x i ) yields the corresponding estimate y i 2 Y .
Consider a source domain D S (and source learning task T S ) and a target domain D T (target learning task T T ).
Let C S
= { ( x 1 S , y 1 S ) , .., ( x nS , y nS ) } and C T = { ( x 1 T , y 1 T ) , .., ( x mT , y mT ) } represent the training data corpus in the source and target domains, respectively, such that | C T | << | C S | , where | · | indicates cardinality of the data corpus (i.e., available training data volume).
Deﬁnition 1 Given a source domain and task ( D S , T S ) and a related but distinct target domain and task ( D T , T T ) such that | C T | << | C S | , transfer learning aims to improve the learned predictive function f T (C T ) by leveraging knowledge from ( D S , T S ) .
Following from Def. 1, a setting in which T S 6 =
T T is speciﬁcally characterized as Inductive Transfer Learning (ITL)
Pan and Yang [2009].
2.2 Fluid Constitution Under Nano-Conﬁnement
The overall structure of a density proﬁle of a ﬂuid conﬁned in a nano-pore comprises of two regions.
Peak Region .
The density distribution of the ﬂuid closest to the pore walls which exhibit increased concentration (i.e., peaks ) due to adsorption onto the pore walls.
Bulk Region .
Intuitively, the bulk region corresponds to any part of the density proﬁle away from the walls where the adsorption effects due to the pore walls is negligible.
For simplicity, we assume that the bulk region corresponds to the density distribution in and near the center of the pore.
Problem Formulation
Setup.
We propose a novel ML emulator that leverages the power of inductive transfer learning (ITL) and scientiﬁc domain knowledge to develop a novel science-guided architecture (NanoNet-SG ) for ﬂuid density estimation of a mixture M of ﬂuids under nano-conﬁnement.
Large scale MD simulations for ﬂuid mixtures are expensive to setup and execute, leading to data paucity in the target domain D T .
To alleviate the effect of data paucity in D T , we leverage ITL and pre-train models to encode knowledge from the relatively simpler context of single ﬂuid nanoconﬁnement for each of the constituent ﬂuids in mixture M .
We now detail the setting of interest.
3.1 Target Domain Description
Our problem comprises of a target domain D T , target task T T and function f T : R
In ! R | M | ⇥ l that estimates ﬂuid density proﬁles for a mixture of ‘k’ constituent ﬂuids M = { F 1 , F 2 , .., F k } .
f T ( · ) is trained on a limited data corpus C T =
{ ( x 1 T , Y 1 T ) , ..., ( x mT , Y mT ) } where x iT corresponds to the i th set of input features and Y iT 2 R | M | ⇥ l (where l is the estimation horizon of the density proﬁles) is the corresponding set of target density proﬁles.
3.1.1 Target Task Feature Description.
Each set of input features x iT = { e i ,  i , ˆ y iS 1 , ˆ y iS 2 , .., ˆ y iS k } consists of three components (i) the experimental setting e i (ii) the spatial pore encoding  i and (iii) the set of predictions
ˆ y iS j from pre-trained single ﬂuid models in the source domain D S j for each constituent ﬂuid F j 2 M .
Experimental Setting.
In e i = { w i , t i , p i , i 1 , .., i k } .
w i 2 R 1 ⇥ 1 indicates the width of the pore being modeled, where the mixture of ﬂuids is conﬁned (measured in Angstroms), t i 2 R 1 ⇥ 1 signiﬁes the temperature in the pore (measured in Kelvins) and p i 2 R 1 ⇥ 1 the total pressure within the pore (measured in Atmospheres).
Each ij 2 R 1 ⇥ 1 value indicates the partial pressure for each constituent ﬂuid F j 2 M such that p i = | M | P [Equation]
=1 ij .
Spatial Pore Encoding.
Each spatial pore encoding  i 2 R d ⇥ l comprises ‘d’ different spatial encodings of every point in the pore at which the density proﬁle is to be estimated (called the estimation horizon of size ‘ l ’) 1 for each constituent ﬂuid.
Note that  i is the same for all ﬂuids in the mixture for a particular instance ‘ i ’.
In our experiments d = 2 , consisting of two separate encodings for every point in the estimation horizon.
First encoding being a binary representation indicating if each point in the estimation horizon is internal to the pore or outside (i.e. pore wall).
The second encoding is a continuous representation of the inverse distance of each point in the pore from the nearest pore wall
Pre-Trained Density Estimates.
ˆ y iS j 2 R 1 ⇥ l represents the prediction (by pre-trained models in the source domain) of ﬂuid density for ﬂuid F j at each point in the estimation horizon for experimental setting e i and spatial encoding  i .
3.2 Source Domain Description
Owing to the data paucity due to simulation cost and complexity of MD simulations in the target domain D T , we leverage knowledge from a related (albeit simpler) paradigm of modeling ﬂuid density under nano-conﬁnement of singular ﬂuids.
For each ﬂuid F j 2 M , our source context is ( D S j , T S j , f S j : R In S ! R Out S ) where D S j is the source domain comprising of the training data
C S j
= { ( x 1 S j , y 1 S j ) , .., ( x nS j , y nS j ) } used to train a learning model to estimate function f S j for the single ﬂuid density estimation task T S j .
x iS j = { e i ,  i } and y iS j 2 R 1 ⇥ l .
The experimental setting for the source domain comprises of e i = { w i , t i , p i } without the partial pressures as the setting comprises only of a single ﬂuid.
i 2 R d ⇥ l is encoded same as in the target domain.
The set of tasks
T S =
{ T S 1 , .., T S | M | } results in | M | pre-trained models, each trained to estimate ﬂuid density for a particular ﬂuid F j 2 M .
3.3 Domain Adaptation with NanoNet
Fig.
1 outlines the inductive transfer learning architecture of our proposed NanoNet-SG model.
The model architecture comprises of four sub-modules (outlined as dotted-red boxes).
The Interaction Encoding module accepts predictions from the pre-trained single ﬂuid models ( f S i ( · ) ) and learns to model the interaction and mixture effects relevant to the target context. The Spatial Encoding module captures the relevant spatial information from the  i spatial input of the current pore context.
The Adaptation module combines outputs of the interaction , spatial encoding to yield the Adapted
Mixture Prediction .
Finally, the Mapper module which is conditioned on experimental setting e i , also comprising partial pressures of each constituent ﬂuid in the mixture being modeled produces outputs used to correct the Adapted Mixture Predictions to yield the ﬁnal output of density proﬁle predictions.
Finally, the adapted mixture prediction is combined with appropriate corrections to yield the ﬁnal predicted output of the ﬂuid density for each constituent ﬂuid in the mixture M .
The
Interaction Encoding, Spatial Encoding & Adaptation modules leverage 1 ⇥ 1 convolutions Lin et al. [2013] while the Mapper module is a set of fully connected blocks.
3.4 Science-Guided Domain Adaptation with NanoNet-SG
The overall NanoNet-SG training loss is a combination of the data-driven and the science guided loss functions and is represented in Eq.
[Equation]
Data Loss ( L Data ).
The NanoNet-SG model is trained on data corpus C T with data-driven meansquared error loss (Eq. 2).
Here, y ijT 2 R 1 ⇥ l represents the density proﬁle of the i th instance for ﬂuid F j 2 M , and ˆ y ijT 2 R 1 ⇥ l represents the corresponding density proﬁle prediction made by
NanoNet-SG model for input x iT .
[Equation]
Science-Guided Loss ( L SG ).
In order to improve model generalization and produce domain consistent representations under data paucity, we augment the NanoNet-SG training pipeline with scienceguidance derived from theoretical knowledge about ﬂuids under nano-conﬁnement.
Speciﬁcally, we adopt the following well understood and accepted phenomenon.
Remark
We draw from thermodynamics that total density ‘ ⇠ ’ of a ﬂuid under nano-conﬁnement is monotonically non-decreasing with increase in pressure (p) for a given pore width (w), temperature (t).
Let ⇠ ij represent the total density, calculated as the area under a predicted density curve ˆ y ijT .
Further, without loss of generality, we represent ⇠ ijp to be the total density for an experimental setting e i with pressure p .
The monotonically non-decreasing relationship between ⇠ and pressure is incorporated directly into the training loss to yield the novel NanoNet-SG model as deﬁned in Eq.
[Equation]
We choose the step-size  resulting from uniformly dividing our entire domain of interest (i.e., pressure range of interest as described in Table. 1) into 100 bins.
Experimental Setup 4.1 Dataset Description
As described in Section.
3, we employ two separate types of datasets.
The ﬁrst type (i.e., the target domain) being the molecular dynamics simulations of ﬂuid mixtures (i.e., a 2 ﬂuid mixture of methane and ethane) and the source domain(s) being data corpuses from MD simulations of each of the constituent ﬂuids (i.e., separate datasets of methane and ethane simulated in isolation).
The pressure and temperature ranges as well as the size of each dataset are described in Table 1.
Our experimental setups comprise distinctly of ﬂuid mixture and single ﬂuid modeling tasks.
In this work, we focus on modeling the two hydrocarbons methane and ethane both in the single ﬂuid and mixture settings.
The experimental setup ranges and data generation processes are primarily governed by domain experts in petroleum engineering.
Although we focus on the task of ﬂuid density estimation, the proposed NanoNet-SG model can be easily adapted to estimate other properties like viscosity .
4.2 Baseline Model Description
Single-Fluid CNN (SF-CNN ) .
A fully convolutional neural network architecture 3 with kernel size of 13 (large enough to capture the spatial span of adsorption phenomena as determined by domain [Table] experts).
This model accepts input features in the source domain D S for a particular ﬂuid and yields the density proﬁle at every point in the estimation horizon as speciﬁed in the inputs.
Two-Fluid CNN (TF-CNN ) .
Similar architecture3 0 to the Single-Fluid CNN model, except the outputs yield density proﬁles for multiple ﬂuids (i.e., all constituent ﬂuids) in a mixture M being modeled.
Additionally the experimental settings added as input to the model also comprise the partial pressures of each constituent ﬂuid in M .
4.3 Training & Evaluation Pipeline
In each of our single ﬂuid and two ﬂuid experimentation settings we employ 64% of the data for training and 36% for testing (data sizes i.e., number of simulations in Table 1).
Each model is trained for 15000 epochs with the Adam optimizer, a learning rate of 1e  4 and mean squared error loss 4 .
Results & Discussion
We inspect our proposed NanoNet-SG framework in multiple qualitative and quantitative contexts to better understand model performance.
Speciﬁcally, we are interested in dissecting the following aspects of model performance.
(1) The need for Domain Adaptation.
(2) Overall generalization performance.
(3) Effect of science guidance on domain consistency
5.1 The Need for Domain Adaptation
To investigate this aspect, we leverage Single-Fluid CNN models (i.e., trained on single ﬂuid simulations) to directly yield predictions in the target domain (i.e., without re-training).
Fig.
2 depicts a qualitative sample 5 (pore width = 20) of the prediction performance of the Single-Fluid CNN models (trained on large corpus of single ﬂuid simulations in similar experimental settings, separately for methane & ethane) tested on ﬂuid density prediction in the target domain.
We can notice distinct deviations in prediction performance in the Single-Fluid CNN model trained on individual ﬂuids.
In the case of the methane prediction curve, we notice that the Single-Fluid
CNN model is able to predict the bulk behavior of the ﬂuid accurately while overestimating the adsorption effects at the pore boundaries.
However, in the case of ethane prediction, we notice that the bulk density behavior of the ﬂuid is overestimated by the corresponding Single-Fluid CNN model while the overestimation of the primary peaks (i [Table] e., the highest peak at each end of the pore boundary) is relatively minimal.
Thus, Fig.
2 reiterates the need for adapting the predictions of the Single-Fluid
CNN model to also account for non-trivial mixture effects on each constituent ﬂuid.
5.2 Overall generalization performance
In an effort to effectively model ﬂuid mixtures by accounting for mixture effects under data paucity [Table] we have developed the NanoNet-SG framework.
Here [Table] we present generalization results of the proposed framework compared to the Single-Fluid CNN and Two-Fluid CNN models (detailed in sec. 4.2).
In Table [Table] we notice that NanoNet (variant of NanoNet-SG without science guided loss) [Table]
[Table]
and NanoNet-SG outperform other baselines thereby exhibiting superior generalization performance.
Speciﬁcally we notice that the models exhibit better generalization performance overall as well as in the context of unseen pore widths and pressures.
We also notice that NanoNet & NanoNet-SG models yield more consistent predictions (i.e., low standard deviations) across all inspected generalization contexts.
From Table 2 , we can infer that inductive transfer has a signiﬁcant effect in domain adaptation and enables improved generalization performance .
We further infer that NanoNet-SG model is the best performing overall (also with the least overall standard deviation - Table. 2) primarily due to science-guidance incorporated during model training leading to improved prediction consistency.
Further baseline comparison (with state of the art regression baselines) are included in the appendix.
Qualitative Performance Characterization
We notice signiﬁcant improvements in bulk and peak overestimation compared to SF-CNN (Fig. 2).
Fig.
3 demonstrates (qualitatively) 6 the improvements in ﬂuid density estimation obtained for each of the two ﬂuids (i.e., methane and ethane).
Speciﬁcally, we notice from Fig.
3a, Fig.
3b, a signiﬁcant improvement in the prediction of the peak densities and bulk densities.
This is in contrast to the prediction results by the SF-CNN models without science guidance or ITL wherein peak and bulk densities were signiﬁcantly overestimated.
Hence, based on results in Fig. 3 and result in Table 2we may infer the effectiveness of our proposed NanoNet-SG model.
5.3 Effect of Science Guidance on Domain Consistency
In addition to prediction performance characterization, we also investigate domain consistency of
NanoNet-SG .
In Fig.
4 we plot the evolution of cumulative 7 total density over pressure (across the entire range of interest of pressure) at different pore widths and temperatures.
Each subplot Fig.
4(a) - (c) compares the total cumulative density derived from the density prediction of NanoNet (yellow curve) and NanoNet-SG (blue curve).
The phenomenon of interest is to ensure that the evolution of total cumulative density depicts monotonically non-decreasing behavior according to known scientiﬁc theory (Remark 1).
In each plot in Fig.
4 we notice that the evolution of the total cumulative density derived from the NanoNet predictions violate the monotonically non-decreasing constraint at multiple points (indicated with dotted red circles) while the corresponding regions exhibit consistency with the domain knowledge for the NanoNet-SG cumulative densities (dotted green circles).
Conclusion
In this work, we have presented NanoNet-SG , a novel model for science-guided learning under data paucity via.
inductive transfer learning.
Our contributions are the novel thermodynamic constraint encoded into NanoNet-SG and the inductive transfer learning framework for modeling density of ﬂuid mixtures under nano-conﬁnement and data paucity.
Through rigorous analysis, we have demonstrated that NanoNet-SG yields signiﬁcantly better predictions than baselines for ﬂuid density estimation.
The superior generalization performance of NanoNet-SG holds in both the bulk and peak regionsof the density proﬁle and also in the case of seen and unseen settings (i.e., unseen pore widths and pressures).
Finally, we also demonstrate that NanoNet-SG yields more domain-consistent predictions as a function of the novel science-guided training mechanism.
Baseline Model Architecture Description
We employ the same basic architecture w.r.t tasks (i.e., density proﬁle shape prediction, density proﬁle bulk prediction) for our Single-Fluid CNN and Two-Fluid CNN models as detailed in Fig.
It must be noted that NanoNet and NanoNet-SG models employ predictions from the Single-Fluid
CNN models as pre-trained baselines in their pipeline.
[Figure]
Spatial Density Proﬁle Component.
In Fig.
1, we notice that our model comprises a multi-layer fully convolutional component (depicted as yellow rectangles) which accepts the spatial pore encoding  i (see section 3.1 for details) as well as the experimental setting
e
i (see section 3.1) as input and yields ˆ ⇢ s 2 R | M | ⇥ l the spatial component of the density proﬁle estimate for each ﬂuid in the mixture
M and each point in the estimation horizon l .
We employ a constant kernel size of 13 and a constant number of channels 32 for both the Single-Fluid CNN and Two-Fluid CNN models.
Bulk Density Proﬁle Component.
Further, the second component of the model (Fig. 1) is a multi-layer fully connected network (depicted as purple boxes) which is conditioned only upon the experimental setup e i to yield an estimate of ˆ ⇢ b 2 R | M | ⇥ 1 the bulk component of the density proﬁle estimate for each ﬂuid in the mixture M .
The outputs ˆ ⇢ s and ˆ ⇢ b from the spatial and bulk components are summed together (point-wise) to yield the ﬁnal density proﬁle estimate y i 2 R | M | ⇥ l .
Note, in the case of Single-Fluid CNN (i.e., single ﬂuid predictions) | M | = 1 and the experimental setup does not comprise of partial pressures (i.e., e i =
[w i , t i , p i ] ).
Both Single-Fluid CNN and
Two-Fluid CNN models have about ⇠ 120,000 learnable parameters.
The Single-Fluid CNN and
Two-Fluid CNN model architectures have been developed as part of a parallel work by our group and the publication (in preparation) will be cited in the current paper upon acceptance.
1.1 NanoNet-SG Model Architecture Details
The Spatial Encoding Module (sec. 3.3) and Interaction Encoding Module (sec. 3.3) both comprise of 2 input channels and 16 output channels.
The outputs of the Spatial Encoding and Interaction
Encoding modules are concatenated (channel-wise) and the 32 channel input is operated on by the
Adaptation Module (sec. 3.3) comprising two convolutional blocks sequentially connected.
First of the two convolutional blocks has 32 input channels and 16 output channels while the second convolutional block has 16 input channels followed by | M | (i.e., the number of ﬂuids in the mixture being predicted) output channels 1 .
The | M | channel output of the Adaptation Module is corrected by the output of the Mapper module (which is a | e i | ⇥ l ⇤ | M | dimensional fully connected network) to produce an | M | channel output to correct each point in the prediction horizon ‘ l ’ obtained from the
Adaptation Module .
Through this process, NanoNet-SG architecture is able to adapt the predictions obtained from pre-trained single ﬂuid models to the complex ﬂuid-mixture modeling context with only 5108 learnable parameters while also yielding improved domain consistency and superior generalization ability relative to models that are agnostic to science guidance.
Qualitative Figures 2.1 Single-Fluid CNN Model on Two-Fluid Prediction Task
We notice from Fig.
2, Fig.
3, that the behavior of the Single-Fluid CNN model is consistent with that presented in the main paper for other sample prediction scenarios in the target domain.
Essentially, we once again notice an overestimation of the bulk density in the context of ethane proﬁle prediction and overestimation of the peak in the methane proﬁle prediction context.
2.2 NanoNet-SG Model Predictions
We also include qualitative samples of the NanoNet-SG density proﬁle predictions.
Fig.
4 and
Fig.
5 demonstrate that NanoNet-SG is able to yield consistent accurate predictions across all the predictions contexts sans any underestimation or overestimation behavior as observed in the case of the Single-Fluid CNN models.
Baseline Performance Comparison 3.1 Baseline Model Description
Random Forest (RF-Reg.).
We train a random forest regressor (with 100 estimators) for the task of density proﬁle prediction.
Speciﬁcally, for a particular experimental setup e i , and a particular pore location ‘q’, the model is trained to predict ˆ y iT ( q ) 2 R 1 ⇥ 1 (i.e., the ﬂuid density at location ‘q’).
The model is conditioned upon e i and  i ( q ) where  i ( q ) represents the spatial inputs at pore location ‘q’.
Gradient Boosting (GB-Reg.).
The problem setup for the GB-Reg.
model (with 100 estimators, squared error loss function, learning rate of 0.1) is identical to that of the RF-Reg.
model (i.e., predicting density of a ﬂuid at each location ‘q’ conditioned upon e i ,  i ( q ) ).
Linear Regression (Lin-Reg.).
The problem setup for the Lin-Reg.
model is identical to the RF-Reg.,
GB-Reg.
setups.
Single-Fluid CNN (SF-CNN ) .
A fully convolutional neural network architecture 2 with kernel size of 13 (large enough to capture the spatial span of adsorption phenomena as determined by domain experts).
This model accepts input features in the source domain D S for a particular ﬂuid and yields the density proﬁle at every point in the estimation horizon as speciﬁed in the inputs.
Two-Fluid CNN (TF-CNN ) .
Similar architecture2 0 to the Single-Fluid CNN model, except the outputs yield density proﬁles for multiple ﬂuids (i.e., all constituent ﬂuids) in a mixture M being modeled.
Additionally the experimental settings added as input to the model also comprise the partial pressures of each constituent ﬂuid in M .
Performance Comparison.
To further investigate the performance of NanoNet-SG , we compare yield the prediction performance to state-of-the-art regression baselines (details in sec. 3.1).
In Fig.
6 we notice that NanoNet and NanoNet-SG models yield signiﬁcantly improved prediction performance on both the methane and ethane proﬁle prediction tasks in the ﬂuid mixture density proﬁle prediction context.
We speciﬁcally report methane and ethane proﬁle prediction errors for each model in Fig.and notice that NanoNet and NanoNet-SG outperform all other models in both prediction tasks (i.e., methane, ethane density proﬁle prediction).
[Table]
We once again notice that NanoNet-SG yields best overall performance due to knowledge transfer coupled with science-guidance.
In Table 1, we further showcase a task breakdown of the generalization performance observed in
Table 1 (main paper) by inspecting generalization errors in the bulk and proﬁle prediction context per ﬂuid for each model.
We can infer from Table 1 that models NanoNet and NanoNet-SG yield signiﬁcant performance improvement owing to the effect of the knowledge transfer architecture and the science guidance in the case of NanoNet-SG .
Speciﬁcally, we notice that science guidance results in improved overall generalization as well as bulk and proﬁle generalization in both the case of Methane and Ethane.
Science guidance leads to NanoNet-SG gaining a 16.26% performance improvement over its variant without science guidance (i.e., NanoNet ) and signiﬁcantly larger gains in performance over SF-CNN and TF-CNN models.
Qualitative Performance Characterization.
We notice from Fig.
7 that NanoNet-SG yields highly accurate density estimates across the entire range of the prediction spectrum of ﬂuid density for each of the ﬂuids in the mixture.
Also apparent from the plots is the ability of the model to ﬁt the multiple distributional modes in the density proﬁle owing to the density in the peak and bulk regions due to the presence and absence of adsorption effects respectively.
These multiple modes are indicated by disjoint clusters of points in each plot in Fig.
We also report the R 2 statistic for density prediction of NanoNet-SG which indicates consistent performance across the two ﬂuids with R 2 =0.991 for methane density prediction and R 2 =0.998 for ethane density prediction.
P o i n t D
en s i
t y
P o i n t D
en s i
t y
Methane
Ethane ( [Figure] [Figure]
3.2 Effect of Science Guidance on Domain Consistency
Further, in Fig.
8 (a), 8(b) we depict evolution of total cumulative densities for NanoNet and NanoNetSG at a sample pore width of 25 at various temperatures.
We once again notice that the evolution for
NanoNet showcases signiﬁcant deviations from the monotonically non-decreasing domain constraint whereas the NanoNet-SG model predictions result in total density curves that adhere to the scientiﬁc domain knowledge much more consistently.
From Fig. ??
and Fig.
8, we can infer that the NanoNet- yields signiﬁcantly domain criterion
Related Work
Transfer Learning -Transfer learning as a model training paradigm has been effectively leveraged in applications like natural language processing Ruder et al.
[2019], computer vision Li et al.
[2020], healthcare Gupta et al.
[2020], Chen et al.
[2019] and many other contexts.
However, few efforts have employed transfer learning under data paucity in scientiﬁc applications.
To the best of our knowledge, our NanoNet-SG is the ﬁrst science-guided model to employ inductive transfer learning to counter the effects of data paucity in scientiﬁc applications.
Fluid-Flow Modeling.
Various ML models have been proposed for modeling ﬂuid ﬂows Brunton et al.
[2019], Wang et al.
[2020], Raissi et al.
However all these models (unlike NanoNet-SG ) either assume complete knowledge of the physics (e.g., PDE, ODE, initial, boundary conditions) governing the system or employ large datasets in target domain.
Modeling Under Nano-conﬁnement.
The works Santos et al.
[2020], Lubbers et al.
[2020] are closest to ours.
In Santos et al.
[2020], authors propose an active-learning framework for training a ML emulator of MD simulations and Lubbers et al.
[2020] develop an ML model to bridge ﬁnegrained (MD) and coarse-grained (Lattice-Boltzman) simulations of nano-conﬁned ﬂuids.
However, neither of these works tackle the complex problem of modeling ﬂuid mixtures and neither work employs explicit science guidance in the loss function for modeling under data paucity. How is our work different? Uses weak scientiﬁc domain knowledge. 2. Modeling under data paucity 3.
No other works leverage inductive transfer learning and science guidance.
Our work is different from Javier’s work as we do not assume access to the MD simulator (only to the scientiﬁc domain knowledge and initial data).