On the feasibility of small-data learning in simulation-driven engineering tasks with known mechanisms and effective data representations
Abstract
The application of machine learning (ML) in scientiﬁc tasks is increasing, especially
ML in simulation-driven engineering tasks.
While previous studies were mostlymodel-centric and required large-data learning, recent studies start to pay attentionto data-centric AI and are investigating small-data learning with effective structuredrepresentations, which is signiﬁcant for industrial application.
This article providesa theoretical discussion for the feasibility of small-data learning with structuredrepresentations, which is then veriﬁed through the surrogate modelling of hotstamping simulations.
Future directions are also discussed.
Introduction
In the past decade, ML, particularly deep learning, has been successfully applied to general Artiﬁcial Intelligence (AI) tasks, including object detection [1], semantic segmentation [2], and imagegeneration [3].
Recently, ML is being extended to scientiﬁc tasks, which can be broadly classiﬁed into mechanism-unknown tasks and mechanism-known tasks, as shown in Figure 1.
Inmechanism-unknown tasks
, ML can be applied to mathematically formulating the mechanisms, suchas ML-assisted material modelling [4][5], or explorative discovery with unknown or partly-unknownmechanisms, such as discovering new materials [6] or drugs [7]; in mechanism-known tasks, alsoknown as simulation-driven engineering tasks, the mechanisms have been mathematically formulatedand integrated with simulation tools, such as ﬁnite element analysis (FEA), while ML is mainlyapplied to surrogate-based optimisation [8].
A frequent observation is that the application of ML ismore convenient and practical in simulation-driven engineering tasks than in mechanism-unknowntasks, since multi-scale simulation tools can generate data containing comprehensive info, such asphysical ﬁelds, that are difﬁcult to obtain in experiments.
Therefore, this article mainly focuses on
ML-assisted surrogate-based optimisation in simulation-driven engineering tasks.
In ML-assisted surrogate-based optimisation, while some studies replaced classical optimisers with
ML, such as reinforcement learning [9], most efforts have been devoted to improving the accuracy,generalisability, and interpretability of surrogate models.
Current ML with scalar representationsof both inputs and outputs, such as Kriging models [10][11], has been widely deployed.
However,these models are only suitable for low-dimension, single-parameterisation, scalar-output use cases,while it is difﬁcult to integrate data from multiple parameterisations or reuse historical data.
Toimprove the performance of ML-assisted surrogate models, including their ﬂexibility, accuracy andgeneralisability, recent studies have intensively investigated ML with structured representations, suchas ﬁelds [12] and graphs [13].
ML with structured representations was initially practiced in computational ﬂuid dynamics (CFD), since Eulerian meshing, which is popular in most CFD tasks, can beconveniently integrated with convolutional neural networks (CNN).
Guo, Xu.
et al.
represented 2D or3D geometries using signed distance functions (SDF) [14].
Based on the SDF ﬁeld representations, a
CNN was trained to predict the full ﬁelds of 2D or 3D non-uniform steady laminar ﬂow.
ML withstructured representations, which integrates advanced architectures such as recurrent neural networks(RNNs) and generative adversarial networks (GANs), has been applied to more complex CFD cases,including temporal prediction [15][16] and inverse design
Based on the success in CFDcases, ML with structured representations has been extended to difﬁcult use cases where non-Eulerianmeshing is applied, including solid mechanics [19], structured optimisation
[20][21], manufacturing[22][23][24], and meta-material design [25][26].
The ML-assisted surrogate models discussed above mainly employed full ﬁeld-based representations,which signiﬁcantly improved the overall performance of surrogate models.
The previous studies discussed above mostly pursued universally-generalised models, large-data training, and high-resolutionfull-ﬁeld representations.
Recent studies have started to investigate more effective structured representations, such as implicit shape parameterisation [27][28][29][30][31][32] and graphs [12][38];surrogate models are trained on datasets with task-speciﬁc sizes [33][34][35].
Despite the emerging trend discussed above, only a few studies attemped to investigate the feasibility of small-datalearning and the beneﬁts of effective representations: Cao et al. showed that non-parametric inputrepresentation using graph neural network (GNN) and ﬁeld-based output representation signiﬁcantlyimproved the performance of surrogate models [12]; Li et al.
signiﬁcantly reduced the dataset size,which was acceptable for industrial practice and signiﬁcantly facilitated multi-query optimisation,using wing mode representations [30][33].
In this article, the feasibility of small-data learning and the beneﬁts of effective representations will beinvestigated based on the surrogate modelling of a case with application to hot stamping.
In section 2,the feasibility and beneﬁts will be discussed based on theoretical considerations.
In section 3, a hotstamping case will be presented for veriﬁcation.
In section 4, current studies will be summarised andfuture directions will be pointed out.
Theoretical discussion
In this section, the feasibility of small-data learning in simulation-driven engineering tasks will betheoretically discussed by comparing these tasks with general tasks, and the beneﬁts and criteriaof effective representations will be discussed.
The feasibility of small-data learning in mechanismunknown tasks remains an open research question due to the lack of physical mechanisms, which isout of the scope of this article and requires further study.
2.1 The feasibility of small-data learning in simulation-driven engineering tasks
The deﬁnition of small-data and the required dataset size largely depends on speciﬁc tasks.
In generaltasks, millions of samples are required to ensure the performance of ML, such as ImageNet and Open Images.
However, simulation-driven engineering tasks are expected to require much less data thanksto the following three attributes:
• From the input side, the design variables and corresponding domain in simulation-drivenengineering tasks can be explicitly deﬁned.
For instance, in a stiffness-driven shape optimisation task, the shape can be completely deﬁned by a set of variables and uniformly
generated using sampling strategies, such as Latin hypercube [37].
This ensures an effectivecoverage of the design space.
However, in general tasks such as a multi-classiﬁcation, it
is nearly impossible to deﬁne the features of a certain class using explicit variables, not to
mention sampling strategies.
In this case, large data is required to better cover the design space of general tasks, while mode collapse still occurs.
• From the output side, in a multi-classiﬁcation task, laborious labelling work is requiredin supervised and semi-supervised learning.
Furthermore, the labels are usually not embedded in a mathematical metric space: it is nonsense to say ’cat’>’dog’ mathematically.
This might lead to highly nonlinear and complicated input-output mapping relationships.
However, the outputs of simulation-driven engineering tasks are mostly ﬁelds with inherent mechanism driven patterns.
Labelling work is rarely required while the outputs are in nature mathematically comparable.
• Data in simulation-driven engineering tasks usually has a higher signal-to-noise ratio (SNR).To clarify, a cat in a multi-classiﬁcation task may be classed under diverse headings because the image contains features such as eyes, irrelevant objects and background.
This has a negative effect on data quality and SNR.
However, data in simulation-driven engineering tasks, as long as the simulation model is veriﬁed and uncertainty is estimated, contains littleirrelevant information.
These advantageous attributes lead to smoother input-
output mapping relationships and signiﬁcantlyreduces the data requirement in simulation-driven engineering
2.2 The beneﬁts and criteria of effective data representations
Besides the natural attributes discussed above, proper representations are required to reduce datasetsizes and improve the performance of surrogate models.
Based on a case with application to stiffenedpanel optimisation, Hao et al.
demonstrated that ﬁeld-based input shapes preserved the topologicalinformation, compared with parametric representations [13].
To clarify, a shape usually consists ofmultiple inter connected features, such as rounded corners, while the interconnection relationshipis lost if the shape is represented merely by the scalar feature parameters, such as radius.
Based ona cold forming case, Zhou et al.
demonstrated that ﬁeld-based output physical ﬁelds preserved thedata structure of physical ﬁelds compared with scalar performance indicators [36].
For example,multiple thinning ﬁelds that have the same maximum thinning might differ in their locations, peaknumber, and patterns.
Overall, ﬁeld-based representations have shown remarkable advantages overconventional scalar representations in academia [12][13][33][36] and industry [38][40].
Case study: A ﬁeld-based Artiﬁcial Intelligence empowered surrogatemodel for a hot-stamped ultrahigh-strength-steel (UHSS) B-pillar
Small-data learning, which has been theoretically discussed in section 2, is veriﬁed based on thesurrogate model of a hot stamped B-pillar used in automotive applications.
The simulation setup in
PAM-STAMP is shown in Figure 2, while the simulation model was experimentally veriﬁed [39].
3.1 Dataset and surrogate model setup
The surrogate model predicted the full blank thinning ﬁelds, which was mapped on the 2D initialblank conﬁguration, given the 2D images of die and blank.
The design variables and domain aredemonstrated in Figure 3.
A training set sized 64, which could be regarded as small-data, and avalidation set sized 256 were sampled from the design domain in Figure 3(b) using Latin hypercube
(LHS).
Two samples in the 64-size training set and nine in the 256-size validation set had excessivethinning due to small draft angles (<1.2°) or large height (>65mm).
The FEA ground truth thinningﬁelds of these samples were not reliable due to numerical errors when excessive thinning occurred.
Since this study was not discussing failure and corresponding representations, these samples wereremoved, which led to a 62-size training set and a 247-size validation set.
The surrogate model was
120 adapted based on a well-developed Res-SE-U-Net that is comprised of residual modules, squeezeexcitation modules, and skip connections [19][21][36].
The details of Res-SE-U-Net were discussedin [36].
For comparison with models that have scalar inputs/outputs, a Gaussian process (GP) modelwith anisotropic radius basis kernels was developed using Python/sklearn [13].
3.2 Results and discussion
Res-SE-U-Net model was trained using combing mean square error (MSE) with batch size 2 and 126 learning rate 0.0004, and GP model was ﬁtted with optimised noise coefﬁcient (0.02).
Both modelswere validated on the 247-size validation set.
As shown in Figure 4(a)(b), the violin plot of maximum 128 thinning predicted by Res-SE-U-Net has a better consistency with the ground truth than that predictedby GP.
For Res-SE-U-Net, the average absolute relative error of the maximum thinning (AREMT)
ARMET =
| ( ML prediction − FEA ground truth ) /FEA ground truth | × 100% (1) was 3.77%, and 82.6% samples (204/247) had AREMT below 6%.
For GP, the average AREMTwas 5.30%, while only 72.3% samples had AREMT below 6%.
In conclusion, Res-SE-U-Net trainedon small data accurately and reliably predicted scalar maximum thinning even though maximumthinning was not explicitly included in MSE loss.
A potential reason is that Res-SE-U-Net predictedthe full thinning ﬁeld that contained intrinsic physics, as shown in Figure 4(c).
Predicting full ﬁeldsalso provides more guiding information to engineers.
Summary and future directions
In this study, small-data learning using ﬁeld-based representations in simulation-driven engineeringtasks was investigated and veriﬁed.
The advantageous attributes that enable small-data learning werediscussed in comparision with general tasks.
In the case study, Res-SE-U-Net outperformed GP byleveraging the intrinsic physics in full thinning ﬁelds.
More studies on data-centric approaches in AI for science and engineering are expected in the future.
Effective representations will be investigated.
For example, a graph representation is preferred in atruss optimisation case since the truss contains only nodes and links [38].
Besides, active sampling,transfer learning, and physics-informed losses are promising to further facilitate small-data learning.