A Survey on Evaluation Metrics for Synthetic Material Micro-Structure Images from Generative Models
Abstract
The evaluation of synthetic micro-structure images is an emerging problem as machine learning and materials science research have evolved together.
Typical state of the art methods in evaluating synthetic images from generative models have relied on the Fréchet Inception Distance.
However, this and other similar methods, are limited in the materials domain due to both the unique features that characterize physically accurate micro-structures and limited dataset sizes.
In this study we evaluate a variety of methods on scanning electron microscope (SEM) images of graphene-reinforced polyurethane foams.
The primary objective of this paper is to report our findings with regards to the shortcomings of existing methods so as to encourage the machine learning community to consider enhancements in metrics for assessing quality of synthetic images in the material science domain.
Introduction
The adoption of artificial intelligence techniques within the material science community has been critical in enabling high throughput research through rapid characterization and discovery of materials.
In particular, various forms of generative models have been proposed to accelerate the design of new materials [1, 2, 3, 4].
Nonetheless, the applications of traditional image based deep learning models in this domain are constrained, among other design choices by the ability to evaluate trained models and synthetic image quality for physically accurate characteristics.
Generative networks are unsupervised learning methods that aim to learn the regularities and patterns in input data such that the model can be used to generate new examples that could have plausibly been drawn from the original dataset [5].
These models are traditionally evaluated using the Fréchet
Inception Distance (FID) score, which leverages the InceptionV3 model pretrained on the ImageNet dataset.
Several studies have shown features from this model to be vision-relevant [6].
By comparing the mean and variance of these features between synthetic and ground truth datasets, the FID score evaluates both the image quality and diversity of the synthetic images.
Though plenty of works have highlighted various pitfalls of the FID score and suggested variations, it is still regarded as the standard for model evaluation.
In this study, we explore the drawbacks of existing methods when working with datasets in the materials domain.
In particular, we outline two main causes of concern:
Dataset Size: Materials scientists have worked with small
noisy datasets with great success in uncovering process-structure-property relationships [3].
However, in small data regimes, the FID score promotes over-fitting, as the distance between distributions is minimized
when matching the smaller variance of our ground truth data.
Refer to Appendix A for an experimental analysis.
Feature Relevance: Unlike well-studied datasets (e.g. ImageNet), many of the features of a realistic micro-structure image cannot be recognized by the human eye, thus limiting the applications of a model pretrained on ImageNet as the base of the metric.
Beyond image quality and diversity, in the material science domain, we aim to generate valid micro-structures which can be synthesized in an experimental setting.
Methods
Unlike traditional machine learning metrics, the evaluation of generative models and synthetic images have been an open subject of study for many years.
Many works [7] have summarized the relationships between various metrics and outlined corresponding limitations.
However, much of this work has been done on large datasets of simple images where population surveys can verify results.
On the contrary, verification of synthetic images in the material science domain is limited to costly experimental trials [8].
Subsequently, this limits the use of metrics using human evaluation such as those provided by Amazon Mechanical Turk [9] and HYPE [10].
Studies in the medical domain [11] have shown how standard metrics can give misleading results when applied to more complex images.
Synthetic micro-structure images have significant potential to accelerate the materials discovery platform.
SEM image data provides researchers with additional information about materials (e.g. connectivity of cells, cell sizes) that cannot be inferred by numerical data measurements.
In particular, end-to-end inverse and forward structure-property models can allow us to replicate experimental testing of new materials computationally.
Generative models, along with advances in methods such as attribute editing, allow us to extrapolate beyond our training dataset and build new variations of cell structures [3].
Corresponding auxiliary networks can then help classify the properties and process parameters of such images.
This places the generation of synthetic data at the center of future materials discovery platforms. For our study, we use a dataset of SEM images of graphene-reinforced polyurethane foams (PUF) with seven unique additive amounts.
The images were acquired under an SE2 secondary electron detector at fixed beam energy and working distance with 25x magnification (approx.
10.3 pix/micron).
The brightness and contrast levels were held constant across all images and samples.
There are three key properties we hope to evaluate in our synthetic micro-structure image datasets:
Image Quality: Are the images of high fidelity?
Image Diversity: Are new images (outside the training data) being formed?
Physically Accurate: Are the synthetic structures experimentally feasible/valid characteristics of materials (e
[Table] g. complete cell walls, inter-connected cell structures)?
Several generative models were trained using our ground truth dataset.
In this study, we focus on the outputs from three particular models, summarized in Table 1 and detailed further in Appendix B.
Resources were not spent to find ideal model parameters/architecture, as the focus was on comparing relative outputs of different generative models.
In particular, outputs from Dataset B, though visually
[Table]
similar to Dataset A, are representative of mode-collapse. Refer to Appendix C for a detailed visualization and explanation of the image quality and diversity in each dataset.
Many alternatives of the FID score have been proposed and were considered during our study.
These include the Inception Score (IS) [14] and the Kernel Inception Distance (KID) [15] to address concerns with the small training dataset.
To differentiate between the image fidelity and diversity we also considered Precision & Recall (P&R) [16, 17] and Density & Coverage [18].
To mitigate the feature relevance concern with the FID score, various other feature extractors have been proposed, from randomly initialized weights [18] to task-specific tuned models [11].
We leveraged transfer learning methods to fine-tune an InceptionV3 model to perform a simple classification task of matching an input PUF image to one of the seven unique additive combinations.
The model was trained for 10 epochs and achieved a >97% accuracy on both the training (2000 images) and test ( images) sets.
Similar to the FID, the output of the final pooling layer was extracted as the 2048-dim feature vector for calculations.
We define this as the FID Modified score.
We additionally considered a wide array of image processing methods, from pixel-wise histograms to Image Quality Assessment (IQA) metrics.
These included BRISQUE [19], LPIPS [20, 21],
Manipulative Precision Metric [22], Optimal Transport [23], along with others.
A key advantage of IQA methods is the ability to score individual images on image naturalness/distortion based on deviation from natural image distributions.
However, experimental results showed these metrics were not suited for our application, with all our synthetic images scoring better than ground truth images.
Results and Discussion
Metric results from the FID score and alternatives are summarized in Table 2.
We can infer Dataset A is consistently slightly higher ranked than Dataset B, and both Datasets A & B are significantly better than Dataset C. Table 3 provides a visual interpretation of each of the datasets for each of our key scoring properties we hope to evaluate.
We observe the vision-relevant features (scoring Dataset A the best) don’t correlate well with our target of physically-accurate .
Similarly, Figure 3 highlights the severe over-fitting in Dataset B resulting in lack of any diversity - yet this dataset is still ranked similar to Dataset A.
It is clear there are flaws in the existing methods.
[Table] [Table] From Table 2 we see our FID Modified score provides the greatest differentiation between datasets, with Dataset A being the best.
However, there is an evident lack of value for the greater diversity seen in Dataset C. To further investigate [Table] we extracted the outputs from the final pooling layer of the
InceptionV3 model (fine-tuned on our materials dataset) to obtain 2048-dim feature vectors for each image.
Using PCA, we were able to visualize these images in 2D, as seen in Figure 1.
Note, for the ground truth dataset, we see the expected clusters for the seven additive types, as this is what the model was trained to distinguish.
The black data points represent the synthetic images distributed on the same two principal components.
The results from Figure 1 are quantified in Table 4 as the euclidean distance between various sets of image feature vectors.
For each of the four datasets, we calculated the same and different distances with respective to each additive type.
The same distance represents the distance between each point in Dataset X classified as Additive Y (using our fine-tuned InceptionV3 model) with every point in the Ground Truth Dataset of Additive Y. Conversely, the different distance compares every point in
[Figure]
Dataset X classified as Additive Y with every point in the Ground Truth Dataset not of Additive Y.
Note these distances are calculated using the complete 2048-dim feature vectors.
Similar to the 2D PCA in Figure 1, Table 4 indicates there is a clear distinction between each category.
For the Ground Truth Dataset, distances within the same category average 12.1 while distances to other category labels average 20.8.
An ideal synthetic dataset should provide same distance values between these endpoints - indicating both realistic images and greater diversity than the original categories.
Results from Datasets A-C support this.
[Table]
Additive
[Table]
[Table]
Additive
[Table]
[Table]
Additive
[Table]
[Table]
[Table]
Additive
[Table]
[Table]
Additive13.
[Table]
[Table]
Additive10.
[Table]
[Table] Additive12. [Table] A key limitation of our method is the model fine-tuned on our ground truth dataset seems unable to distinguish image diversity when image quality is low.
Figure 1 indicates Dataset C is the least diverse, while Table 4 indicates the diversity in Dataset C’s Additive 7 class is much higher than in any other dataset (seen by the greater distance in the same column).
Additionally, each of these metrics lacks the ability to distinguish physically accurate features when scoring.
Dataset C, though of low image quality, exhibits diverse sets of cell structures that still provide valuable information in the materials discovery platforms (visualized in Appendix C).
Conclusion and Future Works
We have identified and evaluated several limitations of current state of the art methods in evaluating generative models from non-traditional problem domains.
In particular, we highlight the limitations from using small dataset sizes, images with features different from those in ImageNet, and the need to score based on physically accurate micro-structure formations that can be synthesized in the lab.
As we progress this work, we hope to leverage additional deep learning tools (e.g. Dragonfly [24, 25]) to statistically quantify image features such as number of cells, cell wall width and size of cells (sample visualization in Appendix D) and evaluate feasibility of synthesized images.
Similarly, we plan to leverage known physics-based correlations to embed the knowledge of physical laws that govern our materials dataset during both the learning and evaluation process of the generative models.
Further exploring model architectures (e.g. diffusion models), hyper-parameter tuning, and custom loss functions will allow us to improve the quality of synthetic images and provide more accurate evaluation criteria. In general, we hope this work will progress the evaluation of synthetic images in various scientific fields beyond micro-structure imaging where traditional methods may be limited. Acknowledgments and Disclosure of Funding We thank Patrick Blanchard, Rachel Couvreur, Janice Tardiff and Alper Kiziltas for their support on the experimental data generation and discussion.
We also thank Ford Motor Company Research &
Advanced Engineering for sponsoring this work. A Datasets Limitations While AI methods require substantially more data than conventional methods, materials scientists have traditionally worked with small & noisy datasets [3], thanks in part to advancements of generative modeling in small data regimes using methods such as differential augmentation [13] and adaptive instance normalization [26].
Yet, there is minimal discussion around the impact of such small training datasets on model evaluation metrics.
The FID score is the Fréchet distance d pq between the Gaussian with mean p m, C q obtained from p p .
q and the Gaussian with mean p m w , C w q obtained from p w pq given by the following [6]:
[Equation]
From Equation 1, we see the FID score compares both the mean and variance of the training dataset to the respective components of the synthetic images.
The FID score is minimized (ideal) when these variables are as similar as possible.
Although any number of synthetic images can be generated, the size of the training dataset can become a limitation.
Smaller training datasets provide a smaller variance, thus resulting in a lower FID score when our synthetic images have a similar variance, subsequently encouraging over-fitting.
We ran several trials to highlight this difference, as seen in Table 5.
The experiments were run using three datasets: CIFAR10, FFHQ* (the thumbnail version), and our internal SEM micro-structure dataset.
For each trial, the dataset was randomly (without repetition) split into two groups of varying sizes.
These groups were representative of a ground truth (Dataset 1) and a synthetic (Dataset 2) dataset.
Note, for the micro-structure trials, Dataset 1 came from our actual ground truth dataset while Dataset 2 was randomly produced using the trained network for Dataset A. As a result, we only see trail results where the ground truth dataset is less than or equal to 1000 images.
Table 5 clearly highlights the impact of small training datasets (Dataset 1), even when both datasets are sampled from the same distribution (CIFAR10, FFHQ*).
In particular, we notice that the FID score is consistently low, no matter the problem domain, when working with large training and synthetic datasets.
However, as the dataset sizes decrease, not only does the FID score diverge quickly, there is a lack of consistency between datasets.
As a result, it becomes difficult to use the
FID score as a standard metric to compare multiple models during training when working in problem domains with limited data.
An alternative metric is needed for scoring generative models.
[Table]
B Model Architecture/Hyper-parameters
Dataset A and B were both derived from a base StyleGAN2 model [12] with Differential Augmentation [13].
The augmentations applied were color, cutout, and translation.
Dataset A was trained for 280 Kimg and Dataset B for 240 Kimg.
Both models were randomly initialized with weights training.
Dataset C is derived from a ResNet18 based Variational Autoencoder. C
G r ound
T r u t h
D a t a s e t D a t a s e t
A
D a t a s e t
B
D a t a s e t
C
G r ound
T r u t h
D a t a s e t D a t a s e t
A
D a t a s e t
B
D a t a s e t
C
G r ound
T r u t h
D a t a s e t
There are clear cell structures and thick walls outline each of the cells.
Note, minimal number of the cells are broken with edges surrounding each intermediate cell hole.
In addition, there is a clear evidence of depth in the micro-structure such that layers of the foam in the front are more reflective of lighting from the SEM.
D a t a s e t
A
The images in this dataset lack complete cell structures.
Several images may include complete cell walls; however, many of the images are representative of broken cells.
This particular image lacks complete enclosed walls that represent the cell structure.
The broken structure is representative of an image on the edge of a foam, a few of which were included in the training dataset.
These are not experimentally valid structures.
D a t a s e t
B
There is a clear set of cell structures with thick walls outlining each of the cells.
However, there is no sense of depth in this image.
Beyond the initial layer of the micro-structure, we see an incomplete formation of cells in the background, thus not showing the dimensionality that is expected of a true micro-structure.
D a t a s e t
C
Despite low image quality, we see clear micro-structures and cell walls in the image.
The image quality impacts the physically accurate scoring due to the inability get accurate measurements of cell wall thickness and cell sizes.
[Figure]