Interdisciplinary Discovery of Nanomaterials Based on Convolutional Neural Networks
Abstract
The material science literature contains up-to-date and comprehensive scientificknowledge of materials.
However, their content is unstructured and diverse, resulting in a significant gap in providing sufficient information for material designand synthesis.
To this end, we used natural language processing (NLP) and computer vision (CV) techniques based on convolutional neural networks (CNN) todiscover valuable experimental-based information about nanomaterials informationand synthesis methods in energy-material-related publications.
Our first system,
TextMaster, extracts opinions from texts and classifies them into challenges andopportunities, achieving 94% and 92% accuracy, respectively.
Our second system, GraphMaster, realizes data extraction of tables and figures from publicationswith 98.3% classification accuracy and 4.3% data extraction mean square erroron average.
Our results show that these systems could assess the suitability ofmaterials for a certain application by evaluation of real synthesis insights and caseanalysis with detailed references.
This work offers a fresh perspective on miningand unifying knowledge from scientific literature, providing a wide swatch toaccelerate nanomaterial research through CNN.
Introduction
The scientific literature is meant to communicate recent advances and keep researchers informed sothey can continue to make breakthroughs and advance their fields.
Since innovative materials havebeen regarded as one of the promising roles leading the fourth industrial revolution, the materialscience field has witnessed a drastic volume increase in publications.
Therefore, comprehensiveselection and reading of the material science scientific papers by hand are not realistic.
A potentialsolution to this issue is to automatize the task using artificial intelligence (AI) technologies.
However, the information in scientific literature is multimodal (text, figures, tables, etc.), which makesidentifying and utilizing the latent knowledge in publications even more challenging.
In this paper, we resort to CNN to discover and summarize knowledge related to nanomaterialsfrom scientific literature, with a novel perspective of combining the NLP and CV techniques toassimilate and integrate the multimodal literature knowledge.
Firstly, we applied the classic NLP task
The system includes four modules:(i) text preparation, (ii) opinion extraction, (iii) opinion classification and (iv) opinion mining forinformation analysis, where CNN with attention mechanism is applied to modules (ii), (iii) and (iv).
This system can assess the suitability of nanomaterials for an application from various aspects, such assynthesis performance and natural properties.
Secondly, we develop a system to assist in compiling a systematic and highly refined performance review of b-Si solar cells.
The system includes (i) content retrieval, (ii) graph classification, (iii) data extraction.
NLP techniques are applied to help screen outirrelevant articles in module (i).
With modules (ii) using a CNN-based neural network, ResNet [1, 2],identifying the different elements, i.e., texts, tables, figures and images, then different NLP and CVmodels are further applied to extract corresponding data in module (iii).
This system indicates thathierarchical textures and inverted-pyramidal black silicon (b-Si) nano-textures and micro-textures arepromising as next-generation texturing techniques for Si solar cell applications.
Related Works
We identify related works for this paper in the area of NLP and CV methods.
NLP methods arecategorized into two domains, namely: NLP methods related to scientific discovery and opinionmining techniques.
CV methods are mainly related to image recognition.
NLP methods related to scientific discovery NLP researchers have implemented different taskson scientific text, such as pre-trained language model for scientific text [3], text classification [4],text summarization [5], named entity recognition (NER) [6] and knowledge graph [7].
In materialsscience, significant effort has been devoted to extracting material mentions [8, 9] or synthesis recipes[10, 11, 12].
These material science applications are based on NER, linking material names to theirco-occurring entities, which help answer large-scale questions, such as "What are the most frequentmethods for TiO 2 synthesis?"
Recent attempts to predict suitable materials for certain applicationshave been made by word embeddings [13] or graph-based machine learning [14], which efficientlyencodes materials science data into high-dimensional numeric representations.
Opinion mining techniques Opinion mining is an NLP task that analyzes people’s opinions,sentiments, evaluations, attitudes, and emotions from written language [15].
It has widespreadapplications, such as categorizing product reviews to support consumers’ decisions [16], analyzingsocial media texts to predict the stock market [17] or election results [18], and fostering effectiveinteractions in smart cities [19].
This technique has been applied to scientific literature but mostlyfocuses on article reviews [20] or citations [21] instead of the article’s body content.
We creativelyapplied sentence-level opinion mining to the body content of energy material-related publications.
Existing sentence-level opinion mining approaches can be divided into lexicon-based and corpus-basedapproaches [22].
Lexicon-based approaches are usually unsupervised, based on the measurementof word sentiment orientation using WordNet or machine learning models [23, 24].
Corpus-basedapproaches are usually supervised, and dominated by feature-based machine learning methods anddeep learning methods.
In comparison, deep learning methods show better performance on most ofthe benchmark datasets.
CV methods related to scientific graph classification The first graph type classification workwas introduced by Zhou and Tan[25] with a modified probabilistic through transform methods.
They achieved 84% accuracy of graph type classification in 1190 total images.
After that, graphclassification research began to attract attention.
Scientists have applied different methods for graphclassification problems, such as hidden Markov models, SVMs and CNNs.
We summarize some ofthe existing graph classifications works in Table 1.
[Table]
Techniques of data extraction from chart Previous research on data extraction from charts canbe divided into two types, interactive and full automated approaches.
Chartsense [2] was a semi- automated data extraction tool that provides the user with an interface to mark graph elements suchas the x-axis and the origin of coordinates.
However, interactive extraction was time-consumingand unsuitable for large data collection.
Automated graph data extraction such as Revision [26],
Chartreader [6], and Chartdecoder [27] used different Optical character recognition (OCR) enginesand CNNs to realize Bar Chart data recovery.
However, for material science publications, a lot ofimportant material information was presented with line charts, such as spectroscopy spectra andwavelength-related graphs.
In this work, we demonstrated how to use CNN to fully automaticallyextract the data from the material science line charts.
Methods
Here we introduce NLP and CV methods that were used to analyze energy-related publications.
Ourpython code (includes TextMaster, GraphMaster and SciCrawler) is available at three repositories inhttps://github.com/for-submission.
3.1 TextMaster
Schematic diagram of TextMaster system is shown in Figure 1.
[Figure]
.1 Text preparation
[Figure]
We used the Web of Science client to collect DOI lists for various nanomaterial scientific articles.
Then we customized a web-scraper, SciCrawler, to automatically download a broad selection ofnano materials-related papers published after 2000 from publishers’ API or websites in PDF format.
The publications on a certain topic were collected by searching keywords from databases suchas Web of Science.
We prepared a mixed-topic dataset formed by three datasets with end-of-lifemanagement (EoL), perovskite solar cells, and atomic layer deposition (ALD) themes, coveringmanagement, material and synthesis.
The EoL dataset was formed by 77 full papers.
Perovskiteand ALD datasets were formed by 34,752 abstracts and 21,276 abstracts respectively, containingall meta fields provided by Web of Science.
Plain texts were extracted, segmented into sentences,and tokenized.
The sentences in EoL dataset were annotated as "opinion" and "non-opinion", and"opinion" sentences were further annotated as "opportunities" and "challenges".
3.1.2 Opinion extraction
The processed sentences were classified as opinions and non-opinions in this module.
In the pilot study, we compared the performance of some typical lexicon-based methods and deep learning models on the EoL data set (see Appendix A).
It can be seen that the CNN model exceeds other methodson distinguishing opinions from non-opinions.
Due to the imbalance of opinion and non-opinioncategories, we also used the Synthetic Minority
Oversampling Technique (SMOTE) [29] to generatedata for opinion category.
To increase data size, the CNN-SMOTE (CNN with SMOTE technique)trained on the EoL dataset was used as raw models to predict the pseudo labels of sentences inabstracts of ALD- and perovskite-related publications.
Each sentence was predicted as opinion ornon-opinion.
Then, annotators were invited to proofread the predictions.
We trained the final CNNmodel on about 22 thousand samples, achieving an accuracy of 94%.
3.1.3 Opinion classification
The input opinions were classified as opportunities and challenges in this module.
Compared with
CNN, a CNN model with attention mechanism [30] had a better performance in the pilot study (see
Appendix A).
Similar to the last module, we used a raw model to increase data size and trainedthe final CNN-Attention model on about 9 thousand mixed-topic samples.
Due to the imbalance ofopportunity and challenge categories, we also used the SMOTE to generate data for the challengecategory, which improved the performance of the CNN-Attention model from 89% to 92%.
3.1.4 Information analysis
With extracted and classified opinions, this module used embedded material names or elements forinformation analysis.
For each opportunity or challenge sentence, ChemDataExtractor [8] was used toidentify the material name in it.
Based on the sentiments of opinions and year of publication, we cancalculate each material’s percentage of opportunities in opinions each year.
This method quantifiedthe sentiment towards elements as sentiment scores across time.
Using opinions, we implementeda case study of ALD elements and a prediction of thermoelectric materials (see the Results and
Discussion Section).
3.2 GraphMaster
[Figure]
Figure 2 presents the graph data recovery pipeline, which mainly consists of three components, i)
[Figure]
content retrieval ii) graph classification and iii) data extraction.
[Figure]
extraction
In this paper, we only proposed the data extraction solution for the line graph, since most of thematerial’s characterization figures are line graphs including Raman spectroscopy, X-ray diffraction,external quantum efficiency graphs, etc.
There are some existing solutions for data recovery of othergraphs, including bar charts and tables [27, 8, 31].
3.2.1 Content Retrival
The paper collection process is similar to the TextMaster (See 3.1.1).
PDFFigure 2 [32, 33] , anexisting algorithm, was used to extract figures from scientific papers.
In material science publications,one figure commonly includes several sub-figures.
However, it is hard for PDFFigure 2.0 to captureeach subfigure.
We decomposed the figure into subfigures through valid axis-aligned splittingrecognition[34].
Finally, we manually assigned them category labels, including line charts, bar charts,and tables.
The figure and metadata like the graphs, journal, and title were stored in the Neo4j datasetfor convenient access (https://neo4j.com/).
After the selection of samples related to the analyzedmaterials, we used 6,000 figures from 3,600 scientific papers.
3.2.2 Graph Classification
We used a 20-layer ResNet [35] with 9 bottleneck blocks to do binary classification on the collectedgraphs.
The purpose of the classification was to extract the specific type of charts that were ready tobe fed to the information detection step in the pipeline to extract the data of the plot.
In this paper,the extraction step focused on the line charts.
However, we tried to make every module as generaland extensible as possible.
Thus, the classification module is not only able to classify line charts, butalso other three types of charts and images that were prevalent in academic literature.
The four typeswere line charts, images of tables, bar charts and images of pseudo code of algorithms.
The modelwas trained on about 20 thousand charts and images with labels using the cross entropy loss.
Weachieved very high accuracy on the classification task in all four categories, 97.9% accuracy on linecharts, 97.33% accuracy on table images, 99.19% on bar charts and 98.78% on algorithm images.
All the accuracies were obtained on a set-aside test dataset.
This high accuracy on the test datasetsuggests that our graph classification module has a very strong generalization performance.
Thissolidly indicates that it would not only perform well on our own dataset but will also perform well onany unseen dataset of the same types, which means that our model would be practical and useful inreal-world using scenarios.
3.2.3 Data Extraction
Axes Detection Inspired by chartreader [36], we convert the line charts to the grey-scale image toidentify the location of the x-axis and y-axis, by replacing all pixels with luminance greater than 200with value ’1’ (black) and all other pixels with value ’0’ (white).
We experimented with differentvalues for grey-scale thresholds and 200 was found to be best for grey-scale image conversion.
Forx-axis identification, we scanned the matrix horizontally the recorded the black pixels’ continuitywithin surrounding rows.
Then we selected all the continuous black pixels with less than ten whitepixels in the row.
Finally, we chose the column with the lowest fluctuation by the highest continuouslyof the adjacent rows.
The reason was that that axis commonly consisted of 2-3 pixels in publicationfor clear reading.
With a similar process, we can also detect the y-axis column.
Text Detection AWS-Rekognition DetectText API was used to detect text within an image.
DetectText also provided rectangular bounding boxes of the detected text.
We selected the detected text withthe highest confidence score.
A high threshold of the confidence score improved precision.
However,it would also reduce the recall rate.
Therefore, in order to mitigate recall rate reduction, we added adual layer for the second pass to release high-quality recognition.
To this end, we set the confidencescore threshold to 82 empirically.
The bonding boxed centre coordinates and height and width werestored in a matrix, { t x , t y , t w , t h } , and they were replaced with white colour.
Text Extraction A line chart contains a lot of different texts, such as figure titles, axes titles, labels,and legends.
The five text types that we classified were listed below:
X-axis value (yellow): x-axis text is the bounding boxes detected below the x-axis, and their centroidsare parallel to the x-axis.
Colour Cluster Since the line charts usually contained many lines, the authors used differentfeatures to distinguish, such as colours and dash lines.
We applied the DBSCAN [37] to classifydifferent figures based on the RGB value of each colour.
We examined the classification results on aset of randomly selected groups, and 16 groups had the best colour classification results.
We selectedthe figures with the most pixels, and the threshold depended on the number of legends boundingboxes detected.
(See Appendix C for an example diagram detailed process)
Data Recovery Based on the clustering results, we recovered data from each line consecutively.
Specifically, for each line, we scanned from left to right horizontally.
At each iteration (at each xvalue), we examined the vertical column of pixels.
There was always one coloured pixel that belongsto the line while all other pixels belong to the white background.
Besides, we can easily locate theposition of that coloured pixel in that column.
By calculating the ratio between the coloured pixel’sheight and the total height of the column, we can recover the y value for this specific x value.
Byrepeating the same process for all x values in the scan and for each line in the plot, we can recover allthe x-y value pairs of the plot.
(See Appendix D for an example detailed process)
Results and Discussion
Here we display the results of two systems on different publication datasets and discuss how they canrender the latent knowledge and accelerate the energy material research.
4.1 CNN assisted trends analysis and insights discovery for nanomaterial synthesis
In this paper, we showed the preliminary findings of ALD, which was a popular nano-film fabricationtechnique.
The ALD dataset had over 2,100 opinions with material names, in which nearly 70% oftheir sentiments are opportunities.
To avoid data fragmentation, we clustered opinions to corresponding periodic table elements of materials.
We visualized scientific categories of the top 10 elementsused in ALD in Figure 3.
It can be summarized that popular chemical elements used in ALD arewidely employed in material science, chemistry, nanoscience and physics, and to a lesser extent,energy and fuels.
In addition, categories like crystallography and pharmacy are unique to the ALD ofspecific elements, namely, titanium (Ti).
The gap between disciplines may hinder the extraction andutilization of shared knowledge about materials.
This figure implies the great potential to transferknowledge in opinions from one field to another.
We visualized the frequency of popular elements used by opinions from 2014 to 2022 in Figure 4(a).
To quantify the sentiment towards elements, we calculated each element’s percentage of opportunitiesin opinions each year based on the sentiments and year of publication.
As shown in Figure 4(b),each cell represents the average sentiment value of the element fabricated with ALD that year.
Weobserved that there is often a drop in frequency after the year when the element experienced a rathernegative sentiment and vice versa.
For example, sentiment scores of Zn (dark blue), Ti (orange),
Mo (pink), and Si (grey) had low levels in 2018, and their frequency fell in the following years.
Incontrast, the sentiment score of Mo (pink) in 2016, Ni (yellow) in 2017 and Al (purple) in 2019 wereall high levels, and their frequency went up after experiencing positive sentiments.
Although thisrelationship cannot be guaranteed, it helps us to predict the future trends of elements: Al, Si and Nigo rather cold; Pt, C and Cu emerge more; Zn, Ti, Li and Mo have relative stable attention.
We also visualized opinion mining results of elements or materials in other domains such as perovskite andthermoelectric (see Appendix B).
[Figure]
Case Study on Li
We examined the scientific facts in extracted opinions of lithium, an element withmarked changes in sentiment score.
As we can see in Figure 4(b), lithium experienced a prominentnegative sentiment in 2018 due to intrinsic properties, including thermal instability [38, 39], shuttlingeffects [40], and electrochemical instability [41].
Fortunately, several improvement measures of thesedrawbacks were found in opinions of the following years.
In 2019, Li [42] provided the possiblesolution for the LiMn 1 · 5 Ni 0 · 4 O 4 (LMNO) nanomaterial cycle time through selectively controlledmethods, integrating ALD with annealing at the nano level and process controlling and optimizationat the macro level.
After that, Lou et al.
[43] showed that the addition of Ti to Li-compoundscan improve its cycling stability which can easily be achieved using ALD.
These drawbacks andimprovements were from diversified categories, such as energy fuels, nanoscience, and metallurgy.
This case study shows that our system can further provide the reasons behind rising or fallingsentiments.
4.2 CNN assisted investigation of potential nanomaterial textures for silicon solar cell
The wide range of complex nanotextured surface morphologies made a comprehensive and systematicempirical investigation impossible.
An unambiguous performance review for reported empiricalresults was necessary to determine the promising materials’ nanotextures.
For generating textual andgraphical information for nanotextures, we implemented GraphMaster to help the data recovery frommaterial nanotextures characterization results, which were usually present as figures, such as Ramanspectroscopy spectra, and wavelength-dependent External Quantum Efficiency (EQE).
Case Study of b-Si Nanotexture for solar cells we used NLP, CV and ML to generate a highlyrefined database of reported b-Si solar cell results from the literature.
The procedure was rapid and straightforward, and the generated database provided a comprehensive insight into b-Si nanotexturesfor solar cell applications.
For example, the solar cell current improvement spectrum in Figure 5 wasgenerated by 45 reported EQE spectrums from 17 independent research groups under certain criteria.
It should be noted that the b-Si nanotextures used in the solar cells of these research works werehighly distinct from each other.
The figure shows that there is a solar cell current increase for the300-600 nm wavelength range when replacing the conventional textures with the b-Si nanotextures.
We can draw a solid conclusion that b-Si nanotextures are promising for solar cell applications assuch textures will contribute to superior optical and electrical performance in the short-wavelengthspectral region.
Conclusions and Future work
This research prepared a large nanomaterial literature collection with about 381,000 sentences and34,000 graphs for model training.
Two CNN models were developed using these data to extractscientific information from unstructured texts and graphs.
Deep learning approaches helped us topredict new thermoelectrical materials and provided valuable solutions to the problems of existingnanomaterial synthesis.
Furthermore, we also used the graph information combined with numericalsimulation to compare the different nanotextures’ performance.
Our future work will address datarecovery for more kinds of material characterization graphs.
In summary, the results of both systemsreported show that a CNN pre-classification process can improve scientific information extractionaccuracy.
The linkage between the text and graph information will provide a foundation to test andverify existing nanomaterial performance, which points to an interesting direction for future work.
Our automated information extraction pipeline is expected to boost nanomaterial data mining researchby replacing the heavy work of manual information extraction and we are also looking forward tobuilding more downstream applications other than designing optimized material composition andsynthesis methods.
B Opinion mining results of perovskite and thermoelectric materials
[Figure] [Figure]
D Data Recovery Results
Here we showed the detailed result of the example in fig.
Since we clustered the figure based ondifferent RGB values.
Therefore, the figure contains not only the data line but also the legends andsome edges of other lines.
For example, in figure 2, the author used green colours from dark to lightto annotate the R1-R3 and purple colours from dark to light to annotate M1 to M3.
Therefore,
The
R3 line’s edge colour was the same as the R1.
To solve that problem, we scanned the figure from left to right and only selected the data with a minimum distance from the previous y-value when therewere two more pixels in one column.
E Graphmaster Classification Experiment Result
The classification step is trained on the DocFigure dataset with 21K various types of charts andimages.
The ResNet model is trained on 1 Nvidia Tesla V100-SXM2 GPU, with 32GB memory size.
[Table]
The text detection process was tested on 120 wavelength-dependent EQE graphs.
[Table]
We showed the mean square error of the example image presented in Fig.2 below:
[Table]