MELT : Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science (2024)

Abstract: We introduce a novel continued pre-training method, MELT (MatEriaLs-aware continued pre-Training), specifically designed to efficiently adapt the pre-trained language models (PLMs) for materials science. Unlike previous adaptation strategies that solely focus on constructing domain-specific corpus, MELT comprehensively considers both the corpus and the training strategy, given that materials science corpus has distinct characteristics from other domains. To this end, we first construct a comprehensive materials knowledge base from the scientific corpus by building semantic graphs. Leveraging this extracted knowledge, we integrate a curriculum into the adaptation process that begins with familiar and generalized concepts and progressively moves toward more specialized terms. We conduct extensive experiments across diverse benchmarks to verify the effectiveness and generality of MELT. A comprehensive evaluation convincingly supports the strength of MELT, demonstrating superior performance compared to existing continued pre-training methods. In-depth analysis also shows that MELT enables PLMs to effectively represent materials entities compared to the existing adaptation methods, thereby highlighting its broad applicability across a wide spectrum of materials science.

## introduction materials science encompasses interdisciplinary studies concerning the behaviors, properties, and applications of materials. given the vast search space in materials science, deep learning-based approaches have emerged as significant avenues to accelerate the entire research pipeline @xcite @xcite . specifically, methods centered on natural language processing (nlp) have provided promising results across a number of materials tasks, such as materials entity recognition @xcite and synthesis action retrieval @xcite . however, the limited and scarce nature of datasets in this domain poses substantial challenges for developing models that generalize well across a broad range of materials entities @xcite . one promising approach to addressing this limitation involves adapting the pre-trained language models (plms) for materials science by continuously pre-training them on materials science corpus @xcite . however, these methods have predominantly focused on constructing the domain-specific corpora used in the continued pre-training process, neglecting the training strategies employed in the adaptation process. this oversight can lead to inefficiencies in capturing domain-specific nuances and knowledge. for example, the chemical formulas (e.g., licoo 2 ), which are fundamental terms in the materials science field, are typically infrequent words (figure 1 ). therefore, these domain-specific terms are often inadequately captured by the random masking strategy used in previous studies, leading to suboptimal adaptation results. in this paper, we propose a novel continued pretraining method, namely melt (materials-aware continued pre-training) that learns the knowledge of materials science through the tailored masking strategy to materials science. to this end, we begin by extracting chemical entities from the materials science corpus. however, these entities alone do not encompass all the fundamental concepts of materials science, specifically the structure-propertyprocessing-performance paradigm @xcite . to bridge this gap, we expand the coverage of materials knowledge by constructing semantic graphs that integrate relevant concepts and missing entities. building on this enhanced materials knowledge, we introduce curriculum into the adaptation process, starting with more familiar and interconnected entities and gradually advancing to more specialized and less common terms. this comprehensive adaptation allows plms to effectively learn materials science knowledge. to verify the efficacy of melt, we perform extensive experiments, spanning a wide range of downstream tasks for materials science on both generation and classification tasks. comprehensive evaluation results clearly support the superiority of melt over strong baselines. moreover, we verify that melt enables plms to effectively learn the materials by thoroughly transforming both the training corpora and adaptation strategies, underscoring the efficacy of the tailored approach to materials science. in summary, the contributions of this paper include the following: ## materials-aware continued pre-training we elaborate on the proposed continued pretraining method, melt. the key strategy is to extract the material entities from the scientific papers and inject the extracted knowledge of the materials into the plms through a masked language modeling (mlm) objective. to this end, we first extract the materials entities from the materials science corpus. from these entities, we expand the materials knowledge by semantically augmenting the related entities and properties. we then start pre-training on the basic materials entities and pro-gressively move more specialized knowledge in a curriculum manner. the overall procedures are described in figure 2 . ## experiments in this section, we verify the efficacy of the melt. specifically, we answer the following four questions through extensive experiments and analysis: q1 (adaptability) does melt enable better adaptation than existing methods across diverse benchmarks? ( §4.2, §4.4 §4.5) q2 (transferability) how the extracted materials knowledge from melt is related and affect to downstream tasks? ( §4.3, §4.6, §4.7) q3 (efficiency) does melt offer better efficiency than existing continued pre-training methods for domain adaptation? ( §4.8) ## conclusion in this paper, we have proposed melt, a novel continued pre-training method to adapt plms for materials science. unlike previous approaches that typically focus on the adaptation corpus, our method comprehensively considers both the corpus and the training strategy in the adaptation process. specifically, we have constructed the materials knowledge base by considering materials science tetrahedron @xcite . this extracted knowledge is then transferred to the plms in a structured, curriculum-based manner. we have conducted extensive experiments across diverse materials science benchmarks. the evaluation results convincingly demonstrate that our tailored approach yields superior performance on downstream tasks compared to domain-agnostic methods. in-depth analysis focusing on generality, efficiency, and insights has further supported the efficacy of melt in continued pre-training, highlighting its broad value in the field of materials science.