SEED : Semantic Knowledge Transfer for Language Model Adaptation to Materials Science (2024)

Abstract: Materials science is an interdisciplinary field focused on studying and discovering materials around us. However, due to the vast space of materials, datasets in this field are typically scarce and have limited coverage. This inherent limitation makes current adaptation methods less effective when adapting pre-trained language models (PLMs) to materials science, as these methods rely heavily on the frequency information from limited downstream datasets. In this paper, we propose Semantic Knowledge Transfer (SEED), a novel vocabulary expansion method to adapt the pre-trained language models for materials science. The core strategy of SEED is to transfer the materials knowledge of lightweight embeddings into the PLMs. To this end, we introduce knowledge bridge networks, which learn to transfer the latent knowledge of the materials embeddings into ones compatible with PLMs. By expanding the embedding layer of PLMs with these transformed embeddings, PLMs can comprehensively understand the complex terminology associated with materials science. We conduct extensive experiments across a broad range of materials-related benchmarks. Comprehensive evaluation results convincingly demonstrate that SEED mitigates the mentioned limitations of previous adaptation methods, showcasing the efficacy of transferring embedding knowledge into PLMs.

## introduction the pre-training and fine-tuning paradigm of language models is widely adopted in natural language processing (nlp). however, since pre-training is typically performed on general-domain corpora, such as wikipedia, the adaptability of pretrained language models (plms) is limited when the target domains differ significantly from the one promising approach to enhance the adaptability of plms is to expand the coverage of vocabulary. for example, previous works have expanded the vocabulary of plms by considering the frequency information of downstream datasets @xcite . however, such a frequency-based approach can be suboptimal in materials science, as downstream datasets in this domain are typically scarce and limited in coverage @xcite . indeed, we experimentally observe that a state-of-the-art optimization method (i.e., avocado @xcite ) rather degrades the performance of the original model foot_0 . figure 1 illustrates that avocado yields poor adaptation results in materials science, while significantly enhancing the performance of plms in other domains (e.g., biomedical, computer science, politics), underscoring the unique challenges of adaptation to the materials science domain. in response, we propose semantic knowledge transfer (seed), a novel method designed to optimize the vocabulary and embedding layer of plms for materials science. specifically, unlike prior works that rely on frequency information from downstream datasets, seed utilizes latent knowledge within the materials science corpus to adapt the plm's vocabulary. given that pre-training models on a large corpus incurs significant adaptation costs, seed leverages @xcite , lightweight word embeddings trained on materials science journals. to bridge these two distinct types of knowledge representations, we introduce bridge networks that transfer the materials knowledge from mat2vec into plms. with the transferred knowledge from mat2vec, plms can be effectively adapted to materials science domains. to verify the efficacy of seed, we conduct extensive experiments across diverse benchmarks in materials science, including materials entity recognition, slot filling, and glass classification, using various plm backbones. the evaluation results demonstrate that seed effectively mitigates the inherent limitations in adapting plms for materials science. additionally, we observe that the transferred embeddings are closely aligned with the original embeddings in plms, confirming the successful knowledge transfer achieved by seed . in summary, the contributions of this paper include the following: ## seed: semantic knowledge transfer in this work, we elaborate on semantic knowledge transfer (seed). the key strategy of seed involves transferring the knowledge from external materials embeddings into plms. to achieve this, we begin with the words shared between the vocabularies of materials embeddings and plms ( §3.1). we then train bridge networks to ensure that the semantic relations of the shared words are transferred to plms ( §3.2). after training, we transfer the materials knowledge only existed in the materials embeddings into the plms through the learned bridge network ( §3.3). the overall procedures of seed are illustrated in the figure 2 . ## experiments in this section, we experimentally demonstrate the efficacy of seed in adapting plms to downstream tasks. specifically, we mainly focus on whether seed mitigates the limitations of vocabulary expansion methods in materials science. ## conclusion in this work, we have proposed semantic knowledge transfer (seed), a novel vocabulary expansion method aimed at adapting plms to materials science. specifically, we have leveraged mat2vec to expand the knowledge of the plms, which are lightweight embeddings trained on largescale scientific papers. the knowledge in the materials embeddings is subsequently transferred to the plms through the learned bridge networks which serve as a mapping function between two different knowledge representations. we have performed extensive experiments to verify the efficacy of the proposed method across diverse benchmarks and various architectures. comprehensive results have convincingly demonstrated that adapting the plms with seed leads to substantial improvements in performance across diverse materials-related tasks compared to existing vocabulary expansion methods, highlighting the broad value of seed in materials science.