M at S ci- NLP : Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling (2023)

Abstract: We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. MatBERT, a model pretrained specifically on materials science journals, generally performs best for most tasks. Moreover, we propose a unified text-to-schema for multitask learning on {pasted macro ‘BENCHMARK’} and compare its performance with traditional fine-tuning methods. In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23 .

## introduction materials science comprises an interdisciplinary scientific field that studies the behavior, properties and applications of matter that make up materials systems. as such, materials science often requires deep understanding of a diverse set of scientific disciplines to meaningfully further the state of the art. this interdisciplinary nature, along with the great technological impact of materials advances and growing research work at the intersection of machine learning and materials science @xcite @xcite , makes the challenge of developing and evaluating natural language processing (nlp) models on materials science text both interesting and exacting. the vast amount of materials science knowledge stored in textual format, such as journal articles, patents and technical reports, creates a tremendous opportunity to develop and build nlp tools to create and understand advanced materials. these tools could in turn enable faster discovery, synthesis and deployment of new materials into a wide variety of application, including clean energy, sustainable manufacturing and devices. understanding, processing, and training language models for scientific text presents distinctive challenges that have given rise to the creation of specialized models and techniques that we review in section 2. additionally, evaluating models on scientific language understanding tasks, especially in materials science, often remains a laborious task given the shortness of high-quality annotated data and the lack of broad model benchmarks. as such, nlp research applied to materials science remains in the early stages with a plethora of ongoing research efforts focused on dataset creation, model training and domain specific applications. the broader goal of this work is to enable the development of pertinent language models that can be applied to further the discovery of new material systems, and thereby get a better sense of how well language models understand the properties and behavior of existing and new materials. as such, we propose matsci-nlp, a benchmark of various nlp tasks spanning many applications in the materials science domain described in section 3. we utilize this benchmark to analyze the performance of various bert-based models for matsci-nlp tasks under distinct textual input schemas described in section 4. concretely, through this work we make the following research contributions: we centered our matsci-mlp analysis on exploring the following questions: q1 how does in-domain pretraining of language models affect the downstream performance on matsci-nlp tasks? we investigate the performance of various models pretrained on different kinds of domain-specific text including materials science, general science and general language (bert @xcite ). we find that matbert generally performs best and that language models pretrained on diverse scientific texts outperform a general language bert. interestingly, scibert @xcite often outperforms materials science language models, such as matscib-ert @xcite and batterybert @xcite . q2 how do in-context data schema and multitasking affect the learning efficiency in lowresource training settings? we investigate how several input schemas shown in figure 1 that contain different kinds of information affect various domain-specific language models and propose a new task-schema method. our experiments show that our proposed task-schema method mostly performs best across all models and that question-answering inspired schema outperform single task and multitask fine-tuning settings. ## background the advent of powerful nlp models has enabled the analysis and generation of text-based data across a variety of domains. bert @xcite was one of the first large-scale transformerbased models to substantially advance the state-ofthe-art by training on large amounts of unlabeled text data in a self-supervised way. the pretraining procedure was followed by task-specific finetuning, leading to impressive results on a variety of nlp task, such as named entity recognition (ner), question and answering (qa), and relation classification @xcite @xcite . a significant collection of large language models spanning millions to billions of parameters followed the success of bert adopting a similar approach of pretraining on vast corpora of text with task-specific fine-tuning to push the state-of-the-art for in natural language processing and understanding @xcite @xcite . ## matsci-nlp benchmark through the creation of matsci-nlp, we aim to bring together some of the fragmented data across multiple research works for a wide-ranging materials science nlp benchmark. as described in section 2, the availability of sizeable, high-quality and diverse datasets remain a major obstacle in applying modern nlp to advance materials science in meaningful ways. this is primarily driven by a high cost of data labeling and the heterogeneous nature of materials science. given those challenges, we created matsci-nlp by unifying various publicly available, high-quality, smaller-scale datasets to form a benchmark for fine-tuning and evaluating modern nlp models for materials science applications. matsci-nlp consists of seven nlp tasks shown in the data in matsci-nlp adheres to a standard json-based data format with each of the samples containing relevant text, task definitions, and annotations. these can in turn be refactored into different input schemas, such as the ones shown in figure 1 consisting of 1) input: primary text jointly with task descriptions and instructions, and 2) output: query and label, which we perform in our text-to-schema modeling described in section 4. next, we describe the tasks in matsci-nlp in greater detail: in the relation classification task, the model predicts the most relevant relation type for a given span pair (s i , s j ). matsci-nlp contains relation classification task data adapted from mysore et al. (2019); yamaguchi et al. (2020); matscire (2022). the tasks contained in matsci-nlp were selected based on publicly available, high-quality annotated materials science textual data, as well as their relevance to applying nlp tools to materials science. conventional nlp tasks (ner, relation classification, event argument extraction, paragraph classification, sentence classification) enable materials science researchers to better process and understand relevant textual data. domain specific tasks (sar, slot filling) enable materials science research to solve concrete challenges, such as finding materials synthesis procedures and realworld experimental planning. in the future, we aim to augment to current set of tasks with additional data and introduce novel tasks that address materials science specific challenges with nlp tools. ## unified text-to-schema language modeling as shown in figure 1 , a given piece of text can include multiple labels across different tasks. given this multitask nature of the matsci-nlp benchmark, we propose a new and unified task-schema multitask modeling method illustrated in figure 2 that covers all the tasks in the matsci-nlp dataset. our approach centers on a unified text-to-schema modeling approach that can predict multiple tasks simultaneously through a unified format. the underlying language model architecture is made up of modular components, including a domainspecific encoder model (e.g. matbert, matscib-ert, scibert), and a generic transformer-based decoder, each of which can be easily exchanged with different pretrained domain-specific nlp models. we fine-tune these pretrained language models and the decoder with collected tasks in matsci-nlp using the procedure described in section 4.3. the unified text-to-schema provides a more structured format to training and evaluating language model outputs compared to seq2seq and text-to-text approaches @xcite . this is particularly helpful for the tasks in matsci-nlp given that many tasks can be reformulated as classification problems. ner and slot filling, for example, are classifications at the tokenlevel, while event arguments extraction entails the classification of roles of certain arguments. without a predefined schema, the model relies entirely on unstructured natural language to provide the answer in a seq2seq manner, which significantly increases the complexity of the task and also makes it harder to evaluate performance. the structure imposed by text-to-schema method also simplifies complex tasks, such as event extraction, by enabling the language model to leverage the structure of the schema to predict the correct answer. we utilize the structure of the schema in decoding and evaluating the output of the language models, as described further in section 4.3 in greater detail. moreover, our unified text-to-schema approach alleviates error propagation commonly found in multitask scenarios @xcite , enables knowledge sharing across multiple tasks and encourages the fine-tuned language model to generalize across a broader set of text-based instruction scenarios. this is supported by our results shown in section 5.2 showing textto-schema outperforming conventional methods. ## evaluation and results our analysis focuses on the questions outlined in section 1: 1) studying the effectiveness of domainspecific language models as encoders, and 2) analyzing the effect of different input schemas in resolving matsci-nlp tasks. concretely, we study the performance of the language models and language schema in a low resource setting where we ## conclusion and future works we proposed matsci-nlp, the first broad benchmark on materials science language understanding tasks constructed from publicly available data. we further proposed text-to-schema multitask modeling to improve the model performance in lowresource settings. leveraging matsci-nlp and text-to-schema modeling, we performed an in-depth analysis of the performance of various scientific language models and compare text-toschema language modeling methods with other input schemas, guided by (q1) addressing the pretrained models and (q2) addressing the textual schema. overall, we found that the choice of pretrained models matters significantly for downstream performance on matsci-nlp tasks and that pretrained language models on scientific text of any kind often perform better than pretrained language models on general text. matbert generally performed best, highlighting the benefits of pretraining with high-quality domain-specific language data. with regards to the textual schema outlined in (q2), we found that significant improvements can be made by improving textual schema showcasing the potential of fine-tuning using structured language modeling. the proposed encoder-decoder architecture, as well as the proposed multitask schema, could also be useful for additional domains in nlp, including both scientific and non-scientific domains. the potential for open-domain transferability of our method is due to: 1) our multitask training method and associated schemas do not depend on any domain-specific knowledge, allowing them to be easily transferred to other domains. 2) the encoder of our proposed model architecture can be exchanged in a modular manner, which enables our model structure to be applied across multiple domains. 3) if the fine-tuning data is diverse across a wide range of domains, our method is likely to learn general language representations for opendomain multitask problems. future work could build upon this paper by applying the model and proposed schema to different scientific domains where fine-tuning data might be sparse, such as biology, physics and chemistry. moreover, future work can build upon the proposed schema by suggesting novel ways of modeling domain-specific or general language that lead to improvements in unified multi-task learning.