Evaluating Extrapolation Ability of Large Language Model in Chemical Domain (2024)

Abstract: Solving a problem outside the training space, i.e. extrapolation, has been a long problem in the machine learning community. The current success of large language models demonstrates the LLM’s extrapolation ability to several unseen tasks. In line with these works, we evaluate the LLM”s extrapolation ability in the chemical domain. We construct a data set measuring the material properties of epoxy polymers depending on various raw materials and curing processes. LLM should predict the material property when novel raw material is introduced utilizing its chemical knowledge. Through experiments, LLM tends to choose the right direction of adjustment but fails to determine the exact degree, resulting in poor MAE on some properties. But LLM can successfully adjust the degree with only a one-shot example. The results show that LLM can extrapolate to new unseen material utilizing its chemical knowledge learned through massive pre-training.

## related works several researchers adopted an llm to the chemical domain by training it on a chemistryrelated corpus. @xcite introduced a data set for instruction tuning including various molecule/protein-oriented tasks. @xcite and @xcite integrated the graph structure of molecules into an llm to improve its representation power. @xcite and @xcite trained a dialogue model on chemical domain. our goal is to verify the chemical ability of an existing llm, not suggesting a new foundation model. @xcite verified existing llms' ability on eight chemical tasks from name prediction to molecule captioning. they showed that gpt-4 (openai, 2024) showed the best performance on most tasks showing comparable performance with sota, task-specific models. our work is an extension of their work while differing on two points: (1) their work focused on molecule-level tasks, while our work is compound-level. as more information should be considered, compound-level tasks require more complex reasoning than the moleculelevel. (2) unlike classic tasks, we focus on the extrapolation ability of an llm, which is also more challenging. ## problem statement let (x , y) be a domain of independent and dependent variables of train data. we have our train data d @xmath0, y i )} n train @xmath1, x @xmath2, y @xmath3. let x ′ be a domain of additional independent variables. set a domain of independent variables of test data as x × x ′ . then we have our test data we measure an extrapolation ability of a model f as e (x,x ′ ,y)∼dtest [l(f (x, x ′ ), y)], when x x ′ = ϕ. a model should infer the relationship between x ′ and other variables to extrapolate successfully. our research hypothesis is can we utilize llm's internal chemical knowledge for extrapolation, by providing additional information, e.g. smiles of an additional raw material? we test this hypothesis through experiments in the next section. ## conclusion in this paper, we evaluate the extrapolation ability of llm in the chemical domain. we focus on regressing three material properties of epoxy compound when a novel raw material is introduced. we build a data set involving various raw materials and curing conditions from lab experiments. compared to baseline regressors, llm shows superior extrapolation ability in predicting cross-link density (v c ), while failing on tan δ peak. by examining the tokens used in llm prediction, we find out that llm tends to capture the right adjustment direction while failing to grasp the exact degree of adjustment. we also show that llm successfully adjusts the degree with only 1-shot example. this result shows the potential applicability of llm's extrapolation ability in chemical knowledge discovery.