AI for Science in the Era of Large Language Models (2024)

Abstract: The capabilities of AI in the realm of science span a wide spectrum, from the atomic level, where it solves partial differential equations for quantum systems, to the molecular level, predicting chemical or protein structures, and even extending to societal predictions like infectious disease outbreaks. Recent advancements in large language models (LLMs), exemplified by models like ChatGPT, have showcased significant prowess in tasks involving natural language, such as translating languages, constructing chatbots, and answering questions. When we consider scientific data, we notice a resemblance to natural language in terms of sequences – scientific literature and health records presented as text, bio-omics data arranged in sequences, or sensor data like brain signals. The question arises: Can we harness the potential of these recent LLMs to drive scientific progress? In this tutorial, we will explore the application of large language models to three crucial categories of scientific data: 1) textual data, 2) biomedical sequences, and 3) brain signals. Furthermore, we will delve into LLMs’ challenges in scientific research, including ensuring trustworthiness, achieving personalization, and adapting to multi-modal data representation.

## tutorial content the impressive capabilities of artificial intelligence (ai) within the realm of science span a wide spectrum, from the atomic level, where it attempts to solve partial differential equations for quantum systems, to the molecular level, where it accurately predicts the structures of chemicals and proteins, and extends even further, encompassing societal predictions like forecasting infectious disease outbreaks @xcite . amidst this landscape of possibilities, recent advancements in large language models (llms), notably exemplified by models like chatgpt 1 , have risen to the forefront, 1 https://chat.openai.com/chat demonstrating significant proficiency in tasks tied to natural language. these tasks include language translation, constructing chatbots, and answering questions @xcite . interestingly, when we turn our attention to scientific data, we discover a striking resemblance to natural language in terms of sequences. scientific literature and health records are laid out as textual narratives, bio-omics data takes the form of molecular sequences, and even sensor data like brain signals is inherently sequential @xcite . this observation prompts a compelling question: can we leverage the potential of these advanced llms to propel scientific advancement? in this tutorial, we embark on a journey to explore precisely this intersection-the fusion of cutting-edge large language models with scientific inquiry. our exploration zooms in on three pivotal categories of scientific data: 1) textual data @xcite @xcite @xcite @xcite , 2) biomedical sequences @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , and 3) brain signals @xcite @xcite . by drawing inspiration from the transformative capabilities of llms, we seek to unravel novel understanding and innovation within each domain. as we move forward, we further discuss the intricate challenges that accompany the infusion of ai into scientific research. the foundation of trustworthiness stands tall-how do we ensure the reliability of ai-enhanced scientific insights? the concept of personalization emerges as a critical consideration, urging us to tailor llms to the specific contours of scientific investigation. furthermore, the multi-dimensional nature of scientific data beckons us to master the art of handling data representations that span across various modalities. ## tutorial type this is a cutting-edge tutorial, bridging the gap between the nlp community and ai for science. ## target audience and prerequisites this tutorial is intended for researchers and practitioners in natural language processing, machine learning, and their applications to science domains. while the audience with a good background in the above areas would benefit most from this tutorial, we believe the material to be presented would give the general audience and newcomers a complete picture of the important research topics in ai for science with large language models. our tutorial is designed as self-contained, so no specific background knowledge is assumed of the audience. however, it would be beneficial for the audience to know about the basics of deep learning technologies and pre-trained language models (e.g., transformer @xcite , bert (kenton and toutanova, 2019), gpt @xcite , and t5 @xcite ) before attending this tutorial. we will provide a reading list of background knowledge on our tutorial website. ## tutorial outline this tutorial is expected to be 3 hours in duration plus a 30-minute break in between. the contents are outlined below. ## diversity consideration we will discuss large language models scaled up to various scientific domains and various data formats (textual data, biomedical sequences, and brain signals). our instructors consist of phd students (zhenyu bi and minghao xu), junior faculty (xuan wang, assistant professor), and senior faculty (jian tang, associate professor). our instructors also came from diverse geographical locations (zhenyu bi and xuan wang from virginia tech in the us, and minghao xu and jian tang from mila -quebec ai institute in canada). we plan to involve inclusive topics, accessible materials, diverse instructors, flexible formats, and targeted outreach to ensure a broad and varied audience engagement. ## reading list we will provide a reading list of background knowledge on our tutorial website. a preliminary reading list can be found as the references. ## tutorial presenters zhenyu bi is a ph.d. student in the computer science department at virginia tech. his research area lies in the field of natural language processing, emphasizing real-world applications of large language models. he is mainly interested in information extraction with weak supervision, especially text mining and event extraction; as well as fact-checking and trustworthy nlp. he received an m.s. degree in intelligent information systems from carnegie mellon university in 2023, a b.s. degree in cognitive science, and a b.s. degree in computer science from the university of california, san diego in 2021. minghao xu is a ph.d. student at mila -quebec ai institute, canada. his research interests mainly lie in protein function understanding and protein design. he aims to understand diverse protein functions with joint guidance from protein sequences, structures, and biomedical text, especially boosted by large-scale multi-modal pre-training. he is also pursuing structure-and sequence-based protein design via generative ai, geometric deep learning and dry-wet experiment closed looping. he has given an oral presentation at the main conference of icml'23. jian tang is an associate professor at mila -quebec ai institute, canada. his long-term interests focus on understanding the language of life (dna, rnas, and proteins) with generative ai and geometric deep learning, with applications in biomedicine and synthetic biology. his group has developed one of the first open-source machine learning frameworks on drug discovery, torchdrug (for small molecules) and torchprotein (for proteins), and developed the first diffusion models for 3d molecular structure generation, geodiff (among the 50 most cited ai paper in 2022). he has given a few tutorials at international ai and data mining conferences including kdd 2017, aaai 2019, aaai 2022. xuan wang is an assistant professor in the computer science department at virginia tech. her research focuses on natural language processing and text mining, emphasizing applications to science and healthcare domains. her current projects include nlp and text mining with extremely weak supervision; text-augmented knowledge graph reasoning; fact-checking and trustworthy nlp, ai for science; and ai for healthcare. she received a ph.d. degree in computer science, an m.s. degree in statistics, and an m.s. degree in biochemistry from the university of illinois urbana-champaign in 2022, 2017, and 2015, respectively, and a b.s. degree in biological science from tsinghua university in 2013. she has delivered tutorials in ieee-bigdata 2019, www 2022, and kdd 2022. ## estimated audience size this is a cutting-edge tutorial that introduces new frontiers in the intersection of nlp and ai for science. the presented topic has not been covered by acl/emnlp/naacl/eacl/coling tutorials in the past four years. it is hard to give an estimate of audience size given no similar tutorials have been delivered before. a rough estimate would be around tens to hundreds of participants. ## preferred venues we prefer the following venues for this tutorial: 1) acl, 2) emnlp, and 3) naacl. ## technique requirement standard equipment will be enough for our tutorial and we don't have specific requirements. we will bring our own laptop and a wireless pointer. ## presentation materials we will provide tutorial materials (e.g., tutorial slides and relevant list of papers) one month prior to the date of the tutorial. the tutorial materials will be publically available for open access. ## ethics statement ethical quandaries frequently confront technological advancements, especially when it comes to dual-use scenarios where an innovation can bring both advantages and disadvantages. the tutorial introduces ie technologies, where the distinction between beneficial and detrimental employment predominantly hinges on data usage. employing this technology responsibly necessitates the lawful and ethical acquisition of input text collections and other forms of input. regulations and standards establish a legal framework to ensure appropriate data utilization, granting individuals the right to request the removal of their data. in the absence of such regulations, the ethical responsibility falls upon technology practitioners to uphold righteous data use. moreover, biases can infiltrate training and evaluation data, potentially diminishing system accuracy for underrepresented groups or in novel domains. this bias can result in performance disparities based on attributes like ethnicity, race, and gender. additionally, systems trained on specific data can experience degradation when confronted with new, dissimilar data. this accentuates the need to thoughtfully contemplate matters of fairness and generalizability when employing ie technologies with particular datasets. to guarantee the conscientious application of dual-use technology, a comprehensive approach involves prioritizing ethical considerations as foundational principles during every phase of system design. transparency and interpretability should remain paramount across data, algorithms, models, and functionality within the system. public verification and auditing can be facilitated by making software open source. furthermore, strategies to safeguard marginalized groups should be explored as a part of ethical technology deployment.