A ction IE : Action Extraction from Scientific Literature with Programming Languages (2024)

Abstract: Extraction of experimental procedures from human language in scientific literature and patents into actionable sequences in robotics language holds immense significance in scientific domains. Such an action extraction task is particularly challenging given the intricate details and context-dependent nature of the instructions, especially in fields like chemistry where reproducibility is paramount. In this paper, we introduce ActionIE, a method that leverages Large Language Models (LLMs) to bridge this divide by converting actions written in natural language into executable Python code. This enables us to capture the entities of interest, and the relationship between each action, given the features of Programming Languages. Utilizing linguistic cues identified by frequent patterns, ActionIE provides an improved mechanism to discern entities of interest. While our method is broadly applicable, we exemplify its power in the domain of chemical literature, wherein we focus on extracting experimental procedures for chemical synthesis. The code generated by our method can be easily transformed into robotics language which is in high demand in scientific fields. Comprehensive experiments demonstrate the superiority of our method. In addition, we propose a graph-based metric to more accurately reflect the precision of extraction. We also develop a dataset to address the scarcity of scientific literature occurred in existing datasets.

## introduction recently, the integration of natural language processing (nlp) techniques into various scientific fields has achieved significant success @xcite @xcite @xcite . among the applications, extracting information from unstructured scientific literature has been one with growing significance @xcite . for example, chemists typically look through a wide range of publications to select candidate protocols for one organic synthesis scene, based on their own reading and repetitive trial-and-error procedures @xcite . therefore, structured chemical data, including reaction formulae, chemical entities, and experiment conditions, facilitates effective utilization and automatic analysis, such as indexing and searching by keywords; discovering and analyzing relations between entities; clustering related objects and discovering potential patterns; automatically executing protocols; and predicting and optimizing experiment conditions. representatively, figure 1 presents a case of structured chemical experimental procedure, essential for guiding practitioners in their laboratory work @xcite . this task involves extracting a sequence of chemical reaction actions from a scientific text passage, where each action is defined by an operation and its corresponding attributes. for instance, in the example "add etoac" shown in figure1, "add" represents the operation, and the chemical "etoac" is the attribute. however, the discovery of new chemical experimental procedures is scattered across unstructured scientific text and described in various writing styles, posing a significant challenge to the automatic creation of reaction action databases. existing chemical databases, predominantly commercial ones such as reaxys @xcite , scifinder (chemical abstracts service (cas), 2023), and pistachio (nextmove software, 2023), depend extensively on the manual contributions of domain experts. analyzing, indexing, and utilizing scientific literature typically requires extensive and costly annotation or labeling by human experts. moreover, this method is prone to errors due to the sheer volume of rapidly expanding scientific data. despite the considerable manual effort, these databases prioritize storing information on the reactants, products, and reaction conditions, rather than the concrete sequences of chemical actions. this is primarily because manually designing these experiment procedures is both time-consuming and costly. to tackle this issue, various studies have employed text mining techniques @xcite @xcite @xcite to automatically extract structured information on procedures from unstructured text, leveraging the advancements in nlp field. however, extracting experimental procedures remains a challenging task. one major hurdle is the complexity and variability of scientific language, which often features intricate sentence structures, domain-specific terminology, abbreviations, and acronyms. these elements pose substantial difficulties for sequential tagging-based approaches. for example, as shown in figure 1 , a text describing a series of chemical reaction actions includes the "wash" operation followed by three chemicals. while sequential tagging-based methods might recognize the chemical compounds, they often struggle to accurately identify the operations and associate them with their corresponding attributes. furthermore, the scarcity of large, annotated datasets poses an additional obstacle to training deep learning models on chemical experimental data effectively. in this paper, we choose chemical experiment procedures as a case study, and explore the potential of large language models (llms) to extract structured data from the complex and domainspecific language in chemical papers and patents. we propose a novel approach that frames the procedure extraction task as a code generation problem, where we express the experimental procedures as a series of pre-defined operations, and utilize the unique features of coding, such as classes, inheritance, and types, to structure this information. our method leverages the capabilities of llms in few-shot in-context learning, reducing the need for large amounts of annotated data, and accelerating the preparation process. moreover, our proposed framework also offers an easy solution to generate protocols for different automated platforms by applying different language configurations. from the perspective of evaluation, we first pinpoint shortcomings within current evaluation metrics for the chemical action extraction task, and propose a novel metric based on graph-matching that substantially improves correlation with human judgments. existing benchmarks largely concentrate on patent documents, which are inherently well-structured. to more accurately meet the realworld demands of practitioners, we meticulously annotate a test set derived from chemistry literature, which offers a more comprehensive evaluation of model performance. notably, our new benchmark is considerably more extensive than previous ones, with an average length of 770.8 characters compared to 158.2 characters, providing a testing environment that mirrors realistic scenarios more closely. experimentally, our method actionie demonstrates consistent superiority over strong baseline models, both against traditional benchmarks and our newly established testbed. ## related work the practice of using nlp in structured scientific data extraction has seen significant advancements, from utilizing traditional nlp techniques to integrating code generation methods into structure extraction, which is especially influenced by the growing capabilities of large language models (llms).