Knowledge Graph Extraction from Total Synthesis Documents (2024)

Abstract: Knowledge graphs (KGs) have emerged as a powerful tool for organizing and integrating complex information, making it a suitable format for scientific knowledge. However, translating scientific knowledge into KGs is challenging as a wide variety of styles and elements to present data and ideas is used. Although efforts for KG extraction (KGE) from scientific documents exist, evaluation remains challenging and field-dependent; and existing benchmarks do not focuse on scientific information. Furthermore, establishing a general benchmark for this task is challenging as not all scientific knowledge has a ground-truth KG representation, making any benchmark prone to ambiguity. Here we propose Graph of Organic Synthesis Benchmark (GOSyBench), a benchmark for KG extraction from scientific documents in chemistry, that leverages the native KG-like structure of synthetic routes in organic chemistry. We develop KG-extraction algorithms based on LLMs (GPT-4, Claude, Mistral) and VLMs (GPT-4o), the best of which reaches 73% recovery accuracy and 59% precision, leaving a lot of room for improvement. We expect GOSyBench can serve as a valuable resource for evaluating and advancing KGE methods in the scientific domain, ultimately facilitating better organization, integration, and discovery of scientific knowledge.

## results the proposed benchmark was used to perform ablations on 3 of the components of the kge algorithm described in section 1.3. namely, we assess the effect of si preprocessing (appendix b), the parsing of pdfs using a rule-based approach, or directly through vision-language models (vlms), and the choice of llm for parsing of reaction descriptions into formatted reaction units. in addition, we evaluate the performance of multiple llms from different providers on the latter task across multiple metrics using a more specific benchmark, aimed at selecting suitable llms for this task, without the need to execute the whole extraction pipeline. ## conclusions we have proposed a novel benchmark for knowledge graph extraction in science from full papers. we exploit the native kg-like structure of synthetic the performance of multiple llm across multiple scales and providers is shown. models are evaluated on 4 metrics: prod ret evaluates the accuracy of retrieving the correct product name from an input paragraph (which involves separating product name from its reference key), react ret evaluates the same, for retrieval of reactants used in the described reaction, wrong inp assesses how good the models are at rejecting inputs that do not describe a chemical reaction, and key exact evaluates the ability of models to output the exact reference key for products.