PAGED : A Benchmark for Procedural Graphs Extraction from Documents (2024)

Abstract: Automatic extraction of procedural graphs from documents creates a low-cost way for users to easily understand a complex procedure by skimming visual graphs. Despite the progress in recent studies, it remains unanswered: whether the existing studies have well solved this task (Q1) and whether the emerging large language models (LLMs) can bring new opportunities to this task (Q2). To this end, we propose a new benchmark PAGED, equipped with a large high-quality dataset and standard evaluations. It investigates five state-of-the-art baselines, revealing that they fail to extract optimal procedural graphs well because of their heavy reliance on hand-written rules and limited available data. We further involve three advanced LLMs in PAGED and enhance them with a novel self-refine strategy. The results point out the advantages of LLMs in identifying textual elements and their gaps in building logical structures. We hope PAGED can serve as a major landmark for automatic procedural graph extraction and the investigations in PAGED can offer insights into the research on logic reasoning among non-sequential elements.

## introduction procedural graphs, though can intuitively represent the execution of actions for goal achievement @xcite , suffer from the high cost of expert-construction @xcite . the automatic extraction of procedural graphs from procedural documents thus has huge potential, as it would enable users to easily understand how to logically perform a goal (e.g, how a restaurant serves the customers) by skimming visual graphs (e.g., figure 1 (a) instead of reading lengthy documents (e.g., figure 1(b) ). * corresponding author however, obtaining optimal procedural graphs is not easy -as shown in figure 1 (a), it requires representing not only sequential actions in the procedure (e.g., i-4 → i-5), but also non-sequentially executed actions (e.g., i-7.1 & i-7.2]) and vital constraints for the actions (e.g., c-2). off-the-shelf attempts only meet part of the requirements. for example, @xcite ; @xcite fail to represent "customers can order only the dishes or drinks, or both" (i-2.1 & i-2.2) due to its inherent limitation for representing complex nonsequential actions. besides, current solutions only focus on hand-written rules @xcite or customized networks @xcite on a small group of cherry-picked instances. this brings up the question of whether the existing studies have well solved the automated extraction of procedural graphs from procedural documents (q1). if the answer is "no", we are also interested in whether the emerging large language models (llms) can bring new opportunities to this task (q2). to answer q1, we propose to construct a standard benchmark for the procedural graphs extraction from documents (paged). as far as we know, there lack of large-scale datasets of document-graph pairs for training and evaluating optimal procedural graph extraction models (cf., to further answer q2, we investigate the performance of three advanced llms @xcite , chatgpt @xcite and llama2 @xcite ) and utilize a selfrefine strategy to improve the ability of llms. in total, we evaluate ten methods in our paged benchmark. extensive experiments on our benchmark reveal that existing studies struggle to accurately extract sequential actions, constraints, and organize non-sequential actions of procedural documents. while llms have shown significant improvement in sequential action and constraint extraction, they still face challenges with non-sequential action organization. our detailed analysis of the results leads us to propose improvement strategies to help large language models better understand nonsequential actions and use correct gateways to represent them. we hope paged can be a key mile-stone for automatic procedural graphs extraction, offering insights into research on logical reasoning among non-sequential elements. in summary, we highlight paged as follows: datasets as shown in ## experiments we conduct systematic experiments to answer q1 and q2 raised in sec. 1. we split the dataset into train, validation and test sets with 3:1:2 ratio. for q1, we collect state-of-the-art baselines and evaluate them in paged. we further introduce three metrics based on the underlying structure of graphs and the surface form of elements. specifically, we use the bleu scores to measure the performance of actor, action and constraint extraction and f1 scores to measure the performance of gateway prediction. besides, the performance of flow prediction is measured via soft f1 scores @xcite , which is computed based on the bleu scores of associated textual elements. for q2, we involve flan-t5 (chung et al., 2022), chatgpt @xcite and llama2 @xcite in paged to show their potentials and improve their performance using a self-refine strategy. ## conclusion and future work we propose the paged benchmark, where we systematically study the progress of current procedural graph extraction methods and explore the potential of emerging llms on this task. we equip paged with a high-quality dataset, which is about ten times larger than the previous largest ones. experimental results of baselines in paged reveal that current methods are far from solving this task well. we further involve three advanced llms in paged to demonstrate their advantages in extracting textual elements and challenges in organizing local structures. to overcome the main challenge, we design a novel self-refine strategy to empower the llms' ability in reasoning gateways. the results show that, with effective strategies, llms have the potential of llms to comprehend the logical structure among non-sequential actions. we hope paged can benefit the research on optimal procedural graphs extraction. there are several directions for further work. first, to improve the performance of llms, we suggest introducing procedural knowledge during the pre-training stage of llms. second, to gain better flow extraction, we will explore more effective methods for handling complex logic structures in the documents. lastly, we plan to find a real-world scenario to investigate, besides accuracy, what else is limiting the practical usage of automatic procedure graph extraction.