Standardizing Genomic Reports: A Dataset, A Standardized Format, and A Prompt-Based Technique for Structured Data Extraction (2024)

Abstract: Extracting information from genomic reports of cancer patients is crucial for both healthcare professionals and cancer research. While Large Language Models (LLMs) have shown promise in extracting information, their potential for handling genomic reports remains unexplored. These reports are complex, multi-page documents that feature a variety of visually rich, structured layouts and contain many domain-specific terms. Two primary challenges complicate the process: (i) extracting data from PDFs with intricate layouts and domain-specific terminology and (ii) dealing with variations in report layouts from different laboratories, making extraction layout-dependent and posing challenges for subsequent data processing. To tackle these issues, we propose GR-PROMPT, a prompt-based technique, and GR-FORMAT, a standardized format. Together, these two convert a genomic report in PDF format into GR-FORMAT as a JSON file using a multimodal LLM. To address the lack of available datasets for this task, we introduce GR-DATASET, a synthetic collection of 100 cancer genomic reports in PDF format. Each report is accompanied by key-value information presented in a layout-specific format, as well as structured key-value information in GR-FORMAT. This is the first dataset in this domain to promote further research for the task. We performed our experiment on this dataset.

## introduction genomic reports provide detailed insights into oncology patients' biomarkers, including specific genetic mutations and associated therapies. these reports primarily assist healthcare professionals in devising treatment strategies for cancer patients. the increasing workload on healthcare professionals heightens the risk of human error, further driving the demand for ai-driven assistance to help them make faster and more accurate treatment decisions. additionally, the information gathered from these reports can contribute to future cancer research. this underscores the need for efficient and accurate systems to extract and interpret data from genomic reports. the problem addressed in this study is the development of an automated system capable of extracting and interpreting key information from genomic reports. input: the pdf file of the genomic report for oncology patients. output: the json file containing information of the report in gr-format. one of the primary challenges in processing genomic reports lies in their inherent complexity. while these reports contain structured data, complexity often arises from the distribution of information across multiple pages, where key data points may be interrupted by page breaks or dispersed throughout non-adjacent sections. this design is presumably intentional, as the layouts of genomic reports in certain laboratories prioritize critical information for human healthcare professionals. by placing the most pertinent data on the first page, the reports aim to facilitate quick access and reduce the time spent searching for essential information. however, this emphasis on accessibility leads to a more intricate pdf layout, complicating the data extraction process. however, for machine understanding, all pages hold equal visibility, and machines prefer data to be organized consistently for effective processing. additionally, using visual elements and domain-specific entities (highly specialized contents that need a better contextual understanding of the domain) adds to this complexity. another challenge is the diversity of report layouts used by different laboratories, as there is no standardized format for these reports. this variability complicates automated information extraction, as subsequent steps should not be biased toward specific laboratory formats. to address these challenges, we propose gr-format, a standardized json format that can be used to convert any genomic report, making it layout-agnostic, regardless of the original layout. we developed this format by analyzing genomic reports with varying layouts. additionally, we propose gr-prompt, a prompt-based technique to extract information from genomic report pdfs in gr-format. gr-prompt has a set of prompt which utilizes the gpt-4 @xcite , a multi-modal closed-source llm to do the task. there is no publicly available dataset for this task, as publishing genomic reports contains privacy and consent issues. in response, we introduce the gr-dataset, a synthetic dataset consisting of 100 cancer genomic reports, key-value information presented in a layout-specific format, as well as structured key-value information in gr-format. this research presents three key contributions: ## related work a related work that processes genomic reports' data is genetex @xcite . it is a tool that converts semi-structured data to structured data using text-mining and regular expressions. however, to the best of our knowledge no work extracts key information from a genomic report and outputs it in a predefined format. however, they did not release the dataset publicly to further facilitate research in research on genomic report information extraction. some works have been done to extract structured information from text using llm @xcite . @xcite highlights the ability of fine-tuned llms, like gpt-3 and llama-2, to extract intricate relationships from scientific literature, enabling flexible output formats for database creation. @xcite introduces a novel entity-centric approach called structured entity extraction (see), which utilizes the approximate entity set over-lap (aesop) metric for performance evaluation and demonstrates enhanced extraction efficiency through a multi-stage model, musee. another direction of work that closely resembles our work is extracting information from pdfs that have complex layouts. donut @xcite is a model designed for document image understanding that leverages a unified architecture to process both text and layout information effectively, enabling it to handle diverse document types without the need for explicit layout annotations. docowl 1.5 @xcite is a versatile tool that combines advanced parsing capabilities with intelligent chunking strategies, ensuring that hierarchical relationships within documents are preserved during extraction, thus enhancing the accuracy and usability of the extracted data. layoutlmv3 @xcite enhances document layout comprehen-sion by integrating unified text and image masking techniques, allowing it to excel in tasks like form understanding and visual question answering. lay-outllms @xcite build their systems by focusing on leveraging large language models for improved contextual understanding of document structures, making them adept at extracting relevant information from intricate layouts. a recent work @xcite emphasizes that while llms are powerful tools, their performance can be significantly affected by the constraints imposed on their output formats, necessitating careful consideration in practical applications. ## methodology in this section, we describe gr-format (the standardised format), gr-prompt (the prompt-based approach), and gr-dataset (the dataset) in detail. ## experimental setup in this section, we provide the experimental details for applying gr-prompt to extract information from genomic reports in gr-format on the gr-dataset. additionally, we describe the process of generating synthetic cancer genomic reports and outline the evaluation strategy used to assess the quality of the output when applying gr-prompt to extract information from genomic reports in gr-format on the gr-dataset. ## results layout-1 layout-2 accuracy 77.62% 68.76% ## summary, conclusion, and future work in this work, we introduced gr-prompt, a novel technique for extracting information from complex cancer genomic reports, and gr-format, a standardized data format, using a multimodal llm to handle intricate layouts and domain-specific terminology. we also introduced gr-dataset, the first synthetic dataset for this task, facilitating future research. while our approach demonstrates a promising trajectory, future work will focus on expanding the dataset to improve the model's performance and adaptability across diverse healthcare scenarios. experiments with open-source llms, such as mistral @xcite and llama @xcite , will be conducted to assess their suitability for on-premise deployment, ensuring compliance with privacy and regulatory requirements. collaboration with healthcare professionals will be prioritized to enhance system credibility, refine domainspecific applicability, and improve dataset quality. additionally, a comprehensive evaluation strategy will be developed to assess fact-correctness and format alignment more effectively. limitations 1. it relies on api-based access to gpt-4, which can be expensive and may introduce latency, limiting scalability in real-time clinical use. additionally, our evaluation is based on a synthetic dataset, which may not fully reflect the diversity of real-world genomic reports, potentially affecting generalizability. the system also assumes that all relevant information is within the report images and may struggle with highly specialized content. finally, relying on ai for critical medical decisions raises ethical concerns, as errors could impact patient care.