Challenges in End-to-End Policy Extraction from Climate Action Plans (2024)

Abstract: Gray policy literature such as climate action plans (CAPs) provide an information-rich resource with potential to inform analysis and decision-making. However, these corpora are currently underutilized due to the substantial manual effort and expertise required to sift through long and detailed documents. Automatically structuring relevant information using information extraction (IE) would be useful for assisting policy scientists in synthesizing vast gray policy corpora to identify relevant entities, concepts and themes. LLMs have demonstrated strong performance on IE tasks in the few-shot setting, but it is unclear whether these gains transfer to gray policy literature which differs significantly to traditional benchmark datasets in several aspects, such as format of information content, length of documents, and inconsistency of document structure. We perform a case study on end-to-end IE with California CAPs, inspecting the performance of state-of-the-art tools for: (1) extracting content from CAPs into structured markup segments; (2) few-shot IE with LLMs; and (3) the utility of extracted entities for downstream analyses. We identify challenges at several points of the end-to-end IE pipeline for CAPs, and we provide recommendations for open problems centered around representing rich non-textual elements, document structure, flexible annotation schemes, and global information. Tackling these challenges would make it possible to realize the potential of LLMs for IE with gray policy literature.

## introduction gray policy literature -non-commercial and nonacademic documents which can include white papers, technical reports, and working papers -is an information-rich resource that is generally difficult to navigate due to the volume and diversity of format @xcite @xcite . paid for by public funds, these documents are usually freely available and often the most timely resource on policy issues @xcite . @xcite found that half of surveyed policymakers would be more likely to use gray policy literature if information were easier to find and access. information extraction (ie) tasks in the nlp space are designed to make it possible to efficiently sift through such information, but gray policy literature poses several challenges for traditional ie. they are distributed as long pdfs with inconsistent document structure, such that relevant sections cannot be easily automatically extracted. they are designed to be visually appealing with crucial information organized in rich non-textual elements such as tables and graphics @xcite . in contrast, the standard datasets that ie is designed to perform well on take the form of short, plain text documents from domains such as webtext or news articles @xcite . in general, ie models that are trained on these standard datasets can be adapted to new domains by finetuning with annotated examples, but this may not be feasible for gray policy documents @xcite . collecting a large number of manually annotated examples for a static set of entities can be prohibitively expensive with gray policy literature due to fast-paced and diverse developments in the field. over standard datasets, large language models (llms) have demonstrated strong performance in entity and relation extraction in the few-shot setting @xcite @xcite . but, it is unclear to what extent llms can be used to extract information from gray policy literature in the few-shot setting. in this work, we present a case study of few-shot ie with llms over climate policy text. we specifically consider climate action plans (caps) from the state of california. caps are distributed as long pdfs, sharing many of the challenging properties of gray policy figure 1 : given a climate action plan in pdf format, the end-to-end ie pipeline includes first extracting the content from pdfs into structured markup segments using extracted headers or the table of contents ( §6.2). then, the segments are filtered for policy relevance ( §6.3). for each structured markup segment, we can perform few-shot re with an llm to extract policies (e.g. "require new homes to install solar photovoltaic systems") and referring policy attributes (e.g. responsible entity, emissions reduction) ( §6.4). we evaluate few-shot ie performance of llms for an end-to-end setup, where the input is a cap in pdf form, and the output is a set of policy entities and relations (figure 1 ). we inspect to what extent existing pdf parsers are able to preserve crucial policy information (raw text recognition), recognize textual and non-textual elements (element recognition), and extract a document structure that would be useful to a domain expert to segment the document (structured segmentation). then, using the parsed segments of the cap, we analyze how well entities can be extracted from informationdense non-textual elements (intra-element extraction), and more generally the relation extraction performance of llms in a few-shot setting (segmentlevel extraction). finally, we experiment with modes of useful representations of the extracted entities and relations (extraction utility). ## climate policy extraction caps generally contain sections describing proposed policies and an inventory of emissions produced by the jurisdiction. climate policies can be classified as adaptation to climate changes or emissions reduction measures. an example cap for the city of encinitas, california can be found here. california municipal and county caps exhibit significant variability in structure but typically contain a similar set of sections. these include front matter, a discussion of the regulatory framework, a description of local and regional climate projections, and, critically for our purposes, chapters on specific and detailed mitigation policies, associated emissions reductions, and sometimes also chapters on implementation and on adaptation policies. climate policies can be broadly categorized as either mitigation policies which reduce greenhouse gas emissions or adaptation policies which build community resilience to the impacts of climate change. the average length of a cap is approximately 62k tokens. 1 caps are distributed by jurisdictions as pdfs. there are at least three user groups could derive significant value from extracting policies from caps. local sustainability officers seek to understand the policies of similar jurisdictions and how their own policies compare. state agencies wish to monitor the progress of cap implementation and to aggregate policy commitments to the state and federal levels. academic researchers seek to understand cap characteristics, credibility, evolution over time, and effectiveness in shaping policy. to date, efforts to systematically extract policies have been expensive and have progressed sporadically @xcite 3 task definition given a cap containing policy information, we want to perform relation extraction, where the entities are policy names p 1 , . . . , p j and a closed set of policy attributes a 1 , . . . , a i . we consider a single relation type of reference between a policy p k and an attribute, indicating that the attribute describes the policy p k . each attribute can refer to at most one policy. for each policy p k , we expect that the referential attributes a 1 , . . . , a i construct a sufficiently informative representation to perform some downstream analysis. ## method given a set of caps in the form of pdf documents, we first parse the pdfs into a structured markup format (e.g. html, markdown, json) using a pdf parser. the resulting file would contain various elements such as headers, paragraphs, tables, and lists. as a part of the raw file content, the parser produces some document structure to mark sections and sub-sections or in some cases the table of contents. we can formalize the document structure as a graph of section headers and segments containing the section content, where headers are linked to corresponding segments, and headers can be subsumed by other headers to reflect the hierarchy induced by the pdf parser. since the average cap in our dataset is 170 pages, it is necessary to divide caps into coherent, 1 as tokenized by the gpt-3.5-turbo-0125 model. topically focused segments. given a fixed level in the document structure hierarchy, the cap can be segmented according to the structure. for example, a cap can be broken down into a series of segments, each corresponding to the content of a sub-section. this results in a corpus of structured markup segments to perform relation extraction over. we perform zero and few-shot relation extraction by prompting autoregressive llms. we draw in-context examples from the set of annotated segments. we use a two-step entity-extraction procedure to extract relations, where the second prompt is dependent on the model output of the first. for each segment, we first prompt the llm to produce the set of policy names that appear in the segment. then, for each policy name p k we prompt the llm to produce the attributes a 1 , . . . a l referring to the policy name. the relation extraction task is linearized as is conventional with ie using llms @xcite . extracted entities are expressed as a json mapping between entity types and mention spans. spans that do not appear in the segment are discarded to reduce the effect of model hallucination. ## related work in contrast to the understudied climate policy domain, there have been extensive studies about the performance of pdf parsing tools for documents with complex layouts over scientific domains @xcite @xcite . pdf parsers generally perform well on scientific text @xcite , but isolating performance on non-textual elements @xcite show that pdf parsers struggle to extract tables more than all other content elements. @xcite has found that processing tables as images using multi-modal llms can be more effective than parsing the tables into text. for few-shot ie using llms, there have been mixed results over biomedical and clinical text domains @xcite @xcite . there has been some limited work to perform fewshot ie in the climate policy domain: @xcite extract features of wind energy systems from pdf ordinances using llms. in our work, we systematically evaluate the end-to-end ie pipeline at both the extraction and pdf parsing stage for a more general climate policy taxonomy. a key limitation in using llms for ie over long, complex documents is limited context length. while there has been work to build models that can process long contexts @xcite , memory and attention constraints result in parts of the context being ignored. with scientific documents, @xcite found that failures occurred when the number of tokens exceeded the model context window. accordingly, we focus on making use of cap structure to produce segments that fit within the context window of llms. historically, nlp tools have been used for climate policy text to identify salient topics using clustering or topic modeling @xcite . there has been some work to classify policy type or targets (e.g. pledge net-zero vs. emissions reduction) @xcite @xcite . more recently, there have been multiple largescale initiatives to extract structured representations of policies. @xcite spent over 600 hours to annotate 412 documents with 42 policy instrument and design types. similarly, berrang-ford et al. (2021) have annotated climate hazards and adaptation efforts in 1,682 articles with the assistance of 126 researchers. accordingly, in our work, we study the capacity of llms to assist and reduce the effort required to collect and maintain information resources about the state of climate policy. ## experiments and analysis to perform an evaluation of state-of-the-art endto-end ie, we annotate the documents at multiple levels of granularity: raw pdfs, structured markup caps, filtered cap segments, and elements of each segment. ## downstream extraction utility a question of interest to policy researchers is how cap focus has shifted from mitigation to adaptation as faith in global mitigation efforts has declined (hoesung lee and josé romero (eds.), 2023). we classify a policy as an adaptation policy using a heuristic string match over the referring attributes with the regular expression "heat|precip|flood|fire|sea level". a keyword search with this regular expression over the entire document could extract mentions of hazards in introductory sections that are not associated with concrete policies. using a gpt-3.5-turbo model, we extracted 47,006 climate policies from 227 jurisdictions. 4.6 percent of the extracted policies mention the five hazards in their "climate-hazard" attribute. the low percentage is expected as caps have traditionally focused on greenhouse gas emission reductions rather than adaptation. linking the extracted set of policies to county and municipal characteristics reveals spatial variability in the fraction of adaptation policies (figure 3 ) with a slight indication that coastal jurisdictions are more focused on adaptation. the share of adaptation policies has increased over the past 16 years. there is a significant positive linear relationship between the fraction of adaptation policies and the logarithm of community per capita income. this may indicate that wealthier communities show more interest in safeguarding their assets, though further investigation is required to rule out potential confounding factors, such as distance to the coast. ## recommendations to improve end-to-end ie over gray policy literature, we propose several directions for future work based on our analysis of california caps. there is a need for more flexible annotation schemes for inconsistently formatted documents. the same entities will frequently appear at different levels of abstraction or detail between documents and within a document. we need mechanisms to specify which mentions to extract. this is distinct from fine-grained entity typing annotation where there are no vertical coreferences between types. ie systems need better representation of nontextual elements such as tables and rich graphics, and for semantic representations of the text to be built from information derived from all modalities. in a text-only modality, it may be useful to build representations that localize information content according to reading order. vague, imprecise, and subjective entity types are crucial for gray policy literature, and current methods to refine the set of extracted entities such as providing icl examples or detailed instructions may be insufficient. in the case of caps, interpretation of policy mentions often require global document information or domain expertise. ie systems should also be designed to leverage rich document structure. in the case of caps, for example, we observed that that the table of contents in isolation is informative enough for annotators to infer which sections are relevant.