A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction (2024)

Abstract: Large language models (LLMs) have demonstrated impressive abilities in generating unstructured natural language according to instructions. However, their performance can be inconsistent when tasked with producing text that adheres to specific structured formats, which is crucial in applications like named entity recognition (NER) or relation extraction (RE). To address this issue, this paper introduces an efficient method, G&O, to enhance their structured text generation capabilities. It breaks the generation into a two-step pipeline: initially, LLMs generate answers in natural language as intermediate responses. Subsequently, LLMs are asked to organize the output into the desired structure, using the intermediate responses as context. G&O effectively separates the generation of content from the structuring process, reducing the pressure of completing two orthogonal tasks simultaneously. Tested on zero-shot NER and RE, the results indicate a significant improvement in LLM performance with minimal additional efforts. This straightforward and adaptable prompting technique can also be combined with other strategies, like self-consistency, to further elevate LLM capabilities in various structured text generation tasks.

## introduction information extraction (ie) is a critical task that involves retrieving specific information, such as named entities and relationships, from unstructured or semi-structured texts, and converting this information into a structured format @xcite . traditionally, ie models have relied heavily on fully supervised learning, necessitating extensive labeled datasets for training. this approach not only demands significant human effort but also restricts the scope of extractable information to a limited set of predefined types, such as "person" for named entity recognition (ner) and "born in" for relation extraction (re). this limitation is particularly prominent in specialized fields like materials science, where resources are scarce. earlier attempts to mitigate these challenges, such as weak supervision @xcite @xcite , have introduced methods that utilize noisy heuristic labeling functions (lfs) to reduce the reliance on manually labeled data. however, the effectiveness of these methods often hinges on the quality of the lfs, which is not always consistent. the advent of large language models (llms) like gpts @xcite has promoted an attention shift towards universal ie approaches. these methods aim to extract a wide range of information without the need for task-specific labels. strategies include directly prompting llms with instructions for specific tasks @xcite @xcite or fine-tuning them on either true labels or pseudo labels generated by gpts @xcite @xcite @xcite . nonetheless, the inherent mismatch between the unstructured data language models are typically trained on and the structured output requirement presents a challenge. previous studies have employed specialized prompts to guide the model in generating structured outputs, such as lists @xcite or tables @xcite . however, integrating task instructions with these organizational prompts has sometimes resulted in formatting issues or compromised ie performance. in response to these challenges, this paper introduces a simple but effective methodology, generate and organize (g&o), designed to enhance the capability of llms in performing structured zero-shot ie tasks, with a focus on ner and re. our approach divides the generation into two distinct components: 1) generating ie responses in a free-form natural language format; followed by 2) structuring these responses into a predefined for- ## related works in the landscape of supervised neural networks, researchers are actively seeking methods to reduce reliance on labeled data for information extraction (ie) tasks, acknowledging the effort and limitations associated with manually labeled data. a prominent approach includes the development of weak and distant supervision techniques @xcite @xcite @xcite @xcite @xcite . these methods aim to lessen the annotation workload through the use of heuristic labeling functions (lfs). these functions, whether singular @xcite @xcite or multiple @xcite @xcite @xcite , are designed to generate noisy labels for unlabeled data. subsequently, models are trained to refine and amalgamate these labels for improved prediction accuracy. however, some critics argue that the efficacy of single-lf methods is highly dependent on the quality of the clean validation set @xcite , and the creation of multiple lfs can be a labor-intensive process @xcite . another research trajectory involves few-shot and zero-shot learning techniques @xcite @xcite @xcite . these methods are directed towards adapting ie models to new domains using minimal labeled examples. in line with the rapid advancements in large language models (llms), some studies have explored directly prompting these models for open-type ie tasks @xcite @xcite . additionally, there is an emerging focus on fine-tuning generative llms to better align with specific prompts or task formats @xcite @xcite . nonetheless, these studies have overlooked the issue of llms' suboptimal performance in structured prediction tasks when using mixed prompts, which is the central topic of our research. ## generate and organize to enhance the capability of llms on zero-shot ie tasks that necessitate structured outputs, our prompting pipeline integrates three key components, as depicted in figure 1 : 1) free-form response generation, which prompts llms to identify the required information from the provided ## conclusion in this paper, we propose a simple yet effective approach-g&o-to improve structured prediction from llms for ie tasks. different from conventional prompting approaches, g&o separates the identification and formatting steps into two stages, which allows the model to focus on each step independently and facilitates the generation of organized results. tested under the zero-shot ie settings with gpt 3.5, this simple adjustment brings significant performance gains, demonstrating the effectiveness of g&o. the improvement is further validated by ablation studies and the generalizability of g&o to other llms, which can be further improved by resolving prediction conflicts using both prompting and fine-tuning methods. we hope our work can bring insights and inspirations for further research on structured prediction from llms and contribute to the development of more effective and interpretable ie systems.