GUIDEX : Guided Synthetic Data Generation for Zero-Shot Information Extraction (2025)

Abstract: Information Extraction (IE) systems are traditionally domain-specific, requiring costlyadaptation that involves expert schema design,data annotation, and model training. WhileLarge Language Models have shown promisein zero-shot IE, performance degrades significantly in unseen domains where label definitions differ. This paper introduces GUIDEX,a novel method that automatically definesdomain-specific schemas, infers guidelines,and generates synthetically labeled instances,allowing for better out-of-domain generalization. Fine-tuning Llama 3.1 with GUIDEXsets a new state-of-the-art across seven zeroshot Named Entity Recognition benchmarks.Models trained with GUIDEX gain up to 7 F1points over previous methods without humanlabeled data, and nearly 2 F1 points higherwhen combined with it. Models trained onGUIDEX demonstrate enhanced comprehension of complex, domain-specific annotationschemas. Code, models, and synthetic datasetsare available at neilus03.github.io/guidex.com

## introduction information extraction (ie) tasks @xcite are structured around two core components: a formal schema specifying target entities/relations and human-readable guidelines defining their interpretation. despite their utility, ie systems face significant scalability challenges due to domain dependence. adapting to new domains requires substantial resources, including (1) domain experts to design schemas and annotation rules, (2) trained annotators to label data accordingly, and (3) machine learning specialists to develop performant models. this complex process creates a bottleneck for real-world applications where label definitions frequently evolve across contexts. early attempts to remove the need for annotated data (zero-shot ie) framed the task through question answering @xcite or natural language inference @xcite paradigms, leveraging supervised data from these auxiliary tasks. while showing initial promise, these methods remained constrained by their reliance on manually crafted schemas and limited cross-domain generalization. more recent advancements leveraging large language models (llms) have streamlined schema definition processes @xcite , facilitating adaptation to new domains. however, performance gaps persist when deploying these systems in unseen domains where label semantics diverge from training distributions @xcite . our work addresses this challenge by automating the generation of domain-specific schemas, guidelines, and annotated data to bridge the seen-unseen domain divide. data augmentation @xcite and synthetic data generation @xcite have proven particularly valuable in the era of llms. in ie, traditional techniques like distant supervision @xcite aim to enhance model performance but often introduce noisy annotations. while effective for achieving high recall, distant supervision frequently generates spurious labels due to its reliance on imperfect heuristics. similarly, llm distillation @xcite -where smaller student models learn from larger teacher models-faces inherent limitations: student performance is constrained by the teacher's capabilities and often fails to capture all valid instances. both methods also require predefined annotation schemas, severely limiting their adaptability to novel domains where label definitions may shift. in this paper, we tackle the aforementioned limitations by introducing guidex, a novel data generation method inspired by the work of domain experts. it is designed to generate schemas, guidelines and annotated examples for any new domain which allows to improve ie performance on new as well as on unseen domains. this approach consists of four main steps. given a set of documents from the target domain, an llm is used to identify the key information within each document, summarizing and synthesizing its content into a set of bulleted ideas. next, the extracted information is structured into a standardized format, typically a json file. then, the model is asked to generate the annotation schema and the corresponding annotation guidelines based on the previously structured annotations. this approach ensures that the annotations align with the guidelines and remain comprehensive. this step is particularly crucial, as it guarantees the correctness of the schema and significantly reduces potential annotation errors in the generated data. finally, we ask the model to generate the final annotations following a standard code-style format. we validated our data generation approach by training state-of-the-art models using the synthetic data produced through our methodology. when fine-tuning base models, such as llama 3.1 @xcite , exclusively on our dataset, we observed an average improvement of 10 f1 points across seven named entity recognition (ner) benchmarks in zero-shot evaluation. furthermore, when leveraging our data to enhance state-of-the-art approaches, we achieved a notable improvement of nearly 2 f1 points on the same benchmarks (see figure 1 ), establishing a new stateof-the-art. ## related work in this section we review the literature related to our work. we begin by highlighting the most significant studies that involve llms for ie. following that, we will focus on methods for generating synthetic data aimed at the same task. ## guidex: guided synthetic data generation in this section, we introduce guidex, a structured synthetic data generation approach designed to enhance ie capabilities of llms. unlike traditional llm distillation approaches or distant supervision, which often rely on predefined annotation schemas, guidex dynamically infers annotation schemas. this approach reduces annotation inconsistencies, enhances flexibility across different ie tasks, and enables high-quality, guideline-driven annotations. as shown in figure 2 , the guidex approach consists of four sequential steps: document summarization, structured representation synthesis, annotation guideline generation, and instance extraction. prior work has shown that llms struggle with information extraction in zero-shot settings, and attempting to perform the entire task in a single step leads to poor results. instead, our structured process progressively refines raw text through these four stages, ensuring high-quality, structured annotations that align with inferred guidelines. complete prompt templates for the four stages are listed in figure 3 in the appendix so that the readers can reproduce every step. below, we describe each step in detail. ## the guidex dataset this section presents the dataset constructed using the proposed method. we begin by detailing the document collection process that served as the foundation for our dataset. then, we provide statistical insights to illustrate its composition and characteristics. ## experimental setting in this section we outline the details of our experiments. we begin by introducing the models used for both synthetic dataset generation and finetuning on manually annotated (gold) data. next, we describe the ie datasets utilized for training and evaluation. finally, we present the baselines and state-of-the-art systems used for comparison. ## results in this section, we discuss our experimental findings by examining guidex's impact on performance and comparing our approach against the state-of-the-art. impact of synthetic data. the third row illustrates the impact of manually annotated data (gold ft ). as anticipated, these sentence-level annotations substantially boost results. unlike our guidex dataset-largely based on documents-both the training and evaluation sets in gold ft focus on sentence-level ie tasks. this alignment phase allows the model to tackle sentence-specific zero-shot tasks more accurately. even so, the improvements we achieve with gold ft are complementary to those achieved with guidex ft . lastly, the bottom row presents a model fine-tuned first with guidex ft and then with gold ft . it can be seen that both steps jointly increase performance by an average of 34 f1 points over the plain llama 3.1 baseline (27 points over guidex ft alone and 1.4 over gold ft ). moreover, applying guidex ft to a model already trained with gold ft improves on five out of the seven datasets. this indicates that the guidex ft data can significantly enhance the model's performance in various specific domains. we analyze the impact on a labelby-label basis in section 7. ## analysis this section examines the impact of guidex across different labels, highlighting its benefits and limitations. do guidelines improve domain-specific labels? guidex effectively mitigates the overgeneralization tendency in zero-shot ie @xcite by teaching models to differentiate between broad and fine-grained entity labels through structured annotation schemas. remaining challenges. some labels, such as other and miscellaneous, remain problematic even with guidex (see ## conclusions in this paper, we introduce guidex, a novel approach for synthetic data generation aimed at ie. we utilize guidex to generate data suitable for a variety of domains using documents from finewebedu. as a demonstration of the method, we generate a dataset with 10,000 annotated documents, featuring a wide range of labels, from generic to highly domain-specific. using this generated data, we train an ie model that surpasses the performance of current state-of-the-art zero-shot ner systems. furthermore, we demonstrate that our method effectively generates domain-specific annotations, which can be utilized to train robust ie systems across multiple domains. guidex paves the way for two key research directions: (1) advancing document-level ie methodologies, and (2) developing automated techniques for handling ill-defined or generic labels like other and miscellaneous. these challenges are critical for ie applications but remain largely unexplored, offering potential for future work. our method provides a foundation to address these open problems dataset label summarized guideline goldft guidex + goldft natural science scientist a person who is studying or has expert knowledge of a natural science field. 38.43 51.21 person individuals that are not scientist. 48.53 53.02 politics politician a person who is actively engaged in politics, holding a public office, involved in political activities or part of a political party. 35.12 44.37 person individuals that are not politician. 59.48 62.86 politics politicalparty an organization that coordinates candidates to compete in a particular country's elections. 58.55 65.30 organization a structured group, institution, company, or association that is not a political party. 54.34 58.38 ai location a specific geographical or structural location. 75.36 75.86 music country a sovereign nation. 81.62 82.45 music other named entities that are not included in any other category. 15.99 13.02 literature other named entities that are not included in any other category. 23.19 20.78