D oc IE @ XLLM 25: In-Context Learning for Information Extraction using Fully Synthetic Demonstrations (2025)

Abstract: Large, high-quality annotated corpora remain scarce in document-level entity and relation extraction in zero-shot or few-shot settings.In this paper, we present a fully automatic, LLM-based pipeline for synthetic data generation and in-context learning for document-level entity and relation extraction.In contrast to existing approaches that rely on manually annotated demonstrations or direct zero-shot inference, our method combines synthetic data generation with retrieval-based in-context learning, using a reasoning-optimized language model.This allows us to build a high-quality demonstration database without manual annotation and to dynamically retrieve relevant examples at inference time.Based on our approach we produce a synthetic dataset of over 5k Wikipedia abstracts with approximately 59k entities and 30k relation triples.Finally, we evaluate in-context learning performance on the DocIE shared task, extracting entities and relations from long documents in a zero-shot setting.The code and synthetic dataset are made available for future research.

## introduction information extraction (ie) is a key task in natural language processing (nlp) research. while significant progress has been made in terms of nlp and ie benchmarks in general, few-shot and long context tasks remain relevant and challenging @xcite @xcite @xcite , even in the age of large language models (llms). in this work, we explore this direction in the context of the document-level information extraction (docie) shared task (organizers), which challenges systems to perform joint entity and relation extraction over long, unstructured documents in a zero-shot setting. specifically, we approach the topic via two recently popular approaches, llms which have been optimized for reasoning-heavy tasks @xcite , as well as synthetic data augmentation, which has become popular for ie in particular @xcite @xcite as scalable data annotation still represents a major challenge. in order to combine the two, we construct a simple, retrieval-based in-context learning setting in which an llm is tasked with extracting entities and relations from a text based on a single example demonstration retrieved based on its similarity to the given text. in order to preserve the zero-shot setting, we add the constraint that the example demonstration must not be manually annotated, but instead is a synthetically generated example. we therefore develop an approach for a synthetic data generation pipeline which produces high quality annotated examples of schema-constrained entity and relation extraction. our evaluations on the shared task, as well as the re-docred @xcite dataset show that in-context joint entity and relation extraction at the document-level remains a challenging task, even for state-of-the-art llms. our contributions in this paper are the following: ## task description the docie shared task (organizers) evaluates long form information extraction, specifically entity and relation extraction, in a zero-shot schema-constrained setting: at test time, a system is provided with only a text document and a schema consisting of strings naming the entity and relation types which are to be extracted. as is typical for few-shot and zero-shot tasks, the supplied training and development data resembles the test data only in structure. it differs in terms of schemata and text domains. ## approach for this paper, we are interested specifically in putting several key technologies to the test in a pipeline, namely retrieval-based in-context learning, synthetic data @xcite @xcite , and reasoning language models @xcite . further, we apply our pipeline to the zero-shot setting using none of the provided training data and foregoing any model fine-tuning. the core idea behind our approach is to construct a fully synthetic demonstration database, from which, given a query, we retrieve the most relevant example and provide it to a reasoning language model as an in-context example. below, we first describe the inference pipeline, as it could also be applied to manually annotated data, before describing the synthetic data generation pipeline in detail in section 4. ## synthetic data generation figure 1 provides an overview of the synthetic data generation pipeline which requires only text data as its input and produces complete, high quality annotations over a two-phase process. ## conclusion we present a fully automatic, llm-driven pipeline for high-quality synthetic data generation for document-level entity and relation extraction, and apply it to create a dataset of over 5k wikipedia abstracts with roughly 59k entities and 30k relation triples. through our two-phase annotation process, zero-shot prompt-based extraction followed by rule-and model-based verification, we demonstrate that automated checks substantially improve annotation fidelity, yielding a dataset that, exhibits consistency and wide coverage of entity and relation types. our evaluations on the docie shared task and re-docred confirm that zero-shot ie remains a hard problem: precision and recall for both entities and relations are modest, and a large number of model outputs still fail to parse cleanly. moreover, ambiguities in type definitions and relation directionality is a challenge for both extraction and evaluation. by releasing our synthetic dataset we aim to provide a valuable resource for future research on fewshot and zero-shot document-level ie. we hope that these tools will catalyze further advances in robust, scalable information extraction methods. language models. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 5495-5505, singapore. association for computational linguistics. shared task organizers. xllm acl 2025 shared task-iv: document-level information extraction. https://xllms.github.io/docie/ . nicholas popovic and michael färber. 2022. few-shot document-level relation extraction. in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5733-5746, seattle, united states. association for computational linguistics. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bertnetworks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992, hong kong, china. association for computational linguistics. steven rogulsky, nicholas popovič, and michael färber. 2024. the effects of hallucinations in synthetic training data for relation extraction. in joint proceedings of the 2nd workshop on knowledge base construction from pre-trained language models (kbc-lm 2024) and the 3rd challenge on language models for knowledge base construction (lm-kbc 2024) co-located with the 23nd international semantic web conference (iswc 2024), baltimore, usa, november 12, 2024, volume 3853 of ceur workshop proceedings. ceur-ws.org. qingyu tan, lu xu, lidong bing, hwee tou ng, and sharifah mahani aljunied. 2022. revisiting do-cred -addressing the false negative problem in relation extraction. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 8472-8487, abu dhabi, united arab emirates. association for computational linguistics. lilong xue, dan zhang, yuxiao dong, and jie tang. 2024. autore: document-level relation extraction with large language models. in proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 211-220, bangkok, thailand. association for computational linguistics. hankiz yilahun, hangtian zhao, and askar hamdulla. 2025. fredc: a few-shot relation extraction dataset for chinese. applied sciences, 15(3).