Enhancing Information Extraction with Large Language Models: A Comparison with Human Annotation and Rule-Based Methods in a Real Estate Case Study (2025)

Abstract: Information Extraction (IE) is a key task in Natural Language Processing (NLP) that transforms unstructured text into structured data. This study compares human annotation, rule-based systems, and Large Language Models (LLMs) for domain-specific IE, focusing on real estate auction documents. We assess each method in terms of accuracy, scalability, and cost-efficiency, highlighting the associated trade-offs. Our findings provide valuable insights into the effectiveness of using LLMs for the considered task and, more broadly, offer guidance on how organizations can balance automation, maintainability, and performance when selecting the most suitable IE solution.

## introduction information extraction (ie) is a fundamental task in natural language processing (nlp), enabling the transformation of unstructured text into structured data. ie involves identifying and extracting relevant information, such as entities, relationships, and events, and organizing it so that machines can process and analyze it effectively @xcite . many industries, such as finance, healthcare, and legal services, rely on ie to process large volumes of documents and extract critical information. in this context, domain-specific ie poses further challenges compared to general-purpose ie @xcite @xcite @xcite ; documents often feature specialized terminology, structured content, considerable length, and various formats that require expert knowledge for accurate processing. human annotation, while serving as the gold standard for accuracy, is costly, time-consuming, and prone to inconsistencies, especially with lengthy or complex documents. traditional rule-based systems have been widely adopted in such settings for their precision, deterministic behavior, and interpretability. however, they require significant effort to develop and maintain, particularly when adapting to new document types. the adoption of large language models (llms) and prompt engineering can offer a flexible alternative for reducing reliance on manually crafted rules. however, despite advancements in ai-driven methods, the transition to ai continues to present significant challenges. widespread adoption remains hindered by several factors, including high implementation costs, integration complexities, data privacy concerns, and a lack of expertise @xcite @xcite . in business environments, where performance, cost, and maintainability must be balanced, careful evaluation of ie approaches is key to selecting effective solutions. to address this challenge and examine key tradeoffs, we conduct an empirical comparison of human annotation, rule-based extraction, and llmbased extraction for domain-specific ie tasks. we focus on the real estate domain in the italian context, using auction documents to extract structured information-such as cadastral data and asset descriptions-crucial for business operations. we develop rule-based and llm-driven models and evaluate their performance against a human-annotated ground truth. this study systematically compares the three approaches in terms of accuracy, scalability, and cost-efficiency, providing insights that can inform ie adoption across various industries. our findings aim to assist organizations in balancing automation, maintainability, and performance when selecting the best ie solution for their needs. ## related work this section reviews the aforementioned ie strategies-human annotation, rule-based, and llmbased methods-highlighting their applications, strengths, limitations, and key trade-offs across distinct domains. several studies have examined the behavior and performance of human annotators. @xcite examine the effects of self-review and peerreview processes among annotators in the realestate domain, highlighting issues related to interannotator agreement and uncertainty. @xcite focus on the pharmaceutical domain and find that crowdsourcing annotators can provide a reliable approximation of expert annotations. similarly, @xcite introduce methods to enhance crowdsourced annotation-such as gamification-achieving expert-level accuracy in the medical domain. these studies underscore challenges such as subjectivity and the need for domain expertise, both of which are crucial for reliable annotation. they suggest that non-expert annotators can often approximate expert performance. however, they do not specifically evaluate annotator performance on tasks involving long documents, where the volume of text may significantly affect annotation quality. rule-based systems represent one of the earliest approaches to ie, relying on handcrafted patterns and domain-specific rules. these systems typically employ regular expressions, as in the seminal work by @xcite , and in some cases support context-free constructs @xcite . they are also frequently implemented as frameworks @xcite @xcite @xcite . rules are effective due to their transparency and the lack of need for training data. however, they struggle to generalize to minor input variations and are sensitive to noise and linguistic diversity @xcite . while rule-based systems can initially boost precision and recall, capturing all linguistic nuances requires excessive manual effort as input complexity increases @xcite . although the rise of machine learning techniques has largely overshadowed rule-based approaches, they remained widely used in industry until recently @xcite and are still employed in academic research today-particularly in the early stages of rapid prototyping @xcite . since the introduction of large language models, many studies on ie have emerged, generally falling into two categories: training-based and training-free. the former involves adapting llms to specific tasks by fine-tuning their parameters us-ing domain-specific labeled datasets. this process enhances the model's ability to accurately identify and extract structured information from unstructured text. for example, deepstruct introduced structural pre-training on task-agnostic corpora to improve llms' structural understanding @xcite . similarly, giellm fine-tuned llms on mixed datasets to exploit mutual reinforcement effects, enhancing performance across multiple tasks @xcite . in contrast, training-free approaches rely on prompt engineering-a technique that guides llm behavior using task-specific prompts, without modifying model parameters. for instance, @xcite highlight the gap between instruction-tuned llms and the structured output requirements of ie. qa-style prompting helps bridge this gap. other methods, such as promptner @xcite , guide llms to generate explanations for entity extraction, while proggen @xcite promotes self-reflection to improve output quality. llm-based techniques have been widely applied across domains such as legal @xcite @xcite , medical @xcite , and fintech @xcite . recently, studies have compared llm-based with rule-based systems @xcite and human annotators @xcite . however, no study has so far comparatively evaluated the three strategies together. additionally, these works focus on performance comparisons, overlooking cost analysis, development efforts, and the length of documents. ## the real estate domain this section provides an overview of the domain we focus on in this study, namely the real estate sector. in particular, the term "real estate" refers to the ownership, management, and trade of properties, including land, buildings, and other structures. in this domain, data primarily consists of information about properties, such as their location, dimensions, ownership details, market value, and legal status. within the italian cadastral system, properties are uniquely identified through cadastral coordinates organized in a hierarchical structure. these include foglio (sheet), particella (parcel), and, where applicable, subalterno (subunit), which together provide a standardized reference for each property. in the real estate market, properties are sold privately, through agencies, or at auctions. this study focuses on real estate auctions, which play a key role in judicial sales and debt recovery, with the goal of extracting structured data from auction documents. ## information extraction solutions in this section, we detail the human annotation process, the rule-based methods, and the llm-based approaches we implemented for the comparative evaluation of the three extraction techniques. ## experimental evaluation this section presents the experimental evaluations conducted to comparatively assess the three proposed ie solutions. first, we provide a detailed you are an information extraction engine. analyze the document and extract the information according to the instructions provided below, following the format indicated in the example. in your response, skip the preamble and provide exclusively the "properties" json list that is being requested as present in input 2. lands and buildings are identified by coordinates in the following hierarchical order: sheet, parcel, and sub. ⟨/fields definitions⟩ ⟨example⟩ input 1: here an example document is provided. response for input 1: here the expected json is provided. ⟨/example⟩ input 2: here we provide the document to be analyzed. response for input 2: description of the construction of the ground truth (gt) and the dataset used for information extraction. next, we outline the llms considered in this work, along with the evaluation metrics employed. finally, we present the results and discuss their implications for the proposed solutions. ## discussion in this section, we summarize the key insights derived from the experiments and the comparative evaluations conducted on the various ie solutions. considering the two key factors of performance and price, we conclude the following: (i) the rule-based model and claude 3 haiku offer the best trade-offs; (ii) the rulebased model is the cheapest overall, while claude 3 haiku delivers the highest performance; (iii) llama 3.1 70b instruct is notable for its strong performance and open-source nature, allowing for on-premise use and fine-tuning to potentially match the other models' performance while reducing costs; (iv) human annotation is neither competitive in terms of performance nor cost. however, when considering the development, maintenance costs, and required skills, llms emerge as the more cost-effective option compared to rule-based approaches. additionally, the ease of performance improvements with newer llm versions, coupled with the rapid advancements in generative models and decreasing costs, makes llms the optimal solution for this case study. ## conclusions and future work in this study, we compared a rule-based system with llm-based approaches for information extraction (ie) in the real estate domain, along with human annotation performance. our findings show that the best-performing llm outperformed both human annotators and the rule-based system in terms of overall performance, particularly in precision and recall. however, the rule-based approach remains a strong contender due to its reliable and consistent performance, largely stemming from the considerable time and effort invested in its development. llms, on the other hand, offer a faster and more scalable development process. with llms, the transition from error analysis to performance improvements is more efficient, and they do not require specialized skills such as knowledge of rules, grammars, or programming. this makes them a more accessible and cost-effective solution compared to rule-based systems. furthermore, the continuous advancements in llms mean that their performance improves over time, often without the need for prompt modifications, making them a sustainable option for long-term applications. human annotation, while flexible and adaptable, was found to be the least effective and most costly approach. despite its high flexibility in handling diverse inputs, human annotation yielded unsatisfactory results compared to automated approaches and proved to be less cost-efficient. however, this may also be due to a suboptimal choice of evaluators or the human evaluation strategy adopted. hence, future work could benefit from a more granular error analysis to identify specific challenges each model faces when extracting particular fields, thereby guiding targeted improvements. to enhance the robustness of human annotations, future studies could incorporate overlapping document sets among annotators to enable the calculation of inter-annotator agreement metrics. this should be complemented by a pilot phase on a small subset of data, followed by thorough error analysis and refinement of annotation guidelines to improve consistency and quality throughout the annotation process. the methodology and findings presented here could be extended to a wider range of ie tasks and domains, including documents of varying lengths, to better understand the effects of document length and potential annotator fatigue. it is worth noting that while llms continue to advance, human input remains valuable-especially when models exhibit uncertainty @xcite . additionally, fine-tuning open-source models like llama 3.1 70b instruct offers a promising and cost-efficient avenue for future research, with the potential to effectively balance performance and scalability.