P atent E val: Understanding Errors in Patent Generation (2024)

Abstract: In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.

## introduction a patent is a legal instrument that grants inventors or entities exclusive rights over their invention for a designated period. this exclusivity is said to stimulate innovation by safeguarding the intellectual property of the inventors. patent drafting refers to the process of writing a detailed description of an invention in a legal document that meets the requirements of patent law. it is a complex and time-consuming task that requires a thorough understanding of the invention and the relevant patent laws, and the cost of obtaining a patent can be significant @xcite , with the drafting process being the biggest part of the cost. with the progression of deep learning technologies, a multitude of complex challenges in the patent domain have been ameliorated. computational techniques have notably enhanced patent prior art searches @xcite @xcite and facilitated efficient patent classification @xcite . however, the generation of patent texts and the assessment of the quality of text produced by neural models remain underexplored areas of research. this can be attributed to the exigent demands for precision and accuracy within the legal domain, and it is clear that the evaluation of machinegenerated patent texts necessitates an extensive domain-specific acumen, attainable only by experts within the field. this additional layer of complexity poses significant hurdles for researchers from allied disciplines, seeking to venture into and assess their contributions to this niche, unlike in more generic text applications such as machine translation or generic dialog systems. nonetheless, recent strides made by large language models, such as openai's gpt-3.5, @xcite , and other open-source variants like llama2 @xcite and falcon @xcite , have demonstrated promising capabilities in generating high-caliber legal texts @xcite . these developments signal a promising horizon for enhanced performance and nuanced evaluation in the domain of patent text generation. in our study, we focus on evaluating and understanding the quality of patent text generation by various language models. we have developed patenteval, a benchmark annotated by human experts, tailored for assessing language models of different sizes and capacities. this includes pairwise comparisons and detailed analysis of error types in each output. our goal is to narrow the gap between human-written and machine-generated patents, offering a clearer view of the potential uses of large language models in this field. ## preliminaries and background a patent is a structured document that typically includes several sections, such as a title, abstract, background, brief summary of the invention, detailed description, one or more claims, drawings, and classification information, among others. in this study, our primary focus is on the generation of patent abstracts and claims. the abstract of a patent is a concise summary that offers a straightforward overview of the invention's main features. it is typically used for informational and search purposes, helping individuals quickly understand the essence of the patented technology without delving into the detailed description found in the patent specification. patent claims stand as the cornerstone of a patent document. claims meticulously define the specific features and associated rights of an invention. written in a unique combination of legal jargon and patent-specific language, these claims serve to concisely and unambiguously detail the novel elements of an invention. this could relate to its construction, composition, or operational methodology. these claims set the boundaries for what others can or cannot do without permission from the patent holder. in this sense, writing the claims is a strategic choice of the patent council that depends on outside, e.g. economic factors, and is not as a whole automatable. nonetheless, a system that proposes subsequent claims can be of great use to the council to ensure the quality and completeness of the claim set. in addition, claim dependency is another variable to look at ( ## related work recent advancements in natural language generation (nlg) have seen significant progress across various domains, yet generating and evaluating patent texts remains a challenge due to the intricacies of legal knowledge. initial strides in this field were made by the patenttransformer project @xcite , which explored adapting the gpt-2 model @xcite to generate patent claims, aiming to assist patent writers with an "augmented inventing" tool. a subsequent version of patent-transformer @xcite expanded this capability to generate different patent sections from given parts (e.g., converting an abstract into a title or claim). to evaluate the effectiveness of these generated claims, @xcite fine-tuned a bert model @xcite for binary classification, assessing the relevance of consecutive claim segments. building on this, @xcite developed a two-transformer model framework for quality control in patent text generation, proposing an "auto-complete" feature to facilitate idea exploration from existing patents. additionally, their study @xcite investigated the origins of generated content by applying priorart search techniques to the training data, laying groundwork for future assessments of text novelty in patents. the ibm research team introduced the patent generative transformer (pgt) @xcite , enhancing the gpt-2 model for multifaceted tasks in the patent domain, such as part generation, text infilling, and coherence checking. to assess pgt, they employed methods like semantic similarity comparison, expert evaluations, 2 and analysis of the model's zero-shot performance on novel generation tasks. more recently, @xcite experimented with various sizes of patentgpt-j, rooted in gpt-j @xcite , and introduced a novel metric that gauges the efficiency of language models in generating patent claims by quantifying the reduction in keystrokes for autocomplete functions. however, the variation in evaluation methodologies and datasets across studies, including those focusing on patent summarization or claim generation, makes it difficult to compare results consistently. several datasets and benchmarks have been developed for patent-related tasks. the big patent dataset @xcite focuses on patent summarization and includes about 1.3 million u.s. patent documents sourced from the google patent public datasets via bigquery. its text coherence and abstractiveness were evaluated using n-gram occurrence rates and entity distribution metrics. the harvard uspto dataset (hupd) @xcite is another crucial resource, comprising english-language utility patent applications filed with the uspto between 2004 and 2014. it features benchmarks for binary patent decision classification, multi-class ipc/cpc classification, masked language modeling, and abstractive summarization, with task-specific metrics like rouge for summarization. furthermore, @xcite emphasized the challenges in ensuring factual consistency in patent texts, suggesting alternative evaluation methods like qags @xcite and factcc @xcite , aligning also with our research focus. recent advancements in large language models (llms), exemplified by openai's gpt-3.5, @xcite , and other open-source alternatives such as llama2 @xcite and falcon @xcite , have showcased their capabilities for zero-shot learning and their adeptness at handling a wide array of tasks when provided with straightforward instructions. in light of these developments, our study aims to comparatively assess the performance of these models, with a particular emphasis on shared tasks like claims generation and abstract generation. we benchmark these models against their contemporary generative counterparts using a dedicated evaluation dataset. moreover, we delve into a nuanced analysis of their outputs by investigating the characteristics and distribution of the errors they produce. ## results and analysis the annotation process involved two primary annotators: a seasoned patent lawyer with over 15 years of experience in relevant domains, and a phd student. in cases of disagreement, a third expert was consulted to reach consensus. our analysis is structured around addressing three key questions: q1: what are the error distributions of each model? figure 1 shows the error distribution in 416 pairs for both task outputs of different models. the distance of the bars to the top represents the proportion of error-free outputs. below that, the segmented bars illustrate the percentages of specific error types occurring. notably, chatgpt excelled in minimizing errors in both quantity and diversity. in the claims2abstract task, chatgpt demonstrates exemplary performance, flawlessly executing the task without any errors. among models within the large language model (llm) family, a discernible trend emerges, wherein larger models tend to exhibit a reduction in the variety of errors. for instance, both llama2-7b and falcon-7b exhibit basic errors, including grammatical mistakes, hallucinations, and other types of errors. in contrast, falcon-40b and llama2-70 are more prone to errors related to the coverage of the invention's scope. falcon-40b frequently generates repetitive and overly verbose abstracts, while llama2-70 struggles to fully capture the scope of the invention. the hupd t5-small model, constrained by its limited encoder and decoder context lengths, often overlooks essential components of claims and incorporates non-factual information. however, its fine-tuning on actual claims-to-abstract data ensures that the abstracts it generates maintain close relevance to the given claims. in contrast, other models exhibit a variety of errors to differing extents. in the next-claim-generation task, improper punctuation usage is common across models, often influenced by the non-standard punctuation usage in the input claims. unlike chatgpt and llama2-70b, which manage to avoid making grammatical errors, the majority of models grapple with grammatical inaccuracy in their drafting. patenttransformer, patentgpt-j, and falcon-7b, in particular, struggle with aligning with the claims formatting, where patenttransformer and patentgpt-j also have trouble maintaining consistent antecedent referencing and avoiding preamble inconsistency, issues that frequently co-occur. large language models (llms) often demonstrated a tendency to rephrase or repeat previous claims without enhancing specificity in the generated dependent claims, thereby disrupting the logical flow of the claims' scope. this issue was particularly pronounced among llama-2 models. additionally, falcon models frequently failed to adhere to the specified dependencies outlined in instructions, resulting in compromised coherence in the generated claims. moreover, patenttransformer exhibited a propensity to misnumber claims or reference non-existent prior claims, further undermining the integrity of claim sequences. excluding chatgpt, llama2-7b, llama2-70b, and patenttransformer, models at times generated content that was either irrelevant or factually incorrect, highlighting the significant challenge of achieving accuracy in patent claim generation. ## conclusions this study marks a pivotal advancement in generating and evaluating patent texts, especially abstracts and claims, created by diverse language models. we aimed to explore the potential of llms in patent drafting. our investigation reveals the strengths of certain llms in generating quality patent texts and also identifies common errors and their frequencies. these insights lay the groundwork for future progress in this area, informing both the enhancement of existing models and the incorporation of ai into patent drafting practices.