PAP 2 PAT : Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs (2025)

Abstract: Dealing with long and highly complex technical text is a challenge for Large Language Models (LLMs), which still have to unfold their potential in supporting expensive and time intensive processes like patent drafting. Within patents, the description constitutes more than 90% of the document on average. Yet, its automatic generation remains understudied. When drafting patent applications, patent attorneys typically receive invention reports (IRs), which are usually confidential, hindering research on LLM-supported patent drafting.Often, pre-publication research papers serve as IRs. We leverage this duality to build PAP2PAT, an open and realistic benchmark for patent drafting consisting of 1.8k patent-paper pairs describing the same inventions. To address the complex long-document patent generation task, we propose chunk-based outline-guided generation using the research paper as invention specification. Our extensive evaluation using PAP2PAT and a human case study show that LLMs can effectively leverage information from the paper, but still struggle to provide the necessary level of detail. Fine-tuning leads to more patent-style language, but also to more hallucination. We release our data and code.

## introduction securing intellectual property is a long and costly process that requires both deep technical knowledge and expertise in patent law. this motivates the use of technology to boost patent attorney productivity. natural language processing (nlp) already assists prior art search @xcite @xcite and patent landscaping @xcite . research on large language models (llms) in the patent domain has recently gained momentum @xcite @xcite , but patent drafting remains a largely manual task. a patent typically consists of claims, which define the invention and the legally relevant scope of protection, and a description, which provides technical details in sections like field of the invention, background, summary, and detailed description. prior work has primarily focused on generating abstracts and claims @xcite @xcite @xcite @xcite . the description makes up over 90% of the document, 1 implying that large productivity gains are expected from writing support for this section. yet, generating them remains a significant challenge for llms due to their length, technical complexity, and specialized language @xcite . existing work on automatic patent generation suffers from a number of shortcomings, such as ill-posed task setups, lack of open benchmarks, and disregard for patent descriptions @xcite . in this paper, we tackle all of these issues: we propose a novel setup, evaluation metrics, and outline-guided models for description generation in a realistic setting using open and human-created data. inventors typically submit invention reports (irs), which patent attorneys formalize into patent applications. in many research labs, it is common to use a pre-publication paper as invention report, which leads to so-called patent-paper pairs @xcite ), an unrecognized treasure for ai research. to facilitate the study of llms on patent drafting, we create pap2pat, a new benchmark of 1.8k carefully identified ppps and patent outlines. we develop and validate a method for reliably matching patents and papers describing the same invention (see figure 1 ). for llm-supported patent drafting, we envision a practical setting in which the attorney, given a paper, provides an outline for the patent. this outline acts as a flexible mechanism to control the document structure and content while keeping manual effort low. a major challenge of the proposed task is document length: patent descriptions in pap2pat are on average 18k tokens long, some exceeding 180k tokens. while current llms increasingly support long context windows, they struggle to generate similarly long outputs @xcite @xcite . as a remedy, we propose chunk-based outline-guided patent generation (copgen), which effectively generates long patent documents in chunks. evaluating generated patents poses significant challenges due to their length, their technical complexity, and the high cost of manual evaluation. no standard evaluation metrics are established in the literature, and prior work @xcite commonly resorts to standard text similarity metrics which do not work well on long documents @xcite . we adapt a suite of metrics based on natural language inference (nli) and authorship attribution to the specific case of evaluating factual correctness, coverage, and language style for long-form patent generation on pap2pat. our main contributions are: outline-guided generation is a paradigm in which llms use an outline to produce longer, more structured and coherent text. outlines are either generated in a planning stage @xcite @xcite @xcite @xcite or provided as input @xcite @xcite . they are created using extraction of key words @xcite , phrases @xcite , or sentences @xcite @xcite , generated using llms @xcite or defined interactively @xcite . in our work, we posit that outlines should be provided by the patent attorney to satisfy the high demand for user control. ## pap2pat benchmark we present the pap2pat dataset containing 1.8k ppps, each annotated with three outlines for generation. we describe the steps taken to construct pap2pat, analyse the obtained corpus, and propose evaluation metrics for our new benchmark. ## copgen: chunk-based outline-guided patent generation since llms cannot yet generate sufficiently long documents in a single call, we instead choose to generate patents in chunks. figure 2 summarizes the approach. we chunk the outline, select the most relevant parts of the paper (paper context) depending on the chunks' outline bullet points, and prompt an llm to generate the patent text for that chunk. the outlines of previous chunks are included for global document context. finally, we concatenate the generated outputs and apply only a lightweight post-processing to remove duplicate headings at chunk boundaries. this framework enables generating a long document in parallel, with customized context per chunk and controllable length. token allocation and chunking. in the default setting, we allocate per chunk 2k tokens for the instruction, 3k tokens for the paper context, and 2k tokens for the output patent (t {@xmath0; @xmath1; @xmath2. we choose this setting because preliminary experiments show that on our task, llms only generate up to roughly 2-3k tokens regardless of the requested amount of content, and because this allows comparing with models that support up to 8k tokens. the chunking procedure segments the outline into chunks depending on the token allocation. in particular, it determines the number of outline bullet points per chunk using the reserved number of patent tokens and the average number of characters per outline bullet point foot_4 . it keeps sections intact whenever possible. length control mechanism. @xcite find that llms' response lengths are constrained within a certain range and only moderately adjust to length requests in the prompt, making instructions unreliable for controlling length. however, as the requested length decreases, the llm is more likely to meet it. hence, in copgen, we decrease the number of allocated patent tokens per chunk, thereby increasing the number of chunks, until the desired total length is reached. in our experimental setting, we do this on a corpus-level, i.e., we search for a setting where the average length matches that of the references, leading to the calibrated token allocation t {@xmath3; @xmath4; @xmath5. ## experiments in this section, we describe our experiments using pap2pat to assess the capabilities and limitations of current llms with copgen. ## human evaluation to gain further insight into the practical applicability beyond the automatic metrics, we conduct a human case study with two patent attorneys for the field of ai. we randomly select 10 samples from the nlp field out of the pap2pat test set and use the outputs of qwen2-72b with copgen and default token allocation. for each of the samples, the attorneys are provided with the paper, the outline, the generated patent and the reference patent. they are then tasked to evaluate the quality of the generated patent based on the hypothetical scenario that they got the paper, wrote the outline and sent both to the llm. we ask about strengths, weaknesses and potential for time savings. the attorneys evaluated 10 and 5 samples, respectively, yielding 15 total evaluations and spending about 30 minutes per sample on average. among the 15 evaluations, the attorneys saw substantial time savings in 8 cases, showing promising potential to increase their productivity. however, they also identify two main limitations that could hinder such time savings, which are consistent with observations from our automatic evaluation, further validating our chosen metrics. these limitations are: ## conclusion in this work, we have presented the first study on the generation of complete patent descriptions in a realistic experimental setting, using papers as real-world invention specifications. we create the pap2pat benchmark, propose targeted evaluation metrics, develop copgen to enable the generation of long patent documents, and conduct a human evaluation. we find that llms can effectively generate patent drafts from research papers, but that there is still headroom for improving the level of detail and linguistic style. promising directions for future research include developing fine-tuning methods that maintain factuality and avoid repetitions, as well as conducting user studies to identify efficient interaction patterns.