Patentformer: A Novel Method to Automate the Generation of Patent Applications (2024)

Abstract: In recent years, Large Language Models (LLMs) have demonstrated impressive performances across various NLP tasks. However, their potential for automating the task of writing patent documents remains relatively unexplored. To address this gap, in this work, we propose a novel method, Patentformer, for generating patent specification by fine-tuning the generative models with diverse sources of information, e.g., patent claims, drawing text, and brief descriptions of the drawings. To enhance the generative models’ comprehension of the complex task of writing patent specification, we introduce a new task, claim+drawing-to-specification, and release a new dataset. We evaluate our proposed method on thousands of patents from the USPTO and show that our method can generate human-like patent specification in legal writing style. Human evaluations by four patent experts further affirm that our proposed method has the potential to generate correct specification, and the quality of generated specification may sometimes be better than the actual specification.

## introduction patents are legal documents that require a very specific writing style where certain words and phrases carry specific meanings, e.g., an "embodiment" of the invention refers to the physical manifestation of the invention or idea. a patent document usually consists of the title, abstract, field of the invention, background, summary of the invention, independent claims, dependent claims, drawings, brief descriptions of the drawings, and a detailed description of the invention which is also referred to as the specification. traditionally, patents are drafted by the patent attorneys who have extensive knowledge of both the law and the patent system, and it costs over $10k on average to draft a moderately complex patent @xcite . usually, the patent attorneys read the invention disclosure documents and interview the inventor(s) to understand the technical details of the invention, and then they draft the claims, drawings, and specification. patent claims protect the boundaries of the invention, and hence, drafting claims requires the expertise of the patent attorneys. the drawings must follow the requirements of the patent office, and label every element of the drawing mentioned in the specification with a unique number. however, the bulk of patent text consists of specification, and the patent attorneys need to spend a significant amount of time and effort in drafting the specification to describe the invention in detail based on the claims and drawings. figure 1 shows an example of a patent claim, relevant drawing, and the specification supporting the claim. in this work, we assume that the patent attorneys can provide their drafted claims and additional drawings as input to our system to automatically generate patent specification. transformer-based large language models (llms) such as bert @xcite , t5 @xcite , gemini @xcite , and gpt-3 @xcite and its successor gpt-4 @xcite have shown impressive performances in the field of natural language generation. however, automating the generation of human-quality patent specification remains challenging for these llms, especially because patents are intricate legal documents that require each claim to be adequately supported in the specification, and the specification must describe the invention in sufficient details using the associated drawings. hence, patents contain much more technical information than, e.g. general web text, making it difficult for the llms to capture all the relevant pieces of information pertaining to an invention to generate a coherent specification. patent specification usually spans several pages, thus presenting another challenge for most of the llms which are limited by their token lengths, e.g., 512, claim 7. the artificial intelligence-based apparatus of claim 2, wherein the processor is further configured to control an operation of the peripheral device according to the predicted average wake-up time or bed time. relevant specification: fig. 4 is a configuration diagram of an artificial intelligence (ai)-based apparatus for providing wake-up and bed time information according to an embodiment of the present invention. referring to fig. 4 , the aibased apparatus for providing wake-up and bed time information may include a communication unit 210a, a processor 260a, and a memory 230a. the processor 260a may control the operation of the peripheral device in correspondence to the predicted wake-up or bed time. a brief description of the drawing: fig. 4 is a configuration diagram of an artificial intelligence-based apparatus for providing wake-up and bed time information according to an embodiment of the present invention. 1024, 2048, or 8192 tokens. moreover, most pretrained llms are not trained on patent data, and thus cannot generate text in legal writing style. in this paper, we present a novel method for generating patent specification to overcome the aforementioned limitations. we introduce two new tasks, claim-to-specification that simply generates specification for the given claim, and claim+drawingto-specification that takes a claim and any associated drawing text as input to generate specification. to the best of our knowledge, our paper is the first work that generates patent specification by using claim and drawing text as inputs. we present new model-agnostic strategies to effectively construct training datasets with enriched context to help the model generate correct specification. we fine-tune two popular llms using our training datasets and show that our method can significantly outperform the pretrained and fine-tuned llms on several years of patent data. we also conduct an extensive user study to evaluate the correctness and quality of the generated specification and show that our proposed method can generate correct specification in 66% of the cases related to neural processor domain, and the quality of generated specification may sometimes be better than the actual, humanwritten, specification. we publicly release the first dataset for claim+drawing-to-specification task at https://github.com/juriand/patentformer . ## related work most prior work related to patent text generation focused on generating specific sections of a patent, for example, lee and hsiang (2020a) generated patent claims by fine-tuning gpt-2, lee (2020c) incorporated an additional bert-based module on the basis of @xcite for personalizing the claims, @xcite presented a span-based approach and a generic framework to measure patent claim generation quantitatively, and @xcite presented an approach to generate patent claims from detailed descriptions. @xcite presented a text-to-text mapping approach for controlling patent text generation by using the structural metadata in patent documents, where the keywords in input indicate different generation tasks. @xcite prior research on patent drafting either focused on generating a small section of text, for example, claims, or simply summarizing patent text to generate title or abstract. the closest related work is the study by @xcite that generated claims from specification. to the best of our knowledge, our paper is the first work that generates patent specification from the claim and drawing text. nent and i num z j is the number of j th component in image i z ; @xmath0, n 2 , ..., n t } corresponding to all images in i. ## experimental setup we used 13,725 patents in this category, representing about 0.69% of the total 2m patents that were granted by the uspto between 2015 and 2023, to construct the patent-2015-2023-g06n dataset consisting of 284,531 quadruplets foot_4 @xmath1, b z , n z , s y >. ## results in this section, we first compare the proposed patentformer with chosen baselines using automatic evaluation metrics. then, we present a user study to evaluate our method from the human's perspective. finally, we perform an ablation study to show the effects of embedding rich context into training data on the performance of patentformer. ## conclusions we proposed a novel method, patentformer, to utilize diverse patent-related information, e.g., patent claims, drawing text, and brief descriptions of the drawings, for generating patent specification. we presented a model-agnostic approach to enrich the training dataset with richer context for the new claim+drawing-to-specification task. we evaluated our approach using both encoder-decoder and decoder-only llms and showed that our proposed method has the potential to generate correct specification in legal writing style. human evaluation of the generated samples by four patent experts further affirmed the effectiveness and practical usefulness of our proposed method.