PGT: a prompt based generative transformer for the patent domain
Abstract
Patents are a valuable source of knowledge, but drafting them is a time-consuming and expensive task.
Methods that assist patent generation can provide a two-fold improvement as they can speed up the generation process and suggest to the inventor ideas and claims.
Herein, influenced by recent advances in language modeling via multitask learning and prompt engineering, we present Patent Generative Transformer (PGT), a transformer-based language model trained to facilitate patent drafting.
Specifically, the model supports three tasks: part-of-patent generation, text infilling, and patent coherence evaluation.
PGT complements inventors and assures the fast and successful transition from their input to a coherent patent disclosure taking advantage of its multitasking nature.
We show how the model outperforms a collection of task-specific baselines on relevant metrics.
We further test the quality of the generated text via blind testing by subject matter experts.
Finally, we explore a zero-shot extension of the model showing how to use PGT for generating domain-specific abstracts.
1. Introduction
A patent is an exclusive right granted for an invention.
It is an essential tool to establish ownership rights and protect the owner’s intellectual property.
Drafting a patent application is challenging and time-consuming.
The process involves several stakeholders and countless iterations to draft a patent using the appropriate jargon.
Besides the time investment, the cost of the drafting process can exceed 15,000$, depending on the subject of the invention.
For the above reasons, natural language processing (NLP) methods that focus on the patent domain (Krestel et al., 2021) have gained a lot of interest recently.
Especially for the patent generation process, methods that can generate parts of a patent in an automated manner have also been proposed in the literature (Lee & Hsiang, 2020; Lee, 2020).
One of the most general and notable approach of this category is the PatentTransformer (Lee & Hsiang, 2020).
This is a GPT-2 (Radford et al., 2018)
based model trained from scratch on patent data to generate part of the patents.
The latest advances in Language Modeling (LM) showed that multitask learning (Collobert & Weston, 2008; Raffel et al., 2020) has drastically improved performance across different NLP applications.
Recently, natural language prompting (Raffel et al., 2020; Sanh et al., 2021) emerged as an effective approach for multitask learning in NLP by reformulating specific tasks in the format of a natural language query/prompt that expects a response formatted accordingly.
Inspired by these seminal works, we present Patent Generative Transformer (PGT), a transformer-based multitask language model (LM) trained to facilitate the patent generation process.
Specifically, we cast three tasks of interest for the patent generation process, namely: part-of-patent generation, text-infilling and patent coherence evaluator as text generative tasks using the proper prompts.
We train all the tasks simultaneously in a multi-task setting.
We examine our model’s quality and actual usage in three ways.
First, we compare the model with other baselines for each designated task.
Second, to highlight PGT’s capabilities in actual scenarios, we perform an evaluation study of our work by consulting subject matter experts.
Lastly, we move to a zero-shot setting and investigate to what extent the model can generate abstracts by providing solely a set of related keywords.
The later use-case underlines the general capabilities of the PGT which are not confined to the patent generation process but can be extended to the broader knowledge discovery and hypothesis generation area.
To facilitate the use of PGT, we have made the model available at https://github.com/GT4SD/gt4sd-core . PGT
This is a patent abstract 
 Given the above abstract, suggest a title
This is a [MASK] abstract with [MASK] masks 
 Replace the [MASK] tokens in the above abstract 
 This is a patent abstract 
 This a patent claim 
 Do the above abstract and claim belong to the same patent? <
|sep|>
This is the suggested patent title patent 
 one or more
Yes
2. Patent Generative Transformer
Automating the patent drafting process is the main goal of our work.
Yet, we deviate from the existing approaches which focus solely on the task of part-of-patent generation and attempt to provide a multitask model that can perform a broader collection of useful tasks which can be used as tools during the patent generation.
We call our model Patent
Generative Transformer (PGT) and it is a multitask and prompt-based model relying on the GPT-2 architecture.
Being an autoregressive model, PGT is trained to predict the most suitable continuation of the given input.
Working with a model that receives and produces text only, there is need to cast all the tasks of interest in a text format.
Subsequently, the task identification is also done based on the input text by adding to it the respective task-specific prompt.
The utilized prompts are small phrases that describe what is the input and what is the expected output, for instance “Given the above abstract suggest a title”.
We place the task-specific prompt at the end of the input as this turns out to be the most effective strategy based on the literature (Reynolds &
McDonell, 2021).
The differentiation point for each task is prompt.
The general approach is presented in Figure 1.
For more information regarding the prompts, we refer the reader to the Appendix.
PGT has been trained simultaneously on three tasks that could be utilized during the patent drafting process, namely: part-of-patent generation, text infilling and patent coherence check.
Part-of-patent generation is the standard generative language modeling task adapted to the patent domain.
The model attempts to generate a part of a patent given as input another, already existing part of it.
We focus on the title-toabstract, abstract-to-claim transitions as well as their inverse.
In the case of the claims, we focus on the first independent claim of the patents and at this point, we completely omit the introduction of the dependent claims in the task.
Patent editing is also desirable as the inventors of the patent may seek to improve a patent’s text or to enhance the patent’s quality by changing some of its parts.
In this direction, the text-infilling task (Lewis et al., 2020) helps by suggesting alternatives for parts of the text that have been masked with the respective special token.
The generated alternatives are not necessarily a single token but they could be a few words or even small phrases depending on the context of the input.
To cast this task as a text generation problem, we reformulate the expected output.
Specifically, the model’s output is a sequence of the suggested mask replacements in the same order as they are appeared in the input text and separated by a special mask separator token.
The previous tasks aim to generate or improve a part of the patent, however, there is also need to assess the quality and the coherence of the generated piece of text.
To this end, the coherence check task treats this case as a binary classification problem and judges whether two given patent parts could belong in the same patent or not.
This task is casted as a text generation problem by teaching the model to produce the output Yes or No depending on the coherence of the two inputs.
Based on this format, we can also convert the output into a coherence score by inspecting the probability that the PGT model assigns to the Yes output.
A large patent dataset is mandatory to train such a model.
In our case, we relied on a corpus of 11,600,000 patents that have been published between 1998-2020.
Using all these patents, we generated 140,000,000 different instances that cover all the three tasks of interest and their different sub-cases.
We refer the reader to the Appendix for further details regarding the dataset generation.
PGT has the standard GPT-2 architecture.
We opted for GPT2 instead of more recent models, such as T5 and GPT-3, due to their size.
GPT-2 has at least one order of magnitude fewer parameters than the other two models, which makes its training feasible even with limited resources.
Additionally, following a low budget approach and given that pretrained GPT-2 has already demonstrated its power in general natural language, we preferred to not train our model from scratch (an approach followed in PatentTransfrormer) but to continue the training in a domain adaptive fashion (Gururangan et al., 2020).
Specifically, we continued the training from the GPT-2 checkpoint using our patent dataset having a training budget time of one week which was translated [Table] into 3 full epochs.
The system that was used for training contains 32 A100 GPUs.
3. Results
To assess the capabilities of our model, we proceeded with three different evaluations.
First, we focused on the three tasks of interest for the model and we compared PGT’s performance with different baselines.
Secondly, we tested the quality of the generated text via blind testing by subject matter experts.
Lastly, we assessed the zero-shot capabilities of the model in an unseen generation task to emphasize the model’s understanding of the broad patent domain.
3.1. Evaluation of Tasks
The first evaluation goal was to compare the PGT’s abilities with other baselines for each task.
Thus, we created an additional test set with 1000 patents.
Using this set, we generated instances for each task following the procedures described above for the training set generation.
For the partof-patent generation, the aim was not to generate the actual part of the patent per se but to generate valid suggestions.
For this reason, we avoided using word-based metrics and focused on the semantic similarity between the actual and the generated part to assess the models’ quality, a metric that has been used extensively in previous studies (Lee & Hsiang, 2020; Lee, 2020).
To measure the semantic similarity, we used the cosine similarity of the text embeddings extracted relying on sentence transformers (Reimers & Gurevych, 2019).
The same motivation stands also for the text-infilling task.
Also, we examined the semantic similarity between the actual and the generated masked content.
Lastly, the patent coherence task is a binary classification problem, and therefore the accuracy of the models was examined in this case.
We compared our model with other baselines for each task.
For the part-of-patent generation task, we used as baselines the standard pretrained GPT-2 model without further task-specific finetuning, PatentTransformer and 4 finetuned
GPT-2 models - one for each of the subtasks of interest.
To finetune these 4 models, we used the respective task instances from our training dataset and trained GPT-2 formore epochs.
For the text-infilling task, we used the pretrained BART (Lewis et al., 2020) and BERT (Devlin et al., 2019).
Note that BERT is limited to replacing each mask with just one token.
This limitation does not hold for BART.
As it concerns the patent coherence check, we selected a
BERT model fine-tuned for the task as a baseline.
To train this model, we followed the standard BERT task-specific finetuning methodology (Devlin et al., 2019) and trained the model for 3 epochs.
Table 1 depicts the mean performance of PGT and the other baselines for the three tasks of interest.
Overall, the multitask training helps PGT to have better or the same performance as the baselines.
In the text-infilling task, PGT significantly outperforms the other baselines.
In the part-of-patent generation, PGT is 2% better than PatentTransformer.
This finding indicates that even though the two extra tasks are not directly relevant to the part-of-patent generation, they assist the model to understand better the patent domain.
In some cases the case-specific finetuned GPT-2 model performs equally or slightly better results, yet the use of different models for each task/case would increase significantly the storage footprint of the patent drafting tool.
Lastly, in the patent coherence check PGT and BERT classification both have the same high level of accuracy.
3.2. Evaluation by Subject Matter Experts
The above investigations focus on the quantitative comparison between PGT and other baselines.
However, such evaluation cannot highlight the value of the model in real cases.
To emphasize this, we performed a subject matter experts evaluation of the model focusing on the part-of-patent generation task.
We focused on a small set of 44 patents related to the broad chemistry domain.
For each of them, we used PGT and performed all the possible generative combi- abstract [Table] abstract 0. [Table]
[Table]
[Table] Generated [Table] A61BA61FA61KA61NA63BB41JB65DG01NG01RG02BG03BG03CG03FG03GG06FG06TG11BG11CH01JH01LH01RH04LH04N
I P C
c o d e s a
[Table] k e y w [Table] r d i n p u t
[Table] [Figure]
[Figure]
nations of the part-of-patent task to perform part of patents. Then, we asked 8 patent experts to select given the input part which of the generated or the actual part of the patent suits better to the given part.
The information regarding which is the actual and generated text was not exposed to the evaluators, so they selected solely based on the structure and the content of the two texts.
Figure 2a presents the results.
Interestingly, more than 37% of the selected texts are generated in the abstract-to-claim, claim-to-abstract and title-to-abstract cases.
Our evaluation was confined to the part-of-patent generation task and there was not any human correction or any other postprocessing step to improve the generated texts before the evaluation, indicating that the generated texts were already in a good state and the needed intervention effort of a human to improve them would be typically low.
Evaluators’ feedback contains also useful information for future directions in the field.
Specifically, it has been highlighted that the model provides quite specific or detailed descriptions in many cases, which is not preferred in a patent document where a more general description could cover a broader invention range.
This is mainly indicated in the results of abstract-to-title generation, where the majority of the generated titles were deemed too specific to be patent titles.
Moreover, it was highlighted that the best starting point of an automated patent generation process is the abstract as it is probably the most descriptive part of the patent.
Yet, the addition of further details, like keywords or a more detailed description of the invention as a starting point, could improve the process.
3.3. Zero-shot evaluation
One reason that LMs have gained a lot of attention recently, is that they depict great performance even in few-shot or zero-shot settings (Radford et al., 2018; Sanh et al., 2021).
Towards this direction, we also examined PGT generation capabilities for an unseen task during training.
Specifically, we attempted to generate patent abstracts based on keywords and check their similarity with existing abstracts.
Keywords are too short, general and lack syntax.
Thus, an invention cannot be described precisely using only them.
Therefore, we examined whether PGT could understand the area of interest and suggest ideas, in the form of patent abstracts, related to this specific domain.
Such attribute would set
PGT as an idea/hypothesis generator method as well.
We used the short descriptions of IPC codes (WIPO, 2022)
(in the level of sections) as keywords and we generatedpatents for each keyword.
Then, we examined the semantic similarity of the generated abstracts and abstracts from our original testing set that hold various IPC codes.
Figure 2b depicts the results.
The generated abstracts are more similar to the abstracts that hold the respective or similar IPC code, which validates PGT’s capabilities.
We refer the reader to the Appendix for further analysis of this investigation.
4. Conclusion
Patent drafting is a tedious and time-consuming process.
In the context of this work, we attempted to leverage state-ofthe-art NLP to present PGT, an LM that can be utilized to automate and speed up this process.
The results indicate that the model holds interesting capabilities and outperforms existing methods.
Patents are a unique language domain with specific syntax, vocabulary and need for different levels of information across the different parts of the document.
Thus, the modeling of this domain is a difficult task.
Both the semantic analysis and the opinion of subject matter experts indicate that our model can generate accurate patent content and assist in the whole patent generation process.
Furthermore, the model holds a strong understanding of the patent domain which that can be leveraged for tasks such as ideas generation in a zero-shot fashion.
Such findings set
PGT as a useful tool that can be utilized either by inventors or researchers in different steps of the discovery process.
Appendix Prompts and output format
Figure 3 presents the format of a model’s input.
The green part (straight line) is the expected input to the model while the orange (dashed line) part is the output that the model learns to generate and append to the input.
Furthermore, in Table 2, the specific prompts that have been used for the tasks of interest are presented.
For the zero-shot keyword-to-abstract generation that has been investigated the prompt for the part-of-patent generation has been used as well where “keywords” are placed in the part of the patent placeholder that refers to the input text type.
Patent dataset
We relied on a corpus of 11,600,000 patents that have been published between 1998-2020 to generate our dataset.
Using all these patents, we generated 140,000,000 different instances that cover all the three tasks of interest and their different sub-cases.
Specifically, for the part-of-patent generation, we generated 4 different instances for each patent based on the generation combinations described above.
For the text infilling task, we generated 2 instances for each patent(one for the abstract and one for the claim).
We selected randomly the number of masks in each instance between 1 and 3.
The number of words per mask follows a binomial distribution with n=5, p=0.3 and a shift of one.
Lastly, we assured that the masked part of the instance is less than 25% of its total number of tokens.
For the patent coherence task, we need both positive and negative examples.
The positive examples were easily created by parts that already belong to the same patent.
To generate negative examples, we used a positive example and we replaced one of the parts with the respective part of a randomly selected patent.
For each patent, we created 1 positive and 1 negative example.
Zero-shot evaluation
Figure 4 presents the non-normalized version of the results presented in Figure 2b as well as the distribution of the similarity scores of the generated abstracts and abstracts that belong to the same domain as the input keyword.
Even if the generated abstracts are more similar to abstracts originating from the same domain, we can observe differences in the achieved similarity between each case.
This is attributed to the broad range of applications that each domain can cover.
The same stands also for the achieved level of similarity which even in the best cases for each row usually lies between 0.35 and 0.40, values way less than the ones achieved in the actual versus generated part of patent comparisons.
Furthermore, based on the similarity scores distribution that we examined, we show that the understanding of the patent domain is attributed to our training and it is not inherited from GPT-2.
Input text 
 Task specific prompt
Expected output
[Table]
Part-of-patent generation
Given the above < part of the patent > suggest a < part of the patent >
Generated part of the patent
Text infilling
Replace the [MASK] tokens in the above < part of the patent > suggestion for 1 st mask < | mask sep | > suggestion for 2 nd mask ....
Patent coherence check
Do the above < part of the patent > and < part of the patent > belong to the same patent?
Yes or No
[Table]
A61BA61F
A61KA61NA63BB41JB65D
G01N G01R
G02BG03B G03C
G03FG03GG06FG06TG11BG11CH01JH01LH01RH04LH04N I P C
c o d e s a s k e y w o r d i n p u t
[Figure]
[Figure]