Towards Comprehensive Patent Approval Predictions:Beyond Traditional Document Classification (2022)

Abstract: Predicting the approval chance of a patent application is a challenging problem involving multiple facets. The most crucial facet is arguably the novelty — 35 U.S. Code § 102 rejects more recent applications that have very similar prior arts. Such novelty evaluations differ the patent approval prediction from conventional document classification — Successful patent applications may share similar writing patterns; however, too-similar newer applications would receive the opposite label, thus confusing standard document classifiers (e.g., BERT). To address this issue, we propose a novel framework that unifies the document classifier with handcrafted features, particularly time-dependent novelty scores. Specifically, we formulate the novelty scores by comparing each application with millions of prior arts using a hybrid of efficient filters and a neural bi-encoder. Moreover, we impose a new regularization term into the classification objective to enforce the monotonic change of approval prediction w.r.t. novelty scores. From extensive experiments on a large-scale USPTO dataset, we find that standard BERT fine-tuning can partially learn the correct relationship between novelty and approvals from inconsistent data. However, our time-dependent novelty features offer a boost on top of it. Also, our monotonic regularization, while shrinking the search space, can drive the optimizer to better local optima, yielding a further small performance gain.

## introduction intellectual property (ip) is an important and integral to the economy. ip-intensive industries directly accounted for 27.9 million jobs in the u.s. (uspto, 2016) theoretical and empirical evidence shows that patents are effective in fostering technological progress. @xcite @xcite securing patent approvals offers a major shot in the arm to inventors * jingbo shang is the corresponding author. and innovators, increasing the chances of obtaining angel and venture capital investments. however, the process of getting a patent approved can cost applicants tens of thousands of dollars in payments to law firms who claim to be helpful in understanding what gets approved and improving the odds of success of a patent application. thus, algorithmic approaches to aid in the patent evaluation process can potentially save precious time and resources for applicants during the patent application phase, as well as benefit patent examiners in government patent offices around the world, accelerating and improving the review process @xcite . the approval of a patent application, according to u.s. patent laws, is determined necessarily and sufficiently by the approval of application claims. patent laws define individual claims as the subject matter of inventions (35 u.s. code §112), on which "patentability" is defined (35 u.s. code § 101, 102, and 103) (refer to appendix b). no overall assessment of a patent application is provisioned. in practice, application claims demarcate the scope of legal protection that an applicant is seeking and are the eventual objects for investigation under legal disputes or transfer of commercial rights. patent examiners from the u.s. patent and trademark office (uspto) make decisions on each application claim individually and independently with other sections as supporting materials. therefore we focus on claim texts and use the term "patent approval" informally and interchangeably referring to "claims approval." in particular, we primarily consider 35 u.s. code § 102, assessing the novelty of application claims. to the best of our knowledge, we are the first to try to predict patent (claim) approval, which is as an extremely challenging problem for multiple reasons. first, patent documents comprise of technically nuanced and challenging to parse language (intricate legalese). patent texts are usually legal and technical descriptions of objects or pro-cesses, which tend to be complex in vocabulary and grammatical structures @xcite . claims are examined not only literally, but also for their legal implications. appendix a provides a few example application claims. second, the patent examination process tends to suffer from subjectivity and inconsistencies (o'neill, 2018a), exemplified by variance across offices and groups, (o'neill, 2018b) and across human examiners. in fy17, only 66% of primary examiners are within a 12.5% delta off the average allowance rate @xcite . third, at the core of patent examination, evaluation of novelty is time-dependent. rejections of claims by 35 u.s. code § 102 require examiners to cite prior approved patent claims, prior art, as evidence. more details about the examination process can be found in @xcite . the united states patent and trademark office (uspto) receives thousands of applications a week; thus a novel application at one time may be dramatically different in the assessment of novelty after a short time period. this means that a classifier can pick up a positive label from an earlier approved application but receive a negative label from an application sometime later with similar technical content, which is deemed no longer novel. such conflicting information can confuse the classifier and undermine its performance. in other words, the data labels are intrinsically noisy and inconsistent due to the nature of the domain problem. although ai/ml approaches are often discussed in the patent domain @xcite such as in the area of information retrieval @xcite @xcite , applications of deep nlp methods are mostly concerned with classifying the content domains of patents @xcite @xcite . in addition, the extant literature usually explores approved patents rather than applications @xcite . even to simply classify the topics of approved patents, state-ofthe-art document classifiers can only achieve an accuracy of about 69.3% -only 2.2% over roberta @xcite . due to these issues, patent approval prediction task is much more challenging than topic classification for document classifiers. to mitigate the issues, we first develop several handcrafted features based on domain knowledge for use alongside the language model for context and control. the time-dependent nature of the novelty also makes traditional document classifiers not suitable here, because they typically assume that similar instances belong to the same label. to address this challenge, we propose a novel framework aiseer as shown in figure 1 . we formulate a time-dependent novelty score for each patent claim with its semantic similarity against prior approved claims from patent grants, which are final versions of approved patents. specifically, inside a comprehensive pool comprising millions of grants, we consider those approved before the filing date of the focal application and then measure the maximum semantic similarity score of the focal patent claim matched with all approved claims in the timedependent sub-pool. to improve computing efficiency, we apply document-level filters to narrow the sub-pool for each claim. integrating such similarity scores with handcrafted features and bert, we conduct experiments on the large-scale uspto dataset and find significant performance gains over fine-tuning a standard bert alone. all else equal, a patent claim with a higher similarity score, i.e., semantically more similar to prior approved claims, should be less likely to be approved. hence we propose to impose monotonic regularization on the novelty score so that the loss function has an additional term of the hinge loss to further penalize non-decreasing predictions in the similarity. this effectively restricts the search space for the optimizer to prediction mechanisms that are reasonably consistent with the novelty measure. from our experiments, this regularization significantly impacts the model outputs. although performance improvements are limited, it can help the optimizer steer away from unfavorable local optima and further improve auroc. we further discuss the experimental findings in depth to illustrate how bert and handcrafted features contribute in overcoming the unconventional data issues. in summary, our contributions are as follows. train validation test applications m 216,101 175,597 153,632 claims n 3.90m 3.07m 2.58m approval % 80.65 80.16 81.68 time range 04/16-02/17 03/17-10/17 11/17-06/19 metadata, office actions, rejections and citations data into a massive dataset; • we develop a series of handcrafted features to aid the prediction of 35 u.s. code § 102 approval decisions. in particular, we design and analyze a time-dependent feature that measures the novelty of patent applications at the time of filing; • we incorporate the handcrafted features and impose monotonic regularization on the novelty features to shed light on how the intrinsic data inconsistency issues in the domain problem can be mitigated. reproducibility. we will release the benchmark dataset and our code on github: ## problem formulation and benchmark in this section, we formally formulate the noveltybased patent approval problem. we describe the experiment setup, the dataset, and baseline results with common document classifiers. ## the aiseer framework our aiseer framework unifies the document classifier, handcrafted features and monotonic regularization, as shown in figure 1 . it is compatible with almost all document classifiers. in this paper, we choose bert as the base document classifier to demonstrate the effects as it is widely adopted and also performs well in our benchmark evaluations. after each application claim text is run through the bert model, the output representation is concatenated with the corresponding handcrafted features. our handcrafted features include a time-dependent claim-level novelty score, claim-level structural features, document-level similarity scores, and other application metadata features. we further impose a monotonic regularization on the impact of the claim-level novelty score so that the loss function has an additional term of the hinge loss. ## experiments in the experiments, we seek to answer a number of questions. to begin with, we evaluate how handcrafted features can help the deep language model, i.e. bert, adapt to a complex domain that differs from typical nlp use cases. in particular we focus on the novelty feature critical to patent approvals. we are interested in the extent to which a standard bert application can learn from highly noisy labels and inconsistent data and find out the novelty pattern, i.e. the significance of novelty in determining patent approval outcomes. in addition, we study if the combination of bert and handcrafted features serves to be adequate in capturing the novelty pattern. we also examine if monotonic regularization boosts the learning process to further overcome the intrinsic data inconsistencies. we mainly compare aiseer with two baseline models, bert and lstm, as they are the best common document classifiers from our benchmark results. for ablation study purpose, we also compare with log. reg. feat. only, a logistics regression model with handcrafted features only, and aiseer w/o regu., which is a bert model integrated with our handcrafted features but not regularized by our monotonic constraints. aiseer is trained with the same set of hyper-parameters as bert: maximum token length as 128, fine-tuning for 5 epochs; batch size as 256; adamw with learning rate being 5e-5 as the optimizer. the monotonic regularization parameter c is 1 2 and λ is 5e-4. the models are trained on a single nvidia quadro rtx 8000 gpu. the results are shown below in ## related work to our knowledge, our work is the first in predicting patent approvals according to the examination procedures at the government patent office. few extant researches attempt to predict decisions in office. @xcite studies ptab (patent trial and appeal board) hearing decisions at uspto. other related work addresses patent quality in a general and broad sense @xcite . more broadly in the ip/patent domain, although ai/ml applications have been often advocated @xcite , studied (for a review see @xcite tietze, 2018)) or implemented in practice @xcite , most work focus on determining patent content classes to save manpower or concern only with patent grants rather than applications @xcite @xcite @xcite . recent studies @xcite ) emerge aiming at predicting patent transfers and the economic value. other streams of related work include those exploring patent similarity. our approach of constructing the novelty feature with a state-of-the-art neural bi-encoder @xcite is significantly more advanced than relatively rudimentary approaches in the extant literature, such as text matching and frequency-based methods @xcite @xcite . studies on semantic analysis and representation of technology @xcite based on patent data are also related. ## conclusions and future work in this paper, we tackle the challenging problem of predicting patent approval decisions as per 35 u.s. code § 102, namely the novelty-based decisions. we have prepared a large-scale benchmark dataset by consolidating different data sources from uspto. from the evaluations of the popular document classifiers, bert and lstm are arguably the most effective ones. we identify the timedependent challenge of the novelty judgement, and therefore propose aiseer, a novel framework going beyond the traditional document classifiers. specifically, we construct a claim-level core novelty feature along with several other handcrafted features and apply them on top of the pre-trained bert model. we further propose to add the monotonic regularization on the core novelty feature to resolve the potential label conflicts caused by the mechanism of the patent examination process. experimental results have verified the superiority of aiseer and also the effectiveness of introducing novelty features and monotonic regularization. we believe that our work is beneficial to various parties, including patent applicants, attorneys, examiners and regulators. while the advantages of our regularization methodology are significant, there is still room for potential metric improvements, thus further developing the work will yield opportunities for promising future research and greater contributions to the communities. in future, it is important to extend the scope from claims to the other sections in the patent applications. relationships among components and entities described in claims and relations among claims are also critical to investigate.