Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph (2024)

Abstract: Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs’ potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply cost-effective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs.

## introduction scaling up language models has demonstrated predictable improvement and unprecedented abilities in many language tasks @xcite @xcite . however, emerging evidence shows that simply scaling up backbone models to large language models (llms) may llm patent applications claim 1 body text claim 2 claim n ... not guarantee success @xcite @xcite . in addition, scaling up models imposes demanding computational costs that prevent it from being widely adopted for realworld applications. such limitations necessitate cost-effective methods beyond scaling, especially for domain-specific tasks that have distinct traits. in this paper, we look into the task of patent approval prediction, a challenging yet straightforward classification task that scaling struggles to address, and explore customized cost-effective solutions. as shown in figure 1 , the objective is to determine if each claim in a patent application will be approved or rejected by the u.s. government patent office (uspto). it is vital for intellectual property (ip) protection, taking up to 40% of the u.s. gdp and over 30% of employment. due to the demanding requirements for knowledge in both technology and law, patent examination is conducted manually, leading to potential inconsistent outcomes across patent examiners @xcite . such inconsistency underscores the necessity for objective and automated computational support. the state-of-the-art (sota) of this task is based on bert embedding @xcite augmented with handcrafted features @xcite . an intuitive idea is to replace its backbone model with modern llms. to this end, we employ llama2 @xcite , mistral @xcite , vicuna @xcite in various sizes (7&13&70b) and apply both lora @xcite and full fine-tuning. surprisingly, they do not live up to expectations, performing on par or worse than bert. to exploit llms' emergent abilities, we utilize prompt-based methods tailored to these open-source llms, as well as closed-source gpt-3.5 @xcite and @xcite , but the results are still unsatisfying. the shattered hope in llms motivates us to dive into patent data analyses, which leads us to the standardized writing of claims and the dependencies nature among them. as depicted in figure 2 , claim 1 compromises multiple sub-components, which are then referenced in subsequent claims. such intricate inner-claim (between sub-components in claim 1) and inter-claim dependencies (between claims 1&2 as well as claims 1&3) have critical implications for the patent approval prediction task, as the patent examination is conducted on each claim and the rejection of one claim can result in the automatic rejection of its dependents. inspired by the observations and domain-specific knowledge acquired from painstakingly extensive data analyses, we propose fine-grained claim de-pendency (flan) graph for patent approval prediction, which represents each claim by a single graph that encapsulates both inner-and inter-claim dependencies. specifically, as shown in figure 1 , we first design a novel algorithm to automatically construct the flan graph at scale, where each node is an informative segment of the claim text. then, the model-agnostic flan graph is fed into a generic graph model for prediction. examples of the flan graphs and the corresponding claims are shown in appendix c. in the experiments, we adopt a variety of cost-effective graph models such as gcn @xcite , gat @xcite to verify the effectiveness of our proposed flan graph for patent approval prediction. all models with flan graph applied outperform the previous claim 1: a system for [...], the system compromising: an authentication component configured to [...]; a tracking component configured to [...]; and a control component configured to: receive authentication information [...]; receive location information from [...]; and deliver a message to the touchpoint authorizing [...]. claim 2: the system of claim 1, where the control component is also configured [...]. claim 3: the system of claim 1, where the authentication component includes [...]. sota, among which graphsage @xcite achieves the highest improvements of 7.4% in auc and 7.8% in macro-f1 scores, achieving absolute scores of 66.04 and 58.22, respectively. to summarize, our contributions are two-fold: (1) we propose a novel algorithm to automatically construct the fine-grained claim dependency (flan) graph at scale that consistently improves the sota by a large margin. (2) we conduct comprehensive experiments and analyses of modern llms on patent approval prediction, which identify the limitations of llms and provide valuable references for developing llm-based solutions in the future. the source code foot_0 and dataset foot_1 are publicly released to facilitate future research. ## problem formulation in this section, we formally introduce the definition of the patent approval prediction task and analyze the dataset we construct for experiments. ## methodology in this section, we delve into the details of our proposed fine-grained claim dependency (flan) graph. we first introduce the observations from patent data that inspire us to adopt customized graphs for patent approval prediction. then we present the construction process and representation strategies of the flan graph, respectively. ## experiments in this section, we elaborate on our experiments and the corresponding results with both: (1) scaling with llms; and (2) customized graph methods using the flan graphs. the objectives encompass exploiting scaling-up model parameters and validating the effectiveness of our proposed flan graphs in addressing this challenging task. ## related work patent documents are receiving increasing attention in the nlp community due to their structured language and extensive content. the survey by @xcite summarized current deep learning work in the patent domain, including subject matter classification @xcite @xcite , retrieval @xcite @xcite , and data generation @xcite . we highlight a few more specifically relevant or more recent works. @xcite utilize sequence tagging techniques to identify text segments within patents that either describe or reference chemical reactions. @xcite tackle the patent retrieval tasks using matrix similarity measures. @xcite introduce the task of unclaimed embodiment extraction (uee) from patent specifications to help the writing process. @xcite explore datacentric strategies to handle the french patent classification task. the state-of-the-art (sota) work of our task, @xcite , first formally proposes the task of patent approval prediction and designs delicate handcrafted features to solve it effectively. there also has been work utilizing graphs on patent data. @xcite form macroscopic graphs to perform patent (content) classification using entire patent documents, inventors, assignees, etc., as nodes. @xcite model published patents (grants) into "<entity, relation, en-@xmath0, but on a single hierarchical level and not constructed on the basis of individual claims. @xcite follow a similar approach to our graph construction, incorporating dependencies among elements in claims. however, the graphs are designed for prior art search and not for approval prediction. ## conclusions and future work in this paper, we delve into a domain-specific task, patent approval prediction, where simply scaling up the backbone model of previous sota falls short and simple customized graph methods work well. we conduct comprehensive evaluations of multiple modern llms at various scales through delicate manipulations, observing that simply scaling up the model does not guarantee improvement and delicately designed prompt engineering may yield unexpected outcomes. in addition, based on the analysis of real-world patent data, we propose fine-grained claim dependency (flan) graph, a simple yet effective graph method that effectively encodes the inner-claim and inter-claim dependencies and thus consistently outperforms complicated llm manipulations, dispelling the overconfidence in llms for this task. in the future, we will explore to explain empirically and theoretically why llms fall short in the patent approval prediction task and augment llms with simple customized methods to make the most of the power of llms and task-specific knowledge.