Three Real-World Datasets and Neural Computational Models for Classification Tasks in Patent Landscaping (2022)

Abstract: Patent Landscaping, one of the central tasks of intellectual property management, includes selecting and grouping patents according to user-defined technical or application-oriented criteria. While recent transformer-based models have been shown to be effective for classifying patents into taxonomies such as CPC or IPC, there is yet little research on how to support real-world Patent Landscape Studies (PLSs) using natural language processing methods. With this paper, we release three labeled datasets for PLS-oriented classification tasks covering two diverse domains. We provide a qualitative analysis and report detailed corpus statistics.Most research on neural models for patents has been restricted to leveraging titles and abstracts. We compare strong neural and non-neural baselines, proposing a novel model that takes into account textual information from the patents’ full texts as well as embeddings created based on the patents’ CPC labels. We find that for PLS-oriented classification tasks, going beyond title and abstract is crucial, CPC labels are an effective source of information, and combining all features yields the best results.

## introduction a patent is a public document granting the exclusive rights to an invention, e.g., a product or a process that provides a new technical solution to a problem. when entering new markets or developing new products, it is of utmost importance for organizations such as companies to be aware of the patent landscape, i.e., the existing patents with regard to their business endeavor in order to ensure their freedom to operate. experiments conducted with expert patent examiners in a context of a feasibility study on prior art search showed that while fully automating the process is infeasible, natural language processing (nlp) methods can help to significantly reduce time and cost by assisting patent examiners @xcite . most prior work on patent classification engages in the task of assigning labels from the international patent classification (ipc) taxonomy and its extension, the co-operative patent classification (cpc) taxonomy @xcite @xcite @xcite . this hierarchical multi-label classification task is challenging due to the large number of cpc codes, but lots of training data exists as each patent gets assigned cpc labels upon submission to the patent office. cpc classification provides a good testbed for developing representations of patents. however, plss are performed on a set of patent applications or granted patents with the aim of categorizing the documents according to a set of application-or business-oriented criteria, which may correspond to cpc categories only to a lim-ited extent. to clearly differentiate from prior work on cpc classification, we call this pls-oriented task the target classification task (see figure 1 ). in our setting, documents are already labeled with cpc labels, and thus these labels can be leveraged as one source of information. despite the expected major impact on the speed and accuracy of plss, nlp research on such pls-oriented target classification tasks has been hindered by the unavailability of public datasets with exemplary tasks. the first important contribution of this work is to release three real-world datasets from two diverse domains, providing a testbed for developing plsoriented classifiers. 1 we release a large manually curated patent corpus, which has been annotated with target labels related to injection valves during a time period of 20 years by domain experts of an industrial collaborator. in addition, we enrich two smaller document collections from wipo, created during real-world plss on hiv drugs, and define benchmark tasks on them. we provide a detailed analysis and corpus statistics, highlighting the difficult nature of the target classification tasks due to class imbalance and multi-label scenarios. our second main contribution is a computational study to tackle the target classification, comparing recently proposed neural and non-neural models that have been shown to be effective for cpc classification. building on the work of @xcite , and inspired by @xcite , we experiment with combinations of content-and label-based feature vectors. we generate scibert-based @xcite embeddings for the patents' title and abstract, claims, and description. to represent the semantics of cpc labels, we compare different approaches to generate embeddings based on a label co-occurrence graph and the label descriptions' texts. we find that using all textual fields as well as the cpc embeddings results in a robust method that works consistently well across our three pls datasets, outperforming all baselines. in sum, our contributions are as follows: ## related work we group our review of related work into patent classification, automated patent landscaping, and metadata-based patent document representations. patent datasets. for cpc classification, various datasets are available @xcite @xcite . similar to our target classification task, @xcite study classification for a patent alert system for the biochemical domain, but the dataset was not open-sourced. @xcite provide a patent dataset with human-written abstractive summaries. in the context of prior-art search, @xcite release a dataset mapping claims to prior-art passages. patent classification. early patent classification systems @xcite @xcite use a tf-idf feature vector exploiting the full document text. cnns @xcite , rnns (gru @xcite , and lstms @xcite ) have also been used to represent patent text. recently, pre-trained transformer-based models have been shown to be effective for patent classification @xcite @xcite . transformer-based models are constrained to a maximum length of 512 input tokens. increasing the maximum sequence length to 4096 tokens, @xcite propose big bird, a long-text transformer, and apply it to cpc classification using the concatenated text of title, abstract, and claims as input. as these approaches are inefficient @xcite and to date have shown only limited improvements over roberta @xcite for patent classification, we leave research on long-transformer methods to future work. ## patent landscaping: task and datasets in this section, we propose a target classification task to support pls, and introduce three new data sets from diverse domains (mechanical systems and biochemistry). we have curated one dataset from in-house annotations of patents in the domain of injection valves and compiled two datasets from freely available wipo patent landscape studies. all datasets are publicly available for future benchmarking in a convenient format. dataset statistics are provided in ## computational models in this section, we describe our computational models for predicting target categories for patents based on their text and cpc labels. we first introduce the representations of the patent text (section 4.1), as well as the generation of embeddings for cpc labels (section 4.2), and then describe the classifier used on top of them (section 4.3). ## experiments in section 4, we have introduced several content and label embeddings for pls target classification tasks. our experiments described in this section aim to identify a patent document representation that works robustly across plss. we analyze the performance of different embeddings when considered individually or in combination (section 5.3), and compare it to strong baselines (section 5.4). finally, as an important analysis in the context of plss, in section 5.5, we address the question of how many labeled training examples are necessary for training a pls target task classifier. details about hyperparameters for our proposed approach are provided in appendix c. ## conclusions and future work to foster research in the field of automating plss, we have introduced the new task of target label classification and released three real-world datasets. we have compared various neural and non-neural methods with different input representations covering the patents' texts and cpc information. as a result, we propose a competitive neural patent classification model, which leverages both patent-text and the cpc label information, and which shows robust performance across all three datasets. we found that an acceptable performance in terms of micro-f1 can be reached with only 200 to 300 training instances, demonstrating the practical applicability of the approach. in order to improve performance for infrequent classes, integrating our methods with active learning of few-shot techniques are potential future directions. our datasets also provide a valuable testbed for future work on neural representations for long and structured text documents.