D esign CLIP : Multimodal Learning with CLIP for Design Patent Understanding (2025)

Abstract: In the field of design patent analysis, traditional tasks such as patent classification and patent image retrieval heavily depend on the image data. However, patent images—typically consisting of sketches with abstract and structural elements of an invention—often fall short in conveying comprehensive visual context and semantic information. This inadequacy can lead to ambiguities in evaluation during prior art searches. Recent advancements in vision-language models, such as CLIP, offer promising opportunities for more reliable and accurate AI-driven patent analysis. In this work, we leverage CLIP models to develop a unified framework DesignCLIP for design patent applications with a large-scale dataset of U.S. design patents. To address the unique characteristics of patent data, DesignCLIP incorporates class-aware classification and contrastive learning, utilizing generated detailed captions for patent images and multi-views image learning. We validate the effectiveness of DesignCLIP across various downstream tasks, including patent classification and patent retrieval. Additionally, we explore multimodal patent retrieval, which provides the potential to enhance creativity and innovation in design by offering more diverse sources of inspiration. Our experiments show that DesignCLIP consistently outperforms baseline and SOTA models in the patent domain on all tasks. Our findings underscore the promise of multimodal approaches in advancing patent analysis. The codebase is available here: https://github.com/AI4Patents/DesignCLIP.

## introduction patents are units for the state of the art innovation, designs and technological advancements as well as they offer legal protection for inventors' intellectual property @xcite . patents grant the owner the authority to prevent others from manufacturing, utilizing, or distributing the patented invention without their permission foot_0 . among the two popular types, utility patents are granted for innovations in processes, machines, manufactures, or compositions of matter, including improvements, design patents are awarded for new, original, and ornamental designs applied to manufactured items. while utility patents have been well studied @xcite @xcite , design patents remain relatively underexplored. one of the most important design patent tasks is patent retrieval. this aims to determine the novelty of the patent and prevent infringements. a patent is only granted if the design significantly differs from existing ones. most previous research @xcite focus primarily on image-to-image retrieval since there is a lack of informative text descriptions in design patents. patent classification @xcite is another task by patent reviewers who classify applications into various subject matters and assign design classification codes. in the us design patent system, there are 33 classes which also contain various subclasses. automating the patent classification process can save an enormous amount of time. solving the above tasks would need sophisticated multimodal techniques due to several challenges. design patent images are sketches that provide detailed design information and differ significantly from natural images (please see the examples of patent designs in figure 2 ). additionally, the class distribution in the design patent dataset is highly imbalanced, with the six most frequent classes accounting for almost half of the total data (figure 1 ). furthermore, the aim is to build one single model that can perform the patent classification as well as retrieval. there are currently no multimodal models that are designed to address these challenges and specifically tailored for design patents. our contributions. in this paper, we develop a new clip-based framework to facilitate multimodal analyses on design patents. our major contributions are as follows. ## background on patent analysis the patent retrieval task focuses on efficiently retrieving relevant patent documents and images based on search queries. in design patents, this task is focused on image-to-image retrieval, where the objective is to find visually similar design images that match a given image query. @xcite implement various models such as resnet50 @xcite , and sketchy rn50 @xcite . their patent-specific models are initially pre-trained on imagenet @xcite and fine-tuned on the deeppatent dataset. similarly, @xcite use a deep metric learning framework, utilizing crossentropy methods like infonce with arcface. on the other hand, @xcite implement several advanced models, such as vit @xcite and swin @xcite , along with more recent multimodal models such as blip-2 @xcite , and gpt-4v @xcite . they introduce a novel approach by proposing a languageinformed strategy for learning features from patent images. please refer to the survey (shomee et al., 2025) for more details. notably, there is currently no specific machine learning or ai-based research focused on the critical constraints (e.g., classes and multi-views) from the design patents and we address this in this paper. in this paper, we mainly focus on design patents using impact dataset and address the two primary tasks associated with the design patents. we demonstrate the design patent retrieval task in three different formats: text-to-image, image-to-text, and image-to-image. additionally, the classification of design patent images into 33 subject matter (class) and various subclasses is a detailed and structured approach to organizing design patents. we address the classification task by categorizing patents into their class level categories using both text-based captions and images. ## designclip in this work, we address the challenges of design patent analysis by adapting the clip visionlanguage model family. the major components of our framework, designclip, are as follows: first, considering the long tail distribution among the classes of design patents, we begin with proposing a class-aware learning method to resample construct pairs in the batch. this helps our method designclip to balance learning from all the classes. then, we pre-train designclip with multiple tasks, including patent classification, imagecaption contrastive loss, and multi-view imageimage contrastive loss. our architecture is illustrated in figure 2 . brief review of clip framework: clip (radford et al., 2021), a multimodal model uses contrastive learning to bridge the gap between text and images. at its foundation is the principle of acquiring perceptual understanding from the guidance of natural language. during its pre-training phase, clip learns to identify if a text snippet and an image are matched in its dataset. the training involves a series of five resnets and three vision transformers to facilitate zero-shot classification. clip has been popular across various fields including medical imaging @xcite @xcite , robotics @xcite @xcite , biodiversity monitoring @xcite , e-commerce (hendriksen et al., 2022), educational technology @xcite and showed great performance. the popular clip @xcite is a pre-trained model to learn image-text pairs with contrastive loss. this approach enables the model to differentiate between semantically similar and dissimilar data points, enhancing its ability to capture meaningful relationships between visual and textual information. clip is widely applied for a variety of multimodal tasks, including zero-shot classification, and multimodal search. specifically, given an image, clip can retrieve the most relevant text descriptions (e.g., captions or labels). given image-text pairs {(v i , t i )} n @xmath0, the vanilla loss function in clip is defined as: ## conclusions in this paper, we introduce designclip to provide a domain-aware multimodal model for design patents. we first consider class-aware sampling and contrastive learning for the long-tail distribution in design patent data. in addition, multi-view image-image contrastive learning provides a comprehensive understanding of the patents from different angles, which can improve the performance in patent retrieval. finally, we pre-train clip-based models on multi-tasks tailored for design patents. our designclip outperforms the baseline models on patent classification, image retrieval and multimodal retrievals. these demonstrate that de-signclip can provide a better understanding and generalization ability for design patents in multimodal scenarios.