Connecting the Dots: Inferring Patent Phrase Similarity with Retrieved Phrase Graphs (2024)

Abstract: We study the patent phrase similarity inference task, which measures the semantic similarity between two patent phrases. As patent documents employ legal and highly technical language, existing semantic textual similarity methods that use localized contextual information do not perform satisfactorily in inferring patent phrase similarity. To address this, we introduce a graph-augmented approach to amplify the global contextual information of the patent phrases. For each patent phrase, we construct a phrase graph that links to its focal patents and a list of patents that are either cited by or cite these focal patents. The augmented phrase embedding is then derived from combining its localized contextual embedding with its global embedding within the phrase graph. We further propose a self-supervised learning objective that capitalizes on the retrieved topology to refine both the contextualized embedding and the graph parameters in an end-to-end manner. Experimental results from a unique patent phrase similarity dataset demonstrate that our approach significantly enhances the representation of patent phrases, resulting in marked improvements in similarity inference in a self-supervised fashion. Substantial improvements are also observed in the supervised setting, underscoring the potential benefits of leveraging retrieved phrase graph augmentation.

## introduction patents are pivotal to the landscape of innovation, safeguarding novel ideas and fostering technological advancements @xcite . consequently, understanding patent phrase similarity becomes essential, offering insights into the nuances of intellectual property and aiding in the patent analytics applications, such as patent document categorization, patent retrieval, patent litigation analysis and so on @xcite @xcite . for example, the united states patent and trademark office (uspto), in its mission to evaluate and grant patents, could leverage the patent phrase similarity task to streamline the examination process, identify prior art more efficiently, and ensure the distinctiveness of newly filed patent applications @xcite . patent documents employ legal and highly technical language, featuring context-dependent terms that can deviate significantly from colloquial usage and may vary between different documents. for instance, while a common term like "smartphone" might be easily recognized in everyday language, in patent documents it might be referred to as a "handheld electronic communication device", "mobile telecommunication apparatus", or even a "portable digital assistant with wireless capabilities". consequently, prevailing semantic textual similarity approaches, such as sentence-bert @xcite or simcse @xcite , which focus on general text and rely on localized contextualized information for text representation, fall short in inferring patent phrase similarity. additionally, obtaining a substantial collection of annotations from experts for supervised training presents significant challenges: the process is not only costly but also demands in-depth domain knowledge of the patent innovation landscape. the intricate interplay between technically nuanced terms and the scarcity of labels renders the task especially challenging. a significant research gap persists: how can one effectively infer phrase similarity within the complex language of patents, especially in the absence of training labels? to address this challenge, we introduce a retrieval-based graph augmentation method to effectively capture phrase representations. our approach is inspired by the strategies employed by real-world patent experts. for a given patent phrase, we begin by extracting a subgraph from the vast patent universe, such as those registered in the uspto patent database. the derived subgraph comprises two node types: the phrase node and the patent node. a connection exists between a phrase node and a patent node if the phrase is present in the patent, and patent nodes interlink based on citation relationships. hence, for a patent phrase such as "handheld electronic communication device", this subgraph might encompass related patents related to "portable telecommunication gadget". concurrently, it could also surface interrelated phrases such as "wireless signal transceiver" and "handheld digital communicator". such related patents and phrases provide a broader context, enabling a deeper understanding of the focal phrase. the information from its focal patent serves as the local context, whereas the extracted phrase graph offers a global context. this extracted phrase graph is subsequently processed through a graph attention network (gat) @xcite to obtain its representative embedding. the final contextualized embedding of a patent phrase is a combination of its textual contextualized embedding and its associated phrase graph embedding. to address the issue of label scarcity, we utilize a self-supervised learning objective that capitalizes on the phrase graph's topology, facilitating the training of both the textual contextualized embedding and the graph learning parameters in an end-to-end manner. we evaluate the proposed retrieval augmented patent phrase similarity (ra-sim) on a large patent phrase similarity dataset @xcite . in comparison with existing textual semantic similarity approaches-such as word embeddings (word2vec @xcite , glove @xcite , fasttext @xcite ), contextualized embeddings (bert @xcite , roberta @xcite ), and semantic similarity embeddings (sentence-bert @xcite , contriever @xcite , simcse @xcite )-our ra-sim method achieves substantial improvements on inferring patent phrase similarities in a self-supervised manner. ablation studies and additional analyses further elucidate how ra-sim enhances the patent phrase similarity inference task. moreover, ra-sim, when evaluated in supervised learning setting, consistently outperforms state-of-the-art methods. this highlights the potential advantages of incorporating retrieved graph for contextualized embedding augmentation. we release the codes for ra-sim, enabling innovation and patent research scholars and practitioners to integrate it into their analytics pipeline. ## related work our work is related to several lines of literature. ## retrieval augmented patent phrase similarity patent phrases typically employ legal and highly technical language and may deviate significantly from colloquial usage. moreover, patent phrases are typically short, containing only a few tokens. this brevity makes it even more challenging to meaningfully represent them due to the lack of contextual information. for example, consider the technical phrases "acid absorption" and "chemically soaked" from the patent phrase similarity dataset released by google @xcite . even though these two phrases don't share any words, patent experts rate them as domainrelated, assigning a similarity score of 0.25 on a scale of [0,1]. conversely, the phrases "acid absorption" and "acid reflux" are rated as "not related" with a similarity score of 0 foot_0 . ## experiment we evaluate proposed ra-sim method on patent phrase similarity dataset @xcite , which is curated by google and rated by domain experts in patents. ## analysis we conduct further analyses to understand the inner workings of self-supervised ra-sim. ## conclusion we explore the challenge of inferring patent phrase similarity, a task distinct from classic semantic textual similarity due to the highly technical language inherent to the patent domain. in this study, we introduce a retrieval-augmented phrase similarity method, termed ra-sim. this method enhances the contextualized textual information of a patent phrase by incorporating a phrase graph that encompasses its focal patent, related patents via the citation network, and associated phrases through these related patents. ra-sim is trained using self-supervised learning objectives centered on the phrase graph. experimental results demonstrate ra-sim's superior performance, highlighting the benefits of retrieval augmentation for contextualized embeddings in domain problems.