Paper: Mining Patents with Large Language Models Demonstrates Congruence of Functional Labels and Chemical Structures (ICLR-2024)

Reviewer comment (Experiments):
It seems this paper only designs a dataset collection pipeline and no benchmark works are involved. For example, a complete benchmark work should include additional evaluations of some baseline methods. This reviewer thinks only the dataset construction contribution is not sufficient for publishing in ICLR.

Author response:
We understand your concerns on the focus of our paper, and we believe they reflect that we did not properly convey our work in the original submitted manuscript. Previously, we had focused too specifically on the dataset creation, when that was just a means to functional label-guided drug discovery which is, to our knowledge, a novel contribution to the field. To correct this shortcoming, we have overhauled the presented story to better explain the results of our contributions. You will find many text changes to the Abstract, Introduction, Results, and Discussion.

We believe our work serves instead to help build the foundations of functional label-guided drug discovery. First, we introduce the importance of focusing on chemical function, rather than structural nomenclature or miscellaneous terms. This is demonstrated with the creation of our Chemical Function dataset. Then, we introduce the concept of the text-based chemical function landscape, which was found to be highly structured by semantic functional categories (Figure 3). This text-based functional landscape aligned with chemical structural relationships, meaning that it adequately approximated the actual chemical function landscape, implicitly capturing complex physical and biochemical interactions (Figs. 2, 4). We then demonstrate that this functional landscape can be harnessed through a model trained to predict entire chemical functional profiles from structure alone. This model was able to comprehensively assign probabilities to every possible function that a given chemical may have, something that we do not believe has been done prior to our work. This has immense utility for experimentalists (hypothesis generation), pharmaceutical companies (discovery & repurposing), environmental agencies (off-target effects), and electronics manufacturers (photochemical discovery) to name a few.

We do realize that some aspects of technical rigor were not present in our original draft. To remedy this, we include additional supplemental figures analyzing the bias of the dataset (Fig. S3) and a benchmarking of molecular fingerprint-based models on the CheF dataset (Fig. S7).

We hope that you will find the revised manuscript and story satisfactory. We firmly believe that the work presented here is beneficial for the ML chemistry / biochemistry researchers attending ICLR, rather than a general chemistry audience.

> [...] a complete benchmark work should include additional evaluations of some baseline methods. 

This is a great suggestion, and the exclusion of such was an oversight on our end as we were focused on the analysis and practical utility of functional label-guided drug discovery rather than dataset creation. To address this, we have added evaluations of several baseline methods on the CheF dataset that may be found in Table S7. From this benchmark process, we found out our previous model was suboptimal. We now use a better model (multi-label logistic regression), and we have redone the relevant downstream analyses.

Reviewer comment (Writing):
Although the problem formulation seems very straightforward (functional label prediction), it still lacks a paragraph to explicitly explain the problem formulation. And need one more paragraph to explain why functional label prediction is an important task (naturally motivated by what application scenarios?).

Author response:
This is a good point. Our revised paper now has an explicit problem formulation paragraph at the end of the Introduction and beginning of Results. Additionally, we have spent considerably more time in the Introduction explaining why functional label-guided drug discovery is important. In particular, we discuss pitfalls in structure-based methods predicting whole-body effects and the impact of Gene Ontology to genome understanding and the lack of an equivalent knowledge source for chemicals. We have also better clarified all of the practical applications exhibited in the section Functional label-guided drug discovery. Additionally, we have added more on applications in the discussions: “Models in this paradigm have the potential to automatically annotate chemical function, examine non-obvious features of drugs such as side effects, and down-select candidates for high-throughput screening”.

Reviewer comment (Experiments):
This reviewer thinks this paper might not be very suitable for the ICLR conference venues (machine learning conference). It is probably more suitable for the domain journal or other conference benchmark tracks if baseline evaluations are further included. This reviewer encourages the authors to make the dataset public. This reviewer thinks probably the most influential dataset format is maintaining the multi-modality, which means including the raw patent text tokens.

Author response:
We understand your concerns on the focus of our paper, and we believe they reflect that we did not properly convey our work in the original submitted manuscript. Previously, we had focused too specifically on the dataset creation, when that was just a means to functional label-guided drug discovery which is, to our knowledge, a novel contribution to the field. To correct this shortcoming, we have overhauled the presented story to better explain the results of our contributions. You will find many text changes to the Abstract, Introduction, Results, and Discussion.

We believe our work serves instead to help build the foundations of functional label-guided drug discovery. First, we introduce the importance of focusing on chemical function, rather than structural nomenclature or miscellaneous terms. This is demonstrated with the creation of our Chemical Function dataset. Then, we introduce the concept of the text-based chemical function landscape, which was found to be highly structured by semantic functional categories (Figure 3). This text-based functional landscape aligned with chemical structural relationships, meaning that it adequately approximated the actual chemical function landscape, implicitly capturing complex physical and biochemical interactions (Figs. 2, 4). We then demonstrate that this functional landscape can be harnessed through a model trained to predict entire chemical functional profiles from structure alone. This model was able to comprehensively assign probabilities to every possible function that a given chemical may have, something that we do not believe has been done prior to our work. This has immense utility for experimentalists (hypothesis generation), pharmaceutical companies (discovery & repurposing), environmental agencies (off-target effects), and electronics manufacturers (photochemical discovery) to name a few.

We do realize that some aspects of technical rigor were not present in our original draft. To remedy this, we include additional supplemental figures analyzing the bias of the dataset (Fig. S3) and a benchmarking of molecular fingerprint-based models on the CheF dataset (Fig. S7).

We hope that you will find the revised manuscript and story satisfactory. We firmly believe that the work presented here is beneficial for the ML chemistry / biochemistry researchers attending ICLR, rather than a general chemistry audience.

> This reviewer encourages the authors to make the dataset public.

We are committed to open science, and believe the best way to increase the development of functional label-guided drug discovery is to have the data accessible for other researchers to build on. The CheF database is already public and may be downloaded at the provided Zenodo link in the Reproducibility Statement section.

> This reviewer thinks probably the most influential dataset format is maintaining the multi-modality, which means including the raw patent text tokens.

The provided Zenodo link contains code to fetch all of the relevant patent data given a SMILES string / Patent ID that the research community may find useful.

However, we believe that high-quality functional labels are at present more powerful than raw patent text. We address this in the rewritten paragraph on text-to-chemical translation in Related Work: “Many existing chemical-to-text translation models have been trained on datasets containing structural nomenclature and irrelevant words mixed with desirable functional information (Edwards et al., 2021; Degtyarenko et al., 2007). Inclusion of structural nomenclature causes inflated prediction metrics for functional annotation or molecular generation tasks, as structure-to-name and name-to-structure is simpler than structure-to-function and function-to-structure. The irrelevant words may cause artifacts during the decoding process depending on the prompt, skewing results in ways irrelevant to the task. In our work, we ensured our model utilized only chemical structure, and not structural nomenclature, when predicting molecular function to avoid data leakage”.

Reviewer comment (Novelty):
Will the functional label prediction be a simple task for deep learning approaches? This reviewer conjectures that different functional molecules can have very different structures, which might be easy for deep learning approaches to distinguish between them.

Author response:
The task of functional label prediction is actually quite nuanced and poses significant challenges for deep learning approaches. The function of a molecule is determined by its interacting partner (e.g., photons, specific receptors, or the organization of a signaling cascade). This means that models trained to predict functional labels need to approximate the chemical function landscape, a massively high-dimensional space encoding complex interaction relationships. This is addressed several times in the updated manuscript as it is central to our work in the revised story.

The diversity and complexity of molecular structures means that molecules with different functions can have subtle and nuanced structural differences, making them challenging to distinguish. And conversely, different molecules can have the same function despite drastically different structure. An example of the former is the drug Thalidomide, which is either therapeutic (prevents morning sickness) or teratogenic (causes birth defects) depending on the stereoisomer (specific handedness of one of the bonds, holding all else constant). And an example of the latter is that there are many distinct mechanisms for a drug to be an antibiotic. This can be seen in beta-lactam antibiotics and tetracyclines both working as antibiotics despite radically different structure and mechanisms. We have added discussion of these points to the manuscript: “This rationale generally holds, but exceptions include stereoisomers with different functions and distinct structures sharing the same function.”

We believe that deep learning approaches provide an opportunity to solve the challenging problem of chemical function prediction, but this requires high-quality uncontaminated data with a model fit suited to the problem, both of which we have provided.

Reviewer comment (Evaluation):
The paper assumes that chemical structures with similar functionality should cluster close to each other in the structural space based on molecular fingerprints; however, this doesn’t necessarily have to be the case—you can have stereoisomers with different functional properties, and you can have chemical compounds with different chemical structure and similar labels. Some of this is visible in the cluster analyses, where molecules belonging to the same functional class are not clustered together. Perhaps a different metric to assess congruence of the two spaces is needed.

Author response:
This is an excellent point. It is indeed true that chemical structures with similar functionalities do not always cluster closely in structural space, and conversely that structurally similar molecules may not always share functionality. An example of the former, as you pointed out, is the label HCV (Hepatitis C Virus) showing distinct clusters depending on mechanism of action (see Figure S3). A well-known example of the latter effect is the drug Thalidomide, which is either therapeutic or teratogenic depending on the stereoisomer. 

Despite this, derivatives of active drugs will often maintain their activity provided that the active moieties are maintained. This can be seen across the broad range of beta-lactam antibiotics. It is because of this generally-true property that patent offices allow for the patenting of molecules and their derivatives. And because the CheF dataset was built from patent data, we should expect to see clustering in structure space above random chance. However, we should not expect to see a single cluster for all molecules containing a given label, as some terms have multiple distinct mechanisms (i.e., beta-lactam antibiotics & tetracyclines both act as antibiotics despite different structure). Our metric accounts for this, as it measures distance to the nearest molecule with a shared label rather than measuring number of clusters. 

We have further clarified the rationale for our introduced hypothesis, in which we add “Due to molecules and their derivatives being patented together, structurally similar molecules should be annotated with similar patent-derived functions. This rationale generally holds, but exceptions include stereoisomers with different functions and distinct structures sharing the same function”.

Reviewer comment (Novelty):
How is this effort different from recent work on molecular generation from chemical subspaces derived from patents containing specific functional keywords (Subramanian, 2023)?

Author response:
We have rewritten the paragraph comparing our work to Subramanian 2023 to better clarify commonalities and differences. The revised paragraph reads as follows:

“Recent work also focused on molecular generation from chemical subspaces derived from patents containing specific functional keywords, for example, all molecules relating to tyrosine kinase inhibitor activity (Subramanian et al., 2023). This allows for a model that can generate potential tyrosine kinase inhibitors but would need to be retrained to predict molecules of a different functional label. In our work, we focus on label classification rather than molecular generation. Further, we integrate multiple functional labels for any given molecule, allowing us to broadly infer molecular functionality given structure. Generative models could be trained on the described dataset, allowing for label-guided molecular generation without re-training for each label.”

Reviewer comment (Reproducibility):
What is the classification model used to go from molecular fingerprint to functional label?

Author response:
In the original paper, we used a feedforward neural network with two hidden layers (dimensions 512, 256 respectively), to output a 1x1,543 vector (size of vocabulary). This was determined using a 5-fold CV.

However, we have since benchmarked additional models on the CheF dataset and found that a multi-label logistic regression model on molecular fingerprints actually worked significantly better. We redid the downstream analyses using this updated model. You can find model training details in the Methods under Model Training. We have also clarified mention of which model is being discussed throughout the paper.

Reviewer comment (Presentation):
Detailed figure of the pipeline (may be with a brief example of a patent) may be easier for readers understanding (Figure 1).

Author response:
We recognize the importance of visual aids for those unfamiliar with patents. To aid this, we have added a figure outlining the summarization process for a specific example which may be found in Figure S2.

Reviewer comment (Experiments):
Can more validations be carried out by searching over the pubmed?

Author response:
In our paper, we use PubMed articles to confirm our model-derived hypothesis of the Hepatitis C Virus antiviral’s mechanism of action (Section 3.4). If you were instead referring to the idea that we should expand our data to PubMed-associated molecules, we completely agree and thank you for the suggestion. To note this in the paper, we have added a sentence to the discussion that reads “Additionally, by restricting the dataset to chemicals with <10 patents, it neglects important well-studied molecules like Penicillin. The inclusion of over-patented chemicals could be accomplished by using only the most abundant k terms for a given molecule, using a fine-tuned LLM to only summarize patents relevant to molecular function (ignoring irrelevant patents on applications like medical devices), or employing other data sources like PubChem or PubMed to fill in these gaps”. We aim to pursue this idea in future work.

Reviewer comment (Evaluation):
In using LLMs, were there any bias in the generated summarizations?

Author response:
There were no obvious biases in the generated summarizations. To confirm this, we analyze the distribution of the generated (uncleaned) labels based on abundance. These appear to reflect underlying biases in the patent literature, and more broadly pharmaceutical incentives, rather than from the LLM. This analysis may be found in Figure S3.

Reviewer comment (Theory):
What was the reason for using the Tanimoto similarity?

Author response:
Tanimoto similarity is the standard method to compute similarity between binary molecular fingerprint vectors, see Dávid Bajusz (2015). This allows us to compute similarity whether or not the molecule contains a particular substructure. If we had been using float-valued vector representations (i.e., learned molecular embeddings), we would have opted for a more fitting similarity metric such as cosine similarity or Euclidean distance. To clarify this, we have added a citation of the metric at its first mention.

Reviewer comment (Reproducibility):
How long (in terms of time) does the whole framework take?

Author response:
The framework is largely limited by the API calls to retrieve a patent and summarize it with ChatGPT. We report the results of this in the Methods section: “Fetching patent information and summarizing with ChatGPT, this method’s bottleneck, took 6 seconds per molecule with 16 CPUs in parallel. This could be sped up to 3.9 seconds by summarizing per-patent rather than per-molecule to avoid redundant summarizations, and even further to 2.6 seconds by using only US and WO patents”.

Reviewer comment (Novelty):
From a technological perspective, the paper's contribution appears somewhat restrained. The use of ChatGPT for text summarization is not a novel concept. As such, the manuscript might find a more fitting audience in journals or conferences with a chemistry focus.

Author response:
We acknowledge your concerns on the scope of our paper and believe it means that we did not properly convey our work in the original submitted manuscript. Previously, we had focused too specifically on the dataset creation, when that was just a means to functional label-guided drug discovery which is, to our knowledge, a novel contribution to the field. To correct this, we have edited each section to better relate our contributions. 

We believe our work helps build the foundations of functional label-guided drug discovery. First, we introduce the importance of focusing on chemical function, rather than structural nomenclature or miscellaneous terms. This is demonstrated with the creation of our Chemical Function dataset. Then, we introduce the concept of the text-based chemical function landscape, which was found to be highly structured by semantic functional categories (Figure 3). This text-based functional landscape aligned with chemical structural relationships, meaning that it adequately approximated the actual chemical function landscape, implicitly capturing complex physical and biochemical interactions (Figs. 2, 4). We then demonstrate that this functional landscape can be harnessed through a model trained to predict entire chemical functional profiles from structure alone. This model was able to comprehensively assign probabilities to every possible function that a given chemical may have, something that we do not believe has been done prior to our work. This has utility for experimentalists (hypothesis generation), pharmaceutical companies (discovery and repurposing), environmental agencies (off-target effects), and electronics manufacturers (photochemical discovery) to name a few.

We do realize that some aspects of technical rigor were not present in our original draft. To remedy this, we now include additional supplemental figures analyzing the bias of the dataset (Fig. S3) and a benchmarking of molecular fingerprint-based models on the CheF dataset (Fig. S7).

We hope that you will find the revised manuscript satisfactory. We believe that the work is beneficial for the ML chemistry / biochemistry researchers attending ICLR, rather than a general chemistry audience.

Reviewer comment (Experiments):
There are concerns regarding the accuracy of labels generated by ChatGPT. While 98.2% of the labels were found valid, solutions for addressing the remaining 1.8% are not discussed. Such inaccuracies could introduce noise into the CheF dataset.

Author response:
We agree that despite the high base accuracy of 98.2%, addressing the 1.8% of inaccuracies is indeed important. It is likely that fine-tuning an LLM on high-quality patent-extract functional data will increase this accuracy even higher, thereby enhancing the overall quality and reliability of the CheF dataset. A sentence has been added to the discussion to note this: “Increasing label quality and ignoring extraneous claims might be achieved through an LLM fine-tuned on high-quality examples”.

Reviewer comment (Experiments):
The selection criterion that omits molecules associated with more than 10 patents suggests the dataset may be missing data on prevalent molecules. The absence of these molecules might limit the dataset's reach and implications.

Author response:
We acknowledge the concern for potential limitations that the patent selection criterion imposes. The selection criterion was designed to maintain dataset quality, and we believe it is not as restrictive as it first appears. The SureChEMBL database we used contains 32,005,106 molecules, and our criterion reduces this to 28,207,428 molecules, only a 12% decrease (these numbers have been added to Results and Methods). This means that a high proportion of chemical space is still included, but this does not solve the problem of omitting the well-studied molecules. This is addressed in the discussion as follows: “The inclusion of over-patented chemicals could be accomplished by using only the most abundant k terms for a given molecule, using a fine-tuned LLM to only summarize patents relevant to molecular function (ignoring irrelevant patents on applications like medical devices), or employing other data sources like PubChem or PubMed to fill in these gaps.”

Reviewer comment (Experiments):
A comparative analysis with other established molecule-text datasets like ChEBI and PubChem would be beneficial. Additionally, the paper could emphasize the practical applications of the CheF dataset, especially its potential role in drug discovery, to underscore its unique advantages.

Author response:
We agree that a comparative analysis with similar datasets would add value to the paper, and thank the reviewer for the good suggestion. This comparison can be found in Table S3 of the revised manuscript. This table highlights the CheF dataset’s unique aspects in comparison to existing chemical function datasets.

We agree that emphasizing practical applications is important in comparison to existing methods. To better emphasize this point, we have made several changes to the paper. In the introduction, we now contrast our text-based paradigm to existing structure-based drug discovery methods, noting that our methodology utilizes the massive corpus of chemical literature. Secondly, we have clarified and re-written the section Functional label-guided drug discovery. In this section, we outline the unique ability of models in our proposed paradigm, using the CheF dataset, in several practical drug discovery roles: antiviral mechanism discovery, identifying potentially novel serotonin drugs, and discovering possible hepatitis C virus antivirals that could be repurposed from existing FDA approved drugs. We hope that this clarifies the practical applications of our work.

Reviewer comment (Evaluation):
In the Section 3.1 (FUNCTIONAL LABELS MAP TO NATURAL CLUSTERS IN CHEMICAL STRUCTURE SPACE) part, the (a) (e), and (g) don't appear to distinctly exhibit clustering. Are there more significant examples of other labels?

Author response:
We acknowledge that Figures 2(a), 2(e), and 2(g) do not visibly exhibit strong clustering. However, the degree of clustering was measured, and it is shown in Figures 2(b), 2(d), 2(f), & 2(h), where each example was found to have significant clustering compared to the background. We believe that lack of visual clustering is an artifact of the t-SNE projection on molecular fingerprint vectors. A closer examination of the clusters in Figure 2(a) is shown in Figure S3, in which we found distinct modes of action for the various clusters of molecules containing the label HCV (Hepatitis C Virus).

To help aid the reader, we have added additional examples of strongly visually clustering labels which may be found in Figure S5. However, we believe that keeping HCV, 5-HT, and Serotonin cluster examples in the main text helps with consistency as analyses on these labels occur throughout the paper. Further cluster visualization may be found at https://chefdb.app/.