Repurformer: Transformers for Repurposing-Aware Molecule Generation (2024)

Abstract: Generating as diverse molecules as possible with desired properties is crucial for drug discovery research, which invokes many approaches based on deep generative models today. Despite recent advancements in these models, particularly in variational autoencoders (VAEs), generative adversarial networks (GANs), Transformers, and diffusion models, a significant challenge known as the sample bias problem remains. This problem occurs when generated molecules targeting the same protein tend to be structurally similar, reducing the diversity of generation. To address this, we propose leveraging multi-hop relationships among proteins and compounds. Our model, Repurformer, integrates bi-directional pretraining with Fast Fourier Transform (FFT) and low-pass filtering (LPF) to capture complex interactions and generate diverse molecules. A series of experiments on BindingDB dataset confirm that Repurformer successfully creates substitutes for anchor compounds that resemble positive compounds, increasing diversity between the anchor and generated compounds.

## introduction the design of valid and novel molecules with desired biological properties, known as de novo molecule generation, is vital to modern drug discovery. recent advancements in deep generative models, particularly variational autoencoders (vaes) @xcite , generative adversarial networks (gans) @xcite , transformers @xcite , and diffusion models @xcite , have significantly enhanced our ability to generate chemically valid and novel molecules. however, these models need to be further refined to generate molecules that interact with specific target proteins. target-specific molecule generation addresses this challenge by producing drug-like molecules that are more likely to bind with specific target proteins @xcite @xcite . nonetheless, there remains a significant issue known as the sample bias problem, where reliance on existing protein-compound pairs results in the generation of structurally similar molecules. this phenomenon limits the diversity of generated molecules and hinders the discovery of novel compounds. to address this, we propose leveraging multi-hop relationships among proteins and compounds to expand the generative space and increase the diversity of the generated molecules. our method introduces the concept of repurposing-aware molecule generation, designed to identify and utilize latent multi-hop relations within the protein-compound interaction network. in this paper, we present repurformer, a novel model that integrates bi-directional pretraining and advanced signal processing techniques to overcome the limitations of existing models. repurformer captures complex relationships between proteins and compounds by pretraining encoders in both @xcite with low-pass filtering (lpf) to the latent space. this approach allows the model to distinguish the different scales of interactions. by focusing on lowfrequency components, which correspond to the longer propagation through the multi-hop proteincompound interaction network, repurformer generates as diverse compounds as possible with desired properties. in summary, the contributions of our work are threefold: ## problem statement the discovery of new compounds often relies on existing protein-compound pairs. this results in that the compounds targeting the same protein exhibit similar structures. in other words, the generative space of models tends to be bounded in limited regions, reducing the diversity of the generation. we refer to this as a sample bias problem. to address this problem, we leverage multi-hop relations among proteins and compounds. specifically, given a pair of protein p and compound c that are known to interact, we assume that the compound relates to p within a 3-hop relation, i.e., a positive compound c + , has a potential interaction with p. definitions from 3.1 to 3.3 describe the key concepts of our approach, and figure 1 visually represents the rationale. note that both protein and compound are represented by 118 amino-acid and smiles sequences, respectively: @xmath0, • • • , p tp ] and @xmath1, • • • , c tc ] with t p and t c being the fixed length of each sequence. definition 3.1 (protein @xcite . the relations between proteins and compounds can be represented as a bipartite graph g(p ∪ c, e), where p and c denote the sets of protein and compound nodes, respectively. specifically, p (i) ∈ p represents the i-th protein and c (j) ∈ c represents the j-th compound, for the presence of an edge e ij indicates a link between the i-th protein and the j-th compound, such that e @xmath2. ## repurformer in this section, we propose repurformer, a novel method designed to address the sample bias problem by leveraging multi-hop relations among proteins and compounds. figure 2 illustrates how repurformer seamlessly integrates the concepts of drug discovery and repurposing. ## experiments experiment setup we collected data from bind-ingdb @xcite which contains over 2.8 million measured binding affinities of interactions between proteins and drug-like molecules. the collected dataset was then preprocessed to filter out missing values, duplicates, and proteins and compounds with excessively long or short sequences. in particular, given the many-to-many nature of protein-compound relationships, we selected compounds that interact with a reasonable number of individual proteins between 10 and 100, to enable our model to learn various compound structures reacting with different proteins. the resulting dataset comprised 60,719 protein-compound pairs derived from 3,006 proteins and 7,803 compounds. we split this dataset into train and test datasets with 8:2 ratio, ensuring that the proteins interacting with each compound did not overlap between the two sets. our model was then trained on proteincompound pairs from the train set, representing proteins with amino acid sequences and compounds with canonicalized smiles strings. we tokenized individual characters from amino acid sequences and smiles strings, resulting in vocabularies containing 30 characters for proteins and 46 characters for compounds. evaluation metrics to thoroughly assess the effectiveness and reliability of repurformer, we employed several evaluation metrics, focusing on the generative performance of the model and physicochemical properties and drug-likeness of the molecules it generated. in terms of generative performance, we applied widely accepted metrics for sequence generation tasks: bleu @xcite , gleu @xcite , and f1 score of rouge @xcite . in particular, we used 1-and 2-gram units as the evaluation basis for these generative metrics. we utilized physicochemical properties, specifically molecular weights and log of octanol-water partition coefficients (logp) (wildman and crippen, 1999), to assess the feasibility of molecular structures as drugs. furthermore, we used other widely used drug-likeness metrics, such as qed @xcite , sa @xcite , and np @xcite , to evaluate the potential effectiveness of the generated molecules as drug-like compounds. configurations this study aims to analyze whether the configuration of repurformer is effective. given that the distinguishing configuration of repurformer is the application of fft with lpf in the embedding space, we conducted comparative experiments with different configuration options: it directly passes h to the compound decoder. in tables 2 and 3 , we can compare the molecular properties of the generated compounds from different configurations. performance comparison to assess the effectiveness of repurformer as a target-specific molecule generation model, we compared its performance with the existing protein-specific generative approaches as external baseline models, including transformer-based model (grechishnikova, 2021) and alphadrug @xcite . transformer-based model utilized the vanilla transformer architecture @xcite to generate compounds based on target proteins. this model viewed the target-specific molecule generation as a translational task, converting amino acid sequence into smiles strings. alphadrug modified the vanilla transformer by introducing skipconnections between its encoders and decoders, facilitating the joint embedding of target proteins and molecules. in addition, it employed a monte carlo tree search algorithm for the conditioned generation of novel molecules based on specific target proteins. to ensure a fair comparison, we trained the external baseline models using our dataset using the same experiment setup and evaluation metrics as for repurformer. ## limitations this study has some limitations. first, due to inconsistencies between the tokens in our dataset and those we borrowed from previous research, some generated outputs @xmath3, which had to be excluded. second, the study lacks experiments on binding affinity, which are necessary to evaluate how strongly the generated compounds bind to proteins. these limitations must be addressed in future research. ## concluding remarks in this study, we introduced repurformer, a novel model designed to address the sample bias problem in de novo molecule generation by leveraging multi-hop relationships. repurformer integrates bidirectional pretraining with fast fourier transform and low-pass filtering, to capture complex interactions between proteins and compounds. this approach focuses on low-frequency components, corresponding to longer propagation through multihop protein-compound interactions. the results show that repurformer successfully generates valid and diverse molecules. building on these positive results, there are several promising directions for future improvement. enhancing the backbone architecture by incorporating advanced models like diffusion or graph neural networks and techniques such as contrastive learning could further improve repurformer's ability to capture multi-hop protein-compound interactions. the results from figures 5 and 6 also suggest promising directions to improve repurformer such as leveraging reinforcement learning to maximize diversity rewards or introducing wasserstein loss to address mode collapse. additionally, while our current experiments have shown the potential of repurformer, it is critical to validate its applicability in real-world scenarios. therefore, we need to verify the performance of repurformer on existing drug repurposing cases. considering these aspects will strengthen the practical implications and utilities of repurformer.