SNP 2 V ec: Scalable Self-Supervised Pre-Training for Genome-Wide Association Study (2022)

Abstract: Self-supervised pre-training methods have brought remarkable breakthroughs in the understanding of text, image, and speech. Recent developments in genomics has also adopted these pre-training methods for genome understanding. However, they focus only on understanding haploid sequences, which hinders their applicability towards understanding genetic variations, also known as single nucleotide polymorphisms (SNPs), which is crucial for genome-wide association study. In this paper, we introduce SNP2Vec, a scalable self-supervised pre-training approach for understanding SNP. We apply SNP2Vec to perform long-sequence genomics modeling, and we evaluate the effectiveness of our approach on predicting Alzheimer’s disease risk in a Chinese cohort. Our approach significantly outperforms existing polygenic risk score methods and all other baselines, including the model that is trained entirely with haploid sequences.

## introduction self-supervised pre-training has become an indispensable step for almost all natural language processing (nlp) tasks @xcite @xcite . pre-trained language models, thanks to the usage of massive text corpora, are effective in handling data scarcity and generalizing to unseen examples @xcite ⇤ these authors contributed equally. @xcite @xcite @xcite . inspired by the success of pre-trained language models, pre-trained genomic models have been proposed to cope with genomic sequence prediction tasks @xcite . however, these models only focus on modeling the four nucleobases (i.e., a, t, c, and g), while ignoring genomic variations in the pre-training stage. although they are effective in haploid pattern analysis, such as promoter region and chromatin-profile prediction, they fail to tackle more complex and challenging tasks, such as genome-wide association study (gwas) @xcite @xcite , which require an in-depth understanding of long genomic sequences and the genomic variation between a homologous chromosome pair. to address these shortcomings, we introduce a self-supervised pre-training approach called snp2vec, which leverages the single-nucleotide polymorphism (snp, pronounced 'snip') information gathered from a large-scale snp database to inject genomic variations in the pre-training stage. snp2vec enables the model to learn the semantics of a diploid sequence (genotype) pattern in a diploid cell. we apply snp2vec to a linearattention model, linformer @xcite , to allow the model to encode long genomic sequences for alzheimer's disease risk prediction in a chinese cohort. we compare snp2vec with non-pretrained models, as well as an existing strong baseline poly-genic risk scoring (prs) model, to demonstrate the effectiveness of our approach. our contributions are summarized as follows: ## snp2vec existing pre-training methods in genomics, such as bigbird @xcite and dnabert @xcite , are only optimized to understand the pattern of a haploid sequence (haplotype) based on the reference genome. this hinders the model from learning genomic variations, which is essential for understanding traits in humans. in contrast to prior works in genomics pre-training, we develop snp2vec to enable pre-training for encoding and understanding patterns of genomic variations in a diploid sequence. figure 1 depicts the overall structure of the snp2vec pre-training method. we elaborate on our snp2vec method in 3 subsections: 1) snp encoding, i.e., how we encode a diploid sequence as a sequence of snp tokens; 2) self-supervised snp dataset, i.e., how we construct a self-supervised dataset using the snp token; 3) self-supervised snp pre-training, i.e., how we perform self-supervised pre-training for learning the sequence pattern of snp tokens. ## results the results of our model and baselines are shown in ## future work for future works, we expect to focus on model explainability by using multiple analysis methods, such as analyzing the attention behaviour, analyzing the gradient saliency map, etc, to gather and verify insights from the model. evaluation on larger scale dataset is also necessary to further demostrate the effectiveness of snp2vec. additionally, adoption of snp2vec to other hereditary disorders and other complex genomics tasks is also an essential direction for future works.