Many time-consuming analyses of next generation sequencing data can be addressed with modern cloud computing. The Apache Hadoop-based solutions have become popular in genomics due to their scalability in a cloud infrastructure. So far, most of these tools have been used for batch data processing rather than interactive data querying. The SparkSeq software has been created to take advantage of a new MapReduce framework, Apache Spark, for next generation sequencing data. SparkSeq is a general-purpose, flexible and easily extendable library for genomic cloud computing. It can be used to build genomic analysis pipelines in Scala and run them in an interactive way. SparkSeq opens up the possibility of customised ad hoc secondary analyses and iterative machine learning algorithms. This paper demonstrates its scalability and overall very fast performance by running the analyses of sequencing datasets. Tests of SparkSeq also prove that the use of cache and HDFS block size can to be tuned for the optimal performance on multiple worker nodes. In recent years next generation sequencing (NGS) has become the essential technology that is producing precise insight into the genomes and transcriptomes of living organisms. A standard sequencer run produces hundreds gigabase pairs of short reads. The experimental results (fastq or alignment files) can be regarded as big data, in particular with the high number of biological samples. This can be less overwhelming when lab techniques of pre-selection are applied or when the data are aggregated. Computationally, the reduction of the datasets is done by counting the reads in genes or finding genome variants. The resulting count tables or variant * to whom correspondence should be addressed lists are then much smaller than the original BAM files and in most cases can be processed in memory. However, aggregation and summarization always result in the loss of information on such phenomena as novel expression regions, alternative splicing or low coverage areas. Sequencing can pinpoint various properties of genomes and transcriptomes (Frazee et al., 2014; Leśniewska and Okoniewski, 2011) with a nucleotide precision but often the size of datasets is prohibitive to study such details. The nucleotidelevel analysis can be a driving force for the many new and additional applications of sequencing such as linc-RNA discovery, verification of gene and UTR boundaries in the species with imprecise annotation or studies of alternative splicing. The developments in computer science, especially in the area of distributed and cloud computing, are trying to keep pace with the rapidly growing amount of experimental data. There are several applications already available, e.g. (Taylor, 2010; Langmead et al., 2010; Schumacher et al., 2014). Currently there are many initiatives within the IT and bioinformatics community that can be utilized for creating a useful and scalable system for sequencing data analysis. One of the most frequently used parallel and distributed programming models is MapReduce which has its open-source implementation -Apache Hadoop (Borthakur, 2007). Since initially it was not possible to analyse NGS data stored in HDFS (Hadoop Distributed File System) without conversion to file formats supported by Apache Hadoop, a library Hadoop-BAM was developed for direct access and manipulation of formats commonly used in bioinformatics: BAM, FASTQ and VCF (Niemenmaa et al., 2012). Many analytical tools that process data within the Hadoop ecosystem have been developed recently in the open-source community: new high-level languages, database engines and application frameworks. They use the same Apache Hadoop execution model, primarily designed and optimized for onepass batch processing of on-disk data but not for interactive and in-memory ad-hoc data exploration. A good example of the successful combination of Hadoop-BAM and the Apache Pig language is SeqPig, an extension for processing of sequencing data (Schumacher et al., 2014) similar to standard samtools (Li 1
Wiewiorka et al. REF have used Spark for genomic analysis and found an almost linear speedup with the number of worker nodes.
SparkSeq: fast, scalable and cloud-ready tool for the interactive genomic data analysis with nucleotide precision
Bioinformatics
Bioinformatics
Computer Science Medicine