Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias identifier Import alias Import alias Import alias identifier Import alias Import alias identifier Import alias identifier ImportFrom alias Assign NameStore NameLoad Import alias identifier Expr Call AttributeLoad NameLoad attr ListLoad Str Assign NameStore AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad Str Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad Call NameLoad Str Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Str attr NameLoad Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad Call NameLoad Str Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Str attr NameLoad Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad TupleLoad NameLoad NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad TupleLoad NameLoad NameLoad NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Call AttributeLoad AttributeLoad NameLoad attr attr Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Call AttributeLoad AttributeLoad NameLoad attr attr Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Call AttributeLoad AttributeLoad NameLoad attr attr Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Call AttributeLoad AttributeLoad NameLoad attr attr Str SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad TupleLoad NameLoad NameLoad NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Print NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpMod Str SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Str Print NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Str Expr Call AttributeLoad NameLoad attr decorator_list Call NameLoad NameLoad Call NameLoad Str Str FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args defaults body Expr Str Pass decorator_list Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr
N/A N/A 
========================
RNAseq Geneset Pipeline
========================

:Author: David Sims 
:Release: $Id: pipeline_rnaseq_genesst.py 2900 2012-03-28 14:38:00Z david $
:Date: |today|
:Tags: Python

The RNAseq geneset pipeline parses multiple GTF files derived from RNAseq experiments in different tissues and produces a consensus geneset

Usage
=====

See :ref:`PipelineSettingUp` and :ref:`PipelineRunning` on general information how to use CGAT pipelines.

Configuration
-------------

The pipeline requires a configured :file:`pipeline.ini` file. 

Configuration files follow the ini format (see the python
`ConfigParser <http://docs.python.org/library/configparser.html>` documentation).
The configuration file is organized by section and the variables are documented within 
the file. In order to get a local configuration file in the current directory, type::

    python <codedir>/pipeline_rnaseq_genest.py config

The sphinxreport report requires a :file:`conf.py` and :file:`sphinxreport.ini` file 
(see :ref:`PipelineDocumenation`). To start with, use the files supplied with the
:ref:`Example` data.


Input
-----

Input are GTF-formatted files. 

Requirements
------------

The pipeline requires the information from the following pipelines:

:doc:`pipeline_annotations`

set the configuration variables:
   :py:data:`annotations_database` 
   :py:data:`annotations_dir`

On top of the default CGAT setup, the pipeline requires the following software to be in the 
path:

+--------------------+-------------------+------------------------------------------------+
|*Program*           |*Version*          |*Purpose*                                       |
+--------------------+-------------------+------------------------------------------------+
|BEDTools            |                   |interval comparison                             |
+--------------------+-------------------+------------------------------------------------+


Pipline Output
==============

The results of the computation are all stored in an sqlite relational
database :file:`csvdb`.


Code
====

 N/A sys N/A tempfile N/A optparse N/A shutil N/A itertools N/A csv N/A math N/A random N/A re N/A glob N/A os N/A shutil N/A collections N/A gzip N/A sqlite3 N/A pysam N/A CGAT.IndexedFasta IndexedFasta N/A CGAT.IndexedGenome IndexedGenome N/A CGAT.FastaIterator FastaIterator N/A CGAT.Genomics Genomics N/A CGAT.IOTools IOTools N/A CGAT.MAST MAST N/A CGAT.GTF GTF N/A CGAT.GFF GFF N/A CGAT.Bed Bed N/A cStringIO N/A numpy N/A CGAT.Masker Masker N/A fileinput N/A CGAT.Experiment E N/A logging L ruffus * N/A USECLUSTER True N/A CGAT.Pipeline P N/A N/A N/A P getParameters N/A pipeline.ini N/A PARAMS N/A P PARAMS convertTxtToGtf N/A N/A infile outfile N/A N/A N/A Convert  txt to Gtf N/A track N/A N/A P snip N/A N/A N/A os path basename infile .txt N/A statement cat %(infile)s 
                   | awk 'OFS="\t" {print "chr"$3,"rnaseq","exon",$4,$5,0,substr($6,0,1),".","transcript_id \""$1"\"; gene_id \""$1"\"; exon_number \""substr($2,20,2)"\"; "}' 
                   | sed s/0\\t1\\t\./0\\t+\\t\./g | sed s/chrUltra/Ultra/ | sed s/chrContig/Contig/ > %(outfile)s N/A N/A N/A P run N/A N/A transform *.txt N/A regex (\S+).txt \1.gtf getGtfStrandedTranscripts N/A N/A infile outfile N/A N/A N/A join exons to get transcripts from GTF file N/A track N/A N/A P snip N/A N/A N/A os path basename infile .gtf N/A statement cat %(infile)s | python %(scriptsdir)s/gtf2gtf.py --join-exons | sort -k1,1 -k4,4n > %(outfile)s N/A N/A N/A P run N/A N/A transform convertTxtToGtf N/A regex (\S+).gtf \1.transcripts.gtf getEnsemblCodingGeneset N/A N/A infile outfile N/A N/A N/A identify transcrpts that overlap an ensembl coding gene  N/A ensembl_genes N/A PARAMS N/A ensembl_genes N/A ensembl_noncoding N/A PARAMS N/A ensembl_noncoding N/A statement cat %(infile)s | intersectBed -a stdin -b %(ensembl_genes)s -u -s 
                   | intersectBed -a stdin -b %(ensembl_noncoding)s -v -s -f 1 -r
                   | sort -k1,1 -k4,4n > %(outfile)s;
                   echo "transcripts with ensembl coding overlap: " > %(outfile)s.count; 
                   cat %(outfile)s | wc -l >> %(outfile)s.count; N/A N/A N/A P run N/A N/A transform getGtfStrandedTranscripts N/A suffix .gtf .coding.gtf renameTranscripts N/A N/A infile outfile N/A N/A N/A systematically rename transcripts to remove duplicate cuffdiff names  N/A statement cat %(infile)s | awk 'OFS="\t" {print $1,$2,$3,$4,$5,$6,$7,$8,"transcript_id \"rnaseq_coding_transcript_"NR"\"; "}' > %(outfile)s; N/A N/A N/A P run N/A N/A transform getEnsemblCodingGeneset N/A suffix .gtf .rename.gtf annotateTranscripts N/A N/A infile outfile N/A N/A N/A  Add ensembl gene id to GTF file N/A ensembl_genes N/A PARAMS N/A ensembl_genes N/A statement cat %(infile)s 
                   | intersectBed -a stdin -b %(ensembl_genes)s -wa -wb -s 
                   | awk -F "\t" 'OFS="\t" {print $1,$2,$3,$4,$5,$6,$7,$8,"gene_id \""$13"\"; "$9}'
                   | sort -k1,1 -k4,4n > %(outfile)s; N/A N/A N/A P run N/A N/A transform renameTranscripts N/A suffix .gtf .ensg.gtf addMissingEnsemblTranscripts N/A N/A infile outfile N/A N/A N/A  Add ensembl gene id to GTF file N/A ensembl_transcripts N/A PARAMS N/A ensembl_transcripts_gtf N/A statement cat %(infile)s 
                   | intersectBed -a %(ensembl_transcripts)s -b stdin -v -s -f 1 -r > missing_ensembl_transcripts.gtf;
                   cat %(infile)s missing_ensembl_transcripts.gtf | sort -k1,1 -k4,4n
                   > %(outfile)s; N/A N/A N/A P run N/A N/A transform annotateTranscripts N/A regex (\S+).ensg.gtf all_transcripts.gtf renameTranscriptToExon N/A N/A infile outfile N/A N/A N/A reformat transcript file for use in proj007 pipeline  N/A statement cat %(infile)s | sed s/\\ttranscript\\t/\\texon\\t/g
                 | gzip > %(outfile)s;  N/A N/A N/A P run N/A N/A follows N/A mkdir geneset N/A transform addMissingEnsemblTranscripts N/A regex all_transcripts.gtf geneset/transcripts.gtf.gz getNoncodingGeneset N/A N/A infile outfile N/A N/A N/A Assume that all transcripts the do not overlap with ensembl coding geneset are noncoding  N/A ensembl_transcripts N/A PARAMS N/A ensembl_transcripts N/A statement cat %(infile)s | intersectBed -a stdin -b %(ensembl_transcripts)s -v -s > %(outfile)s;
                   echo "transcripts without ensembl coding overlap: " > %(outfile)s.count; 
                   cat %(outfile)s | wc -l >> %(outfile)s.count; N/A N/A N/A P run N/A N/A transform getGtfStrandedTranscripts N/A suffix .gtf .noncoding.gtf renameNoncodingTranscripts N/A N/A infile outfile N/A N/A N/A systematically rename transcripts to remove duplicate cuffdiff names  N/A statement cat %(infile)s | awk 'OFS="\t" {print $1,$2,$3,$4,$5,$6,$7,$8,"transcript_id \"rnaseq_noncoding_transcript_"NR"\"; "}' > %(outfile)s; N/A N/A N/A P run N/A N/A transform getNoncodingGeneset N/A suffix .gtf .rename.gtf annotateNoncodingTranscripts N/A N/A infile outfile N/A N/A N/A  Add ensembl gene id to GTF file N/A ensembl_noncoding N/A PARAMS N/A ensembl_noncoding_genes N/A statement intersectBed -a %(infile)s  -b %(ensembl_noncoding)s -wa -wb -s 
                   | awk -F "\t" 'OFS="\t" {print $1,$2,$3,$4,$5,$6,$7,$8,"gene_id \""$13"\"; "$9}' > known_noncoding_exons.gtf;  
                   intersectBed -a %(infile)s -b %(ensembl_noncoding)s -v -s
                   | awk -F "\t" 'OFS="\t" {print $1,$2,$3,$4,$5,$6,$7,$8,"gene_id \"novel_gene_"NR"\"; "$9}' > novel_noncoding_transcripts.gtf; 
                   cat known_noncoding_exons.gtf novel_noncoding_transcripts.gtf | sort -k1,1 -k4,4n
                   > %(outfile)s; N/A N/A N/A P run N/A N/A transform renameNoncodingTranscripts N/A suffix .gtf .ensg.gtf addMissingNoncodingTranscripts N/A N/A infile outfile N/A N/A N/A  Add ensembl gene id to GTF file N/A ensembl_noncoding N/A PARAMS N/A ensembl_noncoding_gtf N/A statement intersectBed -a %(ensembl_noncoding)s -b %(infile)s  -v -s -f 1 -r > missing_ensembl_noncoding_transcripts.gtf;
                   cat %(infile)s missing_ensembl_noncoding_transcripts.gtf | sort -k1,1 -k4,4n
                   > %(outfile)s; N/A N/A N/A P run N/A N/A transform annotateNoncodingTranscripts N/A regex (\S+).ensg.gtf all_noncoding_transcripts.gtf transcriptGtfToTab N/A N/A infile outfile N/A N/A N/A Copy replicated Bed files generated by capseq pipline to geneset-specific output directory N/A statement cat %(infile)s | python %(scriptsdir)s/gtf2tsv.py -f --log=%(outfile)s.log | awk '{if (NR==1) {print $0"\tgene_biotype\tgene_name"} else {print $0"\tprotein_coding\tunknown"}}' > %(outfile)s N/A N/A N/A P run N/A N/A transform addMissingEnsemblTranscripts N/A suffix .gtf .tab noncodingGtfToTab N/A N/A infile outfile N/A N/A N/A Copy replicated Bed files generated by capseq pipline to geneset-specific output directory N/A statement cat %(infile)s | python %(scriptsdir)s/gtf2tsv.py -f --log=%(outfile)s.log | awk '{if (NR==1) {print $0"\tgene_biotype\tgene_name"} else {print $0"\tnoncoding\tunknown"}}' > %(outfile)s N/A N/A N/A P run N/A N/A transform addMissingNoncodingTranscripts N/A suffix .gtf .tab mergeTranscriptInfo N/A N/A infiles outfile N/A N/A N/A Copy replicated Bed files generated by capseq pipline to geneset-specific output directory N/A inlist N/A N/A join infiles N/A statement cat %(inlist)s | awk '{if (NR==1 || $1 != "contig") {print $0}}' > %(outfile)s N/A N/A N/A P run N/A N/A follows transcriptGtfToTab noncodingGtfToTab N/A merge *.tab transcript_info.tab loadTranscripts N/A N/A infile outfile N/A N/A N/A load GTF file into database  N/A headers contig,source,feature,start,end,score,strand,frame,gene_id,transcript_id,gene_biotype N/A statement cat %(infile)s | python ~/src/csv2db.py 
                         --database=%(database)s
                         --table=transcript_info
                         --index=gene_id
                         --index=contig,start
                         --index=transcript_id
                 > %(outfile)s;  N/A N/A N/A P run N/A N/A transform mergeTranscriptInfo N/A suffix .tab .tab.load buildGeneIntervals N/A N/A infile outfile N/A N/A N/A  Merge all transcripts per gene (including utr) to get start and stop 
        coordinates for every protein-coding gene and store in a GTF file N/A ensembl_genes N/A PARAMS N/A ensembl_genes_gtf N/A statement cat %(ensembl_genes)s %(infile)s | mergeBed -i stdin -s -n 
                   | awk 'OFS="\t" {print $1,$2,$3,"gene"NR,$4,$5 }'
                   | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log
                   | sort -k1,1 -k2,2n > %(outfile)s; N/A N/A N/A P run N/A N/A follows N/A mkdir geneset N/A transform getEnsemblCodingGeneset N/A regex (\S+).coding.gtf geneset/\1.genes.bed annotateGeneIntervals N/A N/A infile outfile N/A N/A N/A  Add ensembl gene id to GTF file N/A ensembl_genes N/A PARAMS N/A ensembl_genes N/A statement cat %(infile)s 
                   | intersectBed -a stdin -b %(ensembl_genes)s -wa -wb -s 
                   | awk 'OFS="\t" {print $1,$2,$3,$10,$5,$6}' 
                   | sort -k1,1 -k2,2n > %(outfile)s; N/A N/A N/A P run N/A N/A transform buildGeneIntervals N/A regex geneset/(\S+).genes.bed geneset/genes.bed buildUpstreamFlankBed N/A N/A infile outfile N/A N/A N/A  build interval upstream of gene start for each entry in bed file N/A window N/A PARAMS N/A geneset_flank N/A faidx N/A PARAMS N/A faidx N/A statement flankBed -i %(infile)s -g %(faidx)s -l %(window)s -r 0 -s 
                   | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log > %(outfile)s N/A N/A N/A P run N/A N/A transform annotateGeneIntervals N/A regex N/A PARAMS N/A interface_genic_bed N/A PARAMS N/A interface_upstream_flank_bed buildDownstreamFlankBed N/A N/A infile outfile N/A N/A N/A  build interval downstream of gene start for each entry in bed file N/A window N/A PARAMS N/A geneset_flank N/A faidx N/A PARAMS N/A faidx N/A statement flankBed -i %(infile)s -g %(faidx)s -l 0 -r %(window)s -s 
                   | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log > %(outfile)s N/A N/A N/A P run N/A N/A transform annotateGeneIntervals N/A regex N/A PARAMS N/A interface_genic_bed N/A PARAMS N/A interface_downstream_flank_bed buildIntergenicBed N/A N/A infiles outfile N/A N/A N/A  Genomic regions not associated with any other features N/A inlist N/A N/A join infiles N/A statement cat %(inlist)s | complementBed -i stdin -g %(faidx)s 
                   | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log > %(outfile)s N/A N/A N/A P run N/A N/A merge N/A annotateGeneIntervals buildUpstreamFlankBed buildDownstreamFlankBed N/A N/A N/A os path join geneset N/A PARAMS N/A interface_intergenic_bed convertBedToGtf N/A N/A infile outfile N/A N/A N/A  convert bed files to GTF N/A statement cat %(infile)s | python %(scriptsdir)s/bed2gff.py --as-gtf | grep -v "#" > %(outfile)s N/A N/A N/A P run N/A N/A transform N/A annotateGeneIntervals buildUpstreamFlankBed buildDownstreamFlankBed buildIntergenicBed N/A suffix .bed .gtf buildTranscriptTSS N/A N/A infile outfile N/A N/A N/A annotate transcription start sites from reference gene set.
    Similar to promotors, except that the witdth is set to 1.  N/A statement 
        cat < %(infile)s 
        | python %(scriptsdir)s/gff2gff.py --sanitize=genome --skip-missing --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log  
        | python %(scriptsdir)s/gtf2gff.py --method=promotors --promotor=1 --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gff2bed.py --is-gtf --name=transcript_id --log=%(outfile)s.log 
        | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log
        | gzip
        > %(outfile)s  N/A N/A N/A P run N/A N/A transform addMissingEnsemblTranscripts N/A regex all_transcripts.gtf N/A N/A N/A os path join geneset N/A PARAMS N/A interface_tss_bed buildGeneTSS N/A N/A infile outfile N/A N/A N/A create a single TSS for each gene N/A statement 
        cat %(infile)s 
        | python %(scriptsdir)s/gff2gff.py --sanitize=genome --skip-missing --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gtf2gff.py --method=promotors --promotor=1 --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gff2bed.py --is-gtf --name=gene_id --log=%(outfile)s.log 
        | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log
        | gzip
        > %(outfile)s  N/A N/A N/A P run N/A N/A follows convertBedToGtf N/A files geneset/genes.gtf N/A N/A N/A os path join geneset N/A PARAMS N/A interface_tss_gene_bed buildGeneTSSInterval N/A N/A infile outfile N/A N/A N/A create a single interval that encompasses all annotated TSSs for a given gene N/A statement 
        cat < %(infile)s 
        | python %(scriptsdir)s/gff2gff.py --sanitize=genome --skip-missing --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gtf2gff.py --method=promotors --promotor=1 --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | sed s/\\ttranscript\\t/\\texon\\t/g 
        | python %(scriptsdir)s/gtf2gtf.py --merge-transcripts --log=%(outfile)s.log 
        | python %(scriptsdir)s/gff2bed.py --is-gtf --name=transcript_id --log=%(outfile)s.log 
        | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log
        | gzip
        > %(outfile)s  N/A N/A N/A P run N/A N/A transform addMissingEnsemblTranscripts N/A regex all_transcripts.gtf N/A N/A N/A os path join geneset N/A PARAMS N/A interface_tss_gene_interval_bed buildNoncodingGeneTSS N/A N/A infile outfile N/A N/A N/A Assign a TSS for each non-coding gene N/A statement 
        cat < %(infile)s 
        | python %(scriptsdir)s/gff2gff.py --sanitize=genome --skip-missing --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gtf2gff.py --method=promotors --promotor=1 --genome-file=%(genome_dir)s/%(genome)s --log=%(outfile)s.log 
        | python %(scriptsdir)s/gff2bed.py --is-gtf --name=gene_id --log=%(outfile)s.log 
        | python %(scriptsdir)s/bed2bed.py --method=filter-genome --genome-file=%(genome_dir)s/%(genome)s --log %(outfile)s.log
        | gzip
        > %(outfile)s  N/A N/A N/A P run N/A N/A transform addMissingNoncodingTranscripts N/A regex (\S+).gtf N/A N/A N/A os path join geneset N/A PARAMS N/A interface_tss_gene_noncoding_bed ExtendRegion N/A N/A infile outfile N/A N/A N/A convert bed to gtf N/A statement gunzip < %(infile)s 
                   | slopBed -i stdin -g %(faidx)s -b 1000  
                   | gzip
                   > %(outfile)s  N/A N/A N/A P run N/A N/A transform N/A buildTranscriptTSS buildGeneTSS buildGeneTSSInterval buildNoncodingGeneTSS N/A suffix .bed.gz .extended.bed.gz convertToGTF N/A N/A infile outfile N/A N/A N/A convert bed to gtf N/A statement gunzip < %(infile)s 
                   | python %(scriptsdir)s/bed2gff.py --as-gtf  --log=%(outfile)s.log 
                   > %(outfile)s  N/A N/A N/A P run N/A N/A transform N/A buildTranscriptTSS buildGeneTSS buildGeneTSSInterval buildNoncodingGeneTSS ExtendRegion N/A suffix .bed.gz .gtf copyEnsembl N/A N/A infile outfile N/A N/A N/A convert bed to gtf N/A ensembl_dir N/A PARAMS N/A ensembl_dir N/A statement cp %(ensembl_dir)s/repeats.gtf geneset/ 2> %(outfile)s;
                   cp %(ensembl_dir)s/go.tsv.gz geneset/ 2> %(outfile)s;
                   cp %(ensembl_dir)s/goslim.tsv.gz geneset/ 2> %(outfile)s;
                   cp %(ensembl_dir)s/go_ontology.obo geneset/ 2> %(outfile)s;
                   cp %(ensembl_dir)s/goslim.obo geneset/ 2> %(outfile)s; N/A N/A N/A P run N/A N/A files pipeline.ini ensembl.copy.log copyEnsemblDb N/A N/A infile outfile N/A N/A N/A copy tables from ensembl database to rnaseq database N/A table_list N/A N/A P asList N/A PARAMS N/A ensembl_tables N/A dbhandle N/A N/A sqlite3 connect N/A PARAMS N/A database N/A cc N/A N/A dbhandle cursor N/A query N/A ATTACH "%s" as ensembl; N/A PARAMS N/A ensembl_db N/A N/A N/A cc execute query N/A table table_list N/A N/A cc N/A N/A dbhandle cursor N/A query N/A CREATE TABLE %s AS SELECT * FROM ensembl.%s; N/A table table N/A query N/A N/A N/A cc execute query N/A N/A N/A cc close N/A statement touch %(outfile)s; N/A N/A N/A P run N/A N/A files pipeline.ini ensembldb.copy.log updateGeneName N/A N/A infile outfile N/A N/A N/A update gene name from ensembl database N/A dbhandle N/A N/A sqlite3 connect N/A PARAMS N/A database N/A cc N/A N/A dbhandle cursor N/A query N/A ATTACH "%s" as ensembl; N/A PARAMS N/A ensembl_db N/A N/A N/A cc execute query N/A query UPDATE transcript_info SET gene_name=
               (SELECT gene_name from ensembl.transcript_info 
               WHERE transcript_info.gene_id=ensembl.transcript_info.gene_id); N/A query N/A N/A N/A cc execute query N/A N/A N/A cc close N/A statement touch %(outfile)s; N/A N/A N/A P run N/A N/A transform loadTranscripts N/A regex (\S+).load gene_name.update.log transcripts N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows convertTxtToGtf getGtfStrandedTranscripts getEnsemblCodingGeneset renameTranscripts annotateTranscripts addMissingEnsemblTranscripts renameTranscriptToExon noncoding N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows getNoncodingGeneset renameNoncodingTranscripts annotateNoncodingTranscripts addMissingNoncodingTranscripts loadTranscriptInfo N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows transcriptGtfToTab noncodingGtfToTab mergeTranscriptInfo loadTranscripts genes N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows buildGeneIntervals annotateGeneIntervals buildUpstreamFlankBed buildDownstreamFlankBed buildIntergenicBed convertBedToGtf tss N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows buildTranscriptTSS buildGeneTSS buildNoncodingGeneTSS ExtendRegion convertToGTF ensembl N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows copyEnsembl copyEnsemblDb updateGeneName full N/A N/A N/A N/A N/A build all targets. N/A N/A N/A follows transcripts noncoding loadTranscriptInfo genes tss ensembl N/A N/A __name__ __main__ N/A N/A N/A N/A sys exit N/A N/A P main N/A sys argv