Module Expr Str Import alias Import alias Import alias identifier Import alias identifier ClassDef bases body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return SubscriptLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad attr Index Num decorator_list decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults NameLoad body Expr Str If UnaryOpNot NameLoad body Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword Str keyword SubscriptLoad Call NameLoad Index Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword TupleLoad Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr keyword Str keyword NameLoad keyword NameLoad keyword Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad If CompareNotEq Call NameLoad NameLoad Num body Raise Call NameLoad Str Assign TupleStore NameStore NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign NameStore Call NameLoad AttributeLoad NameLoad attr orelse Assign NameStore NameLoad If AttributeLoad NameLoad attr body Assign NameStore Call NameLoad AttributeLoad NameLoad attr orelse Assign NameStore NameLoad If CompareEq AttributeLoad NameLoad attr Str body Assign NameStore Dict NameLoad NameLoad Num Num FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore AttributeLoad NameLoad attr While NameLoad body Assign NameStore ListComp Call AttributeLoad Call NameLoad attr Str comprehension NameStore Call NameLoad Num If UnaryOpNot SubscriptLoad NameLoad Index Num body Break Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad SubscriptLoad NameLoad Index Num attr Index Num AugAssignAdd SubscriptStore NameLoad Index AttributeLoad NameLoad attr Num If AttributeLoad NameLoad attr body Expr Yield SubscriptLoad NameLoad Slice Num orelse Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore AttributeLoad NameLoad attr While NameLoad body Assign NameStore ListComp Call AttributeLoad Call NameLoad attr Str comprehension NameStore Call NameLoad Num If UnaryOpNot SubscriptLoad NameLoad Index Num body Break Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad SubscriptLoad NameLoad Index Num attr Index Num If AttributeLoad NameLoad attr body Assign NameStore SubscriptLoad NameLoad Slice Num If CompareNotIn NameLoad NameLoad body If CompareIs NameLoad NameLoad body Continue orelse Expr Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad Str attr NameLoad Str orelse Expr Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad Str attr NameLoad Str decorator_list Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Call NameLoad NameLoad If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr BinOpMod AttributeLoad NameLoad attr Str Str orelse Assign NameStore NameLoad With Call AttributeLoad NameLoad attr BinOpMod AttributeLoad NameLoad attr Str Str NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad With Call AttributeLoad NameLoad attr BinOpMod AttributeLoad NameLoad attr Str Str NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr
N/A N/A fastqs2fastqs.py - manipulate (merge/reconcile) fastq files
=============================================================

:Author: Andreas Heger
:Release: $Id$
:Date: |today|
:Tags: Genomics NGS FASTQ FASTQ Manipulation

Purpose
-------

This script manipulates multiple fastq files and outputs
new fastq files. Currently only the method ``reconcile``
is implemented.

reconcile
+++++++++

Reconcile reads from a pair of fastq files.

This method takes two fastq files and outputs two fastq files such
that all reads in the output are present in both output files.

The typical use case is that two fastq files containing the first and
second part of a read pair have been independently filtered, for
example by quality scores, truncation, etc. As a consequence some
reads might be missing from one file but not the other. The reconcile
method will output two files containing only reads that are common to
both files.

The two files must be sorted by read identifier.

Example input, read2 and read3 are only present in either of the
files:

   # File1        # File 2

   @read1         @read1
   AAA            AAA
   +              +
   !!!            !!!
   @read2         @read3
   CCC            TTT
   +              +
   !!!            !!!
   @read4         @read4
   GGG            GGG
   +              +
   !!!            !!!

Example output, only the reads common to both files are output::

   # File1        # File 2

   @read1         @read1
   AAA            AAA
   +              +
   !!!            !!!
   @read4         @read4
   GGG            GGG
   +              +
   !!!            !!!

Usage
-----

Example::

   python fastqs2fastqs.py             --method=reconcile             --output-filename-pattern=myReads_reconciled.%s.fastq             myReads.1.fastq.gz myReads.2.fastq.gz

In this example we take a pair of fastq files, reconcile by read
identifier and output 2 new fastq files named
``myReads_reconciled.1.fastq.gz`` and
``myReads_reconciled.2.fastq.gz``.

Type::

   python fastqs2fastqs.py --help

for command line help.

Command line options
--------------------

 N/A sys N/A re N/A CGAT.IOTools IOTools N/A CGAT.Experiment E PatternGetter N/A N/A __init__ N/A N/A self pattern N/A N/A N/A N/A self pattern N/A N/A re compile pattern N/A __call__ N/A N/A self id N/A N/A N/A N/A N/A N/A N/A N/A N/A self pattern search id groups N/A 0 N/A N/A plain_getter N/A N/A id N/A N/A N/A id N/A main N/A N/A argv N/A None N/A N/A script main.

    parses command line options in sys.argv, unless *argv* is given.
     N/A N/A argv N/A N/A argv N/A sys argv N/A parser N/A N/A E OptionParser version %prog version: $Id$ usage N/A N/A globals N/A __doc__ N/A N/A N/A parser add_option -m --method dest method type choice choices N/A reconcile help method to apply [default=%default]. N/A N/A N/A parser add_option -c --chop-identifier dest chop action store_true help whether or not to trim last character of the  sequence name. For example sometimes ids in the first file in the pair will end with  and the second with . If --chop-identifier is not specified then the results will be wrong [default=%default]. N/A N/A N/A parser add_option -u --unpaired dest unpaired action store_true help whether or not to write out unpaired reads to a separate file N/A N/A N/A parser add_option --id-pattern-1 dest id_pattern_1 help If specified will use the first group from thepattern to determine the ID for the first read default None N/A N/A N/A parser add_option --id-pattern-2 dest id_pattern_2 help As above but for read 2 default None N/A N/A N/A parser add_option -o --output-filename-pattern dest output_pattern type string help pattern for output files [default=%default]. N/A N/A N/A parser set_defaults method reconcile chop False unpaired False output_pattern %s.fastq.gz N/A N/A options args N/A N/A E Start parser argv argv N/A N/A N/A len args 2 N/A N/A N/A ValueError please supply at least two fastq files on the commandline N/A N/A fn1 fn2 args N/A c N/A N/A E Counter N/A N/A options id_pattern_1 N/A N/A id1_getter N/A PatternGetter N/A options id_pattern_1 N/A N/A id1_getter plain_getter N/A N/A options id_pattern_2 N/A N/A id2_getter N/A PatternGetter N/A options id_pattern_2 N/A N/A id2_getter plain_getter N/A N/A N/A options method reconcile N/A N/A id_lengths N/A fn1 fn2 0 0 getIds N/A N/A infile id_getter N/A plain_getter N/A N/A return ids in infile. N/A aread N/A infile readline N/A True N/A N/A l N/A N/A N/A N/A aread rstrip N/A i N/A range 4 N/A N/A N/A l N/A 0 N/A N/A N/A r N/A id_getter N/A N/A N/A N/A l N/A 0 split N/A 0 N/A N/A id_lengths N/A N/A infile name 1 N/A N/A options chop N/A N/A N/A N/A r N/A -1 N/A N/A N/A r N/A write N/A N/A outfile infile take unpaired_file id_getter N/A None plain_getter N/A N/A filter fastq files with ids in take. N/A aread N/A infile readline N/A True N/A N/A l N/A N/A N/A N/A aread rstrip N/A i N/A range 4 N/A N/A N/A l N/A 0 N/A N/A N/A r N/A id_getter N/A N/A N/A N/A l N/A 0 split N/A 0 N/A N/A options chop N/A N/A r N/A r N/A -1 N/A N/A r take N/A N/A N/A unpaired_file None N/A N/A N/A N/A N/A N/A unpaired_file write N/A N/A N/A join l N/A N/A N/A N/A outfile write N/A N/A N/A join l N/A N/A N/A N/A E info reading first in pair N/A inf1 N/A N/A IOTools openFile fn1 N/A ids1 N/A set N/A getIds inf1 id1_getter N/A N/A N/A E info reading second in pair N/A inf2 N/A N/A IOTools openFile fn2 N/A ids2 N/A getIds inf2 id2_getter N/A take N/A N/A ids1 intersection ids2 N/A N/A N/A E info N/A first pair: %i reads, second pair: %i reads, shared: %i reads N/A N/A id_lengths N/A fn1 N/A id_lengths N/A fn2 N/A len take N/A N/A options unpaired N/A N/A unpaired_filename N/A N/A IOTools openFile N/A N/A options output_pattern unpaired w N/A N/A unpaired_filename None N/A N/A N/A IOTools openFile N/A N/A options output_pattern 1 w outf N/A N/A inf N/A N/A IOTools openFile fn1 N/A N/A N/A E info writing first in pair N/A N/A write outf inf take unpaired_filename id1_getter N/A N/A N/A IOTools openFile N/A N/A options output_pattern 2 w outf N/A N/A inf N/A N/A IOTools openFile fn2 N/A N/A N/A E info writing second in pair N/A N/A write outf inf take unpaired_filename id2_getter N/A N/A options unpaired N/A N/A N/A N/A unpaired_filename close N/A N/A N/A E info N/A %s N/A str c N/A N/A N/A E Stop N/A N/A N/A __name__ __main__ N/A N/A N/A N/A sys exit N/A main N/A sys argv