sequenoscope 1.0.0
Description
License: Apache-2.0
Requires Python: >=3.7.0
Keywords: nanopore, ONT, adaptive sampling, sequencing, microbial genomics, metagenomics, visualization, bioinformatics, pipeline, read analysis

[![PyPI](https://img.shields.io/badge/Install%20with-PyPI-blue)](https://pypi.org/project/sequenoscope/#description)
[![Bioconda](https://img.shields.io/badge/Install%20with-bioconda-green)](https://anaconda.org/bioconda/sequenoscope)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/sequenoscope/badges/downloads.svg)](https://anaconda.org/bioconda/sequenoscope)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/sequenoscope/badges/license.svg)](https://anaconda.org/bioconda/sequenoscope)
[![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A523.04.0-23aa62.svg)](https://github.com/ameknas-phac/nf-sequenoscope)


# Sequenoscope

<img height="150" width="400" alt="logo11" src="https://user-images.githubusercontent.com/93303799/225326096-6c9de0f1-9ac0-46a4-914f-a3db51e7e97a.png">

A tool for analyzing sequencing run outputs primarily from adaptive sampling experiments and Oxford Nanopore Technology sequencers.



## Contents <!-- omit from toc -->
- [Introduction](#introduction)
- [Dependencies](#dependencies)
- [Validated Tool Versions](#validated-tool-versions)
- [Python Packages](#python-packages)
- [Tool and Sequencing Platforms Compatibility](#tool-and-sequencing-platforms-compatibility)
- [Installation](#installation)
  - [Option 1: As a conda package (Recomended)](#option-1-as-a-conda-package-recomended)
  - [Option 2: As a PyPI package](#option-2-as-a-pypi-package)
  - [Option 3: Install from source](#option-3-install-from-source)
  - [Option 4: Run with Nextflow](#option-4-run-with-nextflow)
- [Workflow Example](#workflow-example)
  - [Step 1: Filtering Reads with `filter_ONT`](#step-1-filtering-reads-with-filter_ont)
  - [Step 2: Running the `analyze` Module](#step-2-running-the-analyze-module)
  - [Step 3: Visualizing Results with the `plot` Module](#step-3-visualizing-results-with-the-plot-module)
  - [Brief summary](#brief-summary)
- [Limitations](#limitations)
  - [Read mapping ambiguity](#read-mapping-ambiguity)
  - [Post run approximate inference of early adaptive sampling decisions](#post-run-approximate-inference-of-early-adaptive-sampling-decisions)
  - [Reference database dependence](#reference-database-dependence)
- [Use-case Example](#use-case-example)
- [Usage](#usage)
  - [Analyze module options](#analyze-module-options)
  - [Filter\_ONT module options](#filter_ont-module-options)
  - [Plot module options](#plot-module-options)
- [Handling Multiple FASTQ or FASTQ GZ Files (Single End Read Sets)](#handling-multiple-fastq-or-fastq-gz-files-single-end-read-sets)
  - [Concatenating FASTQ Files](#concatenating-fastq-files)
  - [Concatenating FASTQ GZ Files and Uncompressing](#concatenating-fastq-gz-files-and-uncompressing)
- [Quick start](#quick-start)
  - [analyze module](#analyze-module)
  - [filter\_ONT module](#filter_ont-module)
  - [plot module](#plot-module)
- [Outputs](#outputs)
  - [analyze module outputs](#analyze-module-outputs)
  - [sample manifest report format](#sample-manifest-report-format)
  - [sample manifest summary report format](#sample-manifest-summary-report-format)
  - [filter\_ONT module outputs](#filter_ont-module-outputs)
  - [Plot Module Outputs](#plot-module-outputs)
  - [Mapping of terminal `end_reason` values to Adaptive Sampling read decision categories](#mapping-of-terminal-end_reason-values-to-adaptive-sampling-read-decision-categories)
- [Citation](#citation)
- [Legal](#legal)
- [Contacts](#contacts)




## Introduction

Analyzing and interpreting sequencing data is a fundamental task in bioinformatics, and with the advent of ONT adaptive-sampling sequencing, specialized tools are needed to **visualize and assess** the effectiveness of enrichment or depletion in adaptive-sampling sequencing runs. Adaptive sampling data present challenges in effectively visualizing and assessing these sequencing runs in terms of key parameters, necessitating tailored analytical approaches and visual analytics. To assist with these challenges, we have developed a comprehensive bioinformatics pipeline consisting of three modules:  [analyze](#analyze-module), [plot](#plot-module), and [filter_ONT](#filter_ONT-module). Our accessible pipeline aims to provide researchers with a fast and intuitive workflow for easily processing and analyzing sequencing data especially from ONT adaptive sequencing runs, enabling them to gain interpretable insights into their datasets with minimal upfront efforts.

The [analyze](#analyze-module) module serves as the core component of our pipeline. First, It takes an input FASTQ file, a reference FASTA file (which may include multiple taxa or genomes), and an optional text **sequencing summary file** from ONT sequencers or base callers (Guppy/Dorado). Next, Leveraging tools such as `fastp`, `minimap2`, `pysam`, and `mash`, this module performs a series of essential tasks. It filters the input FASTQ file, maps it to the reference FASTA file, and finally, generates a **sequence manifest txt file** and **summary sequence manifest txt file**. These files include key sequencing statistics such as read length, read quality (Q score), mapping efficiency, and coverage depth. For an in-depth explanation of all statistics provided, please refer to the [report format section](#sample-manifest-report-format) below. Sequenoscope infers adaptive sampling per-read outcomes using the **sequencing summary file** `end_reason` field. These inferred categories represent post adaptive sampling run operational approximations and do not correspond one-to-one with true ReadUntil API decisions (`stop_receiving`, `unblock`, `no_decision`). Future versions of Sequenoscope will additionally support direct parsing of adaptive sampling read decision logs generated by the ReadUntil API.

The [plot](#plot-module) module complements the analysis performed by the "analyze" module by using the output to render interactive plots. It takes as input both a "test" and "control" directory, which represent different testing conditions, containing __manifest__ and __manifest summary txt files__ generated by the "analyze" module. With these files, the [plot](#plot-module) module generates visualizations that aid in the interpretation and visualization of the sequencing data. **Please Note:** This module is designed for comparative analysis where two testing conditions are present and can be compared.

The [filter_ONT](#filter_ONT-module) module is designed for for ONT raw reads filtering and subsetting. This module leverages a **sequencing summary file** to allow researchers to precisely filter reads based on customized criteria, including channel, sequencing decisions and [other parameters](#filter_ONT-module).

Our bioinformatics pipeline offers a powerful tool for researchers working with ONT sequencing data. Whether you are exploring metagenomics sample composition, investigating adaptive sampling for your project, or conducting a comparative analysis of different methods in your lab, our pipeline can streamline your analyses and provide valuable insights into your genomic datasets using visual aids and easy to understand outputs.

## Dependencies
- Python: `>=3.7.12, <4`
- fastp: `>=0.22.0` ([`fastplong`](https://github.com/OpenGene/fastplong) support is planned for future releases to enhance long-read processing)
- mash: `>=2.3`
- minimap2: `>=2.26`
- seqtk: `>=1.4`
- samtools: `>=1.6`

## Validated Tool Versions
The analyses presented in the manuscript ([doi:10.1099/acmi.0.001059.v1
](https://doi.org/10.1099/acmi.0.001059.v1)) and corresponding to [Bioproject PRJNA1051081](https://www.ncbi.nlm.nih.gov/bioproject/PRJNA1051081/)
, representing the ZymoBIOMICS LOG and EVEN mock community datasets, were performed using the following tool versions: 
- Python: 3.10.12
- fastp: 0.23.2
- mash: 2.3
- minimap2: 2.24
- seqtk: 1.4
- samtools: 1.18
- pysam: 0.21.0
- plotly: 5.16.1

Other compatible versions may work but were not explicitly tested.

## Python Packages
- pysam: `>=0.16.0`
- plotly: `>=5.16.1`

## Tool and Sequencing Platforms Compatibility
- **fastp vs fastplong**: currently `fastp` is used for quick basic read filtering to ensure broad compatibility across long- and short-read platforms. While `fastplong` is better optimized for ONT long-reads, it was released after initial tool development. Support for `fastplong` is planned for future releases.
- **Illumina short-read support**: Sequenoscope is primarily designed for ONT long-read data. However, Illumina short-read datasets with test and control experimental conditions can be processed for sample manifest generation, comparative summary metrics and visualizations. Because Illumina short-read platform does not support adaptive sampling and does not record pore-level adaptive sampling decision metadata, some ONT-specific outputs would not be vailable for these inputs. Specifically, per-read decision metadata are set to the default read accepted value (`signal_positive`), and adaptive sampling–specific visualizations (including independent and cumulative read-decision bar charts) are not generated. Illumina support is intended for mixed, legacy, or comparative datasets and has not been systematically benchmarked.

## Installation

### Option 1: As a conda package (Recomended)

Install the latest released version from conda:

    conda install -c bioconda sequenoscope

>[!note]
>The `meta.yaml` most recent version is available at https://github.com/bioconda/bioconda-recipes/blob/master/recipes/sequenoscope/meta.yaml

### Option 2: As a PyPI package
Install using `pip` that will fetch the most recent version from [PyPI repo](https://pypi.org/project/sequenoscope/)

    pip install sequenoscope

### Option 3: Install from source

If you wish to install sequenoscope from source, please first ensure these dependencies are installed and configured on your system:
`python>=3.7.12,<4`
`fastp >=0.22.0`
`mash >=2.3`
`minimap2 >=2.26`
`seqtk >=1.4`
`samtools >=1.6`
`pysam >=0.16.0`
`plotly >=5.16.1`

Install the latest commit from the master branch directly from Github:

    pip install git+https://github.com/phac-nml/sequenoscope.git

### Option 4: Run with Nextflow
You can also run Sequenoscope using the Nextflow pipeline available at the following GitHub repository:

👉 [nf-sequenoscope](https://github.com/ameknas-phac/nf-sequenoscope)

This approach ensures better scalability and reproducibility, especially in HPC or cloud environments.

## Workflow Example

In this section, we will walk through a simple workflow using mock data stored in [`mock_data`](./mock_data/) folder to demonstrate how to use each module of **sequenoscope**. The mock data directory contains the following files:

```
./mock_data/
├── mock_adaptive_sampling.fastq
├── mock_adaptive_sampling_summary.txt
├── mock_control.fastq
├── mock.fastq
├── mock_reference.fasta
└── mock_sequencing_summary.txt
```

>[!note]
> - [`mock_adaptive_sampling.fastq`](./mock_data/mock_adaptive_sampling.fastq):
> FASTQ file containing subsampled 1016 reads generated during an ONT adaptive sampling (AS) run (**test sample**). 
>- [`mock_adaptive_sampling_summary.txt`](./mock_data/mock_adaptive_sampling_summary.txt):
Adaptive sampling decision log produced by MinKNOW. This file records only reads that triggered an explicit ReadUntil decision (e.g., `unblock`, `no_decision`, `stop_receiving`) This log contains data exclusively for the **test sample**. Reads from the **control sample** are excluded by design as they do not undergo adaptive filtering.
>- [`mock_control.fastq`](./mock_data/mock_control.fastq): FASTQ file from a non-adaptive (**control sample**) ONT sequencing run, used as a baseline for comparison against adaptive sampling results and contains 1017 reads.
>- [`mock.fastq`](./mock_data/mock.fastq): An overall FASTQ example used for basic pipeline testing and demonstration purposes containing 2023 reads corresponding both to the **test** and **control** samples.
>- [`mock_reference.fasta`](./mock_data/mock_reference.fasta): Reference genome FASTA file with total of 8 reference complete genomes used by the **analyze** module for read mapping and taxonomic assignment.
>- [`mock_sequencing_summary.txt`](./mock_data/mock_sequencing_summary.txt):
this **sequencing summary file** generated by MinKNOW during sequencing or produced post-run by Guppy or Dorado basecaller from POD5 files. This file contains per-read metadata (e.g., `read length`, `channel`, `barcode`, and importantly the `end_reason` fields) for **all reads** generated during the sequencing run, including both control and test samples, and serves as the primary source for inferring per-read adaptive sampling outcomes in Sequenoscope.


**Our goal is to:**

1. Use the `filter_ONT` module to subset raw FASTQ reads into two sets representing different channel ranges.  
2. Run the `analyze` module on both sets (treated as control and adaptive sampling datasets).
3. Use the `plot` module to visualize and compare the results.

This workflow is meant to provide a hands-on example that you can easily follow with your own data.

---

### Step 1: Filtering Reads with `filter_ONT`

First, we will create a dataset that simulates an adaptive sampling scenario by filtering reads by channel. Let’s start by extracting reads from channel 1 to 256 from our `mock.fastq` dataset using the `filter_ONT` module. This will give us a subset similar to `mock_adaptive_sampling.fastq`.

**Command:**
```bash
sequenoscope filter_ONT --input_fastq mock.fastq \
                        --input_summary mock_sequencing_summary.txt \
                        -o mock_filter_ONT \
                        -min_ch 1 \
                        -max_ch 256
```

**What this does:**
- Takes reads from `mock.fastq` that come from channels 1 to 256.
- Outputs a filtered subset in `mock_filter_ONT/sample_filtered_fastq_subset.fastq` which should be identical to `mock_adaptive_sampling.fastq`.

If desired, you could similarly generate the control dataset by adjusting the channel range (e.g., `-min_ch 257 -max_ch 512`) to create a `mock_control.fastq`. However, since we already have `mock_control.fastq` available, we’ll skip that step for now to keep things simple.

**Output Directory Structure:**

#### filter_ONT module (mock_filter_ONT)
```
mock_filter_ONT/
├── filter.log
├── sample_filtered_fastq_subset.fastq
└── sample_read_id_list.csv
```

---

### Step 2: Running the `analyze` Module

Next, we run the `analyze` module on both the control and adaptive sampling datasets. This step will generate various output files including manifest files, BAM alignments, and summary statistics.

**Command for Control Dataset:**
```bash
sequenoscope analyze --input_fastq mock_control.fastq \
                     --input_reference mock_reference.fasta \
                     -seq_sum mock_sequencing_summary.txt \
                     -o mock_control_results \
                     -seq_type SE \
                     -op control
```

**Explanation:**
- `--input_fastq mock_control.fastq`: The control dataset FASTQ file.
- `--input_reference mock_reference.fasta`: Reference genome or sequence.
- `-seq_sum mock_sequencing_summary.txt`: The **sequencing summary file** from ONT.
- `-o mock_control_results`: Output directory.
- `-seq_type SE`: Single-end sequencing.
- `-op control`: A prefix for output files.

**Control Output Directory Structure:**

#### analyze module (mock_control_results)
```
mock_control_results/
├── intermediates
  ├── control_fastp_output.fastp.fastq
  ├── control_fastp_output.html
  ├── control_fastp_output.json
  ├── control_mash_hash.msh
  ├── control_read_list.txt
  ├── control_mapped_bam.bam
  ├── control_mapped_bam.bam.bai
  ├── control_mapped_fastq.fastq
  ├── control_mapped_sam.sam
├── control_manifest_summary.txt
├── control_manifest.txt
├── analyze.log
```

---

**Command for Adaptive Sampling Dataset:**
```bash
sequenoscope analyze --input_fastq mock_adaptive_sampling.fastq \
                     --input_reference mock_reference.fasta \
                     -seq_sum mock_sequencing_summary.txt \
                     -o mock_adaptive_sampling_results \
                     -seq_type SE \
                     -op adaptive_sampling
```

**Explanation:**
- `mock_adaptive_sampling.fastq` represents the dataset filtered by `filter_ONT` (or provided).
- The rest of the parameters are analogous to the control dataset.
- `-op adaptive_sampling` tags output files with "adaptive_sampling" for clarity.

**Adaptive Sampling Output Directory Structure:**

#### analyze module (mock_adaptive_sampling_results)
```
mock_adaptive_sampling_results/
├── intermediates
  ├── adaptive_sampling_fastp_output.fastp.fastq
  ├── adaptive_sampling_fastp_output.html
  ├── adaptive_sampling_fastp_output.json
  ├── adaptive_sampling_mapped_bam.bam
  ├── adaptive_sampling_mapped_bam.bam.bai
  ├── adaptive_sampling_mapped_fastq.fastq
  ├── adaptive_sampling_mapped_sam.sam
  ├── adaptive_sampling_mash_hash.msh
  ├── adaptive_sampling_read_list.txt
├── adaptive_sampling_manifest_summary.txt
├── adaptive_sampling_manifest.txt
└── analyze.log
```

---

### Step 3: Visualizing Results with the `plot` Module

The **plot** module is used to compare control and adaptive sampling datasets. In this example, we use **hours** as the time bin due to truncated data in the mock dataset.


```sh
sequenoscope plot -T mock_adaptive_sampling_results/ \
                  -C mock_control_results/ \
                  -o mock_comparison_plots \
                  -op mock \
                  -AS \
                  -bin hours
```

**Explanation:**
- **`-T mock_adaptive_sampling_results/`**: Test (adaptive sampling) directory.
- **`-C mock_control_results/`**: Control directory.
- **`-o mock_comparison_plots`**: Output directory for plots.
- **`-op mock`**: Prefix for output files.
- **`-AS`**: Enable adaptive sampling decision charts.
- **`-bin hours`**: Use hourly bins for time-based decision charts.

**Plot Output Directory Structure:***
After running the command, the output directory (`mock_comparison_plots/`) will contain standard plots plus a dedicated subdirectory for decision bar charts that reflects the chosen time bin unit.

Example structure:
```
mock_comparison_plots/
├── mock_source_file_taxon_covered_bar_chart.html
├── mock_summary_table.csv
├── mock_taxon_mean_read_length_comparison.html
├── mock_taxon_mean_coverage_comparison.html
├── mock_read_len_violin_comparison_plot.html
├── mock_read_qscore_violin_comparison_plot.html
├── plot.log
└── decision_bar_charts_hours/
    ├── mock_test_independent_decision_bar_chart.html
    ├── mock_control_independent_decision_bar_chart.html
    ├── mock_test_cumulative_decision_bar_chart.html
    └── mock_control_cumulative_decision_bar_chart.html
```
All decision bar charts (both independent and cumulative) are grouped into the `decision_bar_charts_hours/` subdirectory, where the folder name reflects the selected time bin unit.

---

### Brief summary

In this workflow example, we:

1. Used `filter_ONT` to subset reads from a mock dataset by channel number.
2. Applied `analyze` to both the control and adaptive sampling datasets, generating manifest files and alignment statistics.
3. Visualized and compared the results using `plot`, focusing on adaptive sampling decisions and coverage metrics.

By following these steps, you can quickly get started with **sequenoscope** and adapt the workflow to suit your own data and research needs.

## Limitations
### Read mapping ambiguity

In the current release v1.0.0, read mapping is performed using `minimap2` with default primary alignment behavior. Reads that map equally well to multiple reference sequences (i.e., multi-mapped reads) are not explicitly tracked or reported in downstream analyses.

As a result, ambiguous multi-taxa mappings may be assigned to a single taxon, which can affect closely related organisms with shared genomic regions (e.g., Enterococcus/Listeria).

Future versions of sequenoscope will introduce explicit handling of multi-mapped reads, including:

- Optional suppression or reporting of secondary alignments (`minimap2 --secondary=no`)

- Configurable scoring thresholds for ambiguous mappings for read filtering

- Reporting of the proportion of multi-mapped reads per taxon

This enhancement will improve result accuracy, interpretability and quality control when analyzing closely related microbial communities.

### Post run approximate inference of early adaptive sampling decisions
Sequenoscope infers per-read adaptive sampling outcomes using the `end_reason` field recorded in **sequencing summary file**. Because true ReadUntil API decision logs are not always generated or available, these classifications represent post run approximations rather than direct adaptive sampling control signals. Future versions will support direct incorporation of **adaptive sampling summary files**, when available, to leverage per-read ReadUntil API decisions for more precise adaptive sampling read classification.

### Reference database dependence
Taxonomic classification accuracy depends on the completeness and quality of the provided reference database. Sequenoscope does not perform *de novo* taxonomic assignment and organisms absent from the reference set may be underrepresented or assigned at artifically higher taxonomic levels.


## Use-case Example
To demonstrate the practical application of our pipeline, consider a scenario where a researcher conducts adaptive sampling using an ONT sequencer. In this example, the researcher divides the sequencer channels into two sets: one half for adaptive sampling enrichment and the other half for regular sequencing as a control.

- Utilizing our [filter_ONT](#filter_ONT-module) module, the researcher can create two distinct sets of FASTQ files (a 1-256 FASTQ file and a 257-512 FASTQ file), each representing the minimum and maximum channels of the sequencing data.

- These files are then processed separately through our [analyze](#analyze-module) module, generating two datasets – one for the test (adaptive sampling) and one for the control (regular sequencing).

- Finally, by employing the [plot](#plot-module) module, the researcher can visually assess the effectiveness of the adaptive sampling in their experiment. This example shows how Sequenoscope facilitates data processing and analysis, enhancing the researcher's ability to draw meaningful conclusions from their ONT sequencing data.

## Usage
If you run ``sequenoscope``, you should see the following usage statement:

        Usage: sequenoscope <command> <required arguments>

        To get full help for a command use one of:
        sequenoscope <command> -h
        sequenoscope <command> --help


        Available commands:

        analyze     map reads to a target and produce a report with sequencing statistics
        plot        generate plots based on seq manifest files
        filter_ONT  filter reads from a fastq file based on a sequencing summary file

        Other options:

        --check_dependencies  Check if external dependencies (fastp, minimap2, samtools, mash, seqtk) and required Python packages (pysam, plotly) are available
        -v, --version         Show the version and exit
        -h, --help            Show this help message and exit

        
### Analyze module options
If you run ``sequenoscope analyze -h`` or ``sequenoscope analyze --help``, you should see the following options and usage guidleines:

        For help use: sequenoscope analyze -h or --help

        sequenoscope version 1.0.0: a flexible tool for processing multiplatform sequencing data: analyze, subset/filter, compare and visualize.

        optional arguments:
          -h, --help            show this help message and exit
          --force               Force overwrite of existing results directory.
          -v, --version         show program's version number and exit

        USER OPTIONS:
          Direct input files and basic parameters.

          --input_fastq  [ ...]
                                [REQUIRED] Path to 1 (SE) or 2 (PE) FASTQ files to process.
          --input_reference     [REQUIRED] Path to a single reference FASTA file.
          -seq_sum , --sequencing_summary 
                                (Optional) Path to sequencing summary for manifest creation.
          -o , --output         [REQUIRED] Output directory designation.
          -op , --output_prefix 
                                Output file prefix designation. Default is 'sample'.
          -seq_type , --sequencing_type 
                                [REQUIRED] Sequencing type: 'SE' for single-end or 'PE' for paired-end.

        FILTER OPTIONS:
          Parameters to filter/trim FASTQ reads.

          -min_cov , --minimum_coverage 
                                Minimum coverage threshold; default is 1.
          -t , --threads        Number of threads to use.
          -min_len , --minimum_read_length 
                                Minimum read length; default is 15.
          -max_len , --maximum_read_length 
                                Maximum read length; default is 0 (no limit).
          -trm_fr , --trim_front_bp 
                                Bases to trim from the front; default is 0.
          -trm_tail , --trim_tail_bp 
                                Bases to trim from the tail; default is 0.
          -q , --quality_threshold 
                                Quality score threshold; default is 15.

### Filter_ONT module options
If you run ``sequenoscope filter_ONT -h`` or ``sequenoscope filter_ONT --help``, you should see the following options and usage guidleines:

        usage: sequenoscope filter_ONT --input_fastq <file.fq> --input_summary <seq_summary.txt> -o <out.fastq> [options]
        For help use: sequenoscope filter_ONT -h or sequenoscope filter_ONT --help
        
        sequenoscope version 1.0.0: a flexible tool for processing multiplatform sequencing data: analyze, subset/filter, compare and visualize.
        
        Arguments:
          -h, --help            show this help message and exit
          --input_fastq  [ ...]
                                Path to adaptive sequencing fastq files to process. Not required when using --summarize.
          --input_summary       [REQUIRED] Path to ONT sequencing summary file.
          -o , --output         [REQUIRED] Output directory designation
          -op , --output_prefix 
                                Output file prefix designation. default is [sample]
          -cls , --classification 
                                a designation of the adaptive-sampling sequencing decision classification ['unblocked', 'stop_receiving', or 'no_decision']
          -min_ch , --minimum_channel 
                                a designation of the minimum channel/pore number for filtering reads
          -max_ch , --maximum_channel 
                                a designation of the maximum channel/pore number for filtering reads
          -min_dur , --minimum_duration 
                                a designation of the minimum duration of the sequencing run in SECONDS for filtering reads
          -max_dur , --maximum_duration 
                                a designation of the maximum duration of the sequencing run in SECONDS for filtering reads
          -min_start , --minimum_start_time 
                                a designation of the minimum start time of the sequencing run in SECONDS for filtering reads
          -max_start , --maximum_start_time 
                                a designation of the maximum start time of the sequencing run in SECONDS for filtering reads
          -min_q , --minimum_q_score 
                                a designation of the minimum q score for filtering reads
          -max_q , --maximum_q_score 
                                a designation of the maximum q score for filtering reads
          -min_len , --minimum_length 
                                a designation of the minimum read length for filtering reads
          -max_len , --maximum_length 
                                a designation of the maximum read length for filtering reads
          --force               Force overwite of existing results directory
          --summarize           Generate barcode statistics. Must specify an input summary and output directory
          -v, --version         show program's version number and exit

### Plot module options
If you run ``sequenoscope plot -h`` or ``sequenoscope plot --help``, you should see the following options and usage guidleines:

    usage: sequenoscope plot --test_dir <test_dir_path> --control_dir <control_dir_path> --output_dir <out_path>
    For help use: sequenoscope plot -h or sequenoscope plot --help

    sequenoscope version 1.0.0: a flexible tool for processing multiplatform sequencing data: analyze, subset/filter, compare and visualize.

    Optional Arguments:
      -h, --help            show this help message and exit

    Required Paths:
      -T TEST_DIR, --test_dir TEST_DIR
                            Path to test directory.
      -C CONTROL_DIR, --control_dir CONTROL_DIR
                            Path to control directory.
      -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                            Output directory designation.
      --force               Force overwrite of existing results directory.

    Plotting Options:
      -op OUTPUT_PREFIX, --output_prefix OUTPUT_PREFIX
                            Output prefix added before plot names. Default is 'sample'.
      -AS, --adaptive_sampling
                            Generate decision bar charts for adaptive sampling if utilized during sequencing.
      -VP VIOLIN_DATA_PERCENT, --violin_data_percent VIOLIN_DATA_PERCENT
                            Fraction of the data to use for the violin plot.
      -bin {seconds,minutes,5m,15m,hours}, --time_bin_unit {seconds,minutes,5m,15m,hours}
                            Time bin used for decision bar charts.

>[!Note] 
>The options `--single_charts` and `--comparison_metric` have been removed in this version. The module now automatically generates default comparison charts for both taxon mean read length and taxon mean coverage, and the summary table now includes only the columns: `Parameter`, `Test_Value`, `Control_Value`, and `taxon_id`.

## Handling Multiple FASTQ or FASTQ GZ Files (Single End Read Sets)

Typically, ONT sequencing runs produce multiple FASTQ files for each barcode after base calling. Use the following steps to concatenate those files: 

### Concatenating FASTQ Files
To concatenate multiple FASTQ files into a single FASTQ file, you can use the following command:

    cat file1.fastq file2.fastq > combined.fastq

### Concatenating FASTQ GZ Files and Uncompressing
To concatenate multiple FASTQ GZ files and uncompress them into a single FASTQ file, you can use the following commands:

concatenate:

    zcat file1.fastq.gz file2.fastq.gz > combined.fastq.gz

sequenoscope now supports gzip files, so you can directly run your gzip file in sequenoscope

#### Illumina Paired End Read Sets
Typically, paired end read sets will have a forward and a reverse compliment FASTQ that are compressed. Use these steps to run them: 

You should end up with two FASTQ files such as `Illumina_file_R1.fastq` and `Illumina_file_R2.fastq`which can then be run through sequenoscope ``analyze`` module like this:

    sequenoscope analyze --input_fastq Illumina_file_R1.fastq Illumina_file_R2.fastq --input_reference ref.fasta -o output -seq_type PE

>[!Note]
> When using Illumina short-read data, Sequenoscope will generate standard manifest files, summary metrics, and visualizations. However, because Illumina platforms do not support adaptive sampling or pore-level sequencing decisions, adaptive sampling–specific metrics are not available for paired-end short-read inputs.

#### Example Illumina Mock Community Dataset

To facilitate testing of Sequenoscope on short-read data, an Illumina-sequenced mock community dataset is also available. Specifically, we downloaded raw paired-end reads using accessions `ERR2935805` (log-distributed mock community) and `ERR2984773` (evenly distributed mock community). These datasets provide a practical reference for evaluating Sequenoscope’s behavior on non-ONT inputs.

Sequenoscope was applied on the `analyze` and `plot` modules to compare EVEN versus LOG mock community compositions. All expected outputs and visualizations were successfully generated, including sample manifest and sample manifest summary generation, and taxonomic visualizations when applied to Illumina short-read data.  



## Quick start

### analyze module
The analyze module provides specific sequencing statistics based on the reference FASTA file provided. Refer to the [outputs](#Outputs) section below for more details.

To quickly get started with the `analyze` module:

1. Ensure that you have the necessary input files and reference database prepared:
   - **Input FASTQ files:** Provide the path to the FASTQ files you want to process using the `--input_FASTQ` option.
   - **Reference database:** Specify the path to the reference database in FASTA format using the `--input_reference` option.

2. Choose an output directory for the results:
   - Specify the output directory path using the `--output` option.
  
3. Sprcify the sequencing type
   - Specify the sequencing type `-seq_type` as either Paired-end `PE` or Single-end `SE`

4. Run the module with the minimally required options:

       sequenoscope analyze --input_fastq <file.fq> --input_reference <ref.FASTA> -o <output_directory> -seq_type <sr>

This command will initiate the analysis module using the default settings. The input FASTQ file(s) will be processed, and the results will be saved in the specified output directory.

Please note that this is a simplified quick start guide, and additional options are available for advanced usage. For additional customization options and more detailed information on available options please run  `sequenoscope analyze -h` or `sequenoscope analyze --help`.


Remember to replace `<file.fq>` with the actual path to your FASTQ file, `<ref.FASTA>` with the path to your reference database, `<output_directory>` with the desired location for the output files and `<sr>` with your sequencing type (SE for single-end and PE for paired-end).


Taxon IDs are used as a naming convention, reflecting the sequence name in the FASTA file. The pipeline can process genes, subspecies, and other identifiers; it doesn't have to be a taxon.


We plan to incorporate [`fastplong`](https://github.com/OpenGene/fastplong) support in future tool releases, allowing users to select the optimal processing tool and further enhance long-read processing.

### filter_ONT module

To quickly get started with the `filter_ONT` module:

1. Ensure that you have the necessary input files prepared:
   - **Input FASTQ files:** Provide the path to the adaptive sequencing FASTQ files from ONT sequencer you want to process using the `--input_FASTQ` option.
   - **ONT sequencing summary file:** Specify the path to the ONT **sequencing summary file** using the `--input_summary` option that is either generated by MinKnow or base calling tool such as Guppy or Dorado.

2. Choose an output file and directory for the filtered reads:
   - Specify the output file path and directory using the `--output` option.

3. Set the desired filtering criteria:
   - You can optionally apply various filters to the reads based on the following criteria:
     - Read classification status*: Use the `-cls` or `--classification` option to designate the adaptive-sampling sequencing decision classification. Valid options are `'unblocked'`, `'stop_receiving'`, or `'no_decision'`.
     - Channel range/Pore number: Set the minimum and maximum channel/pore number range for filtering using the `-min_ch` and `-max_ch` options.
     - Duration: Define the minimum and maximum duration of the read sequencing time in seconds using the `-min_dur` and `-max_dur` options.
     - Run time range: Specify the minimum and maximum start time of the sequencing run in seconds using the `-min_start` and `-max_start` options.
     - Q score: Determine the minimum and maximum q score for filtering using the `-min_q` and `-max_q` options.
     - Read length range: Set the minimum and maximum read length for filtering using the `-min_len` and `-max_len` options.

  >[!Note]
  >Some sequence summary files lack the field specifying read classification status. A warning will be raised if this is the case.

4. Run the command with the basic required options:

        sequenoscope filter_ONT --input_fastq <file.fq> --input_summary <seq_summary.txt> -o <output.FASTQ>

This command will initiate the filtering process based on the specified criteria and save the filtered reads to the output FASTQ file.

Please note that this is a simplified quick start guide, and additional options are available for advanced usage. For more detailed information on available options, you can run `sequenoscope filter_ONT -h` or `sequenoscope filter_ONT --help`.

>[!Note] 
>Remember to replace `<file.fq>` with the actual path to your ONT sequencing FASTQ file, `<seq_summary.txt>` with the path to your ONT **sequencing summary file**, and `<output.FASTQ>` with the desired path and filename for the filtered reads.

### plot module

The **plot** module is designed for comparative analysis of two conditions (test and control) using outputs from the **analyze** module.

#### What It Does:
- Generates an **interactive Source File Taxon Covered Bar Chart** displaying the percentage of taxon covered bases from source files.
- Creates a **Summary Table (CSV)** listing key parameters (*Parameter, Test_Value, Control_Value, taxon_id*) from the manifest summary files.
- Produces **Violin Plots** comparing read quality scores and read lengths between test and control datasets.
- Automatically generates two default comparison charts:
  - **Taxon Mean Read Length Comparison**: A bar chart comparing `taxon_mean_read_length` values.
  - **Taxon Mean Coverage Comparison**: A bar chart comparing `taxon_mean_coverage` values.
- If **adaptive sampling** is enabled (`-AS` flag), the module produces **decision bar charts** (independent and cumulative), saved in a subdirectory named `decision_bar_charts_<time_bin_unit>`, where `<time_bin_unit>` reflects the user-selected time bin (e.g., minutes, 5m, etc.).

#### Required Paths:
- **Test Directory (`-T/--test_dir`)**: Contains manifest and manifest summary files for the test condition.
- **Control Directory (`-C/--control_dir`)**: Contains manifest and manifest summary files for the control condition.
- **Output Directory (`-o/--output_dir`)**: Directory where plots and summary files will be saved.

#### Plotting Options:
- **Output Prefix (`-op/--output_prefix`)**: Prefix added to output filenames (default: `sample`).
- **Adaptive Sampling (`-AS/--adaptive_sampling`)**: Enable decision bar charts (default: `False`).
- **Violin Data Fraction (`-VP/--violin_data_percent`)**: Fraction of data used for violin plots (default: `0.1`).
- **Time Bin Unit (`-bin/--time_bin_unit`)**: Time bin used for decision bar charts; choices: `seconds`, `minutes`, `5m`, `15m`, `hours` (default: `minutes`).

#### Run Command:
```sh
sequenoscope plot --test_dir <test_dir_path> --control_dir <control_dir_path> --output_dir <out_path>
```
Use `--force` to overwrite an existing output directory if needed.

>[!Note]
>Replace `<test_dir_path>`, `<control_dir_path>`, and `<out_path>` with actual paths.

## Outputs

### analyze module outputs

| F