boldigger3 2.2.0
A python package to query different databases of boldsystems.org v5!
License: MIT
Requires Python: >=3.11

# BOLDigger3
 
![BOLDigger3](https://github.com/user-attachments/assets/99606be9-8ae1-4e90-a5d5-4a6a3311f48c)


[![Downloads](https://pepy.tech/badge/boldigger3)](https://pepy.tech/project/boldigger3)

A Python program to query .fasta files against the databases of www.boldsystems.org v5!

## Introduction
DNA metabarcoding datasets often comprise hundreds of Operational Taxonomic Units (OTUs), requiring querying against databases for taxonomic assignment. The Barcode of Life Data system (BOLD) is a widely used database for this purpose among biologists. However, BOLD's online platform limits users to identifying batches of only 1000, 200 or 100  (depending on operating mode) sequences at a time.

BOLDigger3, the successor to BOLDigger2 and BOLDigger, aims to overcome these limitations. As a pure Python program, BOLDigger3 offers:

- Automated access to BOLD's identification engine
- Downloading of BOLD's latest data package release to access all metadata
- Selection of the best-fitting hit from the returned results

## Overview

**BOLDigger3** is an automated tool designed for DNA sequence identification through **BOLDSystems v5**, supporting integration into bioinformatics pipelines with enhanced functionality and performance. With BOLDigger3, users can identify up to **10,000 sequences per hour** without the need for credentials, using an optimized data storage and queuing system that improves speed and process safety.

## Key Differences Between BOLDigger3 and BOLDigger2

- **Unified Function**: BOLDigger3 consolidates all actions into a single function, `identify`, which automatically performs identification, additional data downloading, and top-hit selection, making it easier to integrate into pipelines.
- **Enhanced Database Accessibility**: Users have access to all databases offered by **BOLDSystems v5** and can select from three different operating modes.
- **Improved Speeds**: Depending on the operating mode, BOLDigger3 can identify up to **10,000 sequences per hour**, significantly faster than BOLDigger2.
- **No Password Required**: Users no longer need credentials to perform identificationsâ€”just select the FASTA file, database, and operating mode to start.
- **Streamlined Data Storage**: Data is stored in an **DuckDB database** for faster processing, with final outputs available in `.xlsx` and `.parquet` formats.
- **Process Safety**: BOLDigger3 can resume interrupted executions, continuing exactly where it left off.
- **Dynamic Queuing**: The tool automatically manages request queuing based on the selected operating mode.

## Features

- **Identify Sequences Automatically**: Run DNA sequence identifications with a single command.
- **Flexible Database Options**: Access to all BOLDSystems v5 databases with user-selected operating modes.
- **High-Performance Processing**: Up to 10,000 identifications per hour, depending on settings.
- **Robust Storage**: Data stored in DuckDB format for efficient processing; results in `.xlsx` and `.parquet`.
- **User-Friendly**: No credentials needed for use.

## Installation and Usage

BOLDigger3 requires Python version 3.11 or higher and can be easily installed using pip in any command line:

`pip install boldigger3`

This command will install BOLDigger3 along with all its dependencies.

To run the ```identify``` function, use the following command:

`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE`

# Databases

The ```--db``` is a number between 1 and 8 corresponding to the eight databases BOLD v5 currently offers:

1: **ANIMAL LIBRARY (PUBLIC)**   
2: **ANIMAL SPECIES-LEVEL LIBRARY (PUBLIC + PRIVATE)**   
3: **ANIMAL LIBRARY (PUBLIC+PRIVATE)**    
4: **VALIDATED CANADIAN ARTHROPOD LIBRARY**   
5: **PLANT LIBRARY (PUBLIC)**   
6: **FUNGI LIBRARY (PUBLIC)**   
7: **ANIMAL SECONDARY MARKERS (PUBLIC)**   
8: **VALIDATED ANIMAL RED LIST LIBRARY**

# Operating modes

The ```--mode``` is a number between 1 and the corresponding to the 3 operating modes BOLD v5 currently offers:

1: **Rapid Species Search**   
2: **Genus and Species Search**   
3: **Exhaustive Search**   

To customize the implemented thresholds for user-specific needs, the thresholds can be passed as an additional (ordered) argument. Up to five different thresholds can be passed for the different taxonomic levels (Species, Genus, Family, Order, Class). Thresholds not passed will be replaced by default, but BOLDigger3 will also inform you about this:

`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE --thresholds 99 97`

Output:

```
19:16:16: Default thresholds changed!
19:16:16: Species: 99, Genus: 97, Family: 90, Order: 85
```

When a new version is released, you can update BOLDigger3 by typing:

`pip install --upgrade boldigger3`

## How to cite

Buchner D, Leese F (2020) BOLDigger â€“ a Python package to identify and organise sequences with the Barcode of Life Data systems. Metabarcoding and Metagenomics 4: e53535. https://doi.org/10.3897/mbmg.4.53535


## The BOLDigger3 Algorithm

The BOLDigger3 algorithm operates as follows:

1. **Check for database updates**: BOLDigger3 will check if there is an updated data package release and download it if needed.

2. **Compile DuckDB database**: BOLDigger3 will parse the TSV from BOLD and save it in a DuckDB database for fast lookups.

3. **Split the FASTA**: The input FASTA file is divided into chunks that fit the limits of the selected operating mode of the identification engine.

4. **Queue the Chunks**: These chunks are then queued in the identification engine for processing.

5. **Check for Results**: The algorithm periodically checks if any results can be downloaded.

6. **Data Download**: Once results are available, the data is downloaded.

7. **Data Validation**: The algorithm ensures that all data has been correctly downloaded.

8. **Retrieve Additional Data**: Additional data is added via the DuckDB database.

9. **Select Top Hit**: Finally, the algorithm selects the top hit backed by the most database entries for the final output.

### Top hit selection

Different thresholds (97%: species level, 95%: genus level, 90%: family level, 85%: order level) for the taxonomic levels are used to find the best fitting hit. After determining the threshold for all hits the most common hit above the threshold will be selected. Note that for all hits below the threshold, the taxonomic resolution will be adjusted accordingly (e.g. for a 96% hit the species-level information will be discarded, and genus-level information will be used as the lowest taxonomic level).

The BOLDigger3 algorithm functions as follows:

1. **Identify Maximum Similarity**: Find the maximum similarity value among the top 100 hits currently under consideration.
   
2. **Set Threshold**: Set the threshold to this maximum similarity level. Remove all hits with a similarity below this threshold. For example, if the highest hit has a similarity of 100%, the threshold will be set to 97%, and all hits below this threshold will be removed temporarily.

3. **Classification and Sorting**: Count all individual classifications and sort them by abundance.

4. **Filter Missing Data**: Drop all classifications that contain missing data. For instance, if the most common hit is "Arthropoda --> Insecta" with a similarity of 100% but missing values for Order, Family, Genus, and Species.

5. **Identify Common Hit**: Look for the most common hit that has no missing values.

6. **Return Hit**: If a hit with no missing values is found, return that hit.

7. **Threshold Adjustment**: If no hit with no missing values is found, increase the threshold to the next higher level and repeat the process until a hit is found.


### BOLDigger3 Flagging System

BOLDigger3 employs a flagging system to highlight certain conditions, indicating a degree of uncertainty in the selected hit. Currently, there are five flags implemented, which may be updated as needed:

1. **Reverse BIN Taxonomy**: This flag is raised if all of the top 100 hits representing the selected match utilize reverse BIN taxonomy. Reverse BIN taxonomy assigns species names to deposited sequences on BOLD that lack species information, potentially introducing uncertainty.

2. **Differing Taxonomic Information**: If the percentage of hits represented by the selected top hit is smaller than 90%, flag 2 will be raised indicating a potential taxonomic conflict. If your top hit is represented by 99 hits and there is 1 hit with differing taxonomy, this flag will not be raised.

3. **Private Data**: If all of the top 100 hits representing the top hit are private hits, this flag is raised, indicating limited accessibility to data.

4. **Unique Hit**: This flag indicates that the top hit result represents a unique hit among the top 100 hits, potentially requiring further scrutiny.

5. **Multiple BINs**: If the selected species-level hit is composed of more than one BIN, this flag is raised, suggesting potential complexities in taxonomic assignment.

Given the presence of these flags, it is advisable to conduct a closer examination of all flagged hits to better understand and address any uncertainties in the selected hit.