NCBImeta 0.8.3
Efficient and comprehensive metadata acquisition from the NCBI databases (includes SRA).
License: MIT
Requires Python: >=3.7,<3.10

# ![ktmeaton/NCBImeta](https://raw.githubusercontent.com/ktmeaton/NCBImeta/master/logo.png)

**Efficient and comprehensive metadata acquisition from NCBI databases (includes SRA).**

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/ktmeaton/NCBImeta/blob/master/LICENSE)
[![Build Status](https://github.com/ktmeaton/NCBImeta/workflows/Building/badge.svg?branch=master)](https://github.com/ktmeaton/NCBImeta/actions?query=workflow%3ABuilding+branch%3Amaster)
[![codecov](https://codecov.io/gh/ktmeaton/ncbimeta/branch/master/graph/badge.svg)](https://codecov.io/gh/ktmeaton/NCBImeta/branch/master)
[![status](https://joss.theoj.org/papers/72376aa12ddf832465c92490b2074e7b/status.svg)](https://joss.theoj.org/papers/72376aa12ddf832465c92490b2074e7b)
[![GitHub issues](https://img.shields.io/github/issues/ktmeaton/NCBImeta.svg)](https://github.com/ktmeaton/NCBImeta/issues)
[![PyPI version](https://badge.fury.io/py/NCBImeta.svg)](https://badge.fury.io/py/NCBImeta)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/ncbimeta/badges/version.svg)](https://anaconda.org/bioconda/ncbimeta)

## Why NCBImeta?

NCBImeta is a command-line application that retrieves and organizes metadata from the National Centre for Biotechnology Information (NCBI). While the NCBI web browser experience allows filtered searches, the output does not facilitate inter-record comparison or bulk record retrieval. NCBImeta tackles this issue by creating a local database of NCBI metadata constructed by user-defined search criteria and customizable metadata columns. The output of NCBImeta, optionally a SQLite database or text files, can then be used by computational biologists for applications such as record filtering, project discovery, sample interpretation, or meta-analyses of published work.

## Requirements

- NCBImeta is written in Python 3 and supported on Linux and macOS.
- Dependencies that will be installed are listed in [requirements.txt](https://github.com/ktmeaton/NCBImeta/blob/master/requirements.txt).
- Python Versions:
  - 3.7
  - 3.8
  - 3.9
- Operating Systems:
  - Ubuntu
  - macOS

[Conda](https://docs.conda.io/en/latest/miniconda.html) is the recommended installation method. To install with pip, ```gcc``` is required.

## Installation

There are three installation options for NCBImeta:

### 1. Conda\*

> \* `mamba` is strongly recommended over `conda`!

```bash
conda env create -f environment.yaml
conda activate ncbimeta
```

### 2. PyPI\*

> \* `gcc` is required.

```bash
pip install ncbimeta
```

### 3. Github

```bash
git clone https://github.com/ktmeaton/NCBImeta.git
cd NCBImeta
pip install .
```

Test that the installation was successful:

```bash
NCBImeta --version
```

## Command-Line Parameters

```text
usage: NCBImeta [-h] --config CONFIGPATH [--flat] [--version]
                   [--email USEREMAIL] [--api USERAPI]
                   [--force-pause-seconds USERFORCEPAUSESECONDS]

NCBImeta: Efficient and comprehensive metadata retrieval from the NCBI
databases.

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIGPATH   Path to the yaml configuration file (ex. config.yaml).
  --flat                Don't create sub-directories in output directory.
  --version             show program's version number and exit
  --email USEREMAIL     User email to override parameter in config file.
  --api USERAPI         User API key to override parameter in config file.
  --force-pause-seconds USERFORCEPAUSESECONDS
                        FORCE PAUSE SECONDS to override parameter in config
                        file.
  --quiet               Suppress logging of each record to the console.  
```

## Quick Start Example

### Access the quick start config file

Download the NCBImeta github repository to get access to the example configuration files:

```bash
git clone https://github.com/ktmeaton/NCBImeta.git
cd NCBImeta
```

### Run the program

Download a selection of genomic metadata pertaining to the plague pathogen *Yersinia pestis*.

```bash
NCBImeta --flat --config test/test.yaml
```

(Note: The 'quick' start config file forces slow downloads to accommodate users with slow internet. For faster record retrieval, please see the [config file docs](https://github.com/ktmeaton/NCBImeta/blob/master/docs/config.md) to start editing config files.)

Example output of the command-line interface (v0.6.1):
[![asciicast](https://asciinema.org/a/289560.svg)](https://asciinema.org/a/289560)

### Annotate the database with the user's custom metadata

```bash
NCBImetaAnnotate \
  --database test/test.sqlite \
  --annotfile test/test_annot.txt \
  --table BioSample
```

Note that the first column of your annotation file MUST be a column that is unique to each record. An Accession number or ID is highly recommended. The column headers in your annotation file must also exactly match the names of your columns in the database.

`NCBImetaAnnotate` by default replaces the existing annotation with the data in your custom metadata file. Alternatively, the flag `--concatenate` can be specified. This will concatenate your custom metadata with the pre-existing value in the database cell (separated by a semi-colon).

```bash
NCBImetaAnnotate \
  --database test/test.sqlite \
  --annotfile test/test_annot.txt \
  --table BioSample \
  --concatenate
```

### Join NCBI tables into a unified master table

```bash
NCBImetaJoin \
  --database test/test.sqlite \
  --final Master \
  --anchor BioSample \
  --accessory "BioProject Assembly SRA Nucleotide" \
  --unique "BioSampleAccession BioSampleAccessionSecondary BioSampleBioProjectAccession"
```

The rows of the output "Master" table will be from the anchor table "BioSample", with additional columns added in from the accessory tables "BioProject", "Assembly", "SRA", and "Nucleotide". Unique accession numbers for BioSample (both primary and secondary) and BioProject allow this join to be unambiguous.

### Export the database to tab-separated text files by table.

```bash
NCBImetaExport \
  --database test/test.sqlite \
  --outputdir test
```

Each table within the database will be exported to its own tab-separated .txt file in the specified output directory.

### Explore!

1. Explore your database text files using a spreadsheet viewer (Microsoft Excel, Google Sheets, etc.)
2. Browse your SQLite database using DB Browser for SQLite (<https://sqlitebrowser.org/>)
3. Use the columns with FTP links to download your data files of interest.

Example database output (a subset of the BioSample table)

<img src="https://raw.githubusercontent.com/ktmeaton/NCBImeta/master/images/NCBImetaDB.gif" alt="NCBImetaDB" width="700px"/>

## Currently Supported NCBI Tables

- Assembly
- BioProject
- BioSample
- Nucleotide
- SRA
- Pubmed

## Recent and Upcoming Features

- [Project "Read The Docs"](https://github.com/ktmeaton/NCBImeta/projects/7): Documentation Overhaul - PLANNED
- [Project v0.8.3 - "Update Dependencies"](https://github.com/ktmeaton/NCBImeta/projects/13): Bugfixes for Installation - DEVELOPMENT
- [Project v0.8.2 - "Annotate Simplicity"](https://github.com/ktmeaton/NCBImeta/projects/12): Simplify the Annotate Command - RELEASED

## Documentation

To get started with customizing the search terms, database, and metadata fields, please read:

1. [Config File Docs](https://github.com/ktmeaton/NCBImeta/blob/master/docs/config.md)
2. [Schema File Docs](https://github.com/ktmeaton/NCBImeta/blob/master/schema/schema.md)

## Issues, Questions, and Suggestions

Please submit your questions, suggestions, and bug reports to the
[Issue Tracker](https://github.com/ktmeaton/NCBImeta/issues).

Please do not hesitate to post any manner of curiosity in the "Issues" tracker :) User-feedback and ideas are the most valuable resource for emerging software.

GitHub not your style? Join the [NCBImeta Slack Group](https://join.slack.com/t/ncbimeta/shared_invite/zt-crbtn51t-bdQKeBqz6Nkmj~hgU8ZZFA) to see release alerts, chat with other users, and get insider perspective on development.

## Contributing

Want to add features and fix bugs? Check out the [Contributor's Guide](https://github.com/ktmeaton/NCBImeta/blob/master/.github/CONTRIBUTING.md) for suggestions on getting started.

## Citation

Eaton, K. (2020). NCBImeta: efficient and comprehensive metadata retrieval from NCBI databases. Journal of Open Source Software, 5(46), 1990, <https://doi.org/10.21105/joss.01990>

## Authors

Author: [Katherine Eaton](https://github.com/ktmeaton) (ktmeaton@gmail.com)  

## Additional Contributors

Those who have filed issues, pull-requests, and participated in discussions.

- [Andreas Sjödin](https://github.com/druvus)
- [hmontenegro](https://github.com/hmontenegro)
- [Matthew Gopez](https://github.com/hellothisisMatt)
- [Philip Mabon](https://github.com/Takadonet)

[![forthebadge made-with-python](http://ForTheBadge.com/images/badges/made-with-python.svg)](https://www.python.org/)