bioidtracker 0.1.4
Biological ID matching between different times and/or databases.
License: MIT
Requires Python: >=3.8,<4.0.0

bioidtracker
===========================

|PyPI| |License| |Read the Docs| |Build| |Tests| |Codecov| |pre-commit| |Black|

.. |PyPI| image:: https://img.shields.io/pypi/v/bioidtracker.svg
   :target: https://pypi.org/project/bioidtracker/
   :alt: PyPI
.. |License| image:: https://img.shields.io/github/license/inecik/bioidtracker
   :target: https://opensource.org/licenses/MIT
   :alt: License
.. |Read the Docs| image:: https://img.shields.io/readthedocs/bioidtracker/latest.svg?label=Read%20the%20Docs
   :target: https://bioidtracker.readthedocs.io/
   :alt: Read the documentation at https://bioidtracker.readthedocs.io/
.. |Build| image:: https://github.com/inecik/bioidtracker/workflows/Build%20bioidtracker%20Package/badge.svg
   :target: https://github.com/inecik/bioidtracker/actions?workflow=Package
   :alt: Build Package Status
.. |Tests| image:: https://github.com/inecik/bioidtracker/workflows/Run%20bioidtracker%20Tests/badge.svg
   :target: https://github.com/inecik/bioidtracker/actions?workflow=Tests
   :alt: Run Tests Status
.. |Codecov| image:: https://codecov.io/gh/inecik/bioidtracker/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/inecik/bioidtracker
   :alt: Codecov
.. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white
   :target: https://github.com/pre-commit/pre-commit
   :alt: pre-commit
.. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black
   :alt: Black

Objectives
~~~~~~~~~~

In the past decades, high throughput sequencing technologies revolutionized the biological research... <Complete this paragraph>

There are many results coming from various kinds of experimental techniques of individual cells or tissues such as microarrays, RNA-seq, single-cell RNA-seq, CITE-seq, ATAC-Seq and so on... <Complete this paragraph>

One of the important aspect of current biological interest is to combine the biological data from various sources into one coherent platform...   For single cell data, explain sfaira... <Complete this paragraph>

One of the challenge behind data integration is the fact that the experimental results were published in different times and potentially using different source annotation files. After next generation sequencing, the raw files (fastq files) is normally mapped to the genome of interest. Then, the mapping is interpretted by using an annotation source (gtf or gff3 files). These annotation files simply indicate which part of the genome belongs to gene, exon, transcript and so on. However, the annotation of genome is also not precise as the new scientific findings result in them to change. As a result, annotation providers (Refseq, Ensembl etc) publish new release of the annotation files, which involves updates to existing one.

The most simple and naive approach to integrate all the data, disregarding the difference of annotation files at the time when the experimental result is published, is to re-map all the raw fastq files using the most updated annotation source. However, this is computationally very costly. Also, experimental details such as the library preparation details or adaptor sequences are sometimes are not clearly shared by the authors of the scientific papers. This makes the re-mapping prone to errors, causes to create uncertanities, and requires a lot of human effort.

This package aims to circumvent the problem. Here, the package aims to match the IDs/names of transcripts, genes, or proteins from various experimental results into the ones in the requested annotation provider and release. The experimental results are often shared with count tables, where the IDs/names of transcripts, genes, or proteins are indicated in the rows. As the main problem is inconsistency of the IDs/names due to use of different annotation source at the time of publication, this package finds the corresponding IDs/names for the ones in the count tables at the requested (it can be the latest) annotation release and requested annotation source (Refseq, Ensembl etc).

The package not only provides a time-travel functionality for IDs/names of transcripts, genes, or proteins but also converts different forms with each other (like transcript to gene) and converts in between different databases (annotation providers or naming conventions). The package matches the dataset identifiers (IDs/names) from various experimental results very fastly, using very limited disk space. It works for all the organisms in the Ensembl database.

Workflow
~~~~~~~~

.. figure:: https://raw.githubusercontent.com/inecik/bioidtracker/development/resources/images/workflow.png
    :width: 650px
    :align: center

There are two main objectives of the package:

1.  Identify a given ID/name list by its provider and corresponding Ensembl release.

2.  Convert them into the provider, release and form of interest.

Below figure provides the summary of the main elements of the package. The workflow starts with the dataset (list of IDs/names) source identification step. Then, the IDs are converted into the Ensembl gene IDs following two conversion procedure: external to Ensembl and then Ensembl form conversion. The graph in the next step, which is the backbone of the package, works for Ensembl gene, transcript and translation IDs but the best practice is to use it with gene IDs. Graph provides the time-travel functionality of the package. Then, the resulting IDs from the graph follows the same two steps followed before the graph in reverse order: form conversion and dataset convertion.

The workflow is an example best practice, but the details of the processing steps and connections between them are not fixed intentionally. The package can be flexible used depending on the needs of the user. The user should read the `<Usage_>`_ to learn the details of how the package is used.

Graph History
~~~~~~~~~~~~~

.. figure:: https://raw.githubusercontent.com/inecik/bioidtracker/development/resources/images/graph.png
    :width: 600px
    :align: center

This feature of the package is called history travel as the matching is between different `Ensembl releases`. Graph is composed of nodes named with an Ensembl ID, connected by edges. Node connections are based on the Ensembl ID history, there are five major edge types:

-   Self-loop: When an Ensembl ID is redefined without changing the ID and version, a self-loop around a node is created.
-   Branching: When an Ensembl ID is branched out into different Ensembl IDs losing/keeping the origin Ensembl ID. If branching keeps origin Ensembl ID, the version can change but not necessarily.
-   Merging: Similar to branching, but in reverse direction. Multiple Ensembl IDs come together to form a single Ensembl ID.
-   Redefinition: When an Ensembl ID is lost and another one is born. The edge is just only one line, no branching or merging happen.
-   Version: When an Ensembl ID is not lost but the version of the ID changes.

The connections between nodes composed of edges in above types can be quite intertwined and unexpectedly complicated. The function below follows the edges origin nodes recursively, collecting the edge weight along the way. When the final `Ensembl release` is reached, then it reduces the collected edge weights. The result is a list of dictionary with keys `Score` and ` Ensembl ID`.

It is possible to indicate the `target release`, and the `target release` can be also smaller than the `origin release`. In this case, the graph is reversed and the same recursive function is run in the reversed graph. It is also possible to omit the `origin release` if it is unknown. In this case, the graph is a bit more error prone in exceptional cases but works robustly anyway.

Edges contain the information by which two nodes are associated: `old release`, `new release` and `weight`. While first two is important for the recursive path finder to function, the `weight` variable (which is based on the pairwise similarity between two nodes) is important when it comes to scoring the found paths and choosing the best one among them.

.. _Usage: https://bioidtracker.readthedocs.io/en/latest/usage.html