ndexstringloader 1.0.3
Loads STRING data into NDEx
License: BSD license
Keywords: ndexstringloader

==========================
NDEx STRING Content Loader
==========================


.. image:: https://img.shields.io/pypi/v/ndexstringloader.svg
        :target: https://pypi.python.org/pypi/ndexstringloader

.. image:: https://img.shields.io/travis/vrynkov/ndexstringloader.svg
        :target: https://travis-ci.org/ndexcontent/ndexstringloader

.. image:: https://coveralls.io/repos/github/ndexcontent/ndexstringloader/badge.svg?branch=master
        :target: https://coveralls.io/github/ndexcontent/ndexstringloader?branch=master

.. image:: https://readthedocs.org/projects/ndexstringloader/badge/?version=latest
        :target: https://ndexstringloader.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status


Python application for loading STRING data into `NDEx <http://ndexbio.org>`_.

This tool downloads and unpacks the `STRING <https://string-db.org/>`_ files below

    `9606.protein.links.full.v11.0.txt.gz <https://stringdb-static.org/download/protein.links.full.v11.0/9606.protein.links.full.v11.0.txt.gz>`_

    `human.name_2_string.tsv.gz <https://string-db.org/mapping_files/STRING_display_names/human.name_2_string.tsv.gz>`_

    `human.entrez_2_string.2018.tsv.gz <https://string-db.org/mapping_files/entrez/human.entrez_2_string.2018.tsv.gz>`__

    `human.uniprot_2_string.2018.tsv.gz <https://string-db.org/mapping_files/uniprot/human.uniprot_2_string.2018.tsv.gz>`_

This loader generates one or more TSV files, converts them to CX, and uploads them to NDEx server.
The number of networks generated is dictated by the :code:`--cutoffscore` parameter which by
default generates two networks, one with all edges `0.0 cutoffscore` and one with edges with score `0.7`
and above

Duplicate edges (edges that have the same Source and Target nodes and the same value of :code:`combined_score`)
are included to the generated TSV and CX files only once.

Name of the newly generated network includes the value of :code:`cutoffscore` argument, for example,
:code:`STRING - Human Protein Links - High Confidence (Score >= 0.7)`.

In case user didn't specify :code:`--update UUID` argument, then the network with this name gets over-written in case if already exists on NDEx server;
otherwise, a new network is created.

Specifying :code:`--update UUID` command line argument will over-write network with this UUID if it is found.
If not, then user is asked if (s)he wants to create a new network. When network is updated, only edges and nodes are
changed; network attributes other then **version** are not modified.


.. note::

    :code:`--stringversion` overrides default version of :code:`11.5` for all files with a version when downloaded

**1\)** Below is an example of a record
from `9606.protein.links.full.v11.0.txt.gz <https://stringdb-static.org/download/protein.links.full.v11.0/9606.protein.links.full.v11.0.txt.gz>`__

.. code-block::

   9606.ENSP00000261819 9606.ENSP00000353549 0 0 0 0 0 102 90 987 260 900 0 754 622 999


To generate a STRING network, the loader reads rows from that file one by one and compares the value of the last
column :code:`combined_score` with the value :code:`cutoffscore` argument.  The row is not added to the network generated in case
:code:`combined_score` is less than the commad-line argument :code:`cutoffscore`.


**2\)** If :code:`combined_score` is no than less :code:`cutoffscore`, the loader process two first columns

.. code-block::

   column 1 - protein1 (9606.ENSP00000261819)
   column 2 - protein2 (9606.ENSP00000353549)

When processing first column :code:`protein1`, the script

replaces :code:`Ensembl Id` with a :code:`display name`, for example :code:`9606.ENSP00000261819` becomes :code:`ANAPC5`. Mapping of
display names to :code:`Enseml Ids` is found in
`human.name_2_string.tsv.gz <https://string-db.org/mapping_files/STRING_display_names/human.name_2_string.tsv.gz>`__

uses `human.uniprot_2_string.2018.tsv.gz <https://string-db.org/mapping_files/uniprot/human.uniprot_2_string.2018.tsv.gz>`__
to create :code:`represents` value.  For example, :code:`represents` for :code:`9606.ENSP00000261819` is :code:`uniprot:Q9UJX4`

uses `human.entrez_2_string.2018.tsv.gz <https://string-db.org/mapping_files/entrez/human.entrez_2_string.2018.tsv.gz>`__
to create list of aliases for the current protein.  Thus, list of aliases for :code:`9606.ENSP00000261819` is
:code:`ncbigene:51433|ensembl:ENSP00000261819`

**3\)** The second column :code:`protein2` is processed the same way as :code:`column 1`.

**4\)**  In the generated tsv file :code:`9606.protein.links.tsv`, :code:`protein1` and :code:`protein2` values from the original file are replaced with

.. code-block::

   protein_display_name_1 represents_1 alias_1 protein_display_name_2 represents_2 alias_2

So, the original

.. code-block::

   9606.ENSP00000261819 9606.ENSP00000353549 0 0 0 0 0 102 90 987 260 900 0 754 622 999

becomes

.. code-block::

   ANAPC5 uniprot:Q9UJX4 ncbigene:51433|ensembl:ENSP00000261819 CDC16 uniprot:Q13042  ncbigene:8881|ensembl:ENSP00000353549 0 0 0 0 0 102 90 987 260 900 0 754 622 999


**5\)**  The generated tsv file :code:`9606.protein.links.tsv` is then transformed to CX :code:`9606.protein.links.cx`.
The default style defined in :code:`style.cx` distributed with this loader is applied to the
generated network in case neither :code:`--style` nor :code:`--template` is specified.
User can specify style template file with either :code:`--style` argument or
style template network UUID :code:`--template UUID_of_style_template_network`.
Specifying both :code:`--template` and :code:`--style` is not allowed.

**6\)**  :code:`9606.protein.links.cx` is then uploaded to NDEx server either replacing
an existing network (in case :code:`--update UUID` is specified),
or creating a new network.


Dependencies
------------

* ndex2
* ndexutil
* networkx
* scipy
* requests
* py4cytoscape
* pandas


Compatibility
-------------

* Python 3.6+

Installation
------------

.. code-block::

   git clone https://github.com/ndexcontent/ndexstringloader
   cd ndexstringloader
   make dist
   pip install dist/ndexloadstring*whl


Run **make** command with no arguments to see other build/deploy options including creation of Docker image

.. code-block::

   make

Output:

.. code-block::

   clean                remove all build, test, coverage and Python artifacts
   clean-build          remove build artifacts
   clean-pyc            remove Python file artifacts
   clean-test           remove test and coverage artifacts
   lint                 check style with flake8
   test                 run tests quickly with the default Python
   test-all             run tests on every Python version with tox
   coverage             check code coverage quickly with the default Python
   docs                 generate Sphinx HTML documentation, including API docs
   servedocs            compile the docs watching for changes
   testrelease          package and upload a TEST release
   release              package and upload a release
   dist                 builds source and wheel package
   install              install the package to the active Python's site-packages
   dockerbuild          build docker image and store in local repository
   dockerpush           push image to dockerhub


Configuration
-------------

The **ndexloadstring.py** requires a configuration file to be created.
The default path for this configuration is :code:`~/.ndexutils.conf` but can be overridden with
:code:`--conf` flag.

**Configuration file**

Networks listed in **[network_ids]** section need to be visible to the **user**

.. code-block::

    [ndexstringloader]
    user = joe123
    password = somepassword123
    server = dev.ndexbio.org


Needed files
------------

Load plan is required for running this script.  **string_plan.json**  found at **ndexstringloader/ndexstringloader** can be used for this purpose.


Usage
-----

For information invoke :code:`ndexloadstring.py -h`

**Example usage**

Here is how this command can be run for **dev** and **prod** targets:

.. code-block::

   ndexloadstring.py --profile dev tmpdir/

   ndexloadstring.py --profile prod tmpdir/ --cutoffscore 0.99 0.95


Credits
-------

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage


=======
History
=======

1.0.3 (2023-09-20)
-------------------

* Updated URL paths to data files because they moved on STRING server

* Set default STRING version to "12.0"

* Version of STRING data used is now appended to the network name.

* Updated default network description and default style template file

1.0.2 (2022-06-29)
-------------------

* Fixed bug where ``--stringversion`` was being ignored when
  downloading data files

* Set default version to ``11.5``

* Fixed bug where ``version`` network attribute was not being updated
  with value of ``--stringversion``

* Changed URL to `human.entrez_2_string.2018.tsv.gz` cause it
  moved on STRING server

* ``--cutoffscore`` parameter can now take multiple values and a network
  for each value will be generated and uploaded to NDEx. The default
  is set to generate a network with all edges (0.0 --cutoffscore) and a
  network with edges 0.7 and above

1.0.0 (2020-11-11)
------------------

* New default behavior: **force-directed-cl** layout is now applied on
  networks via py4cytoscape library and a running instance of Cytoscape.
  Alternate Cytoscape layouts and the networkx "spring" layout can be
  run by setting appropriate value via the new **--layout** flag

0.3.0 (2020-10-28)
------------------

* Added ``--skipupload`` that lets caller skip upload of network to NDEx

* Spring layout applied by default for all networks that have less then 2,000,000
  edges. This can be overridden with new flag ``--layoutedgecutoff``

0.2.4 (2019-12-01)
------------------
* Fixed defect UD-462 Verify new network attributes are correctly set in ndexstringloader (https://ndexbio.atlassian.net/browse/UD-462).

0.2.3 (2019-09-13)
------------------
* If user loads the entire STRING network (i.e., runs the script with --cutoffscore 0), the name of the resulting netwpork should be "STRING - Human Protein Links", not "STRING - Human Protein Links - High Confidence".

0.2.2 (2019-09-12)
------------------
* Added new featured specified by UD-577 Quick improvement for new String loader (added optional --update argument that allows to specify the UUID of a target network to update; added optional --template argument that allows to specify the UUID of a target network to use as style template, the update operation now only changes nodes and edges, but leaves network properties untouched).

0.2.1 (2019-08-23)
------------------
* Improved README file.
* Added new JUnit tests (JUnit test coverage is 87%).

0.2.0 (2019-07-26)
------------------
* Removed duplicate edges. Every pair of connected nodes in STRING networks had the same edge duplicated (one edge going from A to B, and another going from B to A).  Since edges in STRING are not directed, we can safely remove half of them.

* Added new arguments to command line:
   optional --cutoffscore (default is 0.7) - used to filter on combined_score column. To include edges with combined_score of 800 or higher, --cutoffscore 0.8 should be specified

   required --datadir specifies a working directory where STRING files will be downloaded to and processed style.cx file that contains style is supplied with the STRING loader and used by default. It can be overwritten with --style argument.

0.1.0 (2019-03-13)
------------------
* First release on PyPI.