semvec/pypi-packages huggingface dataset:
pypore 0.0.6.dev20180702231556
Pythonic/Cythonic Nanopore Translocation Analysis

**master** branch

.. image:: https://travis-ci.org/parkin/pypore.svg?branch=master 
    :target: https://travis-ci.org/parkin/pypore
    
.. image:: https://coveralls.io/repos/parkin/pypore/badge.png?branch=master
    :target: https://coveralls.io/r/parkin/pypore?branch=master

**develop** branch

.. image:: https://travis-ci.org/parkin/pypore.svg?branch=develop
    :target: https://travis-ci.org/parkin/pypore
    
.. image:: https://coveralls.io/repos/parkin/pypore/badge.png?branch=develop
    :target: https://coveralls.io/r/parkin/pypore?branch=develop
   
Pypore
=======

This project is for nanopore-translocation event finding/analysis.
It's written in Python, with `numpy <http://www.numpy.org/>`_/`scipy <http://www.scipy.org/>`_ as a Matlab replacement.
The gui library is `pyside <http://qt-project.org/wiki/PySide>`_, and `pyqtgraph <http://www.pyqtgraph.org/>`_ is used for plotting.
More information can be found `here <http://parkin.github.io/pypore>`_.

**Note** that Pypore/PyporeGui are pre-release, major changes can happen at any time.

Pull requests are welcome!

Installing
----------

Dependencies
++++++++++++

Pypore has the following dependencies:

* numpy
* scipy
* PyTables

PyporeGui depends on all of the above and:

* PySide
* pyqtgraph

Building from source additionally requires:

* Cython

Running the tests requires:

* nose

Installing with pip
+++++++++++++++++++

Pypore is currently pre-release, so we'll install with the **--pre** tag.

.. code:: shell

    pip install pypore --pre
    
A new dev release is deployed with every push to the **develop** branch, so upgrade often!

.. code:: shell

    pip install pypore --pre --upgrade
    
Installing from source
++++++++++++++++++++++

Clone or download the source. In the directory (with setup.py), run one of the following:

.. code:: shell

    python setup.py install
    
or

.. code:: shell

    pip install .
    
Running the tests:

.. code:: shell

    python setup.py tests
    
PyporeGui
=========

Start PyporeGui with:

.. code:: python

    import pyporegui
    pyporegui.start()


License
=======

    Copyright 2014 Will Parkin
    
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
fanstore 0.0.1a0
Fanstore gathers local storage space in computer clusters to enable distirbuted neural networks training with larger datasets

# Overview 
Fanstore is a shared object store to support parallel neural network training. Fanstore provides a POSIX-compatible file system interface through fusepy, and low latency communication through mpi4py. Fanstore can use main memory, RAM disk, and local storage for transient parallel I/O at run time.

# To start
```
sbatch bin/fanstore.slurm
```

# To manually start fanstore
## The complete ImageNet dataset
```
module load python3
mpiexec.hydra -f ../test/hostfile -ppn 1 python3 fanstore.py /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagenet/16-parts --loadbcast /work/00946/zzhang/imagenet/16-parts-validation &
```

## A quarter of the ImageNet dataset
```
mpiexec.hydra -f ../test/hostfile -ppn 1 python3 fanstore.py /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagen
et/16-parts-test --loadbcast /work/00946/zzhang/imagenet/16-parts-validation &
```


# To run a horovod application
```
module load cuda/9.0 cudnn/7.0
mpiexec.hydra -f /work/00946/zzhang/maverick2/fanstore/test/hostfile -ppn 4  python3 keras_imagenet_resnet50_fanstore.py
```

# Before terminating the job
```
for h in `cat ../test/hostfile`; do   ssh $h "rm -rf /tmp/data; mkdir /tmp/data; mkdir -p /tmp/amfora; rm /tmp/fuse-fanstore.log; fusermount -u /tmp/amfora"; done
```