semvec/pypi-packages huggingface dataset:
pycondusco 0.1.0
pycondusco lets you run a function iteratively, passing it the rows of a dataframe or the results of a query.

# PyCondusco

## Overview

pycondusco lets you run a function iteratively, passing it the rows of a dataframe or the results of a query.

We call the functions pycondusco runs pipelines, and define a pipeline as a function that accepts a list of parameters and does something based on the values of the parameters.

The most common use case for pycondusco are data pipelines.  For data pipelines that primarily run SQL queries, we can template queries with a library (ie. [pystache](https://github.com/defunkt/pystache)), so that parametrized values are separated from the query logic.  We can then render the query with the appropriate values:

```
import pycondusco
from pycondusco.run_pipeline import run_pipeline
import pystache

json_string = '{"first_name": "First", "last_name":"Last"}'

params = [
    {
        'k1':'v1',
        'k2':'v2',
    },
    {
        'k1':'v1',
        'k2': json_string,
    },
]

def pipeline(params):
    print pystache.render('k1 value is {{k1}}, k2 is {{k2}}',params)

run_pipeline(pipeline,params)
```


pycondusco provides the following extensions in functionality to the above design pattern:
 - the user can provide a query and each row of results is iteratively passed to the pipeline
 - any JSON-string parameter will be converted to an object before being passed to the pipeline


## Functions

|function|description|
|:--------------|:--------------|
|run_pipeline(pipeline, parameters)| iteratively pass each row of parameters to a pipeline, converting any JSON parameters to objects|
|run_pipeline_gbq(pipeline, query, project)|calls run_pipeline with the results of query executed via bigquery|


## Installation

```
pip install pycondusco
export GOOGLE_APPLICATION_CREDENTIALS="<FILE_PATH_TO_CREDENTIALS>"
```

## Features

*   Name-based substitution of query-results including JSON into pipelines, iterating through rows of parameters dataframe:
```
import pystache
from google.cloud import bigquery
import pycondusco
from pycondusco.run_pipeline_gbq import run_pipeline_gbq

client = bigquery.Client()

def pipeline(params):
    query = """
      SELECT
        {{#list}}
          SUM(CASE WHEN author.name ='{{name}}' THEN 1 ELSE 0 END) as n_{{name_clean}},
        {{/list}}
        repo_name
      FROM `bigquery-public-data.github_repos.sample_commits`
      GROUP BY repo_name
    """

    query_job = client.query(pystache.render(query, params))
    results = query_job.result()  # Waits for job to complete.
    for row in results:
        print(dict(row.items()))


query = """
   SELECT CONCAT('[',
   STRING_AGG(
     CONCAT('{\"name\":\"',name,'\",'
       ,'\"name_clean\":\"', REGEXP_REPLACE(name, r'[^[:alpha:]]', ''),'\"}'
     )
   ),
   ']') as list
   FROM (
     SELECT author.name,
       COUNT(commit) n_commits
     FROM `bigquery-public-data.github_repos.sample_commits`
     GROUP BY 1
     ORDER BY 2 DESC
     LIMIT 10
   )
"""

run_pipeline_gbq(pipeline, client, query)
```
heyexReader 0.1.3
Python package for reading and parsing Heyex Heidelberg Spectralis OCT files

# heyexReader
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![Documentation Status](https://readthedocs.org/projects/heyexreader/badge/?version=latest)](https://heyexreader.readthedocs.io/en/latest/?badge=latest)
[![Python 2.7](https://img.shields.io/badge/python-2.7-blue.svg)](https://www.python.org/downloads/release/python-270/)
[![Python 3.6](https://img.shields.io/badge/python-3.6-blue.svg)](https://www.python.org/downloads/release/python-360/)

This is a native python implementation for reading Heidelberg Heyex
Spectralis files. The current version only supports the Heyex VOL files.

The full documentation of this project is located at: 
<http://heyexreader.readthedocs.io/>

# Installation

    $ pip install heyexReader

# Requirements

  - Numpy
  - Pillow

# Getting Started

``` python
import heyexReader

vol = heyexReader.volFile("test.vol")

vol.renderIRslo("slo.png", renderGrid = True)
vol.renderOCTscans("oct", renderSeg = True)

print(vol.oct.shape)
print(vol.irslo.shape)
```

# Thanks To

The basis for this code was ported from Markus Mayer's excellent work <https://www5.cs.fau.de/research/software/octseg/>. 


# Funded by
[![Research to Prevent Blindness](/funding/rpb.png?raw=true "RPB")](https://www.rpbusa.org/rpb/)
[![National Eye Institute](/funding/nei.jpg?raw=true "NEI")](https://nei.nih.gov/)
DataSuper 0.10.4
Organization for scientific projects.

DataSuper
=========

.. image:: https://img.shields.io/circleci/project/github/dcdanko/DataSuper/master.svg
    :alt: CircleCI master

.. image:: https://img.shields.io/pypi/v/DataSuper.svg
    :target: https://pypi.python.org/pypi/DataSuper
    :alt: Latest PyPI version

.. image:: https://www.codefactor.io/repository/github/dcdanko/DataSuper/badge
    :target: https://www.codefactor.io/repository/github/dcdanko/DataSuper
    :alt: CodeFactor

.. image:: https://img.shields.io/pypi/pyversions/DataSuper.svg
    :alt: PyPI - Python version

.. image:: https://img.shields.io/pypi/dm/DataSuper.svg
    :alt: PyPI - Downloads

.. image:: https://img.shields.io/github/license/dcdanko/DataSuper.svg
    :alt: Github license


Easy to use pipelines for large biological datasets.

Goals
-----

Bioinformatics pipelines often involve a large number of files with complex organization and metadata. Often researchers keep these files organized with carefully patterned filenames, elaborate directory structures and spreadsheets containing metadata. 

DataSuper builds on this approach. DataSuper provides a system to track files that groups files into related modules (i.e. the two fastq files usually used to represent forward and reverse reads), groups sets of modules with samples that store metadata, and groups sets of samples into projects. All of this information is stored in a simple yet customizable database with an API for programmatic access.

DataSuper is probably overkill for small projects, it has been designed in particular for the MetaSUB project which has thousands of samples and complex analysis pipelines. DataSuper makes it easier to keep track of the huge number of files associated with the analysis of these samples, in particular it helps as a bottom layer that can be accessed by higher level applications. 

MetaSUB is also developing a program called ModuleUltra which builds off DataSuper and SnakeMake to provide easily distributable versioned pipelines.

In summary:
 - DataSuper works without disrupting existing bioinformatic workflows
 - DataSuper tightly packages data with metadata
 - DataSuper groups files in the same project
 - DataSuper packages data that is stored across multiple files
 - DataSuper allows programmatic access and ad-hoc grouping of files

Eventually DataSuper will support peer-to-peer sharing so that data can be more easily shared across academic sites.

Installation
------------

Be aware that DataSuper is still an Alpha. There are likely many bugs, fotunately there is no risk that DataSuper will delete your files since it only exists as a layer on top of your systems filesystem.

DataSuper is currently being used on Ubuntu and RHEL systems. It should work on any *nix system.

To install:


.. code-block:: bash
   
    git clone <url>   

    python setup.py develop


Licence
-------
MIT License

Authors
-------

`DataSuper` was written by `David C. Danko <dcdanko@gmail.com>`_.