semvec/pypi-packages huggingface dataset:
dfpercentile-pkg-chanrl 0.0.1
Package to separate a df by percentile on a column, bootstrap from the upper/lower bounds, create scatter plot and histogram, return p values on ttests and pearsons correlation coefficient
Requires Python: >=3.6

# df_percentile Package

## Background

This package can split a Pandas Dataframe into two groups by a specified percentile and column to split on.

It can be useful for identifying trends in data sets with features such as price or salary. 
E.g. in a NYC rent listing database with the following columns: price, bedrooms, bathrooms
```
In [7]: df.head()

Out[7]: 
   bedrooms  bathrooms  latitude  longitude  price
0         3        1.5   40.7145   -73.9425   3000
1         2        1.0   40.7947   -73.9667   5465
2         1        1.0   40.7388   -74.0018   2850
3         1        1.0   40.7539   -73.9677   3275
4         4        1.0   40.8241   -73.9493   3350
```
If you split the df by price, the scatter method can help you visualize the distribution of features such as bathrooms and bedrooms for the higher rent vs lower rent group.
The create_df will run a ttest on the two groups and whichever columns you specified as you instantiated the object, returning p-values with the means of the features.
It can also bootstrap from the two groups and create a histogram of the distribution of sample means, along with the 95% confidence intervals.

## Instructions

First initialize the object with:

1. **df_percentile(df, col_names, col_to_separate)** *bedrooms and bathrooms are the features you care about, separate by price*
```
In [12]: new_df = df_percentile(df, ['bedrooms','bathrooms'], 'price')
This dataframe is separated by price  
```

- **create_df(percentile)** *p-values were very small for the two groups when you split the rent price by the top 30 percentile*
```
In [13]: new_df.create_df(70)

Out[13]: 
           p-values  upper_bound_means  lower_bound_means
bedrooms        0.0           2.319909           1.159946
bathrooms       0.0           1.522683           1.030201
```

- **bootstrap(percentile, col_name, n_simulations = 10000, ci = 95)**
```
In [16]: new_df.bootstrap(70, 'RAA')    
# was used in a project of mine for splitting MLB relief pitchers by salary and measuring the group performance
```
![Bootstrap](data/Figure_1.png)

- **bootstrap_stats()**
```
      In [6]: data.bootstrap_stats()

      The 95% confidence intervals for the upper bound group ranges from -1.1944029850746265 to 2.716417910447761.
      The lower bound group ranges from -2.529032258064516 and -0.44516129032258067.
      Means of the distribution: 
      Upper Bound Group: 0.7459731343283582
      Lower Bound Group:-1.4869800000000002
```

- **corr(percentile, col_name)**
```
In [25]: new_df.corr(95, 'bathrooms')

For the lower bound group: 
The correlation coefficent is 0.5437659981793554 and the p-value is 0.0
For the higher bound pitcher group: 
The correlation coefficent is 0.03580173165337624 and the p-value is 0.07722030909098343
```

- **scatter(percentile, col_name)**
```
In [39]: new_df.scatter(80, 'bedrooms')
```
![scatter](data/Figure_2.png)
canal-py2 0.5
A canal-cilent for python2

# canal-py2
First, canal-py2 is fork by https://github.com/haozi3156666/canal-python
thx.

## 一.canal-py2 简介

canal-py2 是阿里巴巴开源项目 [Canal](https://github.com/alibaba/canal)是阿里巴巴mysql数据库binlog的增量订阅&消费组件 的 python 客户端。为 python 开发者提供一个更友好的使用 Canal 的方式。Canal 是mysql数据库binlog的增量订阅&消费组件。

基于日志增量订阅&消费支持的业务：

1. 数据库镜像
2. 数据库实时备份
3. 多级索引 (卖家和买家各自分库索引)
4. search build
5. 业务cache刷新
6. 价格变化等重要业务消息

关于 Canal 的更多信息请访问 https://github.com/alibaba/canal/wiki


## 二.工作原理

canal-py2  是 Canal 的 python 客户端，它与 Canal 是采用的Socket来进行通信的，传输协议是TCP，交互协议采用的是 Google Protocol Buffer 3.0。

## 三.工作流程

1.Canal连接到mysql数据库，模拟slave

2.canal-py2 与 Canal 建立连接

2.数据库发生变更写入到binlog

5.Canal向数据库发送dump请求，获取binlog并解析

4.canal-py2 向 Canal 请求数据库变更

4.Canal 发送解析后的数据给canal-py2

5.canal-py2 收到数据，消费成功，发送回执。（可选）

6.Canal记录消费位置。

## 四.快速启动

### 安装Canal

Canal 的安装以及配置使用请查看 https://github.com/alibaba/canal/wiki/QuickStart

### 环境要求
python 2.7 （canal-python支持python3）

### 构建canal python客户端

````shell
pip install canal-py2
````

### 建立与Canal的连接
````python
import time

from canal.client import Client
from canal.protocol import EntryProtocol_pb2
from canal.protocol import CanalProtocol_pb2

client = Client()
client.connect(host='127.0.0.1', port=11111)
client.check_valid(username='canal', password='canal')
client.subscribe(client_id='1001', destination='example', filter='.*\\..*')

while True:
    message = client.get(100)
    entries = message['entries']
    for entry in entries:
        entry_type = entry.entryType
        if entry_type in [EntryProtocol_pb2.EntryType.TRANSACTIONBEGIN, EntryProtocol_pb2.EntryType.TRANSACTIONEND]:
            continue
        row_change = EntryProtocol_pb2.RowChange()
        row_change.MergeFromString(entry.storeValue)
        event_type = row_change.eventType
        header = entry.header
        database = header.schemaName
        table = header.tableName
        event_type = header.eventType
        for row in row_change.rowDatas:
            format_data = dict()
            if event_type == EntryProtocol_pb2.EventType.DELETE:
                for column in row.beforeColumns:
                    format_data = {
                        column.name: column.value
                    }
            elif event_type == EntryProtocol_pb2.EventType.INSERT:
                for column in row.afterColumns:
                    format_data = {
                        column.name: column.value
                    }
            else:
                format_data['before'] = format_data['after'] = dict()
                for column in row.beforeColumns:
                    format_data['before'][column.name] = column.value
                for column in row.afterColumns:
                    format_data['after'][column.name] = column.value
            data = dict(
                db=database,
                table=table,
                event_type=event_type,
                data=format_data,
            )
            print(data)
    time.sleep(1)

client.disconnect()
````
blog 1.0.1
A static website generator
Requires Python: >=3.7.*
Keywords: static website generator

# Blog

This is a simple static site generator. It traverses through a source directory of markdown files and creates a
destination directory with equivalent HTML files.

```
Markdown In -> HTML Out
```

---

[![PyPI](https://img.shields.io/pypi/v/blog.svg)](https://pypi.python.org/pypi/blog)
[![PyPI](https://img.shields.io/pypi/dm/blog.svg)](https://pypi.python.org/pypi/blog)

---

## Installation

```
$ python3.7 -m venv venv
$ source venv/bin/activate
$ pip install blog
```

## Usage

```
(venv) $ python -m blog build /path/to/source-dir /path/to/build-dir
```

## Simple by design

- Nothing smart is done around friendly URLs / generating folders with `index.html` files
    + If you want an `index.html` file, create an `index.md` file in your source directory
- There is no database
- There is no configuration (for now)

**Need something more complex?**

- Consider using Hugo, Gatsby, Django, writing your own server
kelvinnypy 1.0.16
A python script to convert large matrices stored as .h5 to .txt

# kelvinnypy
It is pronounced 'kelvinny-pee'

## Intro
Hi there! I've put together the following python script/package for some work stuff!

## Requirements
python 3.7

## Installation instructions
```bash
# using pip
pip install kelvinnypy

# or using conda
conda install -c kt16 kelvinnypy
```

## Main function (well, only function)
### h5totxt
Together with another R function in zktuong/kelvinny (kelvinny::mtx_to_h5, or in a combined wrapper - kelvinny::mtx_to_h5totxt), this python script helps to parse large matrices in R to .h5 to .txt format. This packages are extremely light weight and should be relatively quick and easy to install and use. Have fun!

## Usage
I've created this as an executable that you can use directly via the terminal/console
```bash
# bash
h5totxt counts.h5

# for help,
h5totxt -h
```

You can also download the file called h5convert.py and call it in terminal via python
```bash
# bash
python ./path/to/kelvinnypy.py counts.h5
```

Or use in R via reticulate
```R
# R
reticulate::repl_python()
```
```python
# python
import kelvinnypy as kp
kp.h5totxt("counts.h5")
# or
kp.h5totxt("counts.h5", "new_counts.txt")
```
or
```R
# R
library(reticulate)
kp <- import("kelvinnypy")
kp$h5totxt("counts.h5")
```

or even as a complete workflow with the the R function
```R
# devtools::install_github('zktuong/kelvinny', dependencies = TRUE)
mat <- matrix(rexp(2e8, rate=.1), ncol=20000, dimnames = list(paste0("gene", 1:10000), paste0("cell", 1:20000)))
dim(mat)
# [1] 10000 20000

# try this!
mtx_to_h5totxt(mat, "mat")

```