semvec/pypi-packages huggingface dataset:
fs.onedatafs 18.2.2.0
Onedata filesystem for PyFilesystem2
License: MIT
Keywords: pyfilesystem

OnedataFS
=========

OnedataFS is a `PyFilesystem <https://www.pyfilesystem.org/>`__
interface to `Onedata <https://onedata.org>`__ virtual file system.

As a PyFilesystem concrete class,
`OnedataFS <https://github.com/onedata/fs-onedatafs/>`__ allows you to
work with Onedata in the same way as any other supported filesystem.

Installing
----------

You can install OnedataFS from pip as follows:

::

   pip install fs-onedatafs

Opening a OnedataFS
-------------------

Open an OnedataFS by explicitly using the constructor:

.. code:: python

   from fs.onedatafs import OnedataFS
   onedata_provider_host = "..."
   onedata_access_token = "..."
   odfs = OnedataFS(onedata_provider_host, onedata_access_token)

Or with a FS URL:

.. code:: python

     from fs import open_fs
     odfs = open_fs('onedatafs://HOST?token=...')

Extended attributes
-------------------

Onedata FS supports in addition to standard PyFilesystem API operations
on metadata via POSIX compatible extended attributes API.

Documentation
-------------

-  `PyFilesystem Wiki <https://www.pyfilesystem.org>`__
-  `OnedataFS
   Reference <http://fs-onedatafs.readthedocs.io/en/latest/>`__
-  `Onedata Homepage <https://onedata.org>`__
comath 0.0.7
Math-related utility functions.
License: MIT
Requires Python: >=3.5
Keywords: python math array metric function segment

comath
######

|PyPI-Status| |PyPI-Versions| |Build-Status| |Codecov| |LICENCE|

A small pure-python package for math-related utility functions.

.. code-block:: python

  >>> from comath.array import percentile
  >>> percentile([4,6,8,9,11], 0.4)
  7.0

.. contents::

.. section-numbering::


Installation
============

Install ``comath`` with:

.. code-block:: bash

  pip install comath


Use
===

``comath`` is divided into four sub-modules.

array
-----

Array operations like percentile and median.


func
----

Utility function like a smooth step function and closest larger power of 2.


metric
------

Computing moving metrics (moving average, precision and variance).


segment
----------

Defines a one-dimensional line segment.


Contributing
============

Package author and current maintainer is Shay Palachy (shay.palachy@gmail.com); You are more than welcome to approach him for help. Contributions are very welcomed.

Installing for development
--------------------------

Clone:

.. code-block:: bash

  git clone git@github.com:shaypal5/comath.git


Install in development mode with test dependencies:

.. code-block:: bash

  cd comath
  pip install -e ".[test]"


Running the tests
-----------------

To run the tests, use:

.. code-block:: bash

  python -m pytest --cov=comath --doctest-modules


Adding documentation
--------------------

This project is documented using the `numpy docstring conventions`_, which were chosen as they are perhaps the most widely-spread conventions that are both supported by common tools such as Sphinx and result in human-readable docstrings (in my personal opinion, of course). When documenting code you add to this project, please follow `these conventions`_.

.. _`numpy docstring conventions`: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
.. _`these conventions`: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt


Credits
=======
Created by Shay Palachy  (shay.palachy@gmail.com).

.. |PyPI-Status| image:: https://img.shields.io/pypi/v/comath.svg
  :target: https://pypi.python.org/pypi/comath

.. |PyPI-Versions| image:: https://img.shields.io/pypi/pyversions/comath.svg
   :target: https://pypi.python.org/pypi/comath

.. |Build-Status| image:: https://travis-ci.org/shaypal5/comath.svg?branch=master
  :target: https://travis-ci.org/shaypal5/comath

.. |LICENCE| image:: https://img.shields.io/pypi/l/comath.svg
  :target: https://pypi.python.org/pypi/comath

.. |Codecov| image:: https://codecov.io/github/shaypal5/comath/coverage.svg?branch=master
   :target: https://codecov.io/github/shaypal5/comath?branch=master
get-cert 19.1.3
Tool for downloading ssl certificates from remote servers.
License: MIT
Requires Python: >=3.6,<4.0
Keywords: ssl,tool

# get_cert

Tool for downloading ssl certificates from remote servers.

## Installation

Just use PIP:

```bash
pip install get_cert
```

## Usage

```bash
$ python -m get_cert --help
Usage: get_cert.py [OPTIONS] URL

  Retrieve and print out the ssl certificate.

  Args:     URL (str): url to be picked up

Options:
  --help  Show this message and exit.
```
bcl2fq-local 0.2.0
Bcl2fastq wrapper
License: MIT

# bcl2fq-local

A bcl2fastq and qc wrapper

# Install
```bash
pip install bcl2fq-local
```
or 

```bash
git clone URL
cd bcl2fq-local
python setup.py install

```

# Usage
For Bcl2fastq
```text
usage: bcl2fq-local [-h] [-i DirPath] [-o DirPath] [--sample-sheet File]
                    [--mismatch N] [--process N] [--io-process Num]
                    [--binpath File] [--dry-run] [--mask MASK]
                    [--mask-reads N] [--mask-index N] [-v]

A bcl2fastq wrapper

optional arguments:
  -h, --help           show this help message and exit
  -i DirPath           Sequence run dir
  -o DirPath           Output dir for fastq files
  --sample-sheet File  Using custom sample sheet file
  --mismatch N         Mismatch for barcode, default: 1
  --process N          Process number for demultiplexing and processing
  --io-process Num     Process number for reading and writing
  --binpath File       bcl2fastq binary file path
  --dry-run            Only print the cmd without running it
  --mask MASK          using self defined mask
  --mask-reads N       set reads mask base count, default is: 0,0
  --mask-index N       set index mask base count, default is: 0,0
  -v, --version        show version info
```

For QC
```text
usage: qc-local [-h] [-r RAW_DIR] [-c CLEAN_DIR] [-q QC_DIR] [-2 ADAPTER_DIR]
                [-t THREADS] [--paired-end] [-v]

A qc wrapper

optional arguments:
  -h, --help      show this help message and exit
  -r RAW_DIR      Raw reads dir input
  -c CLEAN_DIR    Dir to write output clean reads
  -q QC_DIR       Dir to write qc info file, default: <clean_dir>/QC
  -2 ADAPTER_DIR  Dir to write removed reads, default: <clean_dir>/002
  -t THREADS      Threads number for parallel, default:8
  --paired-end    Use qc for paired end reads
  -v, --version   Show version info
```
weibo-preprocess-toolkit 1.1.0
Weibo Preprocess Toolkit.
License: MIT
Requires Python: >=3

# weibo-preprocess-toolkit
Weibo Preprocess Toolkit

## Getting Started

### Installation

```shell
pip install weibo-preprocess-toolkit
```

### Tutorial

```Python
from weibo_preprocess_toolkit import WeiboPreprocess

preprocess = WeiboPreprocess()

test_weibo = "所以我都不喝蒙牛 #南京·大行宫[地点]#，一直不喜欢蒙牛。謝駿毅 赞[122]转发[11] [超话] 收藏09月11日 18:57 "

# traditional2simplified
print(preprocess.traditional2simplified(test_weibo))
# 所以我都不喝蒙牛 #南京·大行宫[地点]#，一直不喜欢蒙牛。谢骏毅 赞[122]转发[11] [超话] 收藏09月11日 18:57

# clean weibo with simplified Chinese
print(preprocess.clean(test_weibo))
# 所以我都不喝蒙牛 一直不喜欢蒙牛 谢骏毅

# clean weibo 
print(preprocess.clean(test_weibo, simplified=False))
# 所以我都不喝蒙牛 一直不喜欢蒙牛 謝駿毅

# seg weibo, keep stop words
print(preprocess.cut(test_weibo))
# ['所以', '我', '都', '不喝', '蒙牛', '#', '南京', '·', '大行宫', '[', '地点', ']', '#', '，', '一直', '不喜欢', '蒙牛', '。', '謝駿毅', '赞', '[', '122', ']', '转发', '[', '11', ']', '[', '超话', ']', '收藏', '09', '月', '11', '日', '18', ':', '57', '\xa0']

# seg weibo, don't keep stop words
print(preprocess.cut(test_weibo, keep_stop_word=False))
# ['都', '不喝', '蒙牛', '#', '南京', '·', '大行宫', '[', '地点', ']', '#', '，', '不喜欢', '蒙牛', '。', '謝駿毅', '赞', '[', '122', ']', '转发', '[', '11', ']', '[', '超话', ']', '收藏', '09', '月', '11', '日', '18', ':', '57', '\xa0']

# clean and cut weibo, keep_stop_words, simplified Chinese
print(preprocess.preprocess(test_weibo))
# 所以 我 都 不喝 蒙牛 一直 不喜欢 蒙牛 谢骏毅
print(preprocess.preprocess(test_weibo, simplified=False, keep_stop_word=False))
# 都 不喝 蒙牛 不喜欢 蒙牛 謝駿毅
```


## Introduction
该工具用于微博文本的预处理：清洗 + 分词。

### Inspiration
在中文 NLP 领域，文本清洗和分词对于模型的性能有着很大的影响，如果语料库和测试集/**线上环境**的文本清洗规则和分词工具不同，就会导致在语料库上训练出来的模型在测试集上效果很差。举例来说，语料库采用了清洗规则 Clean-A 和 分词工具 Seg-A 来清洗和分词微博，而用户在线上环境采用了另一种清洗规则 Clean-B 和另一种分词工具 Seg-B，那么线上环境就会产生很多不在语料库词典中的**未登陆词（Unknown Words）**，这些未登陆词会导致预先训练好的模型，面对线上环境的另一种规则时，性能变差。

本人在对微博进行情感分析的过程中，总结了较多的微博清洗技巧和分词规则，并总结了一份微博情感分析词典用于优化 jieba 分词。所以我在这里尝试对微博的清洗和分词规则进行整理，同时也是为了保持语料库和线上环境的规则同步，为其他研究者和使用我的模型的人，提供一个和语料库匹配的清洗和分词规则。

### Weibo Cleaning

本人对微博文本的清洗规则进行了整理，主要涉及到如下的规则：

1. 中文繁体转简体
2. [微博停用词规则1(正则表达式)](weibo_preprocess_toolkit/dictionary/weibo_stopwords1_regex.csv)，包括 url, email, @某人, 地点，…… 等停用词规则
3. [微博停用词规则2(正则表达式)](weibo_preprocess_toolkit/dictionary/weibo_stopwords2_regex.csv)，包括 时间，数字和微博中常出现的无意义的词等停用词规则
4. [微博特殊字符](weibo_preprocess_toolkit/dictionary/special_chars.csv)
5. 其他细节处理

注意：考虑到停用词在词向量训练中衔接上下文的作用，本工具并没有对微博的停用词进行清洗

### Weibo Seg

基于 jieba 分词对微博文本进行分词优化，优化的地方主要有两点：

1. 扩种 jieba 分词词典，构建情感词典，优化情感分析的分词结果
2. 对否定前缀词进行特殊处理

## Dependencies
```bash
pip install jieba
```

## Acknowledgment
[jieba 结巴中文分词](https://github.com/fxsjy/jieba)

[nstools 中文繁体转简体](https://github.com/skydark/nstools)

[NTUSD 情感词典](https://www.aaai.org/Papers/Symposia/Spring/2006/SS-06-03/SS06-03-020.pdf)

[哈工大停用词表](https://github.com/goto456/stopwords)

## License

MIT