semvec/pypi-packages huggingface dataset:
newspider 0.9.9
分类扒数据的简易框架
License: MIT

##Example.py

# -*- coding: utf-8 -*-
from pyquery import PyQuery as pq

from newspider.interfaces import *
from newspider.spider import Newspider


class DemoFetcher(IntFetcher):
    def __init__(self):
        self.next_page = []

    def fetch_detail_urls(self,html):
        d = pq(html)
        list = []

        for a in d('.post-title a'):
            url = d(a).attr('href')
            extras = {"category": "Test for %s" % url}
            list.append((url,extras))

        for l in d('.page-navigator a'):
            self.next_page.append(d(l).attr('href'))

        return list

    def start_page(self):
        return ['http://www.typechodev.com/','http://www.typechodev.com/index.php/category/questions/']

    def next_pages(self):
        return self.next_page


class DemoParser(IntParser):
    def parse(self,tag,html,extras):
        print "Receive content from url %s for tag %s|%s" % (extras.get('_url'),extras.get('category'),tag)


if __name__ == '__main__':
    sp = Newspider()
    sp.config('GUARD_INTERVAL', 0)

    sp.add_parser(DemoParser())
    sp.add_fetcher(DemoFetcher())

    sp.run()
py2swagger 1.0.0
Swagger schema builder
License: MIT

Python OpenAPI Specification generator
======================================

Tool for automated or semi-automated generate a `OpenAPI
Specification <https://github.com/OAI/OpenAPI-Specification>`__ (aka
Swagger Specification) for Web APIs written with Python and famous
frameworks, such as:

-  `Django REST
   Framework <https://github.com/tomchristie/django-rest-framework>`__
   (DRF>=2; Django>=1.6)
-  `Falcon <https://github.com/falconry/falcon>`__ (falcon>=0.3)
-  flask
-  bottle
-  and ever you own framework

Supported only v2 OpenAPI Specification

Why?
----

This tool is written as part of our Documentation Toolkit which we use
in our job daily. The main idea of toolkit is to make a process of
creating and updating documentation able to be automated

Other parts of our toolkit is:

-  `py2swagger <https://github.com/Arello-Mobile/py2swagger>`__
-  `swagger2rst <https://github.com/Arello-Mobile/swagger2rst>`__
-  `sphinx-confluence <https://github.com/Arello-Mobile/sphinx-confluence>`__
-  `confluence-publisher <https://github.com/Arello-Mobile/confluence-publisher>`__

Install
=======

Install from `PyPI <https://pypi.python.org/pypi/py2swagger>`__ with

::

    $ pip install py2swagger

Usage
-----

::

    usage: py2swagger [-h] [-c CONFIG] [-r ROOT] [-o OUTPUT]
                      {falcon,drf,simple} ...

    Swagger schema builder

    optional arguments:
      -h, --help            show this help message and exit
      -c CONFIG, --config CONFIG
                            Path to config file
      -r ROOT, --root ROOT  Path to project root. Default is current directory or
                            configuration file location
      -o OUTPUT, --output OUTPUT
                            Output file (Default stdout)

    plugins:
      {falcon,drf,simple}
        falcon
        drf
        simple              Plugin for all applications

Run tests
---------

.. code:: bash

    python setup.py test
py-chillog 0.1
SFE Custom Logger based on GELF for python
License: MIT

# py-chillog

Python library for building logging data structure based on [GELF](http://docs.graylog.org/en/2.1/pages/gelf.html)


Installation
============

```shell
$ make install
```


Usage
=====

```python
from chillog.logger import Chillog

logger = Chillog()

# simple log
logger.info("Message")

# log with optional field(s) using kwargs
logger.info("Message", key_one="value_one", key_two="value_two")

# log with optional field(s) using dict
optional_fields = {
    "key_one": "value_one",
    "key_two": "value_two"
}

logger.info("Message", **optional_fields)  # notice `**` sign before `optional_fields` variable

# log with detailed message
logger.info("Message", full_message="More detailed message")
```

By default, this library will read value of environment variable with key `SERVICE_NAME` to fill `service` json field.
This library also read hostname using python native (`socket.gethostname()`).

Service name and hostname can overridden when initialisation Chillog object like example below

```python
from chillog.logger import Chillog

logger = Chillog(service_name="SERVICE_NAME", hostname="HOSTNAME")
```

There are 7 logger you can use:

- ``debug``
- ``info``
- ``notice``
- ``warning``
- ``error``
- ``critical``
- ``alert``

Output log:

```json
{
  "version": 1,
  "host": "HOSTNAME",
  "service": "SERVICE_NAME",
  "short_message": "log description",
  "full_message": "long log description if any",
  "timestamp": 1472009181293,
  "level": 1,
  "_additional_key": "additional value"
}
```

Running Test
============

```shell
$ make test
```