semvec/pypi-packages huggingface dataset:
gspider 0.1.1
A web spider framework based on gevent and requests
License: Apache 2

=======
gspider
=======

.. image:: https://travis-ci.org/jadbin/gspider.svg?branch=master
    :target: https://travis-ci.org/jadbin/gspider

.. image:: https://coveralls.io/repos/jadbin/gspider/badge.svg?branch=master
    :target: https://coveralls.io/github/jadbin/gspider?branch=master

.. image:: https://img.shields.io/badge/license-Apache 2-blue.svg
    :target: https://github.com/jadbin/gspider/blob/master/LICENSE


A web spider framework based on `gevent`_ and `requests`_.


Spider Example
==============

以下是我们的一个爬虫类示例，其作用为爬取 `百度新闻 <http://news.baidu.com/>`_ 的热点要闻:

.. code-block:: python

    from gspider import Spider, HttpRequest, run_spider, Selector


    class BaiduNewsSpider(Spider):
        def start_requests(self):
            yield HttpRequest("http://news.baidu.com/")

        def parse(self, response):
            selector = Selector(response.text)
            hot = selector.css("div.hotnews a").text
            self.log("Hot News:")
            for i in range(len(hot)):
                self.log("%s: %s", i + 1, hot[i])


    if __name__ == '__main__':
        run_spider(BaiduNewsSpider)


在爬虫类中我们定义了一些方法：

- ``start_requests``: 返回爬虫初始请求。
- ``parse``: 处理请求得到的页面，这里借助 ``Selector`` 及CSS Selector语法提取到了我们所需的数据。


Documentation
=============

http://gspider.readthedocs.io/


.. _gevent: https://pypi.org/project/gevent
.. _requests: https://pypi.org/project/requests