semvec/pypi-packages huggingface dataset:
aioscrape 0.0.2
Async scraping library
License: BSD

AioScrape
=========

A scraping library on top of `aiohttp <https://aiohttp.readthedocs.io>`_ and `parsechain <https://github.com/Suor/parsechain>`_. Note that this is **alpha** software.


Installation
-------------

::

    pip install aioscrape


Usage
-----

.. code:: python

    from aioscrape import run, fetch, settings
    from aioscrape.middleware import last_fetch, make_filecache
    from aioscrape.utils import SOME_HEADERS # To not look like a bot

    from urllib.parse import urljoin
    from parsechain import C
    from funcy import lcat, lconcat


    def main():
        # Settings are scoped and can be redefined later with another "with"
        cache = make_filecache('.fcache')
        with settings(headers=SOME_HEADERS, middleware=[cache, last_fetch]):
            print(run(scrape_all()))


    async def scrape_all():
        # All the settings in scope like headers and middleware are applied to fetch()
        start_page = await fetch(START_URL)

        # AioScrape integrates with parsechain to make extracting a breeze
        urls = start_page.css('.pagingLinks a').attrs('href')
        list_urls = [urljoin(start_page.url, page_url) for page_url in urls]

        # Using asyncio.wait() and friends to run requests in parallel
        list_pages = [start_page] + await wait_all(map(fetch, list_urls))

        # Scrape articles
        result = lcat(await wait_all(map(scrape_articles, list_pages)))
        write_to_csv('export.csv', result)


    async def scrape_articles(list_page):
        urls = list_page.css('#headlines .titleLink').attrs('href')
        abs_urls = [urljoin(list_page.url, url) for url in urls]
        return await wait_all(map(scrape_article, abs_urls))


    async def scrape_article(url):
        resp = await fetch(url)
        return resp.root.multi({
            'url': C.const(resp.url),
            'title': C.microdata('headline').first,
            'date': C.microdata('datePublished').first,
            'text': C.microdata('articleBody').first,
            'contacts': C.css('.sidebars .contact p')
                         .map(C.inner_html + html_to_text) + lconcat + ''.join,
        })


    if __name__ == '__main__':
        main()


TODO
----

- Response.follow()
- non-GET requests
- work with forms
parsechain 0.0.3
Making parsing concise
License: BSD

ParseChain 
==========

A way to parse html fluently and as declarative as possible. This is in **alpha stage** now, things will change.


Installation
-------------

::

    pip install parsechain


Usage
-----

.. code:: python

    import requests
    from parsechain import C, Response

    # Fetch html and cast it
    response = Response.cast(requests.get(...))

    # Get a movie title and rating
    title = response.css('h1 .title').text
    rating = response.css('.left-box').inner_text.re(r'IMDb: ([\d.]+)').float

    # Or both
    movie = response.root.multi({
        'title': C.css('h1 .title').text,
        'rating': C.css('.left-box').inner_text.re(r'IMDb: ([\d.]+)').float,
    })


The last example could be extended to show chains reuse:


.. code:: python

    def by_label(label):
        return C.css('.left-box').inner_text.re(fr'{label}: ([\w.]+)')

    parse_movie = C.multi({
        'title': C.css('h1 .title').text,
        'rating': by_label('IMDb').float,
        'status': by_label('Status').trim,
    })

    movie = parse_movie(response.root)  # Pass a root of a tree


The complete list of available ops could be seen in ``parsechain.chains.Ops`` class. Proper documentation to follow, some day ;)


.. |Build Status| image:: https://travis-ci.org/Suor/parsechain.svg?branch=master
   :target: https://travis-ci.org/Suor/parsechain