semvec/pypi-packages huggingface dataset:
scrapy-html-storage 0.4.0
Scrapy downloader middleware that stores response HTML files to disk.
License: MIT

=====
About
=====

.. image:: https://travis-ci.org/povilasb/scrapy-html-storage.svg?branch=master
.. image:: https://coveralls.io/repos/github/povilasb/scrapy-html-storage/badge.svg?branch=master :target: https://coveralls.io/github/povilasb/scrapy-html-storage?branch=master

This is Scrapy downloader middleware that stores response HTMLs to disk.

Usage
=====

Turn downloader on, e.g. specifying it in `settings.py`::

    DOWNLOADER_MIDDLEWARES = {
        'scrapy_html_storage.HtmlStorageMiddleware': 10,
    }

None of responses by default are saved to disk.
You must select for which requests the response HTMLs will be saved::

   def parse(self, response):
        """Processes start urls.

        Args:
            response (HtmlResponse): scrapy HTML response object.
        """
        yield scrapy.Request(
            'http://target.com',
            callback=self.parse_target,
            meta={
              'save_html': True,
            }
        )

The file path where HTML will be stored is resolved with spider method
`response_html_path`. E.g.::

    class TargetSpider(scrapy.Spider):
        def response_html_path(self, request):
            """
            Args:
                request (scrapy.http.request.Request): request that produced the
                    response.
            """
            return 'html/last_response.html'

Configuration
=============

HTML storage downloader middleware supports such options:

* **gzip_output** (bool) - if True, HTML output will be stored in gzip format.
  Default is False.
* **save_html_on_status** (list) - if not empty, sets list of response codes
  whitelisted for html saving. If list is empty or not provided, all response
  codes will be allowed for html saving.

Sample::

    HTML_STORAGE = {
        'gzip_output': True,
        'save_html_on_status': [200, 202]
    }