semvec/pypi-packages huggingface dataset:
scrapy-rabbitmq 0.1.2
RabbitMQ Plug-in for Scrapy
License: MIT

## A RabbitMQ Scheduler for Scrapy Framework.

Scrapy-rabbitmq is a tool that lets you feed and queue URLs from RabbitMQ via Scrapy spiders, using the [Scrapy framework](http://doc.scrapy.org/en/latest/index.html).

Inpsired by and modled after [scrapy-redis](https://github.com/darkrho/scrapy-redis).

## Installation

Using pip, type in your command-line prompt

```
pip install scrapy-rabbitmq
```
 
Or clone the repo and inside the scrapy-rabbitmq directory, type

```
python setup.py install
```

## Usage

### Step 1: In your scrapy settings, add the following config values:

```
# Enables scheduling storing requests queue in rabbitmq.

SCHEDULER = "scrapy_rabbitmq.scheduler.Scheduler"

# Don't cleanup rabbitmq queues, allows to pause/resume crawls.
SCHEDULER_PERSIST = True

# Schedule requests using a priority queue. (default)
SCHEDULER_QUEUE_CLASS = 'scrapy_rabbitmq.queue.SpiderQueue'

# RabbitMQ Queue to use to store requests
RABBITMQ_QUEUE_NAME = 'scrapy_queue'

# Provide host and port to RabbitMQ daemon
RABBITMQ_CONNECTION_PARAMETERS = {'host': 'localhost', 'port': 6666}

# Store scraped item in rabbitmq for post-processing.
ITEM_PIPELINES = {
    'scrapy_rabbitmq.pipelines.RabbitMQPipeline': 1
}

```

### Step 2: Add RabbitMQMixin to Spider.

#### Example: multidomain_spider.py

```
from scrapy.contrib.spiders import CrawlSpider
from scrapy_rabbitmq.spiders import RabbitMQMixin

class MultiDomainSpider(RabbitMQMixin, CrawlSpider):
    name = 'multidomain'

    def parse(self, response):
        # parse all the things
        pass

```

### Step 3: Run spider using [scrapy client](http://doc.scrapy.org/en/1.0/topics/shell.html)

```
scrapy runspider multidomain_spider.py
```

### Step 4: Push URLs to RabbitMQ

#### Example: push_web_page_to_queue.py

```
#!/usr/bin/env python
import pika
import settings

connection = pika.BlockingConnection(pika.ConnectionParameters(
               'localhost'))
channel = connection.channel()

channel.basic_publish(exchange='',
                      routing_key=settings.RABBITMQ_QUEUE_NAME,
                      body='</html>raw html contents<a href="http://twitter.com/roycehaynes">extract url</a></html>')

connection.close()

```

## Contributing and Forking

See [Contributing Guidlines](CONTRIBUTING.MD)

## Releases

See the [changelog](CHANGELOG.md) for release details.

| Version | Release Date |
| :-----: | :----------: |
| 0.1.0 | 2014-11-14 |
| 0.1.1 | 2015-07-02 |



## Copyright & License

Copyright (c) 2015 Royce Haynes - Released under The MIT License.
unleash 0.7.1
Creates release commits directly in git, unleashes them on PyPI and pushes tags to github.
License: MIT

Unleash your code
=================

``unleash`` handles the boring details of cutting from your Python package a
``release`` and publishing it. It assists you by updating version numbers,
making sure the documentation builds, reminding you that you still need to
include a LICENSE file and other things.

Once a release is made, it gets tagged in your git repository. unleash features
a ``publish`` action to push the new tag to github.com and/or upload your
package to `PyPI <http://pypi.python.org>`_

Unleash always works with commits instead of files in your working copy to
ensure all your release are fully committed. It creates temporary checkouts or
verifies directly from the commit's tree. Never again will you have a bad
release because you forgot to check-in that crucial file.


Examples
--------

.. code-block:: shell

   $ unleash --dry-run release
   Updating setup.py and package version (0.6.0)
   Updating documentation version (now 0.6.0)
   Marking release as released by unleash
   Checking documentation builds cleanly
   Verifying release can generate source distribution
   Verifying release can install into a virtualenv
   Running tox tests
   Updating setup.py and package version (0.6.1.dev1)
   Updating documentation version (now 0.6.1.dev1)
   Not saving created commits. Dry-run successful.

Note the ``--dry-run`` option which means that no alterations will be made to
your repository. Otherwise, unleash will prompt you to confirm to create a new
tag ``0.6.0`` for the release and will offer to advance your current branch to
the next commit, in which all version numbers have been increase.

.. code-block:: shell

   $ unleash --dry-run publish

To be written.


Other features
--------------

``unleash`` uses a plugin-based architecture for all of its operations, this
means it is fairly easy to add custom checks and steps for releases or
publications if you desire so.

See the documentation at http://pythonhosted.org/unleash for details.