semvec/pypi-packages huggingface dataset:
shire 0.2.1
Heavy-job queue processor
License: MIT
Keywords: queue job task shire

Shire
========

Heavy-job queue processor


Installation
--------------------

pip install shire


Create config file
--------------------

Run shire-cli create_config, follow instructions.

An example of the shire.cfg::

    [connection]
    redis_url = redis://127.0.0.1:6379/0
    db_url = postgres://peewee:123@127.0.0.1:5432/peewee
    
    [hostler]
    check_time = 5
    
    [shire]
    venv_path = /path/to/your/virtualenv
    sys_path = /path/to/your/project:/path/to/external/library
    host = default

    [scribe]
    per_pool = 1
    log_max_size = 104857600
    log_directory = logs
    max_logs = 5
    default_log = default.log
    
    [pool]
    check_time = 30
    
    [whip]
    limits_update_time = 60
    check_time = 1


Queue management
--------------------

Interactive shire config creation
  shire-cli -c /path/to/your/shire.cfg create_config

Run whip (Enqueue jobs to pools)
  shire-cli -c /path/to/your/shire.cfg run_whip

Run scribe (Shire job logs writer)
  shire-cli -c /path/to/your/shire.cfg run_scribe

Run hostler (Shire failed jobs restarter)
  shire-cli -c /path/to/your/shire.cfg run_hostler

Run pool (Shire job executor)
  shire-cli -c /path/to/your/shire.cfg run_pool --name=pool_name

Run multiple pools with one master-process
  shire-cli -c /path/to/your/shire.cfg start_pools --names=pool_name,another_pool


Example
-------------------
Run shire
  shire-cli -c /path/to/your/shire.cfg run_whip
  shire-cli -c /path/to/your/shire.cfg run_pool --name=default

Create example jobs module::

    # my_jobs.py
    from shire.job import Job
    import requests

    def save(text_len):
        # save - your external function to save result data. Shire not save any results itself.
        pass

    class CountWordsAtUrl(Job):
        def run(url=None):
            resp = requests.get(url)		
            save(len(resp.text.split()))
            
Create example configuration module::

     # shire_conf.py
     from shire.config import Config as ShireConfig
     conf = ShireConfig()
     conf.load('/path/to/your/shire.cfg')

Enqueue job::
	
    from my_jobs import CountWordsAtUrl
    from shire_conf import conf
    CountWordsAtUrl.delay(conf, pool='default', kwargs={'url': 'https://github.com/suguby/shire'})