redis-shard 0.3.7
Redis Sharding API
License: BSD License

Redis Shard
###########

.. image:: https://img.shields.io/travis/zhihu/redis-shard.svg?style=flat
   :target: https://travis-ci.org/zhihu/redis-shard
   :alt: Build Status

.. image:: https://img.shields.io/pypi/v/redis-shard.svg?style=flat
    :target: https://pypi.python.org/pypi/redis-shard
    :alt: Latest Version

.. image:: https://img.shields.io/pypi/pyversions/redis-shard.svg?style=flat
    :target: https://pypi.python.org/pypi/redis-shard
    :alt: Supported Python versions

.. image:: https://img.shields.io/pypi/l/redis-shard.svg?style=flat
    :target: https://pypi.python.org/pypi/redis-shard
    :alt: License

A Redis sharding implementation.

Redis is great. It's fast, lightweight and easy to use. But when we want to store
a mass of data into one single instance, we may encounter some problems such as performance
degradation and slow recovery and we need to scale it.

Usage
=====

First, Create an RedisShardAPI instance with multiple nodes, node ``name`` **must be unique**::

    from redis_shard.shard import RedisShardAPI

    servers = [
        {'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
        {'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0},
        {'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
    ]

    client = RedisShardAPI(servers, hash_method='md5')

Then, you can access the Redis cluster as you use `redis-py <https://github.com/andymccurdy/redis-py>`_::

    client.set('test', 1)
    client.get('test')  # 1
    client.zadd('testset', 'first', 1)
    client.zadd('testset', 'second', 2)
    client.zrange('testset', 0, -1)  # [first, second]


Hash Tags
---------

If you want to store specific keys on one node for some reason (such as you prefer single instance pipeline, or
you want to use multi-keys command such as ``sinter``), you should use Hash Tags::

    client.set('foo', 2)
    client.set('a{foo}', 5)
    client.set('b{foo}', 5)
    client.set('{foo}d', 5)
    client.set('d{foo}e', 5)

    client.get_server_name('foo') == client.get_server_name('a{foo}') == client.get_server_name('{foo}d') \
        == client.get_server_name('d{foo}e')  # True

The string in a braces of a key is the Hash Tag of the key. The hash of a Hash Tag will be treated the hash of the key.
So, keys ``foo``, ``bar{foo}`` and ``b{foo}ar`` will be sotred in the same node.

.. note:: Hash Tags are not supported with ``bytes`` keys in Python 3.

Tag method
~~~~~~~~~~~

Just add ``tag_`` prefix, you can use any of the normal redis method on the same hash tag::

    client.tag_mget("{user:1}question1", "{user:1}question2")
    client.tag_delete("{user:1}question1", "{user:1}question2")


Multi-keys method
~~~~~~~~~~~~~~~~~~
Only support ``mget``, ``mset`` and ``flushdb``.


Config Details
--------------
There are three parameters ``servers``, ``hash_method`` and ``sentinel`` in the class `redis_shard.shard.RedisShardAPI`.

``servers`` is a list.  Each element in it should be a dict or a URL schema.

- dict::

    [
        {'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
        {'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0, 'max_connections': 50},
        {'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
    ]

- URL schema::

    [
        'redis://127.0.0.1:10000/0?name=node1',
        'redis://127.0.0.1:11000/0?name=node2&max_connections=50',
        'redis://127.0.0.1:12000/0?name=node3'
    ]

If the following parameter ``sentinel`` is enabled, only **name** is needed for the ``servers`` config.

``hash_method`` is a string which indicate the method of generating the hash key of the consistent hash ring.
The default value is **crc32**. It also supports **md5** and **sha1**.


``sentinel`` is the config for `Redis Sentinel <http://redis.io/topics/sentinel>`_. With the sentinel support, redis-shard
will do read/write splitting. Config is like this::

    {"hosts": [('localhost', 26379)], "socket_timeout": 0.1}



Limitations
===========

* Redis Shard dose not support all Redis commands.
* As mentioned above, Redis Shard does not support all multi-keys commands crossing different nodes,
  you have to use Hash Tag to work with those commands.
* Redis Shard does not have any replication mechanism.


How it Works
============

Redis Shard is basically inspired by `this article <http://oldblog.antirez.com/post/redis-presharding.html>`_.


Change History
===============

0.3.7 (2021-02-14)
------------------
- Fix hash tags

0.3.6 (2020-10-15)
------------------
- Made Redis URL parsing smarter (@huntrax11 )
- Compact with python 3.8
- Compact with redis-py 3.0.0+

0.3.5 (2017-10-25)
------------------
- Allow for bytes based keys in Python3 (@gmr)
- Fix crc-32 calculation per Python

0.3.4 (2016-07-19)
------------------
- fix md5 hash_method occus NameError on python3

0.3.3 (2016-03-18)
-------------------
- max_connections can now be set when instantiating client instance, and has a default value 100.

0.3.2 (2015-06-02)
--------------------
- add strict redis support

0.3.1 (2015-02-02)
------------------
- transaction support

0.3.0 (2014-11-20)
------------------
- shard key generate function now support **md5** and **sha1** .
- add Redis Sentinel support

0.2.4 (2014-06-26)
------------------
- remove gevent dependency

0.2.3 (2014-06-02)
------------------
- better pipeline support

0.2.1 (2013-08-07)
------------------
- add evel method

0.2.0 (2013-06-02)
------------------
- add python3 support

0.1.11 (2013-05-06)
-------------------
- add mset support

0.1.10 (2013-04-13)
-------------------
- add mget support ,thks to @Yuekui

0.1.9 (2013-03-04)
------------------
- add an reshard example
- tidy the pipeline code
- add more shard methods

0.1.8 (2013-01-21)
------------------
- add `append` and `getrange` method, thks @simon-liu

0.1.7 (2012-11-19)
------------------
- use new redis url config instead of dict

0.1.5 (2012-07-16)
------------------
- Add many new methods, support socket_timeout and password

0.1.4 (2011-07-20)
------------------
- modify hash key algor, support suffix match, thks to dkong
- support more redis methods, include `keys`.

0.1.3 (2011-06-20)
------------------
- support 2.4.X version of redis-py

0.1.2 (2011-06-01)
------------------
- add MANIFEST.in file

0.1.1 (2011-05-29)
------------------
- create hashring use server's name config.

0.1 (2011-05-28)
----------------
- first version