pychbase 0.1.8
A Python wrapper for the libhbase C API to HBase
License: MIT
Keywords: hbase libhbase

pychbase
========

This is a Python C wrapper for MapRDB and HBase using the `libhbase C
API <https://github.com/mapr/libhbase>`__.

``pychbase`` is modeled after the HappyBase API, but it does not use
``thrift``, and is ideal for MapRDB.

``pychbase`` is tested on Python 2.7 and MapR 5.1.

LD\_LIBRARY\_PATH
=================

To compile as well as import ``pychbase``, your ``LD_LIBRARY_PATH`` must
have the directory with ``libjvm.so`` on it, normally in either:

-  $JAVA\_HOME/lib/amd64/server
-  $JAVA\_HOME/jre/lib/amd64/server

If you are using this with MapR, you must also have ``/opt/mapr/lib`` on
your ``LD_LIBRARY_PATH``

E.g,

::

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib/amd64/server:/opt/mapr/lib

Installation
============

Installation on a MapR environment
----------------------------------

Normally, the only environment variable to worry about on a MapR
environment is $PYCHBASE\_LIBJVM\_DIR

::

    export PYCHBASE_LIBJVM_DIR=/usr/lib/jvm/jre-1.7.0/lib/amd64/server
    virtualenv pychbase
    cd pychbase
    source bin/activate
    pip install pychbase

    # Or build it from source
    git clone https://github.com/mkmoisen/pychbase.git
    cd pychbase
    python setup.py install

Installation on a Non-MapR environment
--------------------------------------

Please see the end of the readme for Cloudera intallation notes.

Run the tests
=============

The ``config.py`` file in the ``tests`` directory has two constants,
``ZOOKEEPER`` and ``TABLE_NAME``, that probably won't work if you run
the tests without modification

Create a ``tests/local_config.py`` file like the following:

::

    ZOOKEEPERS = 'localhost:7222'
    TABLE_NAME = 'testpychbase'

To run the tests, make sure to be in the ``tests`` directory, or else
you will face import problems:

::

    cd tests
    python tests.py

Currently ``nosetests`` will not work without facing an import issue.

Usage
=====

I have attempted to mimic the great HappyBase API as closely as
possible.

Make sure to set the LD\_LIBRARY\_PATH environment variable:

::

    # MapR
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server:/opt/mapr/lib

    # Non MapR
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/jvm/jre-1.7.0/lib/amd64/server::/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/lib/native
    export HBASE_LIB_DIR=/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/lib/
    # I've only gotten it to work on CDH4. If you are on CDH5 you'll need to mess around with the classpath some more

Imports:

::

    from pychbase import Connection, Table, Batch

To create a connection:

::

    # On MapR, you don't need to specify the CLDBS/zookeepers:
    connection = Connection()

    # On Non-MapR environments, you'll need to specify the zookeepers:
    connection = Connection('zookeeper-one:2181",zookeeper-two:2181",zookeeper-three:2181"')

To create and delete tables:

::

    # Create a table named 'test_table' with a single column family named 'f' with no additional attributes
    connection.create_table('test_table', {'f': {}})
    connection.delete_table('test_table')

To get a table to operate on:

::

    table = connection.table('test_table')

To put, delete, and get from a table:

::

    # Put a row with the current timestamp
    table.put('rowkey1', {'f:foo': 'bar', 'f:hai': 'bai'})
    data = table.row('rowkey1')
    assert data == {'f:foo': 'bar', 'f:hai': 'bai'}

    # Get a row and only include a single column
    data = table.row('rowkey1', columns=('f:foo',))
    assert data == {'f:foo': 'bar'}

    # Delete the row
    table.delete('rowkey1')
    data = table.row('rowkey1')
    assert data == {}

    # Put a row with a given timestamp
    table.put('rowkey1', {'f:foo': 'bar'}, timestamp=1000)
    table.put('rowkey1', {'f:foo': 'BAR'}, timestamp=10000)

    # Get a row with a given timestamp and include its timestamp
    data = table.row('rowkey1', timestamp=1000, include_timestamp=True)
    assert data == {'f:foo': ('bar', 1000)}

To scan:

::

    # Full table scan:
    for row, data in table.scan():
        pass

    # Scan with a start and stop:
    for row, data in table.scan('foo', 'bar'):
        pass

    # Scan with a row prefix:
    for row, data in table.scan(row_prefix='baz'): # E.g., start='baz', stop='baz~'
        pass

    # Scan with a filter: # Check out tests.py on how to use all of the filters
    for row, data in table.scan(filter="SingleColumnValueFilter('f', 'foo', =, 'binary:foo')":
        pass

    # Scan a table but return only row keys, no rows:
    table.put('foo', {'f:foo': 'foo'}
    table.put('foo1', {'f:foo': 'foo'}
    table.put('foo2', {'f:foo': 'foo'}

    rows = list(table.scan(only_rowkeys=True))
    assert rows == ['foo', 'foo1', 'foo2']

To count the number of rows in a table:

::

    # Full table count:
    count = table.count()

    # Count all rows with a start and stop
    count = table.count('foo', 'bar')

    # Count all rows whose row key starts with a row prefix:
    count = table.count(row_prefix='baz') # E.g., start='baz', stop='baz~'

    # Count all rows with a filter:
    count = table.count(filter="SingleColumnValueFilter('f', 'foo', =, 'binary:foo')")

To batch put:

::

    batch = table.batch()
    datas = [
        ('foo', 'a', 'b'),
        ('foo1', 'a1', 'b1'),
        ('foo2', 'a2', 'b2'),
    ]

    for data in datas:
        batch.put(data[0], {'f:foo': data[1], 'f:bar': data[2]})

    errors = batch.send()

    assert errors == 0

To batch delete:

::

    batch = table.batch()
    rows = ['foo', 'foo1', 'foo2']
    for row in rows:
        batch.delete(row)

    errors = batch.send()

    assert errors == 0

Note that ``batch.send()`` returns the number of errors that occurred,
if any. It is up to the client to ignore this or raise an exception.

::

    batch = table.batch()
    batch.put('foo', {'f:foo', 'bar'})
    batch.put('foo', 'invalid')

    errors = batch.send()
    assert errors == 1

An additional helper method is ``table.delete_prefix(row_prefix)``,
which deletes all rows containing starting with the prefix.

::

    table.put('foo', {'f:foo', 'foo'}
    table.put('foo1', {'f:foo', 'foo'}
    table.put('foo2', {'f:foo', 'foo'}
    table.put('foo3', {'f:foo', 'foo'}
    table.put('bar', {'f:bar', 'bar'}

    number_deleted = table.delete_prefix('foo')
    assert number_deleted == 3

    assert table.count() == 2

Note that attempting to batch/put unescaped null terminators will result
in them being stripped. Attempting to use a row key with an unescaped
null terminator will raise a TypeException. It is the user's duty to
escape null terminators before attempting to batch/put data.

::

    table.put('foo', {'f:foo\0bar': 'baz\0bak'})
    data = table.row('foo')
    assert data == {'f:foo': 'baz'}

    table.put('bar', {'f:foo\\0bar': 'baz\00bak'})
    data = table.row('foo')
    assert data == {'f:foo\\0bar': 'baz\00bak'}

HappyBase compatibility
=======================

One goal of this library is to maintain compatibility with the APIs in
HappyBase.

Check out ``__init__.py`` to understand which features of HappyBase I
have not yet implemented.

In the future, I will force print warnings to stderr in the event a user
uses an unimplemented feature.

Non-MapR Installation and Environment Variables Guide
=====================================================

I have not tested ``pychbase`` heavily on Cloudera. I couldn't get it
working on CDH5 due to a classpath issue with ``libhbase``, and and
while I was able to get it up and running with CDH4, some of the tests
are failing. It seems to me that ``libhbase`` is not fully compatible
outside of MapR.

::

    export PYCHBASE_IS_MAPR=FALSE
    export PYCHBASE_LIBJVM_DIR=/usr/lib/jvm/jre-1.7.0/lib/amd64/server
    export PYCHBASE_INCLUDE_DIR=/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/include
    export PYCHBASE_LIBRARY_DIR=/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/lib/native
    virtualenv pychbase
    cd pychbase
    source bin/activate
    pip install pychbase

    # Or build it from source
    git clone https://github.com/mkmoisen/pychbase.git
    cd pychbase
    python setup.py install

Please note that the following environment variables must be set in
order to install ``pychbase`` correctly:

-  PYCHBASE\_IS\_MAPR
-  PYCHBASE\_LIBJVM\_DIR
-  PYCHBASE\_INCLUDE\_DIR
-  PYCHBASE\_LIBRARY\_DIR

PYCHBASE\_IS\_MAPR
------------------

This defaults to ``TRUE``. IF you are using Cloudera/etc, make sure to:

::

    export PYCHBASE_IS_MAPR=FALSE

PYCHBASE\_LIBJVM\_DIR
---------------------

This is the directory that houses the ``libjvm.so`` file. Normally it is
in either:

-  $JAVA\_HOME/lib/amd64/server
-  $JAVA\_HOME/jre/lib/amd64/server

If ``PYCHBASE_LIBJVM_DIR`` is not set, the installer will check if
``JAVA_HOME`` has been set, and then try each of the above directories.
If ``JAVA_HOME`` is not set, it will attempt to default to
``/usr/lib/jvm/jre-1.7.0/``.

Example:

::

    export PYCHBASE_LIBJVM_DIR=/usr/lib/jvm/jre-1.7.0/lib/amd64/server

PYCHBASE\_INCLUDE\_DIR
----------------------

This houses the ``/hbase/hbase.h`` and other ``libhbase`` C header
files.

If ``PYCHBASE_IS_MAPR`` is true, this defaults to ``/opt/mapr/include``.

For Non-MapR environments, this must be set or the installation will
fail.

Example on Cloudera:

::

    export PYCHBASE_INCLUDE_DIR=/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/include

PYCHBASE\_LIBRARY\_DIR
----------------------

This houses either the ``libMapRClient.so`` file on MapR environments,
or the ``libhbase.so`` file on Non-MapR environments.

If ``PYCHBASE_IS_MAPR`` is true, this defaults to ``/opt/mapr/lib``.

For Non-MapR environments, this must be set or the installation will
fail.

Example on Cloudera:

::

    export PYCHBASE_LIBRARY_DIR=/home/matthew/libhbase/target/libhbase-1.0-SNAPSHOT/lib/native

Author
======

`Matthew Moisen <https://matthewmoisen.com>`__

License
=======

MIT