lovely.gae 1.0.0a2
Appengine related Python Packages from Lovely Systems
License: Apache License 2.0
Keywords: appengine datastore backup batch utilities

Lovely Appengine Packages
*************************

================
lovely.gae.async
================

This package executes jobs asynchronously, it uses the appengine
taskqueue to exectue the jobs.

    >>> from lovely.gae.async import defer, get_tasks

The defer function executes a handler asynchronously as a job. We
create 3 jobs that have the same signature.

    >>> import time
    >>> for i in range(3):
    ...     print defer(time.sleep, [0.3])
    <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
    None
    None

Let us have a look on what jobs are there. Note that there is only one
because all the 3 jobs we added were the same.

    >>> len(get_tasks())
    1

If we change the signature of the job, a new one will be added.

    >>> added = defer(time.sleep, [0.4])
    >>> len(get_tasks())
    2

Normally jobs are automatically executed by the taskqueueapi, we have
a test method which executes the jobs and returns the number of jobs
done.

    >>> run_tasks()
    2

Now we can add the same signature again.

    >>> added = defer(time.sleep, [0.4])
    >>> run_tasks()
    1

We can also set only_once to false to execute a worker many times with
the same signature.

    >>> from pprint import pprint
    >>> defer(pprint, ['hello'], once_only=False)
    <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
    >>> defer(pprint, ['hello'], once_only=False)
    <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
    >>> run_tasks()
    'hello'
    'hello'
    2





==========================
DB Custom Property Classes
==========================

Typed Lists
===========

This property converts model instances to keys and ensures a length.

    >>> from lovely.gae.db.property import TypedListProperty
    >>> from google.appengine.ext import db

Let us create a model

    >>> class Yummy(db.Model): pass
    >>> class Bummy(db.Model): pass

We can now reference Yummy instances with our property. Note that we
can also use the kind name as an argument of the kind.

    >>> class Dummy(db.Model):
    ...     yummies = TypedListProperty(Yummy)
    ...     bummies = TypedListProperty('Bummy', length=3)

The kind arguement needs to be a subclass kind name or a db.Model.

    >>> TypedListProperty(object)
    Traceback (most recent call last):
    ...
    ValueError: Kind needs to be a subclass of db.Model

    >>> dummy = Dummy()
    >>> dummy_key = dummy.put()
    >>> yummy1 = Yummy()
    >>> yummy1_key = yummy1.put()
    >>> dummy.yummies = [yummy1]

We cannot set any other type.

    >>> bummy1 = Bummy()
    >>> bummy1_key = bummy1.put()
    >>> dummy.yummies = [bummy1]
    Traceback (most recent call last):
    ...
    BadValueError: Wrong kind u'Bummy'

The length needs to match if defined (see above).

    >>> dummy.bummies = [bummy1]
    Traceback (most recent call last):
    ...
    BadValueError: Wrong length need 3 got 1

    >>> dummy.bummies = [bummy1, bummy1, bummy1]
    >>> dummy_key == dummy.put()
    True

Case-Insensitive String Property
================================

This property allows searching for the lowercase prefix in a
case-insensitive manner. This is usefull for autocomplete
implementations where we do not want to have a seperate property just
for searching.

    >>> from lovely.gae.db.property import IStringProperty
    >>> class Foo(db.Model):
    ...     title = IStringProperty()

    >>> f1 = Foo(title='Foo 1')
    >>> kf1 = f1.put()
    >>> f2 = Foo(title='Foo 2')
    >>> kf2 = f2.put()
    >>> f3 = Foo(title='foo 3')
    >>> kf3 = f3.put()
    >>> f4 = Foo(title=None)
    >>> kf4 = f4.put()

The property does not allow the special seperator character which is
just one below the  highest unicode character,

    >>> f3 = Foo(title='Foo 3' + IStringProperty.SEPERATOR)
    Traceback (most recent call last):
    ...
    BadValueError: Not all characters in property title

Note that if we want to do an exact search, we have to use a special
filter that can be created by the property instance.

    >>> [f.title for f in Foo.all().filter('title =', 'foo 1')]
    []

The "equal" filter arguments can be computed with a special method on
the property.

    >>> ef = Foo.title.equals_filter('Foo 1')
    >>> ef
    ('title =', u'foo 1\xef\xbf\xbcFoo 1')

    >>> [f.title for f in Foo.all().filter(*ef)]
    [u'Foo 1']

Let us try with inequallity, e.g. prefix search. Prefix search is
normally done with two filters using the highest unicode character.

Search for all that starts with "fo" case-insensitive.

    >>> q = Foo.all()
    >>> q = q.filter('title >=', 'fo')
    >>> q = q.filter('title <', 'fo' + u'\xEF\xBF\xBD')
    >>> [f.title for f in q]
    [u'Foo 1', u'Foo 2', u'foo 3']

Search for all that starts with 'foo 1'

    >>> q = Foo.all()
    >>> q = q.filter('title >=', 'foo 1')
    >>> q = q.filter('title <', 'foo 1' + u'\xEF\xBF\xBD')
    >>> [f.title for f in q]
    [u'Foo 1']

    >>> q = Foo.all()
    >>> q = q.filter('title >=', 'foo 2')
    >>> q = q.filter('title <=', 'foo 2' + u'\xEF\xBF\xBD')
    >>> [f.title for f in q]
    [u'Foo 2']

    >>> q = Foo.all()
    >>> q = q.filter('title >=', 'foo 3')
    >>> q = q.filter('title <=', 'foo 3' + u'\xEF\xBF\xBD')
    >>> [f.title for f in q]
    [u'foo 3']


Pickle Property
===============

A pickle property can hold any pickleable python object.

    >>> from lovely.gae.db.property import PickleProperty

    >>> class Pickie(db.Model):
    ...     data = PickleProperty()

    >>> pickie = Pickie(data={})
    >>> pickie.data
    {}
    >>> kp = pickie.put()
    >>> pickie.data
    {}
    >>> pickie = db.get(kp)
    >>> pickie.data
    {}
    >>> pickie.data = {'key':501*"x"}
    >>> kp = pickie.put()
    >>> pickie.data
    {'key': 'xx...xx'}

If the value is not pickleable we get a validation error.

    >>> pickie = Pickie(data=dict(y=lambda x:x))
    Traceback (most recent call last):
    BadValueError: Property 'data' must be pickleable:
    (Can't pickle <function <lambda> at ...>:
    it's not found as __main__.<lambda>)


Safe ReferenceProperty
======================

    >>> from lovely.gae.db.property import SafeReferenceProperty

We use a new model with a gae reference and our safe reference.

    >>> class Refie(db.Model):
    ...     ref   = db.ReferenceProperty(Yummy, collection_name='ref_ref')
    ...     sfref = SafeReferenceProperty(Yummy, collection_name='sfref_ref')

    >>> refie = Refie()
    >>> refie.sfref is None
    True
    >>> refie.ref is None
    True

An object to be referenced.

    >>> refyummy1 = Yummy()
    >>> ignore = refyummy1.put()

Set the references to our yummy object.

    >>> refie.sfref = refyummy1
    >>> refie.sfref
    <Yummy object at ...>
    >>> refie.ref = refyummy1
    >>> refie.ref
    <Yummy object at ...>

    >>> refieKey = refie.put()

Now we delete the referenced object.

    >>> refyummy1.delete()

And reload our referencing object.

    >>> refie = db.get(refieKey)

The gae reference raises an exception.

    >>> refie.ref
    Traceback (most recent call last):
    Error: ReferenceProperty failed to be resolved

We catch the logs here.

    >>> import logging
    >>> from StringIO import StringIO
    >>> log = StringIO()
    >>> handler = logging.StreamHandler(log)
    >>> logger = logging.getLogger('lovely.gae.db')
    >>> logger.setLevel(logging.INFO)
    >>> logger.addHandler(handler)

Our safe reference returns None.

    >>> pos = log.tell()
    >>> refie.sfref is None
    True

Let's see what the log contains.

    >>> log.seek(pos)
    >>> print log.read()
    Unresolved Reference for "Refie._sfref" set to None

Accessing the stale property once again we will see it was reset to None::

    >>> pos = log.tell()
    >>> refie.sfref is None
    True

    >>> log.seek(pos)
    >>> print log.read() == ''
    True

The property get set to None if the reference points to a dead object but only
if the property is not required::

    >>> class Requy(db.Model):
    ...     sfref = SafeReferenceProperty(Yummy, collection_name='req_sfref_ref',
    ...                                   required=True)

    >>> refyummy1 = Yummy()
    >>> ignore = refyummy1.put()

    >>> requy = Requy(sfref = refyummy1)
    >>> requyKey = requy.put()

    >>> requy.sfref
    <Yummy object at ...>

    >>> refyummy1.delete()

    >>> requy = db.get(requyKey)

    >>> pos = log.tell()
    >>> requy.sfref is None
    True

    >>> log.seek(pos)
    >>> print log.read()
    Unresolved Reference for "Requy._sfref" will remain because it is required

=====================
Batch marker creation
=====================

This packages provides the possibility to create markers for every N
objects of a given query. This is useful to create batched html pages
or to generate jobs for every N objects.

A list of attribute values are created that represent the end of a
batch at any given position in a given query. The result is stored in
memcache and the key is provided to a callback function.

    >>> from lovely.gae import batch

Let us create some test objects.

    >>> from google.appengine.ext import db
    >>> class Stub(db.Model):
    ...     c_time = db.DateTimeProperty(auto_now_add=True, required=True)
    ...     name = db.StringProperty()
    ...     age = db.IntegerProperty()
    ...     state = db.IntegerProperty()
    ...     def __repr__(self):
    ...         return '<Stub %s>' % self.name

    >>> for i in range(1,13):
    ...     s = Stub(name=str(i), age=i, state=i%2)
    ...     sk = s.put()

    >>> Stub.all().count()
    12

First we make sure that we have no tasks in the queue for testing.

    >>> from lovely.gae.async import get_tasks
    >>> len(get_tasks())
    0

So for example if we want to know any 100th key of a given kind we
could calculate it like shown below. Note that we provide the pprint function
as a callback, so we get the memcache key in the output.

The calculate_markers function returns the memcache key that will be
used to store the result when the calculation is completed.

    >>> from pprint import pprint
    >>> mc_key = batch.create_markers('Stub', callback=pprint)
    >>> mc_key
    'create_markers:...-...-...'

A task gets created.

    >>> tasks = get_tasks()
    >>> len(tasks)
    1

Let us run the task.

    >>> run_tasks(1)
    1

We now have another task left for the callback, which is actually the
pprint function.

    >>> run_tasks()
    'create_markers:...-...-...'
    1

We should now have a result. The result shows that we need no batches
for the given batch size (because we only have 12 objects).

    >>> from google.appengine.api import memcache
    >>> memcache.get(mc_key)
    []

Let us use another batch size. This time without callback.

    >>> mc_key = batch.create_markers('Stub', batchsize=1)
    >>> run_tasks()
    1

We now have exatly 12 keys, because the batch size was 1.

    >>> len(memcache.get(mc_key))
    12

The default attributes returned are the keys.

    >>> memcache.get(mc_key)
    [datastore_types.Key.fro...

We can also use other attributes. Let us get items batched by c_time
descending. Note that it is not checked if values are not unique, so
if a non-unique attribute is used it might be the case that batch
ranges contains objects twice.

    >>> mc_key = batch.create_markers('Stub',
    ...                               attribute='c_time',
    ...                               order='desc',
    ...                               batchsize=3)
    >>> run_tasks()
    1
    >>> markers = memcache.get(mc_key)
    >>> markers
    [datetime.datetime(...
    >>> len(markers)
    4
    >>> sorted(markers, reverse=True) == markers
    True

    >>> mc_key = batch.create_markers('Stub',
    ...                               attribute='c_time',
    ...                               order='asc',
    ...                               batchsize=3)
    >>> run_tasks()
    1
    >>> markers = memcache.get(mc_key)
    >>> markers
    [datetime.datetime(...
    >>> len(markers)
    4
    >>> sorted(markers) == markers
    True


We can also pass filters to be applied to the query for the batch like this:

    >>> mc_key = batch.create_markers('Stub',
    ...                               filters=[('state', 0)],
    ...                               attribute='c_time',
    ...                               order='asc',
    ...                               batchsize=3)
    >>> run_tasks()
    1
    >>> markers = memcache.get(mc_key)
    >>> len(markers)
    2