Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias FunctionDef arguments args NameParam NameParam defaults body Expr Str If UnaryOpNot NameLoad body Raise Call NameLoad BinOpMod Str NameLoad orelse If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call NameLoad BinOpMod Str TupleLoad NameLoad NameLoad AttributeLoad NameLoad attr decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Dict Assign NameStore NameLoad Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Expr Call NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call NameLoad NameLoad NameLoad For TupleStore NameStore NameStore NameLoad body Expr Call NameLoad NameLoad NameLoad Assert Call NameLoad NameLoad BinOpMod Str TupleLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Expr Call NameLoad NameLoad TupleLoad NameLoad NameLoad Assert CompareEq Call NameLoad NameLoad Call NameLoad AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call NameLoad AttributeLoad NameLoad attr Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad For TupleStore TupleStore NameStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad body If BoolOpAnd CompareIs NameLoad NameLoad CompareIn Call AttributeLoad NameLoad attr TupleLoad Str Str Str body Assign NameStore NameLoad Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareIsNot NameLoad NameLoad body If UnaryOpNot Call NameLoad NameLoad TupleLoad NameLoad NameLoad body Assign NameStore ListLoad NameLoad For NameStore NameLoad body If UnaryOpNot Call NameLoad NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad BinOpMod Str TupleLoad NameLoad AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args defaults body Expr Str Return Call NameLoad AttributeLoad NameLoad attr decorator_list NameLoad decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign SubscriptStore AttributeLoad AttributeLoad NameLoad attr attr Index Str Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Str AugAssignAdd NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore ListLoad Str Str body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body AugAssignAdd NameStore BinOpMod Str TupleLoad NameLoad AttributeLoad NameLoad attr Break orelse AugAssignAdd NameStore Str AugAssignAdd NameStore Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Num For NameStore NameLoad body Expr Yield TupleLoad NameLoad NameLoad AugAssignAdd NameStore Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad body Expr Str Assign NameStore ListLoad Assign NameStore ListLoad For TupleStore NameStore NameStore NameLoad body Assign NameStore NameLoad If CompareIsNot NameLoad NameLoad body Assign NameStore BinOpMod NameLoad NameLoad If NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return TupleLoad AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return TupleLoad AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Expr Call NameLoad NameLoad NameLoad Expr Call NameLoad NameLoad NameLoad Assign NameStore ListLoad TryExcept body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Index NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Return TupleLoad AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad TryExcept body Expr Call AttributeLoad NameLoad attr Num handlers ExceptHandler type NameLoad body Pass Return Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list decorator_list FunctionDef arguments args defaults vararg body Expr Str If UnaryOpNot NameLoad body Raise Call NameLoad Str For NameStore NameLoad body If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr ListLoad TupleLoad Str NameLoad Expr Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A N/A A mix-in handler for bulk loading data into an application.

For complete documentation, see the Tools and Libraries section of the
documentation.

To use this in your app, first write a script, e.g. bulkload.py, that
instantiates a Loader for each entity kind you want to import and call
bulkload.main(instance). For example:

person = bulkload.Loader(
  'Person',
  [('name', str),
   ('email', datastore_types.Email),
   ('cool', bool), # ('0', 'False', 'No', '')=False, otherwise bool(value)
   ('birthdate', lambda x: datetime.datetime.fromtimestamp(float(x))),
  ])

if __name__ == '__main__':
  bulkload.main(person)

See the Loader class for more information. Then, add a handler for it in your
app.yaml, e.g.:

  handlers:
  - url: /load
    script: bulkload.py
    login: admin

Finally, deploy your app and run bulkloader.py. For example, to load the
file people.csv into a dev_appserver running on your local machine:

./bulkloader.py --filename people.csv --kind Person --cookie ...                      --url http://localhost:8080/load

The kind parameter is used to look up the Loader instance that will be used.
The bulkload handler should usually be admin_only, so that non-admins can't use
the shell to modify your app's data. The bulkload client uses the cookie
parameter to piggyback its HTTP requests on your login session. A GET request
to the URL specified for your bulkload script will give you a cookie parameter
you can use (/load in the example above).  If your bulkload handler is not
admin_only, you may omit the cookie parameter.

If you want to do extra processing before the entities are stored, you can
subclass Loader and override HandleEntity. HandleEntity is called once with
each entity that is imported from the CSV data. You can return one or more
entities from HandleEntity to be stored in its place, or None if nothing
should be stored.

For example, this loads calendar events and stores them as
datastore_entities.Event entities. It also populates their author field with a
reference to the corresponding datastore_entites.Contact entity. If no Contact
entity exists yet for the given author, it creates one and stores it first.

class EventLoader(bulkload.Loader):
  def __init__(self):
    EventLoader.__init__(self, 'Event',
                         [('title', str),
                          ('creator', str),
                          ('where', str),
                          ('startTime', lambda x:
                            datetime.datetime.fromtimestamp(float(x))),
                          ])

  def HandleEntity(self, entity):
    event = datastore_entities.Event(entity.title)
    event.update(entity)

    creator = event['creator']
    if creator:
      contact = datastore.Query('Contact', {'title': creator}).Get(1)
      if not contact:
        contact = [datastore_entities.Contact(creator)]
        datastore.Put(contact[0])
      event['author'] = contact[0].key()

    return event

if __name__ == '__main__':
  bulkload.main(EventLoader())
 N/A Cookie N/A StringIO N/A csv N/A httplib N/A os N/A traceback N/A google N/A wsgiref.handlers google.appengine.api datastore google.appengine.ext webapp google.appengine.ext.bulkload constants Validate N/A N/A value type N/A N/A N/A  Checks that value is non-empty and of the right type.

  Raises ValueError if value is None or empty, TypeError if it's not the given
  type.

  Args:
    value: any value
    type: a type or tuple of types
   N/A N/A value N/A N/A N/A ValueError N/A Value should not be empty; received %s. value N/A N/A N/A N/A isinstance value type N/A N/A N/A TypeError N/A Expected a %s, but received %s (a %s). N/A type value N/A value __class__ N/A Loader N/A object N/A N/A A base class for creating datastore entities from input data.

  To add a handler for bulk loading a new entity kind into your datastore,
  write a subclass of this class that calls Loader.__init__ from your
  class's __init__.

  If you need to run extra code to convert entities from the input
  data, create new properties, or otherwise modify the entities before
  they're inserted, override HandleEntity.

  See the CreateEntity method for the creation of entities from the
  (parsed) input data.
   N/A __loaders N/A N/A __kind None N/A __properties None __init__ N/A N/A self kind properties N/A N/A N/A  Constructor.

    Populates this Loader's kind and properties map. Also registers it with
    the bulk loader, so that all you need to do is instantiate your Loader,
    and the bulkload handler will automatically use it.

    Args:
      kind: a string containing the entity kind that this loader handles

      properties: list of (name, converter) tuples.

      This is used to automatically convert the CSV columns into properties.
      The converter should be a function that takes one argument, a string
      value from the CSV file, and returns a correctly typed property value
      that should be inserted. The tuples in this list should match the
      columns in your CSV file, in order.

      For example:
        [('name', str),
         ('id_number', int),
         ('email', datastore_types.Email),
         ('user', users.User),
         ('birthdate', lambda x: datetime.datetime.fromtimestamp(float(x))),
         ('description', datastore_types.Text),
         ]
     N/A N/A Validate kind basestring N/A N/A self __kind kind N/A N/A Validate properties list N/A N/A name fn properties N/A N/A N/A Validate name basestring N/A N/A callable fn N/A Conversion function %s for property %s is not callable. N/A fn name N/A N/A self __properties properties N/A N/A N/A Loader __loaders N/A kind self N/A kind N/A N/A self N/A N/A N/A  Return the entity kind that this Loader handes.
     N/A N/A self __kind N/A CreateEntity N/A N/A self values key_name N/A None N/A N/A  Creates an entity from a list of property values.

    Args:
      values: list/tuple of str
      key_name: if provided, the name for the (single) resulting Entity

    Returns:
      list of datastore.Entity

      The returned entities are populated with the property values from the
      argument, converted to native types using the properties map given in
      the constructor, and passed through HandleEntity. They're ready to be
      inserted.

    Raises:
      AssertionError if the number of values doesn't match the number
        of properties in the properties map.
     N/A N/A Validate values N/A list tuple N/A N/A N/A len values N/A len N/A self __properties N/A Expected %d CSV columns, found %d. N/A N/A len N/A self __properties N/A len values N/A entity N/A N/A datastore Entity N/A self __kind name key_name N/A N/A N/A name converter val N/A zip N/A self __properties values N/A N/A N/A N/A converter bool N/A N/A N/A val lower N/A 0 false no N/A N/A val False N/A N/A entity N/A name N/A converter val N/A entities N/A N/A self HandleEntity entity N/A N/A entities None N/A N/A N/A N/A isinstance entities N/A list tuple N/A N/A entities N/A entities N/A entity entities N/A N/A N/A N/A isinstance entity N/A datastore Entity N/A N/A N/A TypeError N/A Expected a datastore.Entity, received %s (a %s). N/A entity N/A entity __class__ N/A entities N/A HandleEntity N/A N/A self entity N/A N/A N/A  Subclasses can override this to add custom entity conversion code.

    This is called for each entity, after its properties are populated from
    CSV but before it is stored. Subclasses can override this to add custom
    entity handling code.

    The entity to be inserted should be returned. If multiple entities should
    be inserted, return a list of entities. If no entities should be inserted,
    return None or [].

    Args:
      entity: datastore.Entity

    Returns:
      datastore.Entity or list of datastore.Entity
     N/A entity N/A RegisteredLoaders N/A N/A N/A N/A N/A  Returns a list of the Loader instances that have been created.
     N/A N/A dict N/A Loader __loaders N/A staticmethod N/A BulkLoad N/A N/A webapp RequestHandler N/A N/A A handler for bulk load requests.

  This class contains handlers for the bulkloading process. One for
  GET to provide cookie information for the upload script, and one
  handler for a POST request to upload the entities.

  In the POST request, the body contains the data representing the
  entities' property values. The original format was a sequences of
  lines of comma-separated values (and is handled by the Load
  method). The current (version 1) format is a binary format described
  in the Tools and Libraries section of the documentation, and is
  handled by the LoadV1 method).
   get N/A N/A self N/A N/A N/A  Handle a GET. Just show an info page.
     N/A page N/A N/A self InfoPage N/A N/A self request uri N/A N/A N/A N/A N/A self response out write page N/A post N/A N/A self N/A N/A N/A  Handle a POST. Reads CSV data, converts to entities, and stores them.
     N/A N/A N/A N/A self response headers N/A Content-Type text/plain N/A N/A response output N/A N/A self Load N/A N/A N/A self request get N/A constants KIND_PARAM N/A N/A N/A self request get N/A constants CSV_PARAM N/A N/A N/A N/A self response set_status response N/A N/A N/A N/A N/A self response out write output N/A InfoPage N/A N/A self uri N/A N/A N/A  Renders an information page with the POST endpoint and cookie flag.

    Args:
      uri: a string containing the request URI
    Returns:
      A string with the contents of the info page to be displayed
     N/A page 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html><head>
<title>Bulk Loader</title>
</head><body> N/A page N/A The bulk load endpoint is: <a href="%s">%s</a><br />
 N/A uri uri N/A cookies N/A N/A N/A os environ get HTTP_COOKIE None N/A cookies N/A N/A cookie N/A N/A Cookie BaseCookie cookies N/A param N/A ACSID dev_appserver_login N/A N/A value N/A N/A cookie get param N/A value N/A N/A page N/A Pass this flag to the client: --cookie='%s=%s'
 N/A param N/A value value N/A N/A N/A page No cookie found!
 N/A page </body></html> N/A page N/A IterRows N/A N/A self reader N/A N/A N/A  Yields a tuple of a line number and row for each row of the CSV data.

    Args:
      reader: a csv reader for the input data.
     N/A line_num 1 N/A columns reader N/A N/A N/A N/A line_num columns N/A line_num 1 N/A LoadEntities N/A N/A self iter loader key_format N/A None N/A N/A Generates entities and loads them into the datastore.  Returns
    a tuple of HTTP code and string reply.

    Args:
      iter: an iterator yielding pairs of a line number and row contents.
      key_format: a format string to convert a line number into an
        entity id. If None, then entity ID's are automatically generated.
       N/A entities N/A N/A output N/A N/A N/A line_num columns iter N/A N/A key_name None N/A N/A key_format None N/A N/A key_name N/A key_format line_num N/A columns N/A N/A N/A N/A N/A N/A output append N/A 
Loading from line %d... line_num N/A new_entities N/A N/A loader CreateEntity columns key_name key_name N/A new_entities N/A N/A N/A N/A entities extend new_entities N/A N/A N/A output append done. N/A N/A N/A N/A stacktrace N/A N/A traceback format_exc N/A N/A N/A output append N/A error:
%s stacktrace N/A N/A N/A httplib BAD_REQUEST N/A N/A join output N/A N/A N/A datastore Put entities N/A N/A N/A httplib OK N/A N/A join output N/A Load N/A N/A self kind data N/A N/A N/A Parses CSV data, uses a Loader to convert to entities, and stores them.

    On error, fails fast. Returns a "bad request" HTTP response code and
    includes the traceback in the output.

    Args:
      kind: a string containing the entity kind that this loader handles
      data: a string containing the CSV data to load

    Returns:
      tuple (response code, output) where:
        response code: integer HTTP response code to return
        output: string containing the HTTP response body
     N/A data N/A N/A data encode utf-8 N/A N/A Validate kind basestring N/A N/A Validate data basestring N/A output N/A N/A N/A N/A loader N/A N/A N/A Loader RegisteredLoaders N/A kind N/A N/A N/A KeyError N/A N/A N/A N/A output append N/A Error: no Loader defined for kind %s. kind N/A N/A N/A httplib BAD_REQUEST N/A N/A join output N/A buffer N/A N/A StringIO StringIO data N/A reader N/A N/A csv reader buffer skipinitialspace True N/A N/A N/A N/A N/A csv field_size_limit 800000 N/A N/A N/A AttributeError N/A N/A N/A N/A N/A self LoadEntities N/A N/A self IterRows reader loader N/A N/A main N/A N/A N/A loaders N/A N/A Starts bulk upload.

  Raises TypeError if not, at least one Loader instance is given.

  Args:
    loaders: One or more Loader instance.
   N/A N/A loaders N/A N/A N/A TypeError Expected at least one argument. N/A loader loaders N/A N/A N/A N/A isinstance loader Loader N/A N/A N/A TypeError N/A Expected a Loader instance; received %r loader N/A application N/A N/A webapp WSGIApplication N/A N/A .* BulkLoad N/A N/A N/A N/A N/A N/A wsgiref handlers CGIHandler run application N/A N/A N/A __name__ __main__ N/A N/A N/A main