Module Expr Str Import alias ImportFrom alias ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore AttributeLoad NameLoad attr body Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Call NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr NameLoad keyword Call NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword Call NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore NameLoad If CompareIsNot AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr ListLoad AttributeLoad NameLoad attr NameLoad NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Num If UnaryOpNot NameLoad body Return orelse If CompareLtE Call NameLoad NameLoad AttributeLoad NameLoad attr body Assign NameStore ListComp BoolOpOr Call AttributeLoad NameLoad attr Str comprehension NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad NameLoad orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad keyword Lambda arguments args defaults NameLoad keyword AttributeLoad NameLoad attr Assign NameStore ListComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad comprehension NameStore NameLoad Assign NameStore ListComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad AttributeLoad NameLoad attr comprehension NameStore NameLoad Return ListComp Call NameLoad NameLoad comprehension NameStore NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListComp ListLoad comprehension NameStore Call NameLoad NameLoad For NameStore NameLoad body For TupleStore NameStore NameStore Call NameLoad NameLoad body If NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr NameLoad Assign NameStore ListLoad For NameStore NameLoad body If NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str If CompareEq NameLoad Num body Return ListLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Num Assign NameStore Call AttributeLoad NameLoad attr BinOpMult NameLoad NameLoad If UnaryOpNot NameLoad body Return BinOpAdd ListLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad BinOpMult ListLoad NameLoad BinOpSub NameLoad Num Expr Call AttributeLoad NameLoad attr If CompareGtE Call NameLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore ListLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword SubscriptLoad NameLoad Index Num keyword AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad For NameStore Call NameLoad Num BinOpSub Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword SubscriptLoad NameLoad Index NameLoad keyword SubscriptLoad NameLoad Index BinOpAdd NameLoad Num keyword AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword SubscriptLoad NameLoad Index Num keyword NameLoad keyword AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad If CompareLt Call NameLoad NameLoad NameLoad body AugAssignAdd NameStore BinOpMult ListLoad NameLoad BinOpSub NameLoad Call NameLoad NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assert CompareGtE Call NameLoad NameLoad NameLoad Assign NameStore BinOpDiv Call NameLoad NameLoad Call NameLoad NameLoad Return ListComp SubscriptLoad NameLoad Index Call NameLoad Call NameLoad BinOpMult NameLoad NameLoad comprehension NameStore Call NameLoad Num NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr If CompareNotIn AttributeLoad NameLoad attr NameLoad body Raise Call AttributeLoad NameLoad attr Str If CompareIn AttributeLoad NameLoad attr NameLoad body TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr If CompareLt NameLoad Num body Raise Call AttributeLoad NameLoad attr BinOpMod Str NameLoad handlers ExceptHandler type NameLoad name NameStore body Raise Call AttributeLoad NameLoad attr BinOpMod Str NameLoad TryExcept body Expr Call NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body Raise Call AttributeLoad NameLoad attr Str SubscriptLoad NameLoad Index AttributeLoad NameLoad attr If CompareIn AttributeLoad NameLoad attr NameLoad body If UnaryOpNot Call NameLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad Call NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str If CompareIn AttributeLoad NameLoad attr NameLoad body Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str For NameStore NameLoad body If UnaryOpNot Call NameLoad NameLoad TupleLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str NameLoad If CompareNotEq Call NameLoad NameLoad Num body Raise Call AttributeLoad NameLoad attr Str NameLoad Assign TupleStore NameStore NameStore NameStore NameLoad If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str NameLoad If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list NameLoad decorator_list
N/A N/A Abstract Datastore Input Reader implementation for the map_job API. N/A random google.appengine.api datastore google.appengine.ext key_range google.appengine.ext.mapreduce datastore_range_iterators db_iters google.appengine.ext.mapreduce errors google.appengine.ext.mapreduce key_ranges google.appengine.ext.mapreduce model google.appengine.ext.mapreduce namespace_range google.appengine.ext.mapreduce.api.map_job input_reader AbstractDatastoreInputReader N/A N/A input_reader InputReader N/A N/A Implementation of an abstract base class for a Datastore input reader. N/A _BATCH_SIZE 50 N/A _MAX_SHARD_COUNT 256 N/A MAX_NAMESPACES_FOR_KEY_SHARD 10 N/A _APP_PARAM _app N/A NAMESPACE_PARAM namespace N/A ENTITY_KIND_PARAM entity_kind N/A KEYS_ONLY_PARAM keys_only N/A BATCH_SIZE_PARAM batch_size N/A KEY_RANGE_PARAM key_range N/A FILTERS_PARAM filters N/A _KEY_RANGE_ITER_CLS N/A db_iters AbstractKeyRangeIterator __init__ N/A N/A self iterator N/A N/A N/A Create new AbstractDatastoreInputReader object.

    This is internal constructor. Use split_input to create readers instead.

    Args:
      iterator: an iterator that generates objects for this input reader.
     N/A N/A self _iter iterator N/A __iter__ N/A N/A self N/A N/A N/A Yields whatever the internal iterator yields. N/A o N/A self _iter N/A N/A N/A o N/A __str__ N/A N/A self N/A N/A N/A Returns the string representation of this InputReader. N/A N/A repr N/A self _iter N/A to_json N/A N/A self N/A N/A N/A Inherit doc. N/A N/A N/A N/A self _iter to_json N/A from_json N/A N/A cls state N/A N/A N/A Inherit doc. N/A N/A cls N/A N/A N/A db_iters RangeIteratorFactory from_json state N/A classmethod _get_query_spec N/A N/A cls params N/A N/A N/A Construct a model.QuerySpec from model.MapperSpec. N/A entity_kind N/A params N/A N/A cls ENTITY_KIND_PARAM N/A filters N/A N/A params get N/A cls FILTERS_PARAM N/A app N/A N/A params get N/A cls _APP_PARAM N/A ns N/A N/A params get N/A cls NAMESPACE_PARAM N/A N/A N/A model QuerySpec entity_kind N/A N/A cls _get_raw_entity_kind entity_kind keys_only N/A bool N/A N/A params get N/A cls KEYS_ONLY_PARAM False filters filters batch_size N/A int N/A N/A params get N/A cls BATCH_SIZE_PARAM N/A cls _BATCH_SIZE model_class_path entity_kind app app ns ns N/A classmethod split_input N/A N/A cls job_config N/A N/A N/A Inherit doc. N/A shard_count N/A job_config shard_count N/A params N/A job_config input_reader_params N/A query_spec N/A N/A cls _get_query_spec params N/A namespaces None N/A N/A N/A query_spec ns None N/A N/A k_ranges N/A N/A cls _to_key_ranges_by_shard N/A query_spec app N/A N/A query_spec ns shard_count query_spec N/A N/A ns_keys N/A N/A namespace_range get_namespace_keys N/A query_spec app N/A N/A cls MAX_NAMESPACES_FOR_KEY_SHARD 1 N/A N/A ns_keys N/A N/A N/A N/A N/A N/A len ns_keys N/A cls MAX_NAMESPACES_FOR_KEY_SHARD N/A N/A namespaces N/A N/A N/A N/A ns_key name N/A ns_key ns_keys N/A k_ranges N/A N/A cls _to_key_ranges_by_shard N/A query_spec app namespaces shard_count query_spec N/A N/A ns_ranges N/A N/A N/A namespace_range NamespaceRange split n shard_count contiguous False can_query N/A N/A N/A N/A True _app N/A query_spec app N/A k_ranges N/A N/A N/A N/A key_ranges KeyRangesFactory create_from_ns_range ns_range N/A ns_range ns_ranges N/A iters N/A N/A N/A N/A db_iters RangeIteratorFactory create_key_ranges_iterator r query_spec N/A cls _KEY_RANGE_ITER_CLS N/A r k_ranges N/A N/A N/A cls i N/A i iters N/A classmethod _to_key_ranges_by_shard N/A N/A cls app namespaces shard_count query_spec N/A N/A N/A Get a list of key_ranges.KeyRanges objects, one for each shard.

    This method uses scatter index to split each namespace into pieces
    and assign those pieces to shards.

    Args:
      app: app_id in str.
      namespaces: a list of namespaces in str.
      shard_count: number of shards to split.
      query_spec: model.QuerySpec.

    Returns:
      a list of key_ranges.KeyRanges objects.
     N/A key_ranges_by_ns N/A N/A namespace namespaces N/A N/A ranges N/A N/A cls _split_ns_by_scatter shard_count namespace N/A query_spec entity_kind app N/A N/A N/A random shuffle ranges N/A N/A N/A key_ranges_by_ns append ranges N/A ranges_by_shard N/A N/A N/A _ N/A range shard_count N/A ranges key_ranges_by_ns N/A N/A N/A i k_range N/A enumerate ranges N/A N/A k_range N/A N/A N/A N/A N/A ranges_by_shard N/A i append k_range N/A key_ranges_by_shard N/A N/A ranges ranges_by_shard N/A N/A ranges N/A N/A N/A N/A key_ranges_by_shard append N/A N/A N/A key_ranges KeyRangesFactory create_from_list ranges N/A key_ranges_by_shard N/A classmethod _split_ns_by_scatter N/A N/A cls shard_count namespace raw_entity_kind app N/A N/A N/A Split a namespace by scatter index into key_range.KeyRange.

    TODO: Power this with key_range.KeyRange.compute_split_points.

    Args:
      shard_count: number of shards.
      namespace: namespace name to split. str.
      raw_entity_kind: low level datastore API entity kind.
      app: app id in str.

    Returns:
      A list of key_range.KeyRange objects. If there are not enough entities to
    splits into requested shards, the returned list will contain KeyRanges
    ordered lexicographically with any Nones appearing at the end.
     N/A N/A shard_count 1 N/A N/A N/A N/A N/A key_range KeyRange namespace namespace _app app N/A ds_query N/A N/A datastore Query kind raw_entity_kind namespace namespace _app app keys_only True N/A N/A N/A ds_query Order __scatter__ N/A oversampling_factor 32 N/A random_keys N/A N/A ds_query Get N/A shard_count oversampling_factor N/A N/A random_keys N/A N/A N/A N/A N/A N/A key_range KeyRange namespace namespace _app app N/A N/A None N/A shard_count 1 N/A N/A N/A random_keys sort N/A N/A N/A len random_keys shard_count N/A N/A random_keys N/A N/A cls _choose_split_points random_keys shard_count N/A k_ranges N/A N/A N/A N/A k_ranges append N/A N/A key_range KeyRange key_start None key_end N/A random_keys N/A 0 direction N/A N/A key_range KeyRange ASC include_start False include_end False namespace namespace _app app N/A i N/A range 0 N/A N/A len random_keys 1 N/A N/A N/A N/A k_ranges append N/A N/A key_range KeyRange key_start N/A random_keys N/A i key_end N/A random_keys N/A N/A i 1 direction N/A N/A key_range KeyRange ASC include_start True include_end False namespace namespace _app app N/A N/A N/A k_ranges append N/A N/A key_range KeyRange key_start N/A random_keys N/A -1 key_end None direction N/A N/A key_range KeyRange ASC include_start True include_end False namespace namespace _app app N/A N/A N/A len k_ranges shard_count N/A N/A k_ranges N/A N/A None N/A shard_count N/A len k_ranges N/A k_ranges N/A classmethod _choose_split_points N/A N/A cls sorted_keys shard_count N/A N/A N/A Returns the best split points given a random set of datastore.Keys. N/A N/A N/A len sorted_keys shard_count N/A index_stride N/A N/A len sorted_keys N/A float shard_count N/A N/A N/A sorted_keys N/A N/A int N/A round N/A index_stride i N/A i N/A range 1 shard_count N/A classmethod validate N/A N/A cls job_config N/A N/A N/A Inherit docs. N/A N/A N/A N/A super AbstractDatastoreInputReader cls validate job_config N/A params N/A job_config input_reader_params N/A N/A N/A cls ENTITY_KIND_PARAM params N/A N/A N/A N/A errors BadReaderParamsError Missing input reader parameter 'entity_kind' N/A N/A N/A cls BATCH_SIZE_PARAM params N/A N/A N/A N/A batch_size N/A int N/A params N/A N/A cls BATCH_SIZE_PARAM N/A N/A batch_size 1 N/A N/A N/A N/A errors BadReaderParamsError N/A Bad batch size: %s batch_size N/A N/A N/A ValueError N/A e N/A N/A N/A N/A errors BadReaderParamsError N/A Bad batch size: %s e N/A N/A N/A N/A bool N/A N/A params get N/A cls KEYS_ONLY_PARAM False N/A N/A N/A N/A N/A N/A errors BadReaderParamsError keys_only expects a boolean value but got %s N/A params N/A N/A cls KEYS_ONLY_PARAM N/A N/A N/A cls NAMESPACE_PARAM params N/A N/A N/A N/A isinstance N/A params N/A N/A cls NAMESPACE_PARAM N/A str unicode N/A type None N/A N/A N/A N/A errors BadReaderParamsError Expected a single namespace string N/A N/A N/A cls FILTERS_PARAM params N/A N/A filters N/A params N/A N/A cls FILTERS_PARAM N/A N/A N/A isinstance filters list N/A N/A N/A N/A errors BadReaderParamsError Expected list for filters parameter N/A f filters N/A N/A N/A N/A isinstance f N/A tuple list N/A N/A N/A N/A errors BadReaderParamsError Filter should be a tuple or list: %s f N/A N/A N/A len f 3 N/A N/A N/A N/A errors BadReaderParamsError Filter should be a 3-tuple: %s f N/A N/A prop op _ f N/A N/A N/A isinstance prop basestring N/A N/A N/A N/A errors BadReaderParamsError Property should be string: %s prop N/A N/A N/A isinstance op basestring N/A N/A N/A N/A errors BadReaderParamsError Operator should be string: %s op N/A classmethod _get_raw_entity_kind N/A N/A cls entity_kind_or_model_classpath N/A N/A N/A Returns the entity kind to use with low level datastore calls.

    Args:
      entity_kind_or_model_classpath: user specified entity kind or model
        classpath.

    Returns:
      the entity kind in str to use with low level datastore calls.
     N/A entity_kind_or_model_classpath N/A classmethod N/A