Module Expr Str ImportFrom alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam defaults body Expr Str Return Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr NameLoad decorator_list NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore BinOpMult BinOpMult Num Num Num FunctionDef arguments args NameParam defaults body Assign NameStore ListLoad Assign NameStore Num For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad AugAssignAdd NameStore Call NameLoad NameLoad If CompareGt NameLoad AttributeLoad NameLoad attr body Expr Yield NameLoad Assign NameStore Num Assign NameStore ListLoad Expr Call AttributeLoad NameLoad attr If NameLoad body Expr Yield NameLoad Assign NameStore ListLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpMult ListLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str For NameStore Call NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign SubscriptStore NameLoad Index NameLoad TupleLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd BinOpAdd BinOpAdd AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad With Call AttributeLoad NameLoad attr NameLoad keyword NameLoad NameStore body For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call NameLoad keyword NameLoad keyword Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore ListLoad For NameStore Call NameLoad Call NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore Yield Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad BinOpAdd NameLoad Str BinOpAdd NameLoad Str NameLoad Dict Str Str ListLoad NameLoad Num keyword Num Expr Call AttributeLoad NameLoad attr NameLoad With Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Yield Call AttributeLoad NameLoad attr ListComp AttributeLoad NameLoad attr comprehension NameStore NameLoad Assign NameStore Yield Call NameLoad NameLoad With Call AttributeLoad NameLoad attr NameLoad body Expr Yield Call NameLoad NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr ListComp Call AttributeLoad Call AttributeLoad NameLoad attr attr comprehension NameStore NameLoad Return NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign NameStore ListLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Call AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore NameLoad Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr Index NameLoad If CompareNotEq Call NameLoad NameLoad Call NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Str Assign NameStore ListLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore NameLoad While NameLoad body Assign TupleStore NameStore NameStore NameStore NameStore SubscriptLoad NameLoad Index Num If CompareIsNot NameLoad NameLoad body If BoolOpAnd NameLoad CompareNotEq NameLoad SubscriptLoad NameLoad Index Num body Expr Yield NameLoad If BoolOpOr UnaryOpNot NameLoad CompareNotEq NameLoad SubscriptLoad NameLoad Index Num body Assign NameStore TupleLoad NameLoad ListLoad Expr Call AttributeLoad SubscriptLoad NameLoad Index Num attr NameLoad TryExcept body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr If Call AttributeLoad NameLoad attr body Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Call NameLoad NameLoad Call AttributeLoad NameLoad attr Expr Call Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Call NameLoad BinOpMult BinOpSub Call AttributeLoad NameLoad attr NameLoad Num Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad TupleLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad If NameLoad body Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad SubscriptLoad NameLoad Index Str decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return Dict Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr Return ListComp Call NameLoad BinOpMult ListLoad Num Call NameLoad NameLoad comprehension NameStore NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareNotEq Call AttributeLoad NameLoad attr NameLoad body Raise Call AttributeLoad NameLoad attr Str Assign NameStore AttributeLoad NameLoad attr If UnaryOpNot CompareIn AttributeLoad NameLoad attr NameLoad body Raise Call AttributeLoad NameLoad attr Str decorator_list NameLoad decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareNotEq Call AttributeLoad NameLoad attr NameLoad body Raise Call AttributeLoad NameLoad attr Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore ListLoad For NameStore Call NameLoad NameLoad body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad AttributeLoad NameLoad attr attr Str Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad Assign AttributeStore NameLoad attr Dict Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore NameLoad attr Dict Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad SubscriptLoad NameLoad Index Str decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return Dict Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str If CompareNotEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad NameLoad TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore BinOpMod Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BinOpMod Str NameLoad Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad If CompareIs Call AttributeLoad NameLoad attr NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Call AttributeLoad NameLoad attr decorator_list decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad Expr Yield Call AttributeLoad NameLoad attr decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str BinOpAdd NameLoad Str BinOpAdd NameLoad Str keyword BinOpAdd AttributeLoad NameLoad attr Str keyword Dict Str NameLoad keyword Call NameLoad NameLoad decorator_list decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield TupleLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str BinOpAdd NameLoad Str BinOpAdd AttributeLoad NameLoad attr Str keyword BinOpAdd NameLoad Str keyword Dict Str NameLoad keyword Call NameLoad NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Yield Call NameLoad NameLoad NameLoad Assign NameStore Yield Call NameLoad NameLoad NameLoad Assign NameStore Yield Call NameLoad NameLoad NameLoad With Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Yield Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A Mapreduce shuffler implementation. __future__ with_statement N/A gc N/A heapq N/A logging N/A time appengine_pipeline.src pipeline appengine_pipeline.src.pipeline common pipeline_common google.appengine.api files google.appengine.api.files file_service_pb google.appengine.api.files records google.appengine.ext db google.appengine.ext.mapreduce base_handler google.appengine.ext.mapreduce context google.appengine.ext.mapreduce errors google.appengine.ext.mapreduce input_readers google.appengine.ext.mapreduce mapper_pipeline google.appengine.ext.mapreduce operation google.appengine.ext.mapreduce output_writers _OutputFile N/A N/A db Model N/A N/A Entity to store output filenames of pipelines.

  These entities are always children of key returned by get_root_key().
   kind N/A N/A cls N/A N/A N/A Returns entity kind. N/A _GAE_MR_OutputFile N/A classmethod get_root_key N/A N/A cls job_id N/A N/A N/A Get root key to store output files.

    Args:
      job_id: pipeline's job id.

    Returns:
      root key for a given job id to store output file entities.
     N/A N/A N/A N/A db Key from_path N/A N/A cls kind job_id N/A classmethod N/A _compare_keys N/A N/A key_record1 key_record2 N/A N/A N/A Compare two (key, records) protos by key. N/A N/A cmp N/A key_record1 N/A 0 N/A key_record2 N/A 0 N/A _BatchRecordsReader N/A N/A input_readers RecordsReader N/A N/A Records reader that reads in big batches. N/A BATCH_SIZE N/A N/A 1024 1024 3 __iter__ N/A N/A self N/A N/A N/A records N/A N/A size 0 N/A record N/A N/A N/A input_readers RecordsReader __iter__ self N/A N/A N/A N/A records append record N/A size N/A len record N/A N/A size N/A self BATCH_SIZE N/A N/A N/A records N/A size 0 N/A records N/A N/A N/A N/A gc collect N/A records N/A N/A N/A records N/A records N/A N/A N/A N/A gc collect N/A N/A _sort_records_map N/A N/A records N/A N/A N/A Map function sorting records.

  Converts records to KeyValue protos, sorts them by key and writes them
  into new blobstore file. Creates _OutputFile entity to record resulting
  file name.

  Args:
    records: list of records which are serialized KeyValue protos.
   N/A ctx N/A N/A context get N/A l N/A len records N/A key_records N/A N/A None l N/A N/A N/A logging debug Parsing N/A i N/A range l N/A N/A proto N/A N/A file_service_pb KeyValue N/A N/A N/A proto ParseFromString N/A records N/A i N/A N/A key_records N/A i N/A N/A N/A proto key N/A records N/A i N/A N/A N/A logging debug Sorting N/A N/A N/A key_records sort cmp _compare_keys N/A N/A N/A logging debug Writing N/A blob_file_name N/A N/A N/A N/A N/A ctx mapreduce_spec name - N/A ctx mapreduce_id -output N/A output_path N/A N/A N/A files blobstore create _blobinfo_uploaded_filename blob_file_name N/A N/A N/A output_writers RecordsPool output_path ctx ctx pool N/A N/A key_record key_records N/A N/A N/A N/A pool append N/A key_record N/A 1 N/A N/A N/A logging debug Finalizing N/A N/A N/A files finalize output_path N/A output_path N/A N/A N/A files blobstore get_file_name N/A N/A N/A files blobstore get_blob_key output_path N/A entity N/A _OutputFile key_name output_path parent N/A N/A _OutputFile get_root_key N/A ctx mapreduce_id N/A N/A N/A entity put N/A _SortChunksPipeline N/A N/A base_handler PipelineBase N/A N/A A pipeline to sort multiple key-value files.

  Args:
    job_name: root job name.
    filenames: list of filenames to sort.

  Returns:
    The list of lists of sorted filenames. Each list corresponds to one
    input file. Each filenames contains a chunk of sorted data.
   run N/A N/A self job_name filenames N/A N/A N/A sort_mappers N/A N/A i N/A range N/A len filenames N/A N/A filename N/A filenames N/A i N/A sort_mapper N/A N/A N/A mapper_pipeline MapperPipeline N/A %s-shuffle-sort-%s N/A job_name N/A str i N/A __name__ ._sort_records_map N/A __name__ ._BatchRecordsReader None N/A files processing_rate N/A filename 1000000 shards 1 N/A N/A N/A sort_mappers append sort_mapper N/A N/A N/A pipeline After sort_mappers N/A N/A job_ids N/A N/A N/A pipeline_common Append N/A N/A mapper job_id N/A mapper sort_mappers N/A result N/A N/A _CollectOutputFiles job_ids N/A N/A N/A pipeline After result N/A N/A N/A N/A _CleanupOutputFiles job_ids N/A N/A N/A N/A pipeline_common Return result N/A N/A _CollectOutputFiles N/A N/A base_handler PipelineBase N/A N/A Collect output file names from _OutputFile entities for given jobs.

  Args:
    job_ids: list of job ids to load filenames.

  Returns:
    list of lists of filenames produced by specified job ids.
   run N/A N/A self job_ids N/A N/A N/A result N/A N/A job_id job_ids N/A N/A entities N/A N/A N/A N/A _OutputFile all ancestor N/A N/A _OutputFile get_root_key job_id N/A N/A N/A result append N/A N/A N/A N/A N/A entity key name N/A entity entities N/A result N/A N/A _CleanupOutputFiles N/A N/A base_handler PipelineBase N/A N/A Cleanup _OutputFile entities for given job ids.

  Args:
    job_ids: list of job ids.
   run N/A N/A self job_ids N/A N/A N/A result N/A N/A job_id job_ids N/A N/A N/A N/A db delete N/A N/A N/A N/A _OutputFile all ancestor N/A N/A _OutputFile get_root_key job_id N/A N/A _MergingReader N/A N/A input_readers InputReader N/A N/A Reader which merge-reads multiple sorted KeyValue files.

  Reads list of lists of filenames. Each filename list constitutes one shard
  and is merged together.

  Yields (key, values) tuple.
   N/A expand_parameters True N/A FILES_PARAM files __init__ N/A N/A self offsets N/A N/A N/A Constructor.

    Args:
      offsets: offsets for each input file to start from as list of ints.
     N/A N/A self _offsets offsets N/A __iter__ N/A N/A self N/A N/A N/A Iterate over records in input files.

    self._offsets is always correctly updated so that stopping iterations
    doesn't skip records and doesn't read the same record twice.
     N/A ctx N/A N/A context get N/A mapper_spec N/A N/A ctx mapreduce_spec mapper N/A shard_number N/A N/A ctx shard_state shard_number N/A filenames N/A N/A N/A mapper_spec params N/A N/A self FILES_PARAM N/A shard_number N/A N/A N/A len filenames N/A len N/A self _offsets N/A N/A N/A Exception Files list and offsets do not match. N/A readers N/A N/A N/A i filename N/A enumerate filenames N/A N/A offset N/A N/A self _offsets N/A i N/A reader N/A N/A records RecordsReader N/A N/A files BufferedFile filename N/A N/A N/A reader seek offset N/A N/A N/A readers append N/A None None i reader N/A current_result None N/A readers N/A N/A N/A key value index reader N/A readers N/A 0 N/A N/A key None N/A N/A N/A current_result N/A key N/A current_result N/A 0 N/A N/A N/A current_result N/A N/A N/A current_result N/A key N/A current_result N/A 0 N/A N/A current_result N/A key N/A N/A N/A N/A N/A current_result N/A 1 append value N/A N/A N/A N/A N/A self _offsets N/A index N/A N/A reader tell N/A start_time N/A N/A time time N/A binary_record N/A N/A reader read N/A N/A N/A context get N/A N/A N/A N/A N/A N/A operation counters Increment N/A input_readers COUNTER_IO_READ_BYTES N/A len binary_record N/A N/A context get N/A N/A N/A N/A N/A operation counters Increment N/A input_readers COUNTER_IO_READ_MSEC N/A int N/A N/A N/A N/A time time start_time 1000 N/A N/A context get N/A proto N/A N/A file_service_pb KeyValue N/A N/A N/A proto ParseFromString binary_record N/A N/A N/A heapq heapreplace readers N/A N/A N/A proto key N/A N/A proto value index reader N/A N/A N/A EOFError N/A N/A N/A N/A heapq heappop readers N/A current_result N/A N/A N/A current_result N/A from_json N/A N/A cls json N/A N/A N/A Restore reader from json state. N/A N/A cls N/A json N/A offsets N/A classmethod to_json N/A N/A self N/A N/A N/A Serialize reader state to json. N/A N/A offsets N/A self _offsets N/A split_input N/A N/A cls mapper_spec N/A N/A N/A Split input into multiple shards.

    Only one shard is generated at the moment.
     N/A filelists N/A N/A mapper_spec params N/A N/A cls FILES_PARAM N/A N/A N/A cls N/A N/A 0 N/A len files N/A files filelists N/A classmethod validate N/A N/A cls mapper_spec N/A N/A N/A Validate reader parameters in mapper_spec. N/A N/A N/A N/A mapper_spec input_reader_class cls N/A N/A N/A N/A errors BadReaderParamsError Input reader class mismatch N/A params N/A mapper_spec params N/A N/A N/A N/A cls FILES_PARAM params N/A N/A N/A N/A errors BadReaderParamsError Missing files parameter. N/A classmethod N/A _HashingBlobstoreOutputWriter N/A N/A output_writers BlobstoreOutputWriterBase N/A N/A An OutputWriter which outputs data into blobstore in key-value format.

  The output is tailored towards shuffler needs. It shards key/values using
  key hash modulo number of output files.
   __init__ N/A N/A self filenames N/A N/A N/A Constructor.

    Args:
      filenames: list of filenames that this writer outputs to.
     N/A N/A self _filenames filenames N/A validate N/A N/A cls mapper_spec N/A N/A N/A Validates mapper specification.

    Args:
      mapper_spec: an instance of model.MapperSpec to validate.
     N/A N/A N/A N/A mapper_spec output_writer_class cls N/A N/A N/A N/A errors BadWriterParamsError Output writer class mismatch N/A classmethod init_job N/A N/A cls mapreduce_state N/A N/A N/A Initialize job-level writer state.

    Args:
      mapreduce_state: an instance of model.MapreduceState describing current
      job. State can be modified during initialization.
     N/A shards N/A N/A N/A mapreduce_state mapreduce_spec mapper shard_count N/A filenames N/A N/A i N/A range shards N/A N/A blob_file_name N/A N/A N/A N/A N/A N/A mapreduce_state mapreduce_spec name - N/A N/A mapreduce_state mapreduce_spec mapreduce_id -output- N/A str i N/A N/A N/A filenames append N/A N/A N/A files blobstore create _blobinfo_uploaded_filename blob_file_name N/A N/A mapreduce_state writer_state N/A filenames filenames N/A classmethod finalize_job N/A N/A cls mapreduce_state N/A N/A N/A Finalize job-level writer state.

    Args:
      mapreduce_state: an instance of model.MapreduceState describing current
        job. State can be modified during finalization.
     N/A finalized_filenames N/A N/A filename N/A N/A mapreduce_state writer_state N/A filenames N/A N/A N/A N/A files finalize filename N/A N/A N/A finalized_filenames append N/A N/A N/A files blobstore get_file_name N/A N/A N/A files blobstore get_blob_key filename N/A N/A mapreduce_state writer_state N/A filenames finalized_filenames N/A classmethod from_json N/A N/A cls json N/A N/A N/A Creates an instance of the OutputWriter for the given json state.

    Args:
      json: The OutputWriter state as a dict-like object.

    Returns:
      An instance of the OutputWriter configured using the values of json.
     N/A N/A cls N/A json N/A filenames N/A classmethod to_json N/A N/A self N/A N/A N/A Returns writer state to serialize in json.

    Returns:
      A json-izable version of the OutputWriter state.
     N/A N/A filenames N/A self _filenames N/A create N/A N/A cls mapreduce_state shard_number N/A N/A N/A Create new writer for a shard.

    Args:
      mapreduce_state: an instance of model.MapreduceState describing current
      job. State can be modified.
      shard_number: shard number as integer.
     N/A N/A cls N/A N/A mapreduce_state writer_state N/A filenames N/A classmethod get_filenames N/A N/A cls mapreduce_state N/A N/A N/A Obtain output filenames from mapreduce state.

    Args:
      mapreduce_state: an instance of model.MapreduceState

    Returns:
        list of filenames this writer writes to or None if writer
        doesn't write to a file.
     N/A N/A N/A mapreduce_state writer_state N/A filenames N/A classmethod write N/A N/A self data ctx N/A N/A N/A Write data.

    Args:
      data: actual data yielded from handler. Type is writer-specific.
      ctx: an instance of context.Context.
     N/A N/A N/A len data 2 N/A N/A N/A N/A logging error Got bad tuple of length %d (2-tuple expected): %s N/A len data data N/A N/A N/A key N/A str N/A data N/A 0 N/A value N/A str N/A data N/A 1 N/A N/A N/A TypeError N/A N/A N/A N/A logging error Expecting a tuple, but got %s: %s N/A N/A data __class__ __name__ data N/A file_index N/A N/A N/A key __hash__ N/A len N/A self _filenames N/A pool_name N/A kv_pool%d file_index N/A filename N/A N/A self _filenames N/A file_index N/A N/A N/A N/A ctx get_pool pool_name None N/A N/A N/A N/A ctx register_pool pool_name N/A N/A output_writers RecordsPool filename filename ctx ctx N/A proto N/A N/A file_service_pb KeyValue N/A N/A N/A proto set_key key N/A N/A N/A proto set_value value N/A N/A N/A N/A N/A ctx get_pool pool_name append N/A N/A proto Encode N/A N/A _merge_map N/A N/A k values N/A N/A N/A A map function used in merge phase.

  Stores (k, values) into KeyValues proto and yields its serialization.
   N/A proto N/A N/A file_service_pb KeyValues N/A N/A N/A proto set_key k N/A N/A N/A N/A N/A proto value_list extend values N/A N/A N/A N/A proto Encode N/A _MergePipeline N/A N/A base_handler PipelineBase N/A N/A Pipeline to merge sorted chunks.

  This pipeline merges together individually sorted chunks of each shard.

  Args:
    filenames: list of lists of filenames. Each list will correspond to a single
      shard. Each file in the list should have keys sorted and should contain
      records with KeyValue serialized entity.

  Returns:
    The list of filenames, where each filename is fully merged and will contain
    records with KeyValues serialized entity.
   run N/A N/A self job_name filenames N/A N/A N/A N/A N/A N/A mapper_pipeline MapperPipeline N/A job_name -shuffle-merge N/A __name__ ._merge_map N/A __name__ ._MergingReader output_writer_spec N/A N/A output_writers __name__ .BlobstoreRecordsOutputWriter params N/A files filenames shards N/A len filenames N/A N/A _hashing_map N/A N/A binary_record N/A N/A N/A A map function used in hash phase.

  Reads KeyValue from binary record and yields (key, value).
   N/A proto N/A N/A file_service_pb KeyValue N/A N/A N/A proto ParseFromString binary_record N/A N/A N/A N/A N/A proto key N/A N/A proto value N/A _HashPipeline N/A N/A base_handler PipelineBase N/A N/A A pipeline to read mapper output and hash by key.

  Args:
    job_name: root mapreduce job name.
    filenames: filenames of mapper output. Should be of records format
      with serialized KeyValue proto.

  Returns:
    The list of filenames. Each file is of records formad with serialized
    KeyValue proto. For each proto its output file is decided based on key
    hash. Thus all equal keys would end up in the same file.
   run N/A N/A self job_name filenames N/A N/A N/A N/A N/A N/A mapper_pipeline MapperPipeline N/A job_name -shuffle-hash N/A __name__ ._hashing_map N/A N/A input_readers __name__ .RecordsReader output_writer_spec N/A __name__ ._HashingBlobstoreOutputWriter params N/A files filenames shards N/A len filenames N/A N/A ShufflePipeline N/A N/A base_handler PipelineBase N/A N/A A pipeline to shuffle multiple key-value files.

  Args:
    filenames: list of file names to sort. Files have to be of records format
      defined by Files API and contain serialized file_service_pb.KeyValue
      protocol messages.

  Returns:
    The list of filenames as string. Resulting files contain serialized
    file_service_pb.KeyValues protocol messages with all values collated
    to a single key.
   run N/A N/A self job_name filenames N/A N/A N/A hashed_files N/A N/A _HashPipeline job_name filenames N/A sorted_files N/A N/A _SortChunksPipeline job_name hashed_files N/A merged_files N/A N/A _MergePipeline job_name sorted_files N/A N/A N/A pipeline After merged_files N/A N/A all_temp_files N/A N/A N/A pipeline_common Extend hashed_files sorted_files N/A N/A N/A N/A mapper_pipeline _CleanupPipeline all_temp_files N/A N/A N/A N/A pipeline_common Return merged_files N/A N/A