Module Expr Str ImportFrom alias Import alias ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body If CompareNotEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad NameLoad TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Expr Yield Call NameLoad BinOpAdd NameLoad Str NameLoad NameLoad keyword BinOpAdd NameLoad Str keyword NameLoad keyword NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield TupleLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults body Assign NameStore Call NameLoad BoolOpOr NameLoad Dict Expr Call AttributeLoad NameLoad attr Dict Str NameLoad Expr Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str NameLoad BinOpAdd NameLoad Str NameLoad NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad body Assign NameStore Yield Call NameLoad NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad Assign NameStore Yield Call NameLoad NameLoad NameLoad Assign NameStore Yield Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad With Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Yield Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A Pipelines for mapreduce library. __future__ with_statement N/A google appengine_pipeline.src pipeline appengine_pipeline.src.pipeline common pipeline_common google.appengine.api files google.appengine.api.files file_service_pb google.appengine.ext.mapreduce output_writers google.appengine.ext.mapreduce base_handler google.appengine.ext.mapreduce input_readers google.appengine.ext.mapreduce mapper_pipeline google.appengine.ext.mapreduce shuffler N/A MapperPipeline N/A mapper_pipeline MapperPipeline N/A ShufflePipeline N/A shuffler ShufflePipeline KeyValueBlobstoreOutputWriter N/A N/A output_writers BlobstoreRecordsOutputWriter N/A N/A Output writer for KeyValue records files in blobstore. write N/A N/A self data ctx N/A N/A N/A N/A N/A len data 2 N/A N/A N/A N/A logging error Got bad tuple of length %d (2-tuple expected): %s N/A len data data N/A N/A N/A key N/A str N/A data N/A 0 N/A value N/A str N/A data N/A 1 N/A N/A N/A TypeError N/A N/A N/A N/A logging error Expecting a tuple, but got %s: %s N/A N/A data __class__ __name__ data N/A proto N/A N/A file_service_pb KeyValue N/A N/A N/A proto set_key key N/A N/A N/A proto set_value value N/A N/A N/A N/A output_writers BlobstoreRecordsOutputWriter write self N/A N/A proto Encode ctx N/A N/A MapPipeline N/A N/A base_handler PipelineBase N/A N/A Runs the map stage of MapReduce.

  Iterates over input reader and outputs data into key/value format
  for shuffler consumption.

  Args:
    job_name: mapreduce job name as string.
    mapper_spec: specification of map handler function as string.
    input_reader_spec: input reader specification as string.
    params: mapper and input reader parameters as dict.
    shards: number of shards to start as int.

  Returns:
    list of filenames list sharded by hash code.
   run N/A N/A self job_name mapper_spec input_reader_spec params shards N/A None N/A N/A N/A N/A MapperPipeline N/A job_name -map mapper_spec input_reader_spec output_writer_spec N/A __name__ .KeyValueBlobstoreOutputWriter params params shards shards N/A N/A KeyValuesReader N/A N/A input_readers RecordsReader N/A N/A Reader to read KeyValues records files from Files API. N/A expand_parameters True __iter__ N/A N/A self N/A N/A N/A binary_record N/A N/A N/A input_readers RecordsReader __iter__ self N/A N/A proto N/A N/A file_service_pb KeyValues N/A N/A N/A proto ParseFromString binary_record N/A N/A N/A N/A N/A proto key N/A N/A proto value_list N/A N/A ReducePipeline N/A N/A base_handler PipelineBase N/A N/A Runs the reduce stage of MapReduce.

  Merge-reads input files and runs reducer function on them.

  Args:
    job_name: mapreduce job name as string.
    reader_spec: specification of reduce function.
    output_writer_spec: specification of output write to use with reduce
      function.
    params: mapper parameters to use as dict.
    filenames: list of filenames to reduce.

  Returns:
    filenames from output writer.
   run N/A N/A self job_name reducer_spec output_writer_spec params filenames N/A N/A N/A new_params N/A dict N/A params N/A N/A N/A N/A new_params update N/A files filenames N/A N/A N/A N/A mapper_pipeline MapperPipeline N/A job_name -reduce reducer_spec N/A __name__ .KeyValuesReader output_writer_spec new_params N/A N/A MapreducePipeline N/A N/A base_handler PipelineBase N/A N/A Pipeline to execute MapReduce jobs.

  Args:
    job_name: job name as string.
    mapper_spec: specification of mapper to use.
    reader_spec: specification of reducer to use.
    input_reader_spec: specification of input reader to read data from.
    output_writer_spec: specification of output writer to save reduce output to.
    mapper_params: parameters to use for mapper phase.
    reducer_params: parameters to use for reduce phase.
    shards: number of shards to use as int.

  Returns:
    filenames from output writer.
   run N/A N/A self job_name mapper_spec reducer_spec input_reader_spec output_writer_spec mapper_params reducer_params shards N/A None None None None N/A N/A map_pipeline N/A N/A MapPipeline job_name mapper_spec input_reader_spec params mapper_params shards shards N/A shuffler_pipeline N/A N/A ShufflePipeline job_name map_pipeline N/A reducer_pipeline N/A N/A ReducePipeline job_name reducer_spec output_writer_spec reducer_params shuffler_pipeline N/A N/A N/A pipeline After reducer_pipeline N/A N/A all_temp_files N/A N/A N/A pipeline_common Extend map_pipeline shuffler_pipeline N/A N/A N/A N/A mapper_pipeline _CleanupPipeline all_temp_files N/A N/A N/A N/A pipeline_common Return reducer_pipeline N/A N/A