Module Expr Str Assign NameStore TupleLoad Str Str Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore ListLoad Str Str ClassDef bases NameLoad body Expr Str decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Dict Str Str Str Str Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Dict Str Str Str Str Str Str Str Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Dict Str Str Str Str Str NameLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Dict Str Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore AttributeLoad NameLoad attr body Assign NameStore Dict Str Str Str AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr If AttributeLoad AttributeLoad NameLoad attr attr body Assign SubscriptStore NameLoad Index Str AttributeLoad AttributeLoad NameLoad attr attr If AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore Dict For NameStore AttributeLoad AttributeLoad NameLoad attr attr body Assign SubscriptStore NameLoad Index AttributeLoad NameLoad attr BoolOpOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index Str NameLoad If AttributeLoad NameLoad attr body Assign NameStore Dict For NameStore AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index AttributeLoad NameLoad attr BoolOpOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index Str NameLoad If AttributeLoad AttributeLoad NameLoad attr attr body Assign SubscriptStore NameLoad Index Str AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list NameLoad decorator_list FunctionDef arguments args NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad If UnaryOpNot NameLoad body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore NameLoad While CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Return NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr handlers ExceptHandler type TupleLoad NameLoad AttributeLoad NameLoad attr name NameStore body Raise Call AttributeLoad NameLoad attr NameLoad If CompareLt Call NameLoad NameLoad Num body Raise Call AttributeLoad NameLoad attr Str If CompareGt Call NameLoad NameLoad Num body Raise Call AttributeLoad NameLoad attr BinOpMod Str Call NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad GeneratorExp AttributeLoad NameLoad attr comprehension NameStore AttributeLoad NameLoad attr If CompareNotEq Call NameLoad AttributeLoad NameLoad attr Call NameLoad NameLoad body Raise Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad If UnaryOpNot NameLoad body Raise Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad TryFinally body Return Call NameLoad Call AttributeLoad NameLoad attr finalbody Expr Call AttributeLoad NameLoad attr decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Dict Str Str Str Str Str Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str FunctionDef arguments args NameParam NameParam defaults body If CompareNotIn NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Num Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str Return Assign TupleStore NameStore NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str NameLoad If CompareIn BinOpAdd Str AttributeLoad NameLoad attr NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad attr BinOpAdd Str AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str attr NameLoad orelse TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad BinOpAdd Str NameLoad handlers ExceptHandler type NameLoad body Assign NameStore NameLoad Assign NameStore BoolOpOr NameLoad Call AttributeLoad Call NameLoad NameLoad Str attr Assign SubscriptStore AttributeLoad AttributeLoad NameLoad attr attr Index Str Str Assign SubscriptStore AttributeLoad AttributeLoad NameLoad attr attr Index Str NameLoad Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Call NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body Expr Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd NameLoad Num If CompareEq Call NameLoad NameLoad BinOpAdd NameLoad Num body Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad Call AttributeLoad SubscriptLoad NameLoad Index Num attr Assign NameStore SubscriptLoad NameLoad Slice Num Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Dict Str Str Str Str Str Str Str Str Str AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Call NameLoad BinOpMult Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Num Call NameLoad BinOpMult Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Num AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr If AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str If UnaryOpNot NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If CompareIs NameLoad NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Call NameLoad BinOpMult Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Num Call NameLoad BinOpMult Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Num AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr Assign NameStore ListLoad For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Dict Str Str Str Str Str Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call NameLoad BinOpMult Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Num AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults SubscriptLoad NameLoad Index Str Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad decorator_list decorator_list
N/A N/A Status page handler for mapreduce framework. N/A __author__ N/A aizatsky@google.com (Mike Aizatsky) bslatkin@google.com (Brett Slatkin) N/A os N/A pkgutil N/A time N/A zipfile google.appengine.api validation google.appengine.api yaml_builder google.appengine.api yaml_errors google.appengine.api yaml_listener google.appengine.api yaml_object google.appengine.ext db google.appengine.ext webapp mapreduce base_handler mapreduce errors mapreduce model N/A MR_YAML_NAMES N/A mapreduce.yaml mapreduce.yml BadStatusParameterError N/A Exception N/A N/A A parameter passed to a status handler was invalid. N/A UserParam N/A N/A validation Validated N/A N/A A user-supplied parameter to a mapreduce job. N/A ATTRIBUTES N/A name default value [a-zA-Z0-9_\.]+ N/A N/A validation Optional .* N/A N/A validation Optional .* N/A MapperInfo N/A N/A validation Validated N/A N/A Configuration parameters for the mapper part of the job. N/A ATTRIBUTES N/A handler input_reader output_writer params params_validator .+ .+ N/A N/A validation Optional .+ N/A N/A validation Optional N/A N/A validation Repeated UserParam N/A N/A validation Optional .+ N/A MapreduceInfo N/A N/A validation Validated N/A N/A Mapreduce description in mapreduce.yaml. N/A ATTRIBUTES N/A name mapper params params_validator .+ MapperInfo N/A N/A validation Optional N/A N/A validation Repeated UserParam N/A N/A validation Optional .+ N/A MapReduceYaml N/A N/A validation Validated N/A N/A Root class for mapreduce.yaml.

  File format:

  mapreduce:
  - name: <mapreduce_name>
    mapper:
      - input_reader: google.appengine.ext.mapreduce.DatastoreInputReader
      - handler: path_to_my.MapperFunction
      - params:
        - name: foo
          default: bar
        - name: blah
          default: stuff
      - params_validator: path_to_my.ValidatorFunction

  Where
    mapreduce_name: The name of the mapreduce. Used for UI purposes.
    mapper_handler_spec: Full <module_name>.<function_name/class_name> of
      mapper handler. See MapreduceSpec class documentation for full handler
      specification.
    input_reader: Full <module_name>.<function_name/class_name> of the
      InputReader sub-class to use for the mapper job.
    params: A list of optional parameter names and optional default values
      that may be supplied or overridden by the user running the job.
    params_validator is full <module_name>.<function_name/class_name> of
      a callable to validate the mapper_params after they are input by the
      user running the job.
   N/A ATTRIBUTES N/A mapreduce N/A N/A validation Optional N/A N/A validation Repeated MapreduceInfo to_dict N/A N/A mapreduce_yaml N/A N/A N/A Converts a MapReduceYaml file into a JSON-encodable dictionary.

    For use in user-visible UI and internal methods for interfacing with
    user code (like param validation). as a list

    Args:
      mapreduce_yaml: The Pyton representation of the mapreduce.yaml document.

    Returns:
      A list of configuration dictionaries.
     N/A all_configs N/A N/A config N/A mapreduce_yaml mapreduce N/A N/A out N/A name mapper_input_reader mapper_handler N/A config name N/A N/A config mapper input_reader N/A N/A config mapper handler N/A N/A N/A config mapper params_validator N/A N/A N/A out N/A mapper_params_validator N/A N/A config mapper params_validator N/A N/A N/A config mapper params N/A N/A param_defaults N/A N/A param N/A N/A config mapper params N/A N/A N/A param_defaults N/A N/A param name N/A N/A param default N/A param value N/A N/A out N/A mapper_params param_defaults N/A N/A config params N/A N/A param_defaults N/A N/A param N/A config params N/A N/A N/A param_defaults N/A N/A param name N/A N/A param default N/A param value N/A N/A out N/A params param_defaults N/A N/A N/A config mapper output_writer N/A N/A N/A out N/A mapper_output_writer N/A N/A config mapper output_writer N/A N/A N/A all_configs append out N/A all_configs N/A staticmethod N/A find_mapreduce_yaml N/A N/A status_file N/A __file__ N/A N/A Traverse directory trees to find mapreduce.yaml file.

  Begins with the location of status.py and then moves on to check the working
  directory.

  Args:
    status_file: location of status.py, overridable for testing purposes.

  Returns:
    the path of mapreduce.yaml file or None if not found.
   N/A checked N/A set N/A yaml N/A _find_mapreduce_yaml N/A N/A N/A os path dirname status_file checked N/A N/A yaml N/A N/A yaml N/A _find_mapreduce_yaml N/A N/A os getcwd checked N/A yaml N/A _find_mapreduce_yaml N/A N/A start checked N/A N/A N/A Traverse the directory tree identified by start until a directory already
  in checked is encountered or the path of mapreduce.yaml is found.

  Checked is present both to make loop termination easy to reason about and so
  that the same directories do not get rechecked.

  Args:
    start: the path to start in and work upward from
    checked: the set of already examined directories

  Returns:
    the path of mapreduce.yaml file or None if not found.
   N/A dir start N/A N/A dir checked N/A N/A N/A N/A checked add dir N/A mr_yaml_name MR_YAML_NAMES N/A N/A yaml_path N/A N/A N/A os path join dir mr_yaml_name N/A N/A N/A N/A os path exists yaml_path N/A N/A yaml_path N/A dir N/A N/A N/A os path dirname dir N/A None N/A parse_mapreduce_yaml N/A N/A contents N/A N/A N/A Parses mapreduce.yaml file contents.

  Args:
    contents: mapreduce.yaml file contents.

  Returns:
    MapReduceYaml object with all the data from original file.

  Raises:
    errors.BadYamlError: when contents is not a valid mapreduce.yaml file.
   N/A N/A N/A builder N/A N/A yaml_object ObjectBuilder MapReduceYaml N/A handler N/A N/A yaml_builder BuilderHandler builder N/A listener N/A N/A yaml_listener EventListener handler N/A N/A N/A listener Parse contents N/A mr_info N/A N/A handler GetResults N/A N/A N/A N/A ValueError N/A yaml_errors EventError N/A e N/A N/A N/A N/A errors BadYamlError e N/A N/A N/A len mr_info 1 N/A N/A N/A N/A errors BadYamlError No configs found in mapreduce.yaml N/A N/A N/A len mr_info 1 N/A N/A N/A N/A errors MultipleDocumentsInMrYaml N/A Found %d YAML documents N/A len mr_info N/A jobs N/A mr_info N/A 0 N/A job_names N/A set N/A N/A j name N/A j N/A jobs mapreduce N/A N/A N/A len N/A jobs mapreduce N/A len job_names N/A N/A N/A N/A errors BadYamlError Overlapping mapreduce names; names must be unique N/A jobs N/A get_mapreduce_yaml N/A N/A parse N/A parse_mapreduce_yaml N/A N/A Locates mapreduce.yaml, loads and parses its info.

  Args:
    parse: Used for testing.

  Returns:
    MapReduceYaml object.

  Raises:
    errors.BadYamlError: when contents is not a valid mapreduce.yaml file or the
    file is missing.
   N/A mr_yaml_path N/A find_mapreduce_yaml N/A N/A mr_yaml_path N/A N/A N/A N/A errors MissingYamlError N/A mr_yaml_file N/A open mr_yaml_path N/A N/A N/A N/A parse N/A N/A mr_yaml_file read N/A N/A N/A N/A mr_yaml_file close N/A ResourceHandler N/A N/A webapp RequestHandler N/A N/A Handler for static resources. N/A _RESOURCE_MAP N/A status detail base.css jquery.js jquery-json.js jquery-url.js status.js N/A overview.html text/html N/A detail.html text/html N/A base.css text/css N/A jquery-1.6.1.min.js text/javascript N/A jquery.json-2.2.min.js text/javascript N/A jquery.url.js text/javascript N/A status.js text/javascript get N/A N/A self relative N/A N/A N/A N/A relative N/A self _RESOURCE_MAP N/A N/A N/A N/A N/A self response set_status 404 N/A N/A N/A N/A N/A self response out write Resource not found. N/A N/A N/A real_path content_type N/A N/A self _RESOURCE_MAP N/A relative N/A path N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ static real_path N/A N/A N/A .zip N/A os sep path N/A N/A N/A zip_file zip_path N/A N/A N/A N/A N/A os path relpath path split N/A .zip N/A os sep 1 N/A content N/A N/A N/A N/A zipfile ZipFile N/A zip_file .zip read zip_path N/A N/A N/A N/A data N/A N/A pkgutil get_data __name__ N/A static/ real_path N/A N/A N/A AttributeError N/A N/A data None N/A content N/A data N/A N/A N/A open path rb read N/A N/A N/A N/A self response headers N/A Cache-Control public; max-age=300 N/A N/A N/A N/A self response headers N/A Content-Type content_type N/A N/A N/A N/A N/A self response out write content N/A N/A ListConfigsHandler N/A N/A base_handler GetJsonHandler N/A N/A Lists mapreduce configs as JSON for users to start jobs. handle N/A N/A self N/A N/A N/A N/A N/A self json_response N/A configs N/A N/A MapReduceYaml to_dict N/A get_mapreduce_yaml N/A N/A ListJobsHandler N/A N/A base_handler GetJsonHandler N/A N/A Lists running and completed mapreduce jobs for an overview as JSON. handle N/A N/A self N/A N/A N/A cursor N/A N/A N/A self request get cursor N/A count N/A int N/A N/A N/A self request get count 50 N/A query N/A N/A N/A model MapreduceState all N/A cursor N/A N/A N/A N/A query filter __key__ >= N/A N/A db Key cursor N/A N/A N/A query order __key__ N/A jobs_list N/A N/A query fetch N/A count 1 N/A N/A N/A len jobs_list N/A count 1 N/A N/A N/A N/A self json_response N/A cursor N/A str N/A N/A N/A jobs_list N/A -1 key N/A jobs_list N/A jobs_list N/A -1 N/A all_jobs N/A N/A job jobs_list N/A N/A out N/A name mapreduce_id active start_timestamp_ms updated_timestamp_ms chart_url chart_width active_shards shards N/A N/A job mapreduce_spec name N/A N/A job mapreduce_spec mapreduce_id N/A job active N/A int N/A N/A N/A time mktime N/A N/A N/A job start_time utctimetuple 1000 N/A int N/A N/A N/A time mktime N/A N/A N/A job last_poll_time utctimetuple 1000 N/A job sparkline_url N/A job chart_width N/A job active_shards N/A N/A N/A job mapreduce_spec mapper shard_count N/A N/A job result_status N/A N/A N/A out N/A result_status N/A job result_status N/A N/A N/A all_jobs append out N/A N/A N/A self json_response N/A jobs all_jobs N/A N/A GetJobDetailHandler N/A N/A base_handler GetJsonHandler N/A N/A Retrieves the details of a mapreduce job as JSON. handle N/A N/A self N/A N/A N/A mapreduce_id N/A N/A N/A self request get mapreduce_id N/A N/A mapreduce_id N/A N/A N/A BadStatusParameterError 'mapreduce_id' was invalid N/A job N/A N/A N/A model MapreduceState get_by_key_name mapreduce_id N/A N/A job None N/A N/A N/A KeyError N/A Could not find job with ID %r mapreduce_id N/A N/A N/A N/A self json_response update N/A N/A N/A job mapreduce_spec to_json N/A N/A N/A N/A self json_response update N/A N/A N/A job counters_map to_json N/A N/A N/A N/A self json_response update N/A active start_timestamp_ms updated_timestamp_ms chart_url chart_width N/A job active N/A int N/A N/A N/A time mktime N/A N/A N/A job start_time utctimetuple 1000 N/A int N/A N/A N/A time mktime N/A N/A N/A job last_poll_time utctimetuple 1000 N/A job chart_url N/A job chart_width N/A N/A N/A self json_response N/A result_status N/A job result_status N/A all_shards N/A N/A shard N/A N/A N/A model ShardState find_all_by_mapreduce_state job N/A N/A out N/A active result_status shard_number shard_id updated_timestamp_ms shard_description last_work_item N/A shard active N/A shard result_status N/A shard shard_number N/A shard shard_id N/A int N/A N/A N/A time mktime N/A N/A N/A shard update_time utctimetuple 1000 N/A shard shard_description N/A shard last_work_item N/A N/A N/A out update N/A N/A N/A shard counters_map to_json N/A N/A N/A all_shards append out N/A N/A N/A all_shards sort key N/A N/A N/A x N/A N/A x N/A shard_number N/A N/A N/A self json_response N/A shards all_shards N/A N/A