Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias FunctionDef arguments args NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list ClassDef bases NameLoad body Expr Str decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Str Assign NameStore Str Assign NameStore Str decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Dict Str Str ListLoad Str Str ListLoad Str Str Assign NameStore Str Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad Str Str Str Str Str Str Str Num Str body Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr NameLoad If NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad For NameStore TupleLoad Str Str Str Str Str Str Str Str Str body If AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad NameLoad Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad orelse Expr Call NameLoad NameLoad NameLoad Call AttributeLoad Call NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Dict decorator_list FunctionDef arguments args NameParam defaults body If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Return BinOpMod Str AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str Str Str Str Str Str Str BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareNotIn NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad BinOpMod Str NameLoad For NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad body Assign SubscriptStore NameLoad Index NameLoad Num Expr Call AttributeLoad NameLoad attr Str NameLoad If UnaryOpNot Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword ListLoad Str keyword NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr If CompareEq NameLoad Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults Str body Expr Str Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Return Call AttributeLoad NameLoad attr NameLoad keyword Num keyword Dict Str Str decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return BinOpAdd Str Call AttributeLoad NameLoad attr Dict Str Str NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults vararg body Expr Str Assign NameStore Call AttributeLoad Str attr NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str For NameStore Call NameLoad AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad If NameLoad body TryExcept body Return SubscriptLoad SubscriptLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str Index Num Index Str handlers ExceptHandler type TupleLoad NameLoad NameLoad body Pass Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Raise Call NameLoad BinOpMod Str NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad For NameStore Call NameLoad AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr For NameStore Call NameLoad AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Str If CompareEq AttributeLoad NameLoad attr Num body If CompareNotEq Call AttributeLoad AttributeLoad NameLoad attr attr Str Num body Expr Call AttributeLoad NameLoad attr Str Return orelse Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Raise Call NameLoad Str handlers ExceptHandler type AttributeLoad NameLoad attr body Pass Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return BinOpMod AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore Call NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMod Str TupleLoad AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str While NameLoad body Assign NameStore Call NameLoad NameLoad Assign NameStore ListComp SubscriptLoad NameLoad Index Str comprehension NameStore NameLoad If UnaryOpNot NameLoad body Break For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call NameLoad NameLoad For NameStore Call NameLoad AttributeLoad NameLoad attr body Assign NameStore ListLoad For NameStore NameLoad body If Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad orelse Expr Call AttributeLoad NameLoad attr Str NameLoad If UnaryOpNot NameLoad body Break Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore BinOpMod Str TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad Call AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad Call AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list decorator_list
N/A N/A Manipulate Hadoop cluster on Google Compute Engine. N/A logging N/A os N/A os.path N/A time N/A urllib N/A uuid google.appengine.api urlfetch src.hadoop datastore src.hadoop gce_api src.model appconfig MakeScriptRelativePath N/A N/A relative_path N/A N/A N/A Converts file path relative to this script to valid path for OS. N/A N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ relative_path N/A ClusterSetUpError N/A Exception N/A N/A Error during Hadoop cluster set-up. N/A PostprocessStatus N/A object N/A N/A Enum for postprocess status. N/A WARMING_UP WARMING_UP N/A POSTPROCESSING POSTPROCESSING N/A READY READY N/A HadoopCluster N/A object N/A N/A Class to start Compute Engine server farm for Hadoop cluster.

  This class starts up Compute Engines with appropriate configuration for
  Hadoop cluster.  The server farm consists of 1 'master' and multiple
  'workers'.  Hostnames are set by /etc/hosts so that master and workers
  can recognize each other by hostnames.  The common SSH key is distributed
  so that user hadoop can ssh with each other without password.  (SSH is
  the way Hadoop uses for communication.)
   N/A COMPUTE_STARTUP_SCRIPT startup-script.sh N/A GENERATED_FILES_DIR generated_files N/A MASTER_NAME hadoop-master N/A WORKER_NAME_CORE hadoop-worker N/A DONE_FILE /var/log/STARTUP_SCRIPT_DONE N/A INSTANCE_ROLES N/A master worker N/A NameNode JobTracker N/A DataNode TaskTracker N/A HADOOP_PATCH hadoop-1.2.1.patch N/A INSTANCE_IP_ADDRESS_CHECK_INTERVAL 5 N/A INSTANCE_IP_ADDRESS_MAX_CHECK_TIMES 60 N/A MASTER_WARM_UP_CHECK_INTERVAL 15 N/A MASTER_WARM_UP_MAX_CHECK_TIMES 20 N/A CLUSTER_SHUTDOWN_CHECK_INTERVAL 5 N/A CLUSTER_SHUTDOWN_MAX_CHECK_TIMES 60 __init__ N/A N/A self http cluster_id project name prefix zone machinetype image network num_workers custom_command N/A None managed 5 N/A N/A N/A self postprocess_rpcs N/A N/A N/A self authorized_http http N/A N/A self api None N/A N/A self enable_hbase 0 N/A N/A self cluster_info None N/A cluster_id N/A N/A N/A self cluster_info N/A N/A N/A datastore ClusterInfo get_by_id cluster_id N/A item N/A prefix project name zone machinetype image network num_workers custom_command N/A N/A N/A self cluster_info N/A N/A N/A setattr self item N/A getattr N/A self cluster_info item None N/A N/A N/A setattr self item N/A N/A N/A locals get item None N/A N/A self master_name N/A N/A N/A self prefix - N/A self MASTER_NAME N/A N/A self worker_name_template N/A %s-%s-%%03d N/A N/A self prefix N/A self WORKER_NAME_CORE N/A N/A self worker_name_pattern N/A ^%s-%s-\d+$ N/A N/A self prefix N/A self WORKER_NAME_CORE N/A N/A self instances N/A N/A _GetApi N/A N/A self N/A N/A N/A N/A N/A self api N/A N/A N/A self api N/A N/A gce_api GceApi N/A self project N/A self zone authorized_http N/A self authorized_http N/A N/A self api N/A _GetTmpCloudStorage N/A N/A self N/A N/A N/A N/A %s/hadoop N/A N/A N/A N/A appconfig AppConfig GetAppConfig cloud_storage_bucket N/A _StartInstance N/A N/A self instance_name role N/A N/A N/A Starts single Compute Engine instance with a new persistent boot disk.

    Args:
      instance_name: Name of the new instance.
      role: Instance role name.  Must be one of the keys of INSTANCE_ROLES.
    Raises:
      ClusterSetUpError: if the method fails to start an instance.
     N/A N/A N/A logging info Starting instance: %s instance_name N/A rpckey N/A str N/A N/A uuid uuid4 N/A metadata N/A startup-script-url rpckey hostname-prefix num-workers hadoop-master hadoop-worker-template tmp-cloud-storage custom-command hadoop-patch N/A %s/%s N/A N/A N/A self _GetTmpCloudStorage N/A self COMPUTE_STARTUP_SCRIPT rpckey N/A self prefix N/A self num_workers N/A self master_name N/A self worker_name_template N/A N/A self _GetTmpCloudStorage N/A self custom_command N/A self HADOOP_PATCH N/A N/A role N/A self INSTANCE_ROLES N/A N/A N/A ClusterSetUpError N/A Invalid instance role name: %s role N/A command N/A N/A self INSTANCE_ROLES N/A role N/A N/A N/A metadata N/A command 1 N/A N/A N/A logging info Create instance: %s instance_name N/A N/A N/A N/A N/A N/A self _GetApi CreateInstanceWithNewPersistentBootDisk instance_name N/A self network N/A self machinetype N/A self image service_accounts N/A https://www.googleapis.com/auth/devstorage.full_control metadata metadata N/A N/A N/A ClusterSetUpError N/A Failed to start instance %s instance_name N/A instance_status N/A N/A datastore InstanceInfo parent N/A N/A self cluster_info key name instance_name role role rpckey rpckey N/A N/A N/A instance_status put N/A N/A role master N/A N/A N/A N/A N/A self cluster_info SetMasterInstance instance_status N/A N/A N/A self instances N/A instance_name instance_status N/A _HttpRequest N/A N/A self host path N/A N/A N/A Sends HTTP request to the instance.

    Args:
      host: Hostname or IP address.
      path: URL path.
    Returns:
      HTTP response object.
     N/A request_url N/A http://%s/%s N/A host path N/A N/A N/A logging info Request: %s request_url N/A N/A N/A urlfetch fetch request_url deadline 300 headers N/A Cache-Control max-age=0 N/A _BuildRpcUrl N/A N/A self rpckey command N/A N/A N/A Builds URL path for RPC command execution.

    Args:
      rpckey: RPC key for the instance.
      command: Command to execute on the instance in string.
    Returns:
      Encoded URL path for RPC.
     N/A N/A call? N/A N/A urllib urlencode N/A command key command rpckey N/A _RunCommandAtRemote N/A N/A self instance_name N/A command_args N/A N/A Runs specified script at remote instance. N/A command N/A N/A join command_args N/A N/A N/A logging debug Executing on %s: %s instance_name command N/A instance_info N/A N/A N/A datastore InstanceInfo GetByName instance_name N/A url_path N/A N/A self _BuildRpcUrl N/A instance_info rpckey command N/A N/A N/A self _HttpRequest N/A instance_info external_ip url_path N/A _WaitForExternalIp N/A N/A self instance_name N/A N/A N/A Waits for the instance to get external IP and returns it.

    Args:
      instance_name: Name of the Compute Engine instance.
    Returns:
      External IP address in string.
    Raises:
      ClusterSetUpError: External IP assignment times out.
     N/A _ N/A xrange N/A self INSTANCE_IP_ADDRESS_MAX_CHECK_TIMES N/A N/A instance N/A N/A N/A N/A self _GetApi GetInstance instance_name N/A instance N/A N/A N/A N/A N/A N/A N/A N/A N/A instance N/A networkInterfaces N/A 0 N/A accessConfigs N/A 0 N/A natIP N/A N/A N/A N/A KeyError IndexError N/A N/A N/A N/A N/A time sleep N/A self INSTANCE_IP_ADDRESS_CHECK_INTERVAL N/A N/A ClusterSetUpError N/A External IP address time out for %s instance_name N/A _WaitForInstances N/A N/A self N/A N/A N/A Waits for instances in the cluster to start. N/A master_ip N/A N/A self _WaitForExternalIp N/A self master_name N/A N/A N/A N/A self cluster_info SetMasterIpAddress master_ip N/A i N/A xrange N/A self num_workers N/A N/A worker_name N/A N/A self _WorkerName i N/A external_ip N/A N/A self _WaitForExternalIp worker_name N/A instance_info N/A N/A N/A datastore InstanceInfo GetByName worker_name N/A N/A instance_info external_ip external_ip N/A N/A N/A instance_info put N/A _ N/A xrange N/A self MASTER_WARM_UP_MAX_CHECK_TIMES N/A N/A N/A N/A response N/A N/A self _RunCommandAtRemote N/A self master_name cat /var/log/STARTUP_SCRIPT_DONE N/A N/A N/A response status_code 200 N/A N/A N/A N/A N/A N/A response content find success -1 N/A N/A N/A N/A logging info Hadoop cluster started successfully. N/A N/A N/A N/A N/A logging error N/A response content N/A N/A ClusterSetUpError Hadoop master failed to start N/A N/A N/A N/A urlfetch DownloadError N/A N/A N/A N/A N/A time sleep N/A self MASTER_WARM_UP_CHECK_INTERVAL N/A N/A ClusterSetUpError Hadoop master set up timed out N/A _WorkerName N/A N/A self index N/A N/A N/A Returns Hadoop worker name with spedified worker index. N/A N/A N/A self worker_name_template index N/A _StartMaster N/A N/A self N/A N/A N/A Starts Hadoop master Compute Engine instance. N/A N/A N/A self _StartInstance N/A self master_name master N/A _StartWorkers N/A N/A self N/A N/A N/A Starts Hadoop worker Compute Engine instances. N/A i N/A xrange N/A self num_workers N/A N/A N/A N/A self _StartInstance N/A N/A self _WorkerName i worker N/A StartHadoopCluster N/A N/A self N/A N/A N/A Starts Hadoop cluster on Compute Engine. N/A N/A N/A N/A self cluster_info N/A N/A datastore ClusterInfo name N/A self name project N/A self project prefix N/A self prefix zone N/A self zone machinetype N/A self machinetype image N/A self image network N/A self network num_workers N/A self num_workers custom_command N/A self custom_command N/A N/A N/A N/A self cluster_info put N/A N/A N/A self _StartMaster N/A N/A N/A self _StartWorkers N/A N/A N/A self _WaitForInstances N/A N/A N/A ClusterSetUpError N/A e N/A N/A N/A N/A N/A self cluster_info SetStatus N/A %s: %s N/A N/A N/A datastore ClusterStatus ERROR N/A str e N/A _DeleteResource N/A N/A self filter_string list_method delete_method get_method N/A N/A N/A Deletes Compute Engine resource that matches the filter.

    Args:
      filter_string: Filter string of the resource.
      list_method: Method to list the resources.
      delete_method: Method to delete the single resource.
      get_method: Method to get the status of the single resource.
     N/A True N/A N/A list_of_resources N/A list_method filter_string N/A resource_names N/A N/A i N/A name N/A i list_of_resources N/A N/A resource_names N/A N/A N/A name resource_names N/A N/A N/A N/A logging info   %s name N/A N/A delete_method name N/A _ N/A xrange N/A self CLUSTER_SHUTDOWN_MAX_CHECK_TIMES N/A N/A still_alive N/A N/A name resource_names N/A N/A N/A get_method name N/A N/A N/A N/A still_alive append name N/A N/A N/A N/A logging info Deletion complete: %s name N/A N/A still_alive N/A N/A N/A resource_names still_alive N/A N/A N/A time sleep N/A self CLUSTER_SHUTDOWN_CHECK_INTERVAL N/A TeardownCluster N/A N/A self N/A N/A N/A Deletes Compute Engine instances with likely names. N/A N/A N/A N/A self cluster_info SetStatus N/A N/A datastore ClusterStatus TEARING_DOWN N/A name_filter N/A name eq "%s|%s" N/A N/A self master_name N/A self worker_name_pattern N/A N/A N/A logging info Delete instances: N/A N/A N/A self _DeleteResource name_filter N/A N/A N/A self _GetApi ListInstances N/A N/A N/A self _GetApi DeleteInstance N/A N/A N/A self _GetApi GetInstance N/A N/A N/A logging info Delete disks: N/A N/A N/A self _DeleteResource name_filter N/A N/A N/A self _GetApi ListDisks N/A N/A N/A self _GetApi DeleteDisk N/A N/A N/A self _GetApi GetDisk N/A N/A N/A N/A N/A self cluster_info key delete N/A N/A