Module Expr Str Import alias Import alias Import alias Import alias Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore NameLoad TryExcept body Import alias Import alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore Call AttributeLoad NameLoad attr Str Str keyword NameLoad keyword Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str FunctionDef arguments args defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Global identifier Assign NameStore NameLoad decorator_list ClassDef bases AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword Str keyword NameLoad keyword NameLoad Assign NameStore Call NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad attr Assign AttributeStore NameLoad attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults ListLoad body Assign NameStore Dict Str Dict Str Str Dict Str AttributeLoad NameLoad attr Dict Str Str Str NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults ListLoad ListLoad body Assign NameStore Dict Str Dict Str Str Dict Str AttributeLoad NameLoad attr Dict Str Str Str NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body If CompareIs AttributeLoad NameLoad attr NameLoad body Raise Call NameLoad Str While NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr attr Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareEq NameLoad Str body Break If CompareEq NameLoad Str body Raise Call NameLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Expr Call AttributeLoad NameLoad attr Num decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword IfExp AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str ListLoad keyword IfExp AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str ListLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr keyword Str keyword IfExp AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str ListLoad keyword IfExp AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str ListLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str Assign NameStore Call AttributeLoad NameLoad attr keyword Str FunctionDef arguments args NameParam defaults body Return Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body TryExcept body Expr Call AttributeLoad NameLoad attr Return NameLoad handlers ExceptHandler type NameLoad name NameStore body If CompareEq AttributeLoad AttributeLoad NameLoad attr attr Num body Return NameLoad orelse Raise NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Dict Str Str Str Str Str Str Dict Str Str Str BinOpAdd BinOpAdd NameLoad Str AttributeLoad NameLoad attr BinOpAdd BinOpAdd NameLoad Str AttributeLoad NameLoad attr ListLoad Str Dict Str Str Str Num BinOpAdd BinOpAdd BinOpAdd BinOpAdd NameLoad Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Dict Str Str AttributeLoad NameLoad attr Num Dict Str Str Str AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd NameLoad Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Dict Str Str AttributeLoad NameLoad attr Num Dict Str Str AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad attr While NameLoad body Expr Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr NameLoad If CompareEq NameLoad Str body Break If CompareEq NameLoad Str body Raise Call NameLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults body TryExcept body Return Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Str attr handlers ExceptHandler type NameLoad name NameStore body If CompareEq AttributeLoad AttributeLoad NameLoad attr attr Num body Return NameLoad orelse Raise NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return CompareIs Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr attr While NameLoad body Expr Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Break Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad NameLoad Index Str Index Str decorator_list decorator_list
N/A N/A luigi bindings for Google Dataproc on Google Cloud N/A os N/A time N/A logging N/A luigi N/A logger N/A N/A logging getLogger luigi-interface N/A _dataproc_client None N/A N/A N/A httplib2 N/A oauth2client.client googleapiclient discovery googleapiclient.errors HttpError N/A DEFAULT_CREDENTIALS N/A N/A N/A N/A oauth2client client GoogleCredentials get_application_default N/A _dataproc_client N/A N/A discovery build dataproc v1 credentials DEFAULT_CREDENTIALS http N/A N/A httplib2 Http N/A N/A N/A ImportError N/A N/A N/A N/A logger warning Loading Dataproc module without the python packages googleapiclient & oauth2client.         This will crash at runtime if Dataproc functionality is used. get_dataproc_client N/A N/A N/A N/A N/A _dataproc_client N/A set_dataproc_client N/A N/A client N/A N/A N/A _dataproc_client N/A _dataproc_client client N/A _DataprocBaseTask N/A N/A luigi Task N/A N/A gcloud_project_id N/A N/A luigi Parameter significant False positional False N/A dataproc_cluster_name N/A N/A luigi Parameter significant False positional False N/A dataproc_region N/A N/A luigi Parameter default global significant False positional False N/A dataproc_client N/A get_dataproc_client N/A DataprocBaseTask N/A _DataprocBaseTask N/A N/A 
    Base task for running jobs in Dataproc. It is recommended to use one of the tasks specific to your job type.
    Extend this class if you need fine grained control over what kind of job gets submitted to your Dataproc cluster.
     N/A _job None N/A _job_name None N/A _job_id None submit_job N/A N/A self job_config N/A N/A N/A N/A self _job N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions jobs submit projectId N/A self gcloud_project_id region N/A self dataproc_region body job_config execute N/A N/A self _job_id N/A N/A N/A self _job N/A reference N/A jobId N/A N/A self _job N/A submit_spark_job N/A N/A self jars main_class job_args N/A N/A N/A N/A job_config N/A job N/A placement sparkJob N/A clusterName N/A self dataproc_cluster_name N/A args mainClass jarFileUris job_args main_class jars N/A N/A N/A self submit_job job_config N/A N/A self _job_name N/A N/A N/A os path basename N/A N/A N/A self _job N/A sparkJob N/A mainClass N/A N/A N/A logger info N/A N/A Submitted new dataproc job:{} id:{} format N/A self _job_name N/A self _job_id N/A N/A self _job N/A submit_pyspark_job N/A N/A self job_file extra_files job_args N/A N/A N/A N/A N/A job_config N/A job N/A placement pysparkJob N/A clusterName N/A self dataproc_cluster_name N/A mainPythonFileUri pythonFileUris args job_file extra_files job_args N/A N/A N/A self submit_job job_config N/A N/A self _job_name N/A N/A N/A os path basename N/A N/A N/A self _job N/A pysparkJob N/A mainPythonFileUri N/A N/A N/A logger info N/A N/A Submitted new dataproc job:{} id:{} format N/A self _job_name N/A self _job_id N/A N/A self _job N/A wait_for_job N/A N/A self N/A N/A N/A N/A N/A self _job None N/A N/A N/A Exception You must submit a job before you can wait for it N/A True N/A N/A job_result N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions jobs get projectId N/A self gcloud_project_id region N/A self dataproc_region jobId N/A self _job_id execute N/A status N/A N/A job_result N/A status N/A state N/A N/A N/A logger info N/A N/A Current dataproc status: {} job:{} id:{} format status N/A self _job_name N/A self _job_id N/A N/A status DONE N/A N/A N/A N/A status ERROR N/A N/A N/A Exception N/A N/A job_result N/A status N/A details N/A N/A N/A time sleep 5 N/A N/A DataprocSparkTask N/A DataprocBaseTask N/A N/A 
    Runs a spark jobs on your Dataproc cluster
     N/A main_class N/A N/A luigi Parameter N/A jars N/A N/A luigi Parameter default N/A job_args N/A N/A luigi Parameter default run N/A N/A self N/A N/A N/A N/A N/A self submit_spark_job main_class N/A self main_class jars N/A N/A self jars N/A N/A N/A self jars split , N/A job_args N/A N/A self job_args N/A N/A N/A self job_args split , N/A N/A N/A N/A self wait_for_job N/A N/A DataprocPysparkTask N/A DataprocBaseTask N/A N/A 
    Runs a pyspark jobs on your Dataproc cluster
     N/A job_file N/A N/A luigi Parameter N/A extra_files N/A N/A luigi Parameter default N/A job_args N/A N/A luigi Parameter default run N/A N/A self N/A N/A N/A N/A N/A self submit_pyspark_job job_file main_job.py extra_files N/A N/A self extra_files N/A N/A N/A self extra_files split , N/A job_args N/A N/A self job_args N/A N/A N/A self job_args split , N/A N/A N/A N/A self wait_for_job N/A N/A CreateDataprocClusterTask N/A _DataprocBaseTask N/A N/A  Task for creating a Dataproc cluster.  N/A gcloud_zone N/A N/A luigi Parameter default europe-west1-c N/A gcloud_network N/A N/A luigi Parameter default default N/A master_node_type N/A N/A luigi Parameter default n1-standard-2 N/A master_disk_size N/A N/A luigi Parameter default 100 N/A worker_node_type N/A N/A luigi Parameter default n1-standard-2 N/A worker_disk_size N/A N/A luigi Parameter default 100 N/A worker_normal_count N/A N/A luigi Parameter default 2 N/A worker_preemptible_count N/A N/A luigi Parameter default 0 _get_cluster_status N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions clusters get projectId N/A self gcloud_project_id region N/A self dataproc_region clusterName N/A self dataproc_cluster_name execute N/A complete N/A N/A self N/A N/A N/A N/A N/A N/A N/A self _get_cluster_status N/A True N/A N/A N/A HttpError N/A e N/A N/A N/A N/A N/A e resp status 404 N/A N/A False N/A N/A e N/A run N/A N/A self N/A N/A N/A base_uri N/A N/A https://www.googleapis.com/compute/v1/projects/{} format N/A self gcloud_project_id N/A cluster_conf N/A clusterName projectId config N/A self dataproc_cluster_name N/A self gcloud_project_id N/A configBucket gceClusterConfig masterConfig workerConfig secondaryWorkerConfig N/A networkUri zoneUri serviceAccountScopes N/A N/A base_uri /global/networks/ N/A self gcloud_network N/A N/A base_uri /zones/ N/A self gcloud_zone N/A https://www.googleapis.com/auth/cloud-platform N/A numInstances machineTypeUri diskConfig 1 N/A N/A N/A N/A base_uri /zones/ N/A self gcloud_zone /machineTypes/ N/A self master_node_type N/A bootDiskSizeGb numLocalSsds N/A self master_disk_size 0 N/A numInstances machineTypeUri diskConfig N/A self worker_normal_count N/A N/A N/A N/A base_uri /zones/ N/A self gcloud_zone /machineTypes/ N/A self worker_node_type N/A bootDiskSizeGb numLocalSsds N/A self worker_disk_size 0 N/A numInstances isPreemptible N/A self worker_preemptible_count True N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions clusters create projectId N/A self gcloud_project_id region N/A self dataproc_region body cluster_conf execute N/A True N/A N/A N/A N/A time sleep 10 N/A cluster_status N/A N/A self _get_cluster_status N/A status N/A N/A cluster_status N/A status N/A state N/A N/A N/A logger info N/A N/A Creating new dataproc cluster: {} status: {} format N/A self dataproc_cluster_name status N/A N/A status RUNNING N/A N/A N/A N/A status ERROR N/A N/A N/A Exception N/A N/A cluster_status N/A status N/A details N/A N/A DeleteDataprocClusterTask N/A _DataprocBaseTask N/A N/A 
    Task for deleting a Dataproc cluster.
    One of the uses for this class is to extend it and have it require a Dataproc task that does a calculation and have
    that task extend the cluster creation task. This allows you to create chains where you create a cluster,
    run your job and remove the cluster right away.
    (Store your input and output files in gs://... instead of hdfs://... if you do this).
     _get_cluster_status N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions clusters get projectId N/A self gcloud_project_id region N/A self dataproc_region clusterName N/A self dataproc_cluster_name fields status execute N/A N/A N/A HttpError N/A e N/A N/A N/A N/A N/A e resp status 404 N/A N/A None N/A N/A e N/A complete N/A N/A self N/A N/A N/A N/A N/A N/A self _get_cluster_status None N/A run N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_client projects regions clusters delete projectId N/A self gcloud_project_id region N/A self dataproc_region clusterName N/A self dataproc_cluster_name execute N/A True N/A N/A N/A N/A time sleep 10 N/A status N/A N/A self _get_cluster_status N/A N/A status None N/A N/A N/A N/A logger info N/A N/A Finished shutting down cluster: {} format N/A self dataproc_cluster_name N/A N/A N/A N/A logger info N/A N/A Shutting down cluster: {} current status: {} format N/A self dataproc_cluster_name N/A N/A status N/A status N/A state N/A N/A