Module Import alias Import alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore NameLoad Assign NameStore NameLoad ClassDef bases body FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr attr keyword AttributeLoad NameLoad attr keyword NameLoad attr Assign AttributeStore NameLoad attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Call NameLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str decorator_list FunctionDef arguments args NameParam defaults body While NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr attr If CompareEq Str SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body Print Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Return NameLoad If CompareEq Str SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body Print Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Return NameLoad If CompareEq Str SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body Return NameLoad Expr Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Call NameLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str Expr Call AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body If CompareEq Str SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body If CompareIs NameLoad NameLoad body Assign NameStore Str Raise Call NameLoad BinOpAdd BinOpAdd NameLoad Str Call NameLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str decorator_list FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return BinOpAdd BinOpAdd Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Global identifier Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad If CompareIsNot NameLoad NameLoad body If BoolOpAnd UnaryOpNot Call AttributeLoad NameLoad attr Str CompareEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str orelse If Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr If CompareIsNot Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Dict Str Dict Str Str Str Dict Str Str Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Dict Str Call AttributeLoad NameLoad attr Str Str Dict Str Str Str NameLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr If CompareIsNot NameLoad NameLoad body Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str NameLoad Print NameLoad Assign NameStore Call NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If UnaryOpNot Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return BinOpAdd BinOpAdd Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Raise NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Global identifier Assign NameStore ListLoad Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad If CompareIsNot NameLoad NameLoad body If BoolOpAnd UnaryOpNot Call AttributeLoad NameLoad attr Str CompareEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str orelse If Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr If CompareIsNot Call AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad NameLoad attr Str Str Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Dict Str Dict Str Str Str Dict Str Str Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Dict Str Call AttributeLoad NameLoad attr Str Str Dict Str Str Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr If CompareIsNot NameLoad NameLoad body Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Str NameLoad Print NameLoad Assign NameStore Call NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If UnaryOpNot Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str decorator_list decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Global identifier Assign NameStore Dict Str Str NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Global identifier Assign NameStore Dict Str Str NameLoad NameLoad decorator_list
N/A N/A logging N/A time luigi_gcloud.gcore _GCloudTask luigi_gcloud.storage GCSFileSystem N/A logger N/A N/A logging getLogger luigi-gcloud N/A default_pig_udf None N/A default_spark_udf None _DataProcJob N/A N/A __init__ N/A N/A self dataproc_api project_id job N/A N/A N/A N/A self dataproc_api dataproc_api N/A N/A self project_id project_id N/A N/A self job N/A N/A N/A N/A N/A N/A N/A N/A dataproc_api projects jobs submit projectId N/A self project_id body job execute N/A N/A self job_id N/A N/A N/A self job N/A reference N/A jobId N/A N/A N/A logger info DataProc job %s is %s N/A self job_id N/A str N/A N/A N/A self job N/A status N/A state N/A wait_for_done N/A N/A self N/A N/A N/A True N/A N/A N/A self job N/A N/A N/A N/A N/A N/A N/A N/A N/A self dataproc_api projects jobs get projectId N/A self project_id jobId N/A self job_id execute N/A N/A ERROR N/A N/A N/A self job N/A status N/A state N/A N/A N/A str N/A self job N/A N/A N/A logger error DataProc job %s has errors N/A self job_id N/A N/A N/A logger error N/A N/A N/A self job N/A status N/A details N/A N/A N/A logger debug N/A str N/A self job N/A False N/A N/A CANCELLED N/A N/A N/A self job N/A status N/A state N/A N/A N/A str N/A self job N/A N/A N/A logger warning DataProc job %s is cancelled N/A self job_id N/A N/A N/A logger warning N/A N/A N/A self job N/A status N/A details N/A N/A N/A logger debug N/A str N/A self job N/A False N/A N/A DONE N/A N/A N/A self job N/A status N/A state N/A N/A True N/A N/A N/A logger debug DataProc job %s is %s N/A self job_id N/A str N/A N/A N/A self job N/A status N/A state N/A N/A N/A time sleep 5 N/A raise_error N/A N/A self message N/A None N/A N/A N/A ERROR N/A N/A N/A self job N/A status N/A state N/A N/A N/A message None N/A N/A message Google DataProc job has error N/A N/A Exception N/A N/A message :  N/A str N/A N/A N/A self job N/A status N/A details N/A get N/A N/A self N/A N/A N/A N/A self job N/A N/A DataProcPigTask N/A _GCloudTask N/A N/A service_name dataproc query_file N/A N/A self N/A N/A N/A None N/A query_uri N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A self client project_staging N/A N/A self resolved_name .pig N/A lib_uris N/A N/A self N/A N/A N/A N/A N/A _jar_file_uris N/A N/A self N/A N/A N/A default_pig_udf N/A artifacts N/A N/A uris N/A N/A self lib_uris N/A N/A N/A artifacts extend uris N/A N/A default_pig_udf None N/A N/A N/A N/A N/A N/A default_pig_udf get append N/A N/A len uris 0 N/A N/A N/A N/A artifacts extend N/A N/A default_pig_udf get uris N/A N/A N/A N/A default_pig_udf get append N/A N/A N/A N/A artifacts extend N/A N/A default_pig_udf get uris N/A artifacts N/A run N/A N/A self N/A N/A N/A N/A N/A self start_job N/A http N/A N/A N/A self client http_authorized N/A dataproc_api N/A N/A N/A self client dataproc_api http N/A name N/A N/A self job_name N/A N/A N/A N/A self query_file None N/A N/A fs N/A GCSFileSystem N/A N/A N/A fs put N/A N/A self query_file N/A N/A self query_uri N/A job N/A job N/A reference placement pigJob N/A projectId jobId N/A N/A N/A self client project_id name N/A clusterName N/A N/A self get_service_value clusterName cluster-1 N/A continueOnFailure queryFileUri jarFileUris False N/A N/A self query_uri N/A N/A self _jar_file_uris N/A variables N/A N/A self variables N/A N/A variables None N/A N/A N/A N/A N/A job N/A job N/A pigJob N/A scriptVariables variables N/A job N/A submitted N/A _DataProcJob dataproc_api N/A N/A N/A self client project_id job N/A N/A N/A N/A submitted wait_for_done N/A N/A N/A N/A submitted raise_error DataProcTask has errors N/A N/A DataProcSparkTask N/A _GCloudTask N/A N/A service_name dataproc job_file N/A N/A self N/A N/A N/A None N/A job_uri N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A self client project_staging N/A N/A self resolved_name .jar N/A lib_uris N/A N/A self N/A N/A N/A N/A N/A main N/A N/A self N/A N/A N/A NotImplementedError N/A args N/A N/A self N/A N/A N/A N/A N/A _jar_file_uris N/A N/A self N/A N/A N/A default_spark_udf N/A artifacts N/A N/A N/A self job_uri N/A uris N/A N/A self lib_uris N/A N/A N/A artifacts extend uris N/A N/A default_spark_udf None N/A N/A N/A N/A N/A N/A default_spark_udf get append N/A N/A len uris 0 N/A N/A N/A N/A artifacts extend N/A N/A default_spark_udf get uris N/A N/A N/A N/A default_spark_udf get append N/A N/A N/A N/A artifacts extend N/A N/A default_spark_udf get uris N/A artifacts N/A run N/A N/A self N/A N/A N/A N/A N/A self start_job N/A http N/A N/A N/A self client http_authorized N/A dataproc_api N/A N/A N/A self client dataproc_api http N/A name N/A N/A self job_name N/A N/A N/A N/A self job_file None N/A N/A N/A N/A logger warning Copying job artifact to storage. Consider uploading outside of job. N/A fs N/A GCSFileSystem N/A N/A N/A fs put N/A N/A N/A self get_service_value basePath . N/A N/A self job_file N/A N/A self job_uri N/A job N/A job N/A reference placement sparkJob N/A projectId jobId N/A N/A N/A self client project_id name N/A clusterName N/A N/A self get_service_value clusterName cluster-1 N/A mainClass jarFileUris N/A N/A self main N/A N/A self _jar_file_uris N/A args N/A N/A self args N/A N/A args None N/A N/A N/A N/A N/A job N/A job N/A sparkJob N/A args args N/A job N/A submitted N/A _DataProcJob dataproc_api N/A N/A N/A self client project_id job N/A N/A N/A N/A submitted wait_for_done N/A N/A N/A N/A submitted raise_error DataProcTask has errors N/A N/A set_default_pig_udf N/A N/A uris append N/A False N/A N/A default_pig_udf N/A default_pig_udf N/A append uris append uris N/A set_default_spark_udf N/A N/A uris append N/A False N/A N/A default_spark_udf N/A default_spark_udf N/A append uris append uris N/A