Module Expr Str Import alias TryExcept body Import alias Import alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore Call AttributeLoad NameLoad attr Str Str keyword NameLoad keyword Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Raise Call AttributeLoad NameLoad attr Str Import alias Import alias Import alias ImportFrom alias Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str ClassDef bases AttributeLoad NameLoad attr body FunctionDef arguments args NameParam defaults body Pass decorator_list FunctionDef arguments args NameParam defaults body Pass decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpSub Call AttributeLoad NameLoad attr NameLoad Num decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr attr attr attr keyword NameLoad keyword NameLoad keyword NameLoad attr Assign NameStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Str Str Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr attr attr attr keyword NameLoad keyword NameLoad keyword NameLoad attr Assign NameStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str ListLoad Str Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str ListLoad Str Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Str Str Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr attr attr attr keyword NameLoad keyword NameLoad keyword NameLoad attr Assign NameStore SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str ListLoad Str Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str ListLoad Str Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr ListLoad Str Str Str BinOpAdd Str NameLoad BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpSub Call AttributeLoad NameLoad attr NameLoad Num decorator_list decorator_list Call NameLoad Str
N/A N/A This is an integration test for the Dataproc-luigi binding.

This test requires credentials that can access GCS & access to a bucket below.
Follow the directions in the gcloud tools to set up local credentials.
 N/A unittest N/A N/A N/A oauth2client N/A httplib2 luigi.contrib dataproc googleapiclient discovery N/A default_credentials N/A N/A N/A N/A oauth2client client GoogleCredentials get_application_default N/A default_client N/A N/A discovery build dataproc v1 credentials default_credentials http N/A N/A httplib2 Http N/A N/A N/A dataproc set_dataproc_client default_client N/A N/A N/A ImportError N/A N/A N/A N/A unittest SkipTest Unable to load google cloud dependencies N/A luigi N/A os N/A time nose.plugins.attrib attr N/A PROJECT_ID N/A N/A N/A os environ get DATAPROC_TEST_PROJECT_ID your_project_id_here N/A CLUSTER_NAME N/A N/A N/A os environ get DATAPROC_TEST_CLUSTER unit-test-cluster N/A REGION N/A N/A N/A os environ get DATAPROC_REGION global _DataprocBaseTestCase N/A N/A unittest TestCase N/A setUp N/A N/A self N/A N/A N/A N/A tearDown N/A N/A self N/A N/A N/A N/A N/A DataprocTaskTest N/A _DataprocBaseTestCase N/A test_1_create_cluster N/A N/A self N/A N/A N/A success N/A N/A luigi run N/A --local-scheduler --no-lock CreateDataprocClusterTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME N/A N/A N/A self assertTrue success N/A test_2_create_cluster_should_notice_existing_cluster_and_return_immediately N/A N/A self N/A N/A N/A job_start N/A N/A time time N/A success N/A N/A luigi run N/A --local-scheduler --no-lock CreateDataprocClusterTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME N/A N/A N/A self assertTrue success N/A N/A N/A self assertLess N/A N/A N/A time time job_start 3 N/A test_3_submit_minimal_job N/A N/A self N/A N/A N/A N/A N/A luigi run N/A --local-scheduler --no-lock DataprocSparkTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME --main-class=my.MinimalMainClass N/A response N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A dataproc get_dataproc_client projects regions jobs list projectId PROJECT_ID region REGION clusterName CLUSTER_NAME execute N/A lastJob N/A N/A N/A response N/A jobs N/A 0 N/A sparkJob N/A N/A N/A self assertEquals N/A lastJob N/A mainClass my.MinimalMainClass N/A test_4_submit_spark_job N/A N/A self N/A N/A N/A N/A N/A luigi run N/A --local-scheduler --no-lock DataprocSparkTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME --main-class=my.MainClass --jars=one.jar,two.jar --job-args=foo,bar N/A response N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A dataproc get_dataproc_client projects regions jobs list projectId PROJECT_ID region REGION clusterName CLUSTER_NAME execute N/A lastJob N/A N/A N/A response N/A jobs N/A 0 N/A sparkJob N/A N/A N/A self assertEquals N/A lastJob N/A mainClass my.MainClass N/A N/A N/A self assertEquals N/A lastJob N/A jarFileUris N/A one.jar two.jar N/A N/A N/A self assertEquals N/A lastJob N/A args N/A foo bar N/A test_5_submit_pyspark_job N/A N/A self N/A N/A N/A N/A N/A luigi run N/A --local-scheduler --no-lock DataprocPysparkTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME --job-file=main_job.py --extra-files=extra1.py,extra2.py --job-args=foo,bar N/A response N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A dataproc get_dataproc_client projects regions jobs list projectId PROJECT_ID region REGION clusterName CLUSTER_NAME execute N/A lastJob N/A N/A N/A response N/A jobs N/A 0 N/A pysparkJob N/A N/A N/A self assertEquals N/A lastJob N/A mainPythonFileUri main_job.py N/A N/A N/A self assertEquals N/A lastJob N/A pythonFileUris N/A extra1.py extra2.py N/A N/A N/A self assertEquals N/A lastJob N/A args N/A foo bar N/A test_6_delete_cluster N/A N/A self N/A N/A N/A success N/A N/A luigi run N/A --local-scheduler --no-lock DeleteDataprocClusterTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME N/A N/A N/A self assertTrue success N/A test_7_delete_cluster_should_return_immediately_if_no_cluster N/A N/A self N/A N/A N/A job_start N/A N/A time time N/A success N/A N/A luigi run N/A --local-scheduler --no-lock DeleteDataprocClusterTask N/A --gcloud-project-id= PROJECT_ID N/A --dataproc-cluster-name= CLUSTER_NAME N/A N/A N/A self assertTrue success N/A N/A N/A self assertLess N/A N/A N/A time time job_start 3 N/A N/A N/A attr gcloud